이 블로그 검색

2017년 11월 20일 월요일

deb package 삭제 - Ubuntu

Ubuntu 에서 설치된 package(.deb) 를 삭제

기존 설치된 cuda 9.0 삭제를 예시
u0018670@sys-89985:~$ dpkg -l |grep cuda
ri  cuda-9-0                               9.0.176-1                                  ppc64el      CUDA 9.0 meta-package
ii  cuda-command-line-tools-9-0            9.0.176-1                                  ppc64el      CUDA command-line tools
ii  cuda-core-9-0                          9.0.176-1                                  ppc64el      CUDA core tools
ii  cuda-cublas-8-0                        8.0.61.2-1                                 ppc64el      CUBLAS native runtime libraries
ii  cuda-cublas-9-0                        9.0.176-1                                  ppc64el      CUBLAS native runtime libraries
ii  cuda-cublas-dev-9-0                    9.0.176-1                                  ppc64el      CUBLAS native dev links, headers
ii  cuda-cudart-8-0                        8.0.61-1                                   ppc64el      CUDA Runtime native Libraries
ii  cuda-cudart-9-0                        9.0.176-1                                  ppc64el      CUDA Runtime native Libraries
ii  cuda-cudart-dev-9-0                    9.0.176-1                                  ppc64el      CUDA Runtime native dev links, headers
ii  cuda-cufft-8-0                         8.0.61-1                                   ppc64el      CUFFT native runtime libraries
ii  cuda-cufft-9-0                         9.0.176-1                                  ppc64el      CUFFT native runtime libraries
ii  cuda-cufft-dev-9-0                     9.0.176-1                                  ppc64el      CUFFT native dev links, headers
...



u0018670@sys-89985:~$ sudo apt-get --purge remove cuda
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package 'cuda' is not installed, so not removed
The following packages were automatically installed and are no longer required:
  acpid bbswitch-dkms bumblebee cuda-9-0 cuda-command-line-tools-9-0 cuda-core-9-0 cuda-cublas-9-0 cuda-cublas-dev-9-0
  cuda-cudart-9-0 cuda-cudart-dev-9-0 cuda-cufft-9-0 cuda-cufft-dev-9-0 cuda-curand-9-0 cuda-curand-dev-9-0 cuda-cusolver-9-0
  cuda-cusolver-dev-9-0 cuda-cusparse-9-0 cuda-cusparse-dev-9-0 cuda-documentation-9-0 cuda-driver-dev-9-0 cuda-drivers
  cuda-libraries-9-0 cuda-libraries-dev-9-0 cuda-license-9-0 cuda-misc-headers-9-0 cuda-npp-9-0 cuda-npp-dev-9-0 cuda-nvgraph-9-0
  cuda-nvgraph-dev-9-0 cuda-nvml-dev-9-0 cuda-nvrtc-9-0 cuda-nvrtc-dev-9-0 cuda-runtime-9-0 cuda-samples-9-0 cuda-toolkit-9-0
  cuda-visual-tools-9-0 dkms libcuda1-384 libjansson4 libxfont1 libxkbfile1 libxnvctrl0 nvidia-384 nvidia-384-dev nvidia-modprobe
  nvidia-settings policykit-1-gnome python3-xkit screen-resolution-extra x11-xkb-utils xfonts-base xfonts-encodings xfonts-utils
  xserver-common xserver-xorg-core xserver-xorg-legacy
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 86 not upgraded.


다음 command 를 통해 삭제 수행.
u0018670@sys-89985:~$ sudo apt autoremove
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  acpid bbswitch-dkms bumblebee cuda-9-0 cuda-command-line-tools-9-0 cuda-core-9-0 cuda-cublas-9-0 cuda-cublas-dev-9-0
  cuda-cudart-9-0 cuda-cudart-dev-9-0 cuda-cufft-9-0 cuda-cufft-dev-9-0 cuda-curand-9-0 cuda-curand-dev-9-0 cuda-cusolver-9-0
  cuda-cusolver-dev-9-0 cuda-cusparse-9-0 cuda-cusparse-dev-9-0 cuda-documentation-9-0 cuda-driver-dev-9-0 cuda-drivers
  cuda-libraries-9-0 cuda-libraries-dev-9-0 cuda-license-9-0 cuda-misc-headers-9-0 cuda-npp-9-0 cuda-npp-dev-9-0 cuda-nvgraph-9-0
  cuda-nvgraph-dev-9-0 cuda-nvml-dev-9-0 cuda-nvrtc-9-0 cuda-nvrtc-dev-9-0 cuda-runtime-9-0 cuda-samples-9-0 cuda-toolkit-9-0
  cuda-visual-tools-9-0 dkms libcuda1-384 libjansson4 libxfont1 libxkbfile1 libxnvctrl0 nvidia-384 nvidia-384-dev nvidia-modprobe
  nvidia-settings policykit-1-gnome python3-xkit screen-resolution-extra x11-xkb-utils xfonts-base xfonts-encodings xfonts-utils
  xserver-common xserver-xorg-core xserver-xorg-legacy
0 upgraded, 0 newly installed, 56 to remove and 85 not upgraded.
After this operation, 2,236 MB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 190184 files and directories currently installed.)
Removing cuda-9-0 (9.0.176-1) ...
Removing cuda-runtime-9-0 (9.0.176-1) ...
Removing cuda-drivers (384.81-1) ...
Removing nvidia-384-dev (384.81-0ubuntu1) ...
...
u0018670@sys-89985:~$ sudo apt-get clean


추가로, CUDA 8.0 을 설치하고 싶은데 NVIDIA 사이트에는 최신인 CUDA 9.0 링크 제공
https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=ppc64le&target_distro=Ubuntu&target_version=1604&target_type=debnetwork

그냥 apt-get 으로 설치.
u0018670@sys-89985:~$ sudo apt-get install cuda-8-0
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  acpid bbswitch-dkms bumblebee cuda-command-line-tools-8-0 cuda-core-8-0 cuda-cublas-dev-8-0 cuda-cudart-dev-8-0
  cuda-cufft-dev-8-0 cuda-curand-dev-8-0 cuda-cusolver-dev-8-0 cuda-cusparse-dev-8-0 cuda-documentation-8-0 cuda-driver-dev-8-0
  cuda-drivers cuda-misc-headers-8-0 cuda-npp-8-0 cuda-npp-dev-8-0 cuda-nvgraph-8-0 cuda-nvgraph-dev-8-0 cuda-nvml-dev-8-0
  cuda-nvrtc-8-0 cuda-nvrtc-dev-8-0 cuda-runtime-8-0 cuda-samples-8-0 cuda-toolkit-8-0 cuda-visual-tools-8-0 dkms libcuda1-384
  libjansson4 libxfont1 libxkbfile1 libxnvctrl0 nvidia-384 nvidia-384-dev nvidia-modprobe nvidia-settings policykit-1-gnome
  python3-xkit screen-resolution-extra x11-xkb-utils xfonts-base xfonts-encodings xfonts-utils xserver-common xserver-xorg-core
  xserver-xorg-legacy
Suggested packages:
......


아니면, 다음 link 를 통해 다운로드
https://developer.nvidia.com/cuda-toolkit-archive 

2017년 11월 19일 일요일

NFS 서버, 클라이언트 구성 - Ubuntu

NFS 서버(10.10.10.10), NFS 클라이언트(20.20.20.20)

서버(10.10.10.10)에서 NFS 관련 package 설치
root@ubuntu:~#apt-get install nfs-common nfs-kernel-server rpcbind

/data/nfs 디렉토리 생성 후 chmod 777 권한 변경
root@ubuntu:/data$ ls -al
total 28
drwxr-xr-x  4 root root  4096 Nov 19 21:41 .
drwxr-xr-x 24 root root  4096 Nov 19 18:02 ..
drwx------  2 root root 16384 Nov 19 18:02 lost+found
drwxrwxrwx  2 root root  4096 Nov 19 21:41 nfs


nfs 설정 파일에 공유할 폴더와 설정 추가
root@ubuntu:~#vi /etc/exports
# /etc/exports: the access control list for filesystems which may be exported
#               to NFS clients.  See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes       hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4        gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes  gss/krb5i(rw,sync,no_subtree_check)
#
/data/nfs       *(rw,sync,no_subtree_check)


root@ubuntu:~#service nfs-kernel-server restart
root@ubuntu:~#service rpcbind stop
root@ubuntu:~#service rpcbind start

클라이언트(20.20.20.20)에서 nfs-common 패키지 설
root@ubuntu:~#apt-get install nfs-common

서버에서 export 된 디렉토리 /root/v7000 에 mount
root@ubuntu:~# mount -t nfs -o nolock 10.10.10.10:/data/nfs /root/v7000
root@ubuntu:~/v7000# df -h
Filesystem               Size  Used Avail Use% Mounted on
udev                     475M     0  475M   0% /dev
tmpfs                     99M  4.8M   95M   5% /run
/dev/sda1                 19G   11G  7.4G  59% /
tmpfs                    495M  4.0K  495M   1% /dev/shm
tmpfs                    5.0M     0  5.0M   0% /run/lock
tmpfs                    495M     0  495M   0% /sys/fs/cgroup
tmpfs                     99M     0   99M   0% /run/user/0
10.10.10.10:/data/nfs   50G   52M   47G   1% /root/v7000

iSCSI 스토리지 디스크 연결 - Ubuntu

설치 환경
root@ubuntu:~# more /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS"


먼저, iSCSI initiator 설치
root@ubuntu:~# sudo apt-get install open-iscsi
root@ubuntu:~# dpkg -l |grep iscsi
ii  open-iscsi                         2.0.873+git0.3b4b4500-14ubuntu3.3  amd64        iSCSI initiator tools


initiator 설치되면 다음과 같이 디렉토리에서 iqn 정보 확인 가능
root@ubuntu:/etc/iscsi# pwd
/etc/iscsi
root@ubuntu:/etc/iscsi# ls -al
total 32
drwxr-xr-x  4 root root  4096 Nov 19 17:27 .
drwxr-xr-x 89 root root  4096 Nov 19 17:06 ..
-rw-------  1 root root   349 Nov  7 21:03 initiatorname.iscsi
-rw-------  1 root root 11788 Nov 19 17:27 iscsid.conf
drw-------  4 root root  4096 Nov 19 17:32 nodes
drw-------  4 root root  4096 Nov  8 16:12 send_targets
root@ubuntu:/etc/iscsi# more initiatorname.iscsi
## DO NOT EDIT OR REMOVE THIS FILE!
## If you remove this file, the iSCSI daemon will not start.
## If you change the InitiatorName, existing access control lists
## may reject this initiator.  The InitiatorName must be unique
## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.
InitiatorName=iqn.1993-08.org.debian:01:aeff61f6e0e3


iscsid.conf 파일을 다음과 같이 수정

이제 스토리지(Storwize IBM V7000)에서 iSCSI 환경 설정, 볼륨 생성 및 해당 서버에 볼륨 할당

iSCSI 볼륨을 사용하기 위해서는 네트워크 - 이더넷 포트 메뉴에서 IP 설정을 해주어야 한다. 당연히 V7000 GUI 접속을 위한 management IP 와는 다른 IP를 설정하는데 UTP 케이블은 물리적으로 동일한 port 를 사용한다.

스토리지에서 호스트 생성시 서버의 iqn 정보를 입력하고 생성한다.

스토리지 볼륨(50GB) 생성된 호스트에 할당


이제 다시 서버에서 스토리지 정보 인식. 스토리지에서 설정한 iSCSI 용 IP 입력
root@ubuntu:~# iscsiadm -m discovery -t st -p 9.187.227.230
9.187.227.230:3260,1 iqn.1986-03.com.ibm:2145.stg-v7000.node1
root@ubuntu:~# iscsiadm -m discovery -t st -p 9.187.227.240
9.187.227.240:3260,1 iqn.1986-03.com.ibm:2145.stg-v7000.node2



스토리지 연결. 다음과 같이 successful 이 나오면 성공
root@ubuntu:~# iscsiadm -m node --login
Logging in to [iface: default, target: iqn.1986-03.com.ibm:2145.stg-v7000.node2, portal: 9.187.227.240,3260] (multiple)
Logging in to [iface: default, target: iqn.1986-03.com.ibm:2145.stg-v7000.node1, portal: 9.187.227.230,3260] (multiple)
Login to [iface: default, target: iqn.1986-03.com.ibm:2145.stg-v7000.node2, portal: 9.187.227.240,3260] successful.
Login to [iface: default, target: iqn.1986-03.com.ibm:2145.stg-v7000.node1, portal: 9.187.227.230,3260] successful.


그리고, dmesg 로 연결된 스토리지 볼륨 확인
root@ubuntu:~# dmesg | tail -100 | grep sd
[    5.273551] EXT4-fs (sda1): re-mounted. Opts: errors=remount-ro
[    7.079363] Adding 4191228k swap on /dev/sda5.  Priority:-1 extents:1 across:4191228k FS
[597769.393140] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 39s! [vmtoolsd:985]
[597769.569910] CPU: 0 PID: 985 Comm: vmtoolsd Not tainted 4.4.0-62-generic #83-Ubuntu
[953286.712454] sd 4:0:0:0: [sdb] 104857600 512-byte logical blocks: (53.7 GB/50.0 GiB)
[953286.732047] sd 4:0:0:0: [sdb] Write Protect is off
[953286.732051] sd 4:0:0:0: [sdb] Mode Sense: 97 00 10 08
[953286.738440] sd 4:0:0:0: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
[953286.738663] sd 3:0:0:0: [sdc] 104857600 512-byte logical blocks: (53.7 GB/50.0 GiB)
[953286.746023] sd 3:0:0:0: [sdc] Write Protect is off
[953286.746027] sd 3:0:0:0: [sdc] Mode Sense: 97 00 10 08
[953286.746522] sd 3:0:0:0: [sdc] Write cache: disabled, read cache: enabled, supports DPO and FUA
[953286.808293] sd 3:0:0:0: Attached scsi generic sg2 type 0
[953286.808727] sd 4:0:0:0: Attached scsi generic sg3 type 0
[953286.857392] sd 3:0:0:0: [sdc] Attached SCSI disk
[953286.857563] sd 4:0:0:0: [sdb] Attached SCSI disk


스토리지의 컨트롤러가 2개이여서 sdb, sdc 로 볼륨 확인.

이제 multipath 구성을 위해 multipath-tool package 설치
root@ubuntu:~# apt-get install multipath-tools

설치후 다음과 같이 스토리지 볼륨을 확인 할 수 있으나,
root@ubuntu:~# multipath -ll
36005076802820c032000000000000055 dm-0 IBM,2145
size=50G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 3:0:0:0 sdc 8:32 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  `- 4:0:0:0 sdb 8:16 active ready running


/etc/multipath.conf 파일을 수정하여 볼륨을 alias 정의 후 사용. 기본적으로 multipath.conf 을 없으니 vi 로 파일 생성 후 다음과 같이 수정.
multipaths {
multipath {
wwid 36005076802820c032000000000000055
alias vol01
path_grouping_policy multibus
path_selector "round-robin 0"
failback manual
rr_weight priorities
no_path_retry 5
}
}


multipath 의 자세한 내용은 다음 링크 또는 각 스토리지 벤더에서 제공하는 방법을 참고.
https://help.ubuntu.com/lts/serverguide/multipath-dm-multipath-config-file.html

root@ubuntu:~# service multipathd restart
root@ubuntu:~# multipath -ll
vol01 (36005076802820c032000000000000055) dm-0 IBM,2145
size=50G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=30 status=active
  |- 3:0:0:0 sdc 8:32 active ready running
  `- 4:0:0:0 sdb 8:16 active ready running


생성된 볼륨 포멧 및 마운트
root@ubuntu:~# fdisk -l |grep -i vol01
Disk /dev/mapper/vol01: 50 GiB, 53687091200 bytes, 104857600 sectors

root@ubuntu:~# mkfs.ext4 /dev/mapper/vol01
mke2fs 1.42.13 (17-May-2015)
Creating filesystem with 13107200 4k blocks and 3276800 inodes
Filesystem UUID: 3a2d7e7d-580a-4fd1-baaf-66bdb688ecb0
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done


root@ubuntu:~# mount /dev/mapper/vol01 /data/
root@ubuntu:~# df -h
Filesystem         Size  Used Avail Use% Mounted on
udev               2.0G     0  2.0G   0% /dev
tmpfs              396M  5.8M  390M   2% /run
/dev/sda1           26G  2.0G   23G   9% /
tmpfs              2.0G     0  2.0G   0% /dev/shm
tmpfs              5.0M     0  5.0M   0% /run/lock
tmpfs              2.0G     0  2.0G   0% /sys/fs/cgroup
tmpfs              396M     0  396M   0% /run/user/1000
/dev/mapper/vol01   50G   52M   47G   1% /data

2017년 11월 18일 토요일

Linux 기본

호스트 버전 확인
[root@localhost ~]# uname -a
Linux localhost 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

[root@localhost ~]# more /etc/redhat-release
Red Hat Enterprise Linux Server release 7.3 (Maipo)

[root@localhost ~]# hostnamectl
Static hostname: vm-01
Icon name: computer-vm
Chassis: vm
Machine ID: 55e73fcc6f8e433c84e7c7c4d15cf270
Boot ID: 60073b5ad7a4433c8388bebf008aaa4f
Virtualization: vmware
Operating System: Red Hat Enterprise Linux Server 7.3 (Maipo)
CPE OS Name: cpe:/o:redhat:enterprise_linux:7.3:GA:server
Kernel: Linux 3.10.0-514.el7.x86_64
Architecture: x86-64

-.Ubuntu
root@ubuntu:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.3 LTS
Release:        16.04
Codename:       xenial

root@ubuntu:~# more /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"


호스트명 변경
-.RHEL6 or earlier
/etc/sysconfig/network, /etc/hosts 에 IP 및 host name 등록 후
[root@localhost ~]# more /etc/hosts
192.168.0.1   vm-01
192.168.0.2   vm-02
[root@localhost ~]# hostname -v ‘hostname’ 후 logout login
 

-.RHEL7
[root@localhost ~]# hostnamectl set-hostname vm-01


IP 설정 및 변경
-.RHEL
[root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
# Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)
DEVICE=eth0
BOOTPROTO=static
DHCPCLASS=
HWADDR=00:0C:29:6C:28:37
ONBOOT=yes
#yes 는 부팅시 on
IPADDR=192.168.189.128
NETMASK=255.255.255.0
[root@localhost ~]# ifdown eth0
[root@localhost ~]# ifup eth0
[root@localhost ~]# service network restart


-.Ubuntu
root@ubuntu:~# cd /etc/network/
root@ubuntu:/etc/network# ls
if-down.d  if-post-down.d  if-pre-up.d  if-up.d  interfaces  interfaces.d
root@ubuntu:/etc/network# more interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
address 10.77.77.60
netmask 255.255.255.0
gateway 10.77.77.2


root@ubuntu:/etc# more resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 8.8.8.8 #google DNS


User 생성 및 Password 설정
[root@localhost ~]#useradd shlee     # shlee User 생성
[root@localhost ~]#passwd shlee     # shlee 의 password 설정
Changing password for user ftpuser.
New UNIX password:
[root@localhost ~]#tail /etc/passwd
avahi:x:70:70:Avahi mDNS/DNS-SD Stack:/var/run/avahi-daemon:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
dovecot:x:97:97:Dovecot IMAP server:/usr/libexec/dovecot:/sbin/nologin
dovenull:x:380:378:Dovecot's unauthorized user:/usr/libexec/dovecot:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
tcpdump:x:72:72::/:/sbin/nologin
oprofile:x:16:16:Special user account to be used by OProfile:/var/lib/oprofile:/sbin/nologin
rhelvm1:x:1000:1000:rhelvm1:/home/rhelvm1:/bin/bash
postgres:x:26:26:PostgreSQL Server:/var/lib/pgsql:/bin/bash
shlee:x:1001:1001::/home/shlee:/bin/bash


Root User 패스워드 설정
testuser01@ubuntu:~$ sudo passwd root
[sudo] password for testuser01:
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
testuser01@ubuntu:~$ su -
Password:
root@ubuntu:~#


명령어 경로 확인
root@ubuntu:~# which ifconfig
/sbin/ifconfig
root@ubuntu:~# locate ifconfig
/sbin/ifconfig
/usr/share/man/de/man8/ifconfig.8.gz
/usr/share/man/fr/man8/ifconfig.8.gz
/usr/share/man/man8/ifconfig.8.gz
/usr/share/man/pt_BR/man8/ifconfig.8.gz
/usr/src/linux-headers-4.4.0-62/tools/hv/hv_set_ifconfig.sh


CD-ROM 마운트
[root@localhost ~]# mount /dev/cdrom /mnt
[root@localhost ~]# df
Filesystem            1K-blocks     Used Available Use% Mounted on
/dev/mapper/rhel-root  36805060 12893944  23911116  36% /
devtmpfs                1925356        0   1925356   0% /dev
tmpfs                   1941156       92   1941064   1% /dev/shm
tmpfs                   1941156     9176   1931980   1% /run
tmpfs                   1941156        0   1941156   0% /sys/fs/cgroup
/dev/sda1               1038336   178992    859344  18% /boot
/dev/fs1               41943040   787584  41155456   2% /gpfs
tmpfs                    388232       16    388216   1% /run/user/0
/dev/sr0                3704296  3704296         0 100% /run/media/root/RHEL-7.3 Server.x86_64


ISO 이미지 마운트
[root@localhost ~]# mount -o loop rhel7.3-x86_64.iso /mnt/isodevice

디렉토리별 용량 확인
[root@localhost /]# du -hs /usr
4.4G    /usr
[root@localhost /]# du -hsc /usr
4.4G    /usr
4.4G    total
[root@localhost /]# du -hsc /usr /home /bin
4.4G    /usr
4.6M    /home
0       /bin
4.4G    total


SCP 활용
ubuntu01(10.10.10.10) 에서 10.10.1.20 서버의 /home/testuser01 디렉토리에 파일 복사를 할 경우
root@ubuntu01:# scp sg3-utils-udev_1.40-0ubuntu1_all.deb  testuser01@10.10.10.20:/home/testuser01/
testuser01@10.10.10.20's password:
sg3-utils-udev_1.40-0ubuntu1_all.deb                                                               100% 5252     5.1KB/s   00:00

하위 디렉토리 모두 복사(-r 옵션)
root@ubuntu01:# scp -r /tmp testuser01@10.10.10.20:/tmp

시간 설정
root@ubuntu:~# timedatectl
      Local time: Sun 2017-11-19 19:13:54 KST
Universal time: Sun 2017-11-19 10:13:54 UTC
      RTC time: Sun 2017-11-19 10:13:54
      Time zone: Asia/Seoul (KST, +0900)
Network time on: yes
NTP synchronized: no
RTC in local TZ: no

root@ubuntu:~# timedatectl set-timezone Asia/Seoul

PATH 적용
.bashrc 파일 마지막 줄에 다음과 같이 입력
export PATH="/opt/anaconda3/bin:$PATH" 

2017년 11월 9일 목요일

Jupyter Notebook 설치 및 사용 - Ubuntu

IBM PowerAI toolkit 만 설치를 해도 python 2.7 이 자동으로 설치되고 기본적이 라이브러리들이 설치된다.
그래서 Jupyter notebook 도 그냥 실행이 된다. 다만 config 파일을 생성해서 접속 IP 부분을 "*" 로 수정해야 한다.

u0018670/@sys-89538:/$ whcih jupyter
/home/u0018670/.local/bin/jupyter
u0018670/@sys-89538:/$ jupyter notebook --generate-config​
Writing default config to: /home/u0018670/.jupyter/jupyter_notebook_config.py
u0018670/@sys-89538:/$ vi /home/u0018670/.jupyter/jupyter_notebook_config.py
....................
## The IP address the notebook server will listen on.
#c.NotebookApp.ip = 'localhost'
c.NotebookApp.ip = '*'
....................
u0018670/@sys-89538:/$ jupyter notebook
[W 03:12:17.412 NotebookApp] WARNING: The notebook server is listening on all IP addresses and not using encryption. This is not recommended.
[I 03:12:17.419 NotebookApp] Serving notebooks from local directory: /
[I 03:12:17.419 NotebookApp] 0 active kernels
[I 03:12:17.419 NotebookApp] The Jupyter Notebook is running at:
[I 03:12:17.419 NotebookApp]
http://[all
ip addresses on your system]:8888/?token=14bbfe40057c79cc5150369de56280394095aa32fde6eda5
[I 03:12:17.419 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 03:12:17.420 NotebookApp] No web browser found: could not locate runnable browser.
[C 03:12:17.420 NotebookApp]
    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://localhost:8888/?token=14bbfe40057c79cc5150369de56280394095aa32fde6eda5

위와 같이 되었다면, 웹브라우저를 통해 해당 서버의 IP주소:port 를 통해 웹브라우저에서 접속이 가능하다. 당연히 위의 token 을 password 를 입력해야한다.



실행된 후, 기본적인 python 코드를 작성 하기위해 new 버튼으로 파일을 한다.
jupyter notebook 따로 config 파일을 설정하지 않으면 실행 된 위치의 디렉토리에 파일을 생성하는데 이때 user 권한의 영향을 받게 되니 주의.
Jupyter notebook 을 통해 tensorflow 소스 코드 작성 및 수행을 위해서는 notebook 을 실행하는 세션에서 tensorflow 를 activate 시키고 notebook 을 실행해야 한다.

Hadoop, Spark 소스코드 building - Ubuntu

Hadoop 소스코드를 build 하기 위해 필요한 packages 들 중 다음 3가지는 기본.
java, maven, protobuf

-. java 설치
u0018670@sys-89986:~$ sudo apt-get install openjdk-8-jdk
u0018670@sys-89986:~$ java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-2ubuntu1.16.04.3-b11)
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)



다음 명령어로 java 설치된 상세 정보 확인
u0018670@sys-89986:~$ java -XshowSettings:properties -version
만약 여러 버전의 java 설치치 다음 명령어로 선택적 설정
u0018670@sys-89986:~$ sudo update-alternatives --config java

java 홈디렉토리 설정
u0018670@sys-89986:~$ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-ppc64el

-. maven 설치
다음 명령어로 설치가능 한 버전을 확인
u0018670@sys-89986:~$ aptitude show maven maven2 | egrep ^'(Package|Version)'
Package: maven
Version: 3.3.9-3
Package: maven2
u0018670@sys-89986:~$ sudo apt-get install maven
u0018670@sys-89986:~$ mvn -version
Apache Maven 3.3.9
Maven home: /usr/share/maven
Java version: 1.8.0_131, vendor: Oracle Corporation  ==> 해당 버전은 error
(Could not find artifact jdk.tools:jdk.tools:jar:1.8 at specified path /usr/lib/jvm/java-8-openjdk-ppc64el/jre/../lib/tools.jar -> [Help 1])
Java home: /usr/lib/jvm/java-8-openjdk-ppc64el/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.4.0-97-generic", arch: "ppc64le", family: "unix"
mvn 을 이용해서 build 중 상세 log 확인
u0018670@sys-89986:~$ mvn package -e -X


-.protobuf 설치
protobuf(Protocol BUffer compiler) 2.5.0 을 설치해야 하는데 ppc64le 환경에서 source 를 새로 build 해야 한다.
source build 를 위해 다음 필수 package 먼저 설치
u0018670@sys-89986:~$ sudo apt-get install cmake automake autoconf
그리고, git-hub 에서 source gathering
u0018670@sys-89986:~$ git clone --recursive https://github.com/ibmsoe/Protobuf.git
u0018670@sys-89986:/home/u0018670$ cd Protobuf
u0018670@sys-89986:/home/u0018670/Protobuf$ ./configure
u0018670@sys-89986:/home/u0018670/Protobuf$ make
u0018670@sys-89986:/home/u0018670/Protobuf$ make check
u0018670@sys-89986:/home/u0018670/Protobuf$ sudo make install
다음과 같이 나오면 설치 완료.
u0018670@sys-89986:~/Protobuf$ /usr/local/bin/protoc --version
libprotoc 2.5.0


만약 다음과 같은 error 가 날 경우,
u0018670@sys-89986:~/Protobuf$ /usr/local/bin/protoc --version
/usr/local/bin/protoc: error while loading shared libraries: libprotoc.so.8: cannot open shared object file: No such file or directory
아래를 수행
u0018670@sys-89986:~/Protobuf$ find /usr/local -name libprotoc.so.8
/usr/local/lib/libprotoc.so.8
u0018670@sys-89986:~/Protobuf$ export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

-. hadoop (v2.7.4) build
u0018670@sys-89986:~$ wget http://www-eu.apache.org/dist/hadoop/common/hadoop-2.7.4/hadoop-2.7.4-src.tar.gz
u0018670@sys-89986:~$ tar -zxvf hadoop-2.7.4-src.tar.gz
u0018670@sys-89986:~$ mvn clean package -Pdist -Pdoc -Psrc -Dtar -DskipTests -Dmaven.javadoc.skip=true

output 파일은,
u0018670@sys-89985:~$ cd /home/u0018670/hadoop-2.7.4-src/hadoop-dist/target/hadoop-2.7.4.tar.gz

-.spark (v1.6.3) build
u0018670@sys-89985:~$ git clone https://github.com/apache/spark
u0018670@sys-89985:~$ cd spark
u0018670@sys-89985:~/spark$ git checkout tags/v1.6.3
Checking out files: 100% (6361/6361), done.
Note: checking out 'tags/v1.6.3'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
  git checkout -b <new-branch-name>
HEAD is now at 1e86074... Preparing Spark release v1.6.3-rc2

spark build 중 sparkR 을 위한 package 를 먼저 설치
u0018670@sys-89985:~/spark$ sudo apt-get install r-base r-base-dev

다음 명령어로 spark build
u0018670@sys-89985:~/spark$ ./make-distribution.sh --name spark-ppc64le --tgz -Psparkr -Phive -Phive-thriftserver

output 파일은 spark 디렉토리안에 spark-1.6.3-bin-spark-ppc64le.tgz 으로 생성된다.

spark v2.0.0 은 dev 디렉토리 밑에 make-distribution.sh 파일 있다.
================================================================================

source build 중 생긴 error 의 대부분은 java 관련된 내용이다.
위 hadoop 과 spark 을 build 하면서 사용한 java 는,

java version "1.7.0_95"
OpenJDK Runtime Environment (IcedTea 2.6.4) (7u95-2.6.4-3)
OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)

openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)



*. 참고 페이지
https://hwengineer.blogspot.kr/2017/02/linux-on-powerppc64le-spark-sparkr.html