설치된 docker engine 확인 u0018670@sys-90572:~$ dpkg -l |grep docker ii docker.io 1.13.1-0ubuntu1~16.04.2 ppc64el Linux container runtime ii runc 1.0.0~rc2+docker1.13.1-0ubuntu1~16.04.1 ppc64el Open Container Project - runtime 아무런 docker image 가 없는 것을 확인 u0018670@sys-90572:~$ sudo docker images REPOSITORY TAG IMAGE ID CREATED SIZE 그래서, ppc64le/ubuntu docker image 를 실행시키면 latest 로 docker hub 로 부터 docker tag 가 latest 인 것을 download 후 docker 실행 u0018670@sys-90572:~$ sudo docker run -it ppc64le/ubuntu Unable to find image 'ppc64le/ubuntu:latest' locally latest: Pulling from ppc64le/ubuntu 586a07d977d9: Pull complete 2bf911559ec0: Pull complete 48999c060756: Pull complete fa5e5494889f: Pull complete 3bd4f80d03d3: Pull complete Digest: sha256:79d11934f87adb78614d5ca93092f6681ca154c2bd9df56a7027615c56d53e41 Status: Downloaded newer image for ppc64le/ubuntu:latest root@b916c38bbbb7:/# ls <== docker container 안 bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var root@b916c38bbbb7:/# exit exit 생성된 docker image 확인 u0018670@sys-90572:~$ sudo docker images REPOSITORY TAG IMAGE ID CREATED SIZE ppc64le/ubuntu latest 44c2eaa37868 3 days ago 150 MB 수행중인 container 확인 u0018670@sys-90572:~$ sudo docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES b916c38bbbb7 ppc64le/ubuntu "/bin/bash" 54 seconds ago Exited (0) 17 seconds ago epic_cori
수행중인 container 를 image 로 commit u0018670@sys-90727:~$ sudo docker commit 58d0d814ae31 ppc64le/ubuntu:latest sha256:4de2af599de3a7a0aa2351a9fcb6e9b8cbaa8f3a345d6fed03291477097da82d u0018670@sys-90727:~$ sudo docker images REPOSITORY TAG IMAGE ID CREATED SIZE ppc64le/ubuntu latest 4de2af599de3 About a minute ago 260MB ppc64le/ubuntu 16.04 44c2eaa37868 3 weeks ago 150MB 수행중인 container 삭제 u0018670@sys-90572:~$ sudo docker rm epic_cori epic_cori u0018670@sys-90572:~$ sudo docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES u0018670@sys-90572:~$ 한꺼번에 모든 container 삭제는 docker rm $(docker ps -a -q) 다음은 2개의 container 가 시작된 상황. (ppc64le/ubuntu 이미지로 eager_hugle, quirky_pike) u0018670@sys-90572:~$ sudo docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES eaf515d4a2c6 ppc64le/ubuntu "/bin/bash" 16 seconds ago Exited (0) 12 seconds ago eager_hugle 4e3e7338019b ppc64le/ubuntu "/bin/bash" About a minute ago Exited (0) 21 seconds ago quirky_pike container 수행 중이기에 삭제 되지 않는 docker image, rm 옵션으로 container 를 먼저 삭제 후 image 삭제 u0018670@sys-90572:~$ sudo docker rmi ppc64le/ubuntu Error response from daemon: conflict: unable to remove repository reference "ppc64le/ubuntu" (must force) - container 4e3e7338019b is using its referenced image 44c2eaa37868 u0018670@sys-90572:~$ docker hub 에 생성된 이미지 upload u0018670@sys-90461:~/data$ sudo docker images REPOSITORY TAG IMAGE ID CREATED SIZE leearmy/cudnn6 ppc64le de632816acda About an hour ago 6.58GB leearmy/ubuntu 16.04 23296ffbcbfc 4 hours ago 359MB ppc64le/ubuntu 16.04 44c2eaa37868 3 days ago 150MB 다음 명령어를 수행하기 위해서는 먼저 https://hub.docker.com/ 에 계정과 repository 를 생성하여야 함 u0018670@sys-90461:~/data$ sudo docker login --username=leearmy Password: Login Succeeded u0018670@sys-90461:~/data$ sudo docker push leearmy/cudnn6:ppc64le The push refers to a repository [docker.io/leearmy/cudnn6] 026ec1f6d915: Pushed c963a10dda0d: Pushed 4a08cdc31ed0: Pushed 49076433d00c: Pushed 8788fee75149: Pushed d9561c8b4bf3: Pushed 54ac221998f0: Pushed b10475677e3b: Pushed adf3a2526761: Pushed e8449de7ace1: Pushed 0c5b7df1df7c: Pushed b1dc141f6073: Mounted from ppc64le/ubuntu 448bb1e132fc: Mounted from ppc64le/ubuntu e7f4651b678d: Mounted from ppc64le/ubuntu ee8f1c150908: Mounted from ppc64le/ubuntu 258cc4d5502c: Mounted from ppc64le/ubuntu ppc64le: digest: sha256:3df7ee55f48c08fe7dc836c60053867dac28ada8b7eb9b9413b09e5361e4937c size: 3690 다음과 같이 https://hub.docker.com/ 에서 올려진 이미지 확인
docker hub 에 올려진 이미지는 다른 서버에서 pull u0018670@sys-90572:~$ sudo docker images REPOSITORY TAG IMAGE ID CREATED SIZE ppc64le/ubuntu latest 44c2eaa37868 3 days ago 150 MB u0018670@sys-90572:~$ sudo docker pull leearmy/cudnn6:ppc64le ppc64le: Pulling from leearmy/cudnn6 586a07d977d9: Already exists 2bf911559ec0: Already exists 48999c060756: Already exists fa5e5494889f: Already exists 3bd4f80d03d3: Already exists 3d482c0c4d09: Pull complete 9e891e52327a: Pull complete a40daf5d9d7f: Pull complete 2ca681ce2118: Pull complete 111379f9235b: Pull complete 9db8b0fb9301: Pull complete b2b39482e9b8: Pull complete 93f5e3ec7d59: Pull complete 38e0ece0a1d4: Pull complete dbc7427ab4ea: Pull complete f1cc2b939ac9: Pull complete Digest: sha256:3df7ee55f48c08fe7dc836c60053867dac28ada8b7eb9b9413b09e5361e4937c Status: Downloaded newer image for leearmy/cudnn6:ppc64le u0018670@sys-90572:~$ sudo docker images REPOSITORY TAG IMAGE ID CREATED SIZE leearmy/cudnn6 ppc64le de632816acda 20 hours ago 6.58 GB ppc64le/ubuntu latest 44c2eaa37868 3 days ago 150 MBu0018670@sys-90572:~$
nvidia-docker 가 설치될 환경은 -.ubuntu 16.04 -.CUDA 8.0 + cudnn 6.0.21 먼저 docker engine 설치 test_svr@ubuntu:~$ test_svr@ubuntu:~$ sudo vi /etc/apt/sources.list.d/xenial-docker.list deb http://ftp.unicamp.br/pub/ppc64el/ubuntu/16_04/docker-17.06.0-ce-ppc64el/ xenial main test_svr@ubuntu:~$ sudo apt-get update test_svr@ubuntu:~$ sudo apt-get install docker-ce test_svr@ubuntu:~$ dpkg -l |grep -i docker ii docker 1.5-1 ppc64el System tray for KDE3/GNOME2 docklet applications ii docker-ce 17.06.0~ce-0~ubuntu ppc64el Docker: the open-source application container engine
docker engine 이 설치되었으면 nvidia-docker 설치 test_svr@ubuntu:~$curl -s-L https://nvidia.github.io/nvidia-docker/gpgkey | \ > sudo apt-key add - OK test_svr@ubuntu:~$ curl -s-L https://nvidia.github.io/nvidia-docker/ubuntu16.04/ppc64el/nvidia-docker.list | \ > sudo tee /etc/apt/sources.list.d/nvidia-docker.list deb https://nvidia.github.io/nvidia-docker/ubuntu16.04/ppc64el / test_svr@ubuntu:~$ sudo apt-get update Get:1 file:/var/cuda-repo-8-0-local-ga2v2 InRelease Ign:1 file:/var/cuda-repo-8-0-local-ga2v2 InRelease Get:2 file:/var/nvidia-driver-local-repo-tr6 InRelease Ign:2 file:/var/nvidia-driver-local-repo-tr6 InRelease Get:3 file:/var/cuda-repo-8-0-local-ga2v2 Release [574 B] Get:4 file:/var/nvidia-driver-local-repo-tr6 Release [574 B] Get:3 file:/var/cuda-repo-8-0-local-ga2v2 Release [574 B] Get:4 file:/var/nvidia-driver-local-repo-tr6 Release [574 B] Get:7 http://ports.ubuntu.com/ubuntu-ports xenial-security InRelease [102 kB] Hit:8 http://kr.ports.ubuntu.com/ubuntu-ports xenial InRelease Get:9 https://nvidia.github.io/nvidia-docker/ubuntu16.04/ppc64el InRelease [1,098 B] Get:10 http://kr.ports.ubuntu.com/ubuntu-ports xenial-updates InRelease [102 kB] Get:11 https://nvidia.github.io/nvidia-docker/ubuntu16.04/ppc64el Packages [704 B] Ign:12 http://ftp.unicamp.br/pub/ppc64el/ubuntu/16_04/docker-17.06.0-ce-ppc64el xenial InRelease Get:13 http://kr.ports.ubuntu.com/ubuntu-ports xenial-backports InRelease [102 kB] Hit:14 http://ftp.unicamp.br/pub/ppc64el/ubuntu/16_04/docker-17.06.0-ce-ppc64el xenial Release Ign:15 http://ftp.unicamp.br/pub/ppc64el/ubuntu/16_04/docker-17.06.0-ce-ppc64el xenial Release.gpg Fetched 308 kB in 2s (111 kB/s) Reading package lists... Done W: The repository 'http://ftp.unicamp.br/pub/ppc64el/ubuntu/16_04/docker-17.06.0-ce-ppc64el xenial Release' is not signed. N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use. N: See apt-secure(8) manpage for repository creation and user configuration details. test_svr@ubuntu:~$ sudo apt-get install -y nvidia-docker Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: nvidia-docker 0 upgraded, 1 newly installed, 0 to remove and 84 not upgraded. Need to get 2,267 kB of archives. After this operation, 16.0 MB of additional disk space will be used. Get:1 https://nvidia.github.io/nvidia-docker/ubuntu16.04/ppc64el nvidia-docker 1.0.1-1 [2,267 kB] Fetched 2,267 kB in 2s (964 kB/s) Selecting previously unselected package nvidia-docker. (Reading database ... 122052 files and directories currently installed.) Preparing to unpack .../nvidia-docker_1.0.1-1_ppc64el.deb ... Unpacking nvidia-docker (1.0.1-1) ... Processing triggers for ureadahead (0.100.0-19) ... ureadahead will be reprofiled on next reboot Setting up nvidia-docker (1.0.1-1) ... Configuring user Setting up permissions Processing triggers for ureadahead (0.100.0-19) ... test_svr@ubuntu:~$
framework 버전 확인 anaconda 를 통해 pip install 로 설치된 tensorflow 1.3.1 로 동작 확인 u0018670@sys-90257:~$ env | grep PATH PATH=/home/u0018670/bin:/home/u0018670/.local/bin:/home/u0018670/anaconda2/bin:/opt/ibm/xlC/current/bin:/opt/ibm/xlf/current/bin:/opt/at10.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games u0018670@sys-90257:~$ python Python 2.7.13 |Anaconda, Inc.| (default, Sep 15 2017, 20:54:27) [GCC 4.8.4] on linux2 Type "help", "copyright", "credits" or "license" for more information. Anaconda is brought to you by Continuum Analytics. Please check out: http://continuum.io/thanks and https://anaconda.org >>> import tensorflow as tf >>> print(tf.__version__) 1.3.1 >>> powerAI toolkit 에 포함된 tensorflow 로 동작 u0018670@sys-90257:~$ . /opt/DL/tensorflow/bin/tensorflow-activate u0018670@sys-90257:~$ env | grep PATH LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64:/usr/local/cuda-8.0/extras/CUPTI/lib64:/opt/DL/tensorflow/lib PATH=/home/u0018670/bin:/home/u0018670/.local/bin:/home/u0018670/anaconda2/bin:/opt/ibm/xlC/current/bin:/opt/ibm/xlf/current/bin:/opt/at10.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/opt/DL/tensorflow/bin PYTHONPATH=/opt/DL/tensorflow/lib/python2.7/site-packages u0018670@sys-90257:~$ python Python 2.7.13 |Anaconda, Inc.| (default, Sep 15 2017, 20:54:27) [GCC 4.8.4] on linux2 Type "help", "copyright", "credits" or "license" for more information. Anaconda is brought to you by Continuum Analytics. Please check out: http://continuum.io/thanks and https://anaconda.org >>> import tensorflow as tf >>> print(tf.__version__) 1.1.0 >>> python system path 확인 u0018670@sys-90257:~$ python3 Python 3.5.2 (default, Nov 23 2017, 16:37:01) [GCC 5.4.0 20160609] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path ['', '/usr/lib/python35.zip', '/usr/lib/python3.5', '/usr/lib/python3.5/plat-powerpc64le-linux-gnu', '/usr/lib/python3.5/lib-dynload', '/home/u0018670/.local/lib/python3.5/site-packages', '/usr/local/lib/python3.5/dist-packages', '/usr/lib/python3/dist-packages'] >>> native python3 설치 후 link 생성 native python3 를 다음과 같이 설치 후, u0018670@sys-90257:~$ sudo apt-get install python3 jupyter 도 pip install 설치 u0018670@sys-90254:~$ pip list DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning. bleach (2.1.1) decorator (4.1.2) entrypoints (0.2.3) html5lib (1.0b10) ipykernel (4.6.1) ipython (6.2.1) ipython-genutils (0.2.0) ipywidgets (7.0.5) jedi (0.11.0) Jinja2 (2.10) jsonschema (2.6.0) jupyter (1.0.0) jupyter-client (5.1.0) jupyter-console (5.2.0) jupyter-core (4.4.0) MarkupSafe (1.0) 그런데 jupyter 가 실행이 안됨. u0018670@sys-90257:~$ jupyter -bash: /home/u0018670/.local/bin/jupyter: /usr/bin/python: bad interpreter: No such file or directory 안되는 이유는 /usr/bin/python 파일이 없어서 그래서 link 시켜주면 잘 됨. u0018670@sys-90257:~$ sudo ln -s /usr/bin/python3 /usr/bin/python u0018670@sys-90257:~$ python Python 3.5.2 (default, Nov 23 2017, 16:37:01) [GCC 5.4.0 20160609] on linux Type "help", "copyright", "credits" or "license" for more information. >>> u0018670@sys-90257:~$ jupyter usage: jupyter [-h] [--version] [--config-dir] [--data-dir] [--runtime-dir] [--paths] [--json] [subcommand] theano 기본 테스트 root@3498b2e65eb1:/tmp# python Python 3.5.4 |Anaconda, Inc.| (default, Sep 19 2017, 14:42:07) [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import theano >>> import theano.tensor as T >>> import numpy as np >>> x = T.dscalar('x') >>> y = x ** 2 >>> f = theano.function(inputs=[x], outputs=y) >>> f(1) array(1.0) >>> f(2) array(4.0) >>> f(3) array(9.0) >>> jupyter notebook 에서는 다음과 같이 출력 확인
/opt/at10.0 경로 IBM 패키지 중 advance-toolchain, xlc 등이 설치되는 경로 여기에 pip3 도 있고 gcc 도 있고... 과감히 삭제 u0018670@sys-90257:/opt/at10.0/bin$ dpkg -l |grep advance ii advance-toolchain-at10.0-devel 10.0-2 ppc64el Advance Toolchain ii advance-toolchain-at10.0-mcore-libs 10.0-2 ppc64el Advance Toolchain ii advance-toolchain-at10.0-perf 10.0-2 ppc64el Advance Toolchain ii advance-toolchain-at10.0-runtime 10.0-2 ppc64el Advance Toolchain
Anaconda2 로 python 2.7 이 설치된 환경에서 root 에 영향이 없이, 다시말해 Anaconda3 를 설치하지 않아도 python 3.5 를 사용하기 위해 다음과 같이 수행 현재 설치된 anaconda2 확인 u0018670@sys-90257:~$ which python /home/u0018670/anaconda2/bin/python jupyter kernal 확인 u0018670@sys-90257:~$ jupyter kernelspec list Available kernels: python2 /home/u0018670/anaconda2/share/jupyter/kernels/python2 jupyter notebook 을 실행해봐도 Python2 만 확인
u0018670@sys-90257:~$ conda update conda Fetching package metadata ........... Solving package specifications: . # All requested packages already installed. # packages in environment at /home/u0018670/anaconda2: # conda 4.3.27 py27_0 python 3.5 환경으로 업데이트 u0018670@sys-90257:~$ conda create -n py35 python=3.5 anaconda Fetching package metadata ........... Solving package specifications: . Package plan for installation in environment /home/u0018670/anaconda2/envs/py35: The following NEW packages will be INSTALLED: alabaster: 0.7.10-py35_0 anaconda: 5.0.0-py35h419c926_0 anaconda-client: 1.6.3-py35_0 anaconda-project: 0.6.0-py35_0 asn1crypto: 0.22.0-py35_0 astroid: 1.5.3-py35_0 astropy: 2.0.1-np113py35_1 babel: 2.5.0-py35_0 backports: 1.0-py35_0 beautifulsoup4: 4.6.0-py35_0 bitarray: 0.8.1-py35_0 ...................... Proceed ([y]/n)? y python-3.5.4-0 100% |#####################################################################################| Time: 0:00:01 14.46 MB/s alabaster-0.7. 100% |#####################################################################################| Time: 0:00:00 15.39 MB/s asn1crypto-0.2 100% |#####################################################################################| Time: 0:00:00 16.71 MB/s backports-1.0- 100% |#####################################################################################| Time: 0:00:00 3.98 MB/s beautifulsoup4 100% |#####################################################################################| Time: 0:00:00 18.22 MB/s bitarray-0.8.1 100% |#####################################################################################| Time: 0:00:00 25.68 MB/s boto-2.48.0-py 100% |#####################################################################################| Time: 0:00:00 3.83 MB/s ...................... anaconda-5.0.0 100% |#####################################################################################| Time: 0:00:00 12.86 MB/s # # To activate this environment, use: # > source activate py35 # # To deactivate an active environment, use: # > source deactivate # u0018670@sys-90257:~$ source activate py35 (py35) u0018670@sys-90257:~$ jupyter notebook
(py35) u0018670@sys-90257:~/anaconda2$ which python /home/u0018670/anaconda2/envs/py35/bin/python (py35) u0018670@sys-90257:~/anaconda2$ source deactivate u0018670@sys-90257:~/anaconda2$
추가적으로, 특정 버전의 python 패키지를 설치할때는 다음과 같이 그냥 keras 설치를 해버리면 최신 버전이 설치됨. nimbix@JARVICENAE-0A0A183D:~$ pip install keras Collecting keras Downloading Keras-2.1.2-py2.py3-none-any.whl (304kB) 100% |████████████████████████████████| 307kB 4.3MB/s Requirement already satisfied: scipy>=0.14 in ./anaconda3/lib/python3.6/site-packages (from keras) Requirement already satisfied: six>=1.9.0 in ./anaconda3/lib/python3.6/site-packages (from keras) Requirement already satisfied: pyyaml in ./anaconda3/lib/python3.6/site-packages (from keras) Requirement already satisfied: numpy>=1.9.1 in ./anaconda3/lib/python3.6/site-packages (from keras) Installing collected packages: keras Successfully installed keras-2.1.2
그런데, nimbix@JARVICENAE-0A0A183D:~$ pip install keras==1.2 Collecting keras==1.2 Downloading Keras-1.2.0.tar.gz (167kB) 100% |████████████████████████████████| 174kB 6.4MB/s Collecting theano (from keras==1.2) Downloading Theano-1.0.1.tar.gz (2.8MB) 100% |████████████████████████████████| 2.8MB 506kB/s Requirement already satisfied: pyyaml in ./anaconda3/lib/python3.6/site-packages (from keras==1.2) Requirement already satisfied: six in ./anaconda3/lib/python3.6/site-packages (from keras==1.2) Requirement already satisfied: numpy>=1.9.1 in ./anaconda3/lib/python3.6/site-packages (from theano->keras==1.2) Requirement already satisfied: scipy>=0.14 in ./anaconda3/lib/python3.6/site-packages (from theano->keras==1.2) Building wheels for collected packages: keras, theano Running setup.py bdist_wheel for keras ... done Stored in directory: /home/nimbix/.cache/pip/wheels/f8/2c/8e/27d952128220b1acf6fcd5e692269c6e8d60744c15fd160c2c Running setup.py bdist_wheel for theano ... done Stored in directory: /home/nimbix/.cache/pip/wheels/46/a2/7d/b4cac381d5151daa9f9e0b3e4e4b65edaea6355ae296c97cf2 Successfully built keras theano Installing collected packages: theano, keras Found existing installation: Keras 2.1.2 Uninstalling Keras-2.1.2: Successfully uninstalled Keras-2.1.2 Successfully installed keras-1.2.0 theano-1.0.1
기존 설치된 cuda 9.0 삭제를 예시 u0018670@sys-89985:~$ dpkg -l |grep cuda ri cuda-9-0 9.0.176-1 ppc64el CUDA 9.0 meta-package ii cuda-command-line-tools-9-0 9.0.176-1 ppc64el CUDA command-line tools ii cuda-core-9-0 9.0.176-1 ppc64el CUDA core tools ii cuda-cublas-8-0 8.0.61.2-1 ppc64el CUBLAS native runtime libraries ii cuda-cublas-9-0 9.0.176-1 ppc64el CUBLAS native runtime libraries ii cuda-cublas-dev-9-0 9.0.176-1 ppc64el CUBLAS native dev links, headers ii cuda-cudart-8-0 8.0.61-1 ppc64el CUDA Runtime native Libraries ii cuda-cudart-9-0 9.0.176-1 ppc64el CUDA Runtime native Libraries ii cuda-cudart-dev-9-0 9.0.176-1 ppc64el CUDA Runtime native dev links, headers ii cuda-cufft-8-0 8.0.61-1 ppc64el CUFFT native runtime libraries ii cuda-cufft-9-0 9.0.176-1 ppc64el CUFFT native runtime libraries ii cuda-cufft-dev-9-0 9.0.176-1 ppc64el CUFFT native dev links, headers ...
u0018670@sys-89985:~$ sudo apt-get --purge remove cuda Reading package lists... Done Building dependency tree Reading state information... Done Package 'cuda' is not installed, so not removed The following packages were automatically installed and are no longer required: acpid bbswitch-dkms bumblebee cuda-9-0 cuda-command-line-tools-9-0 cuda-core-9-0 cuda-cublas-9-0 cuda-cublas-dev-9-0 cuda-cudart-9-0 cuda-cudart-dev-9-0 cuda-cufft-9-0 cuda-cufft-dev-9-0 cuda-curand-9-0 cuda-curand-dev-9-0 cuda-cusolver-9-0 cuda-cusolver-dev-9-0 cuda-cusparse-9-0 cuda-cusparse-dev-9-0 cuda-documentation-9-0 cuda-driver-dev-9-0 cuda-drivers cuda-libraries-9-0 cuda-libraries-dev-9-0 cuda-license-9-0 cuda-misc-headers-9-0 cuda-npp-9-0 cuda-npp-dev-9-0 cuda-nvgraph-9-0 cuda-nvgraph-dev-9-0 cuda-nvml-dev-9-0 cuda-nvrtc-9-0 cuda-nvrtc-dev-9-0 cuda-runtime-9-0 cuda-samples-9-0 cuda-toolkit-9-0 cuda-visual-tools-9-0 dkms libcuda1-384 libjansson4 libxfont1 libxkbfile1 libxnvctrl0 nvidia-384 nvidia-384-dev nvidia-modprobe nvidia-settings policykit-1-gnome python3-xkit screen-resolution-extra x11-xkb-utils xfonts-base xfonts-encodings xfonts-utils xserver-common xserver-xorg-core xserver-xorg-legacy Use 'sudo apt autoremove' to remove them. 0 upgraded, 0 newly installed, 0 to remove and 86 not upgraded. 다음 command 를 통해 삭제 수행. u0018670@sys-89985:~$ sudo apt autoremove Reading package lists... Done Building dependency tree Reading state information... Done The following packages will be REMOVED: acpid bbswitch-dkms bumblebee cuda-9-0 cuda-command-line-tools-9-0 cuda-core-9-0 cuda-cublas-9-0 cuda-cublas-dev-9-0 cuda-cudart-9-0 cuda-cudart-dev-9-0 cuda-cufft-9-0 cuda-cufft-dev-9-0 cuda-curand-9-0 cuda-curand-dev-9-0 cuda-cusolver-9-0 cuda-cusolver-dev-9-0 cuda-cusparse-9-0 cuda-cusparse-dev-9-0 cuda-documentation-9-0 cuda-driver-dev-9-0 cuda-drivers cuda-libraries-9-0 cuda-libraries-dev-9-0 cuda-license-9-0 cuda-misc-headers-9-0 cuda-npp-9-0 cuda-npp-dev-9-0 cuda-nvgraph-9-0 cuda-nvgraph-dev-9-0 cuda-nvml-dev-9-0 cuda-nvrtc-9-0 cuda-nvrtc-dev-9-0 cuda-runtime-9-0 cuda-samples-9-0 cuda-toolkit-9-0 cuda-visual-tools-9-0 dkms libcuda1-384 libjansson4 libxfont1 libxkbfile1 libxnvctrl0 nvidia-384 nvidia-384-dev nvidia-modprobe nvidia-settings policykit-1-gnome python3-xkit screen-resolution-extra x11-xkb-utils xfonts-base xfonts-encodings xfonts-utils xserver-common xserver-xorg-core xserver-xorg-legacy 0 upgraded, 0 newly installed, 56 to remove and 85 not upgraded. After this operation, 2,236 MB disk space will be freed. Do you want to continue? [Y/n] y (Reading database ... 190184 files and directories currently installed.) Removing cuda-9-0 (9.0.176-1) ... Removing cuda-runtime-9-0 (9.0.176-1) ... Removing cuda-drivers (384.81-1) ... Removing nvidia-384-dev (384.81-0ubuntu1) ... ... u0018670@sys-89985:~$ sudo apt-get clean 추가로, CUDA 8.0 을 설치하고 싶은데 NVIDIA 사이트에는 최신인 CUDA 9.0 링크 제공 https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=ppc64le&target_distro=Ubuntu&target_version=1604&target_type=debnetwork 그냥 apt-get 으로 설치. u0018670@sys-89985:~$ sudo apt-get install cuda-8-0 Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: acpid bbswitch-dkms bumblebee cuda-command-line-tools-8-0 cuda-core-8-0 cuda-cublas-dev-8-0 cuda-cudart-dev-8-0 cuda-cufft-dev-8-0 cuda-curand-dev-8-0 cuda-cusolver-dev-8-0 cuda-cusparse-dev-8-0 cuda-documentation-8-0 cuda-driver-dev-8-0 cuda-drivers cuda-misc-headers-8-0 cuda-npp-8-0 cuda-npp-dev-8-0 cuda-nvgraph-8-0 cuda-nvgraph-dev-8-0 cuda-nvml-dev-8-0 cuda-nvrtc-8-0 cuda-nvrtc-dev-8-0 cuda-runtime-8-0 cuda-samples-8-0 cuda-toolkit-8-0 cuda-visual-tools-8-0 dkms libcuda1-384 libjansson4 libxfont1 libxkbfile1 libxnvctrl0 nvidia-384 nvidia-384-dev nvidia-modprobe nvidia-settings policykit-1-gnome python3-xkit screen-resolution-extra x11-xkb-utils xfonts-base xfonts-encodings xfonts-utils xserver-common xserver-xorg-core xserver-xorg-legacy Suggested packages: ......
아니면, 다음 link 를 통해 다운로드 https://developer.nvidia.com/cuda-toolkit-archive
NFS 서버(10.10.10.10), NFS 클라이언트(20.20.20.20) 서버(10.10.10.10)에서 NFS 관련 package 설치 root@ubuntu:~#apt-get install nfs-common nfs-kernel-server rpcbind
/data/nfs 디렉토리 생성 후 chmod 777 권한 변경 root@ubuntu:/data$ ls -al total 28 drwxr-xr-x 4 root root 4096 Nov 19 21:41 . drwxr-xr-x 24 root root 4096 Nov 19 18:02 .. drwx------ 2 root root 16384 Nov 19 18:02 lost+found drwxrwxrwx 2 root root 4096 Nov 19 21:41 nfs
nfs 설정 파일에 공유할 폴더와 설정 추가 root@ubuntu:~#vi /etc/exports # /etc/exports: the access control list for filesystems which may be exported # to NFS clients. See exports(5). # # Example for NFSv2 and NFSv3: # /srv/homes hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check) # # Example for NFSv4: # /srv/nfs4 gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check) # /srv/nfs4/homes gss/krb5i(rw,sync,no_subtree_check) # /data/nfs *(rw,sync,no_subtree_check) root@ubuntu:~#service nfs-kernel-server restart root@ubuntu:~#service rpcbind stop root@ubuntu:~#service rpcbind start
클라이언트(20.20.20.20)에서 nfs-common 패키지 설치 root@ubuntu:~#apt-get install nfs-common
설치 환경 root@ubuntu:~# more /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=16.04 DISTRIB_CODENAME=xenial DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS" 먼저, iSCSI initiator 설치 root@ubuntu:~# sudo apt-get install open-iscsi root@ubuntu:~# dpkg -l |grep iscsi ii open-iscsi 2.0.873+git0.3b4b4500-14ubuntu3.3 amd64 iSCSI initiator tools initiator 가 설치되면 다음과 같이 디렉토리에서 iqn 정보 확인 가능 root@ubuntu:/etc/iscsi# pwd /etc/iscsi root@ubuntu:/etc/iscsi# ls -al total 32 drwxr-xr-x 4 root root 4096 Nov 19 17:27 . drwxr-xr-x 89 root root 4096 Nov 19 17:06 .. -rw------- 1 root root 349 Nov 7 21:03 initiatorname.iscsi -rw------- 1 root root 11788 Nov 19 17:27 iscsid.conf drw------- 4 root root 4096 Nov 19 17:32 nodes drw------- 4 root root 4096 Nov 8 16:12 send_targets root@ubuntu:/etc/iscsi# more initiatorname.iscsi ## DO NOT EDIT OR REMOVE THIS FILE! ## If you remove this file, the iSCSI daemon will not start. ## If you change the InitiatorName, existing access control lists ## may reject this initiator. The InitiatorName must be unique ## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames. InitiatorName=iqn.1993-08.org.debian:01:aeff61f6e0e3 iscsid.conf 파일을 다음과 같이 수정
이제 스토리지(Storwize IBM V7000)에서 iSCSI 환경 설정, 볼륨 생성 및 해당 서버에 볼륨 할당
iSCSI 볼륨을 사용하기 위해서는 네트워크 - 이더넷 포트 메뉴에서 IP 설정을 해주어야 한다. 당연히 V7000 GUI 접속을 위한 management IP 와는 다른 IP를 설정하는데 UTP 케이블은 물리적으로 동일한 port 를 사용한다. 스토리지에서 호스트 생성시 서버의 iqn 정보를 입력하고 생성한다.
스토리지 볼륨(50GB) 생성된 호스트에 할당
이제 다시 서버에서 스토리지 정보 인식. 스토리지에서 설정한 iSCSI 용 IP 입력 root@ubuntu:~# iscsiadm -m discovery -t st -p 9.187.227.230 9.187.227.230:3260,1 iqn.1986-03.com.ibm:2145.stg-v7000.node1 root@ubuntu:~# iscsiadm -m discovery -t st -p 9.187.227.240 9.187.227.240:3260,1 iqn.1986-03.com.ibm:2145.stg-v7000.node2
스토리지 연결. 다음과 같이 successful 이 나오면 성공 root@ubuntu:~# iscsiadm -m node --login Logging in to [iface: default, target: iqn.1986-03.com.ibm:2145.stg-v7000.node2, portal: 9.187.227.240,3260] (multiple) Logging in to [iface: default, target: iqn.1986-03.com.ibm:2145.stg-v7000.node1, portal: 9.187.227.230,3260] (multiple) Login to [iface: default, target: iqn.1986-03.com.ibm:2145.stg-v7000.node2, portal: 9.187.227.240,3260] successful. Login to [iface: default, target: iqn.1986-03.com.ibm:2145.stg-v7000.node1, portal: 9.187.227.230,3260] successful. 그리고, dmesg 로 연결된 스토리지 볼륨 확인 root@ubuntu:~# dmesg | tail -100 | grep sd [ 5.273551] EXT4-fs (sda1): re-mounted. Opts: errors=remount-ro [ 7.079363] Adding 4191228k swap on /dev/sda5. Priority:-1 extents:1 across:4191228k FS [597769.393140] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 39s! [vmtoolsd:985] [597769.569910] CPU: 0 PID: 985 Comm: vmtoolsd Not tainted 4.4.0-62-generic #83-Ubuntu [953286.712454] sd 4:0:0:0: [sdb] 104857600 512-byte logical blocks: (53.7 GB/50.0 GiB) [953286.732047] sd 4:0:0:0: [sdb] Write Protect is off [953286.732051] sd 4:0:0:0: [sdb] Mode Sense: 97 00 10 08 [953286.738440] sd 4:0:0:0: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA [953286.738663] sd 3:0:0:0: [sdc] 104857600 512-byte logical blocks: (53.7 GB/50.0 GiB) [953286.746023] sd 3:0:0:0: [sdc] Write Protect is off [953286.746027] sd 3:0:0:0: [sdc] Mode Sense: 97 00 10 08 [953286.746522] sd 3:0:0:0: [sdc] Write cache: disabled, read cache: enabled, supports DPO and FUA [953286.808293] sd 3:0:0:0: Attached scsi generic sg2 type 0 [953286.808727] sd 4:0:0:0: Attached scsi generic sg3 type 0 [953286.857392] sd 3:0:0:0: [sdc] Attached SCSI disk [953286.857563] sd 4:0:0:0: [sdb] Attached SCSI disk 스토리지의 컨트롤러가 2개이여서 sdb, sdc 로 볼륨 확인. 이제 multipath 구성을 위해 multipath-tool package 설치 root@ubuntu:~# apt-get install multipath-tools 설치후 다음과 같이 스토리지 볼륨을 확인 할 수 있으나, root@ubuntu:~# multipath -ll 36005076802820c032000000000000055 dm-0 IBM,2145 size=50G features='1 queue_if_no_path' hwhandler='0' wp=rw |-+- policy='round-robin 0' prio=50 status=active | `- 3:0:0:0 sdc 8:32 active ready running `-+- policy='round-robin 0' prio=10 status=enabled `- 4:0:0:0 sdb 8:16 active ready running /etc/multipath.conf 파일을 수정하여 볼륨을 alias 정의 후 사용. 기본적으로 multipath.conf 을 없으니 vi 로 파일 생성 후 다음과 같이 수정. multipaths { multipath { wwid 36005076802820c032000000000000055 alias vol01 path_grouping_policy multibus path_selector "round-robin 0" failback manual rr_weight priorities no_path_retry 5 } } multipath 의 자세한 내용은 다음 링크 또는 각 스토리지 벤더에서 제공하는 방법을 참고. https://help.ubuntu.com/lts/serverguide/multipath-dm-multipath-config-file.html root@ubuntu:~# service multipathd restart root@ubuntu:~# multipath -ll vol01 (36005076802820c032000000000000055) dm-0 IBM,2145 size=50G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=30 status=active |- 3:0:0:0 sdc 8:32 active ready running `- 4:0:0:0 sdb 8:16 active ready running 생성된 볼륨 포멧 및 마운트 root@ubuntu:~# fdisk -l |grep -i vol01 Disk /dev/mapper/vol01: 50 GiB, 53687091200 bytes, 104857600 sectors
