Hadoop, Spark 소스코드 building - Ubuntu

Hadoop 소스코드를 build 하기 위해 필요한 packages 들 중 다음 3가지는 기본.
java, maven, protobuf

-. java 설치
u0018670@sys-89986:~$ sudo apt-get install openjdk-8-jdk
u0018670@sys-89986:~$ java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-2ubuntu1.16.04.3-b11)
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)


다음 명령어로 java 설치된 상세 정보 확인
u0018670@sys-89986:~$ java -XshowSettings:properties -version
만약 여러 버전의 java 설치치 다음 명령어로 선택적 설정
u0018670@sys-89986:~$ sudo update-alternatives --config java

java 홈디렉토리 설정
u0018670@sys-89986:~$ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-ppc64el

-. maven 설치
다음 명령어로 설치가능 한 버전을 확인
u0018670@sys-89986:~$ aptitude show maven maven2 | egrep ^'(Package|Version)'
Package: maven
Version: 3.3.9-3
Package: maven2
u0018670@sys-89986:~$ sudo apt-get install maven
u0018670@sys-89986:~$ mvn -version
Apache Maven 3.3.9
Maven home: /usr/share/maven
Java version: 1.8.0_131, vendor: Oracle Corporation  ==> 해당 버전은 error
(Could not find artifact jdk.tools:jdk.tools:jar:1.8 at specified path /usr/lib/jvm/java-8-openjdk-ppc64el/jre/../lib/tools.jar -> [Help 1])
Java home: /usr/lib/jvm/java-8-openjdk-ppc64el/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.4.0-97-generic", arch: "ppc64le", family: "unix"
mvn 을 이용해서 build 중 상세 log 확인
u0018670@sys-89986:~$ mvn package -e -X

-.protobuf 설치
protobuf(Protocol BUffer compiler) 2.5.0 을 설치해야 하는데 ppc64le 환경에서 source 를 새로 build 해야 한다.
source build 를 위해 다음 필수 package 먼저 설치
u0018670@sys-89986:~$ sudo apt-get install cmake automake autoconf
그리고, git-hub 에서 source gathering
u0018670@sys-89986:~$ git clone --recursive https://github.com/ibmsoe/Protobuf.git
u0018670@sys-89986:/home/u0018670$ cd Protobuf
u0018670@sys-89986:/home/u0018670/Protobuf$ ./configure
u0018670@sys-89986:/home/u0018670/Protobuf$ make
u0018670@sys-89986:/home/u0018670/Protobuf$ make check
u0018670@sys-89986:/home/u0018670/Protobuf$ sudo make install
다음과 같이 나오면 설치 완료.
u0018670@sys-89986:~/Protobuf$ /usr/local/bin/protoc --version
libprotoc 2.5.0

만약 다음과 같은 error 가 날 경우,
u0018670@sys-89986:~/Protobuf$ /usr/local/bin/protoc --version
/usr/local/bin/protoc: error while loading shared libraries: libprotoc.so.8: cannot open shared object file: No such file or directory
아래를 수행
u0018670@sys-89986:~/Protobuf$ find /usr/local -name libprotoc.so.8
/usr/local/lib/libprotoc.so.8
u0018670@sys-89986:~/Protobuf$ export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

-. hadoop (v2.7.4) build
u0018670@sys-89986:~$ wget http://www-eu.apache.org/dist/hadoop/common/hadoop-2.7.4/hadoop-2.7.4-src.tar.gz
u0018670@sys-89986:~$ tar -zxvf hadoop-2.7.4-src.tar.gz
u0018670@sys-89986:~$ mvn clean package -Pdist -Pdoc -Psrc -Dtar -DskipTests -Dmaven.javadoc.skip=true

output 파일은,
u0018670@sys-89985:~$ cd /home/u0018670/hadoop-2.7.4-src/hadoop-dist/target/hadoop-2.7.4.tar.gz

-.spark (v1.6.3) build
u0018670@sys-89985:~$ git clone https://github.com/apache/spark
u0018670@sys-89985:~$ cd spark
u0018670@sys-89985:~/spark$ git checkout tags/v1.6.3
Checking out files: 100% (6361/6361), done.
Note: checking out 'tags/v1.6.3'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
  git checkout -b <new-branch-name>
HEAD is now at 1e86074... Preparing Spark release v1.6.3-rc2

spark build 중 sparkR 을 위한 package 를 먼저 설치
u0018670@sys-89985:~/spark$ sudo apt-get install r-base r-base-dev

다음 명령어로 spark build
u0018670@sys-89985:~/spark$ ./make-distribution.sh --name spark-ppc64le --tgz -Psparkr -Phive -Phive-thriftserver

output 파일은 spark 디렉토리안에 spark-1.6.3-bin-spark-ppc64le.tgz 으로 생성된다.

spark v2.0.0 은 dev 디렉토리 밑에 make-distribution.sh 파일 있다.
================================================================================
source build 중 생긴 error 의 대부분은 java 관련된 내용이다.
위 hadoop 과 spark 을 build 하면서 사용한 java 는,

java version "1.7.0_95"
OpenJDK Runtime Environment (IcedTea 2.6.4) (7u95-2.6.4-3)
OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)

openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)


*. 참고 페이지
https://hwengineer.blogspot.kr/2017/02/linux-on-powerppc64le-spark-sparkr.html