At Apache: Big Data North America in May 2017, I'll show you how much the situation has changed since I talked about Apache Hadoop not currently supporting Java 9 and that's a lot of work.
Presentation material at Hadoop source code reading in March 2017 (Japanese)
Let's compile it on your MacBook Pro (High Sierra). Try with the following version.
mbpaa:hadoop$ java -version
java version "9.0.1"
Java(TM) SE Runtime Environment (build 9.0.1+11)
Java HotSpot(TM) 64-Bit Server VM (build 9.0.1+11, mixed mode)
It's my first time to touch Java 9 after its official release.
Revert HADOOP-14986
Hmmm, I was falling like I had never seen before ...
mbpaa:hadoop$ mvn install -DskipTests
[INFO] Scanning for projects...
(snip)
[INFO] --- animal-sniffer-maven-plugin:1.16:check (signature-check) @ hadoop-annotations ---
[INFO] Checking unresolved references to org.codehaus.mojo.signature:java18:1.0
[ERROR] /Users/ajisaka/git/hadoop/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/RootDocProcessor.java:56: Undefined reference: com.sun.javadoc.RootDoc
[ERROR] /Users/ajisaka/git/hadoop/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/RootDocProcessor.java:104: Undefined reference: com.sun.javadoc.Doc
[ERROR] /Users/ajisaka/git/hadoop/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/RootDocProcessor.java:106: Undefined reference: com.sun.javadoc.Doc
Apparently, the animal-sniffer-maven-plugin added in HADOOP-14986. Enforce JDK limitations doesn't work in Java 9. I will try to revert at hand for the time being.
mbpaa:hadoop$ git revert b50def36970e0afa65bd3fd3f40d8c4f81119200
Apply HADOOP-12760
The next error is here.
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hadoop-common: Compilation failure: Compilation failure:
[ERROR] /Users/ajisaka/git/hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java:[333,17] cannot find symbol
[ERROR] symbol: class Cleaner
[ERROR] location: package sun.misc
[ERROR] /Users/ajisaka/git/hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoStreamUtils.java:[40,21] cannot find symbol
[ERROR] symbol: class Cleaner
[ERROR] location: package sun.misc
This is a familiar error. HADOOP-12760. sun.misc.Cleaner has moved to a new location in OpenJDK 9 can be solved by applying the patch. Let's do it. dev-support / bin / smart-apply-patch is useful. I'm adding dev-support / bin to PATH.
mbpaa:hadoop$ smart-apply-patch HADOOP-12760
Processing: HADOOP-12760
HADOOP-12760 patch is being downloaded at Sat Dec 2 23:19:19 JST 2017 from
https://issues.apache.org/jira/secure/attachment/12850964/HADOOP-12760.03.patch -> Downloaded
Applying the patch:
Sat Dec 2 23:19:20 JST 2017
cd /Users/ajisaka/git/hadoop
patch -p1 -E
patching file hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoStreamUtils.java
patching file hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
Hunk #1 succeeded at 38 with fuzz 1 (offset 1 line).
Hunk #2 succeeded at 316 (offset 3 lines).
Hunk #3 succeeded at 330 (offset 3 lines).
patching file hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/CleanerUtil.java
Apply HDFS-11610
The next error is here.
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project hadoop-hdfs: Compilation failure: Compilation failure:
[ERROR] /Users/ajisaka/git/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java:[66,31] package sun.net.spi.nameservice does not exist
This is also a familiar error. Apply the patch HDFS-11610. Sun.net.spi.nameservice.NameService has moved to a new location.
mbpaa:hadoop$ smart-apply-patch HDFS-11610
Processing: HDFS-11610
HDFS-11610 patch is being downloaded at Sat Dec 2 23:27:05 JST 2017 from
https://issues.apache.org/jira/secure/attachment/12879343/HDFS-11610.001.patch -> Downloaded
Applying the patch:
Sat Dec 2 23:27:06 JST 2017
cd /Users/ajisaka/git/hadoop
git apply --binary -v --stat --apply -p1 /tmp/yetus-30092.22999/patch
Hunk #1 succeeded at 600 (offset 5 lines).
Applied patch hadoop-hdfs-project/hadoop-hdfs/pom.xml cleanly.
hadoop-hdfs-project/hadoop-hdfs/pom.xml | 19 ++++++++++++++++++-
1 file changed, 18 insertions(+), 1 deletion(-)
I was able to compile successfully.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 08:05 min
[INFO] Finished at: 2017-12-02T23:35:42+09:00
[INFO] Final Memory: 185M/616M
[INFO] ------------------------------------------------------------------------
Merge HADOOP-11875. Add a second copy of Hamlet without _ as a one-character identifier or some Maven plugins Thanks to the version upgrade, there are fewer steps to compile.
Also, thanks to the update on the Java side, it is no longer necessary to add the compile option --add-opens.
I would like to merge the above three fixes into the community as soon as possible. Also update the library. Log4J and JUnit are especially hard. Lately, I've been busy with work other than activities in the Apache Hadoop community and haven't had time. .. ..
For busy people, I've put a patch of these three on gist.
Next, I would like to run Apache Hadoop.
Recommended Posts