Resolved the error that occurred when trying to use Spark in an environment where Java 8 and Java 11 coexist.

Resolved the error that occurred when trying to use Spark in an environment where Java 8 and Java 11 coexist.


I got the same error as below when I used spark from spark-shell or pyspark, so I fixed it.

'Unsupported class file major version 55'


Ubuntu18.04 Spark 2.4.3

Solution (with trial and error)

The cause seems to be using Java 11. Check the version of Java you are using and do the following to choose another version:

$ sudo update-alternatives --config java
alternative java (/usr/bin/Provide java)Has three choices.

Choice Path Priority Status
  0            /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1101 automatic mode
  1            /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1101 manual mode
* 2            /usr/lib/jvm/java-11-oracle/bin/java 1091 manual mode
  3            /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java 1081 manual mode

Current choice[*]To hold<Enter>, Otherwise press the key of the choice number:3

I was wondering if I could use Java 8 with this, but it didn't happen as shown below.

$ java --version
openjdk 11.0.2 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)

Let's find out where we are going to see in the first place. Then

$ which java

There is evidence that Java (and Kotlin) was installed using SDKMAN.

At this point, I learned that SDKMAN can install spark, so I decided to try it.

$ sdk install spark

I installed it with and tried to check the operation from $ spark-session, but I still get the same error like'Unsupported class file major version 55'.

$ sdk list java

When I looked at whether Java 8 could be installed with SDKMAN, it turned out that Java 8 or less could not be installed. In that case, I decided that it was impossible to solve it here.

Therefore, uninstall java that was put in SDKMAN below.

$ sdk uninstall java 11.0.2-open

with this,

$ java -version
openjdk version "1.8.0_212"
OpenJDK Runtime Environment (build 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03)
OpenJDK 64-Bit Server VM (build 25.212-b03, mixed mode)

$ which java

Then, it will read java under / usr / bin /.

Try launching spark installed with SDKMAN as $ spark-shell. Then

$ spark-shell
scala> val textFile ="")
textFile: org.apache.spark.sql.DataFrame = [value: string]

scala> textFile.count()
res0: Long = 109

And the error disappeared safely.

As far as I can tell, it's too early to use Java 11 with Spark and Hadoop, and if you want to use Java 11, expect Spark 3.x. This seems to solve the problem when introducing Hadoop the other day.

Recommended Posts

Resolved the error that occurred when trying to use Spark in an environment where Java 8 and Java 11 coexist.
Summary when trying to use Solr in Java and getting an error (Solr 6.x)
Solution that gives an error when trying to connect to DB (MySQL) in Java
About the solution of the error that occurred when trying to create a Japanese file of devise in the Docker development environment
407 error when trying to access an HTTPS site in Java via an authenticated proxy
What happened in "Java 8 to Java 11" and how to build an environment
An embarrassing story that was treated as the same day when trying to compare dates on 3/31 and 4/1 [Java / Calendar]
[Docker] The story that an error occurred in docker-compose up
How to change Java version and execute in an environment where Java cannot be installed freely
[jOOQ] How to CASE WHEN in the WHERE / AND / OR clause
How to solve the unknown error when using slf4j in Java
[Programming beginner] What to do when rails s becomes an error in the local development environment
Error when trying to use heroku command
What to do and how to install when an error occurs in DXRuby 1.4.7
I want to display an error message when registering in the database
An error occurred in the free course of RubyOnRails Udemy, solved it, and went through to the end
I get an error when I try to use "^" or "$" in ruby ​​regular expression
[Error resolution] Occurs when trying to build an environment for spring with docker
An error occurred when executing a function from MyBatis with the OUT parameter set to CURSOR in PostgreSQL.
When using a list in Java, java.awt.List comes out and an error occurs
A solution to an error that makes you angry that you are not following the MySQL default setting ONLY_FULL_GROUP_BY in a production environment and it is not unique.
Error logging and exception handling that you can't often see in the Java area
How to use git with the power of jgit in an environment without git commands
I got an error when trying to install sbt to build a Scala development environment
Summary of how to use the proxy set in IE when connecting with Java
Verification value error that occurred in a clustering environment
Java classes and instances to understand in the figure
What to do when an error occurs in rails db: migrate ((StandardError: An error has occurred, this and all later migrations canceled :))
A story that solved the problem that the Java (jdk) version was too high to use the h2o library in R and R Studio.
What to do when an error (StandardError: An error has occurred, this and all later migrations canceled:) appears in rails db: migrate
A program (Java) that outputs the sum of odd and even numbers in an array
About the error that occurred when adding the column name in rails (rails db: migrate, rails db: rollback, add)
Use JLine when you want to handle keystrokes on the console character by character in Java
When trying to insert a specific gem (levenshtein) with Docker, an error occurred without ruby-dev
Resolve CreateProcess error = 206 when running Java in a Windows environment
Use Java Web Start in an OpenJDK environment on Windows
[Docker] List of errors that occurred when building the environment
How to call and use API in Java (Spring Boot)
A validation error occurred when saving to the intermediate table.
Reasons to use Servlet and JSP separately in Java development
[Java] Use ResolverStyle.LENIENT to handle the date and time nicely
[Rails] What to do when the error No database selected and Unknown database appears in db: migrate
Organize communication in an environment that uses various resources on the host machine and Docker container
Throw an exception and catch when there is no handler corresponding to the path in spring
[Java] Adding an element to the Collection causes a compile error
I got an IllegalAccessError when trying to use PowerMock with JUnit
How to output the value when there is an array in the array
I tried to translate the error message when executing Eclipse (Java)
After switching between Java 8 and 11 versions, an error occurs in confirmation
[Java] Shallow copy and deep copy when converting an array to List
How to install the language used in Ubuntu and how to build the environment
When log data accumulates in Rails and the environment stops working
The operator that was born to be born, instanceof (Java) ~ How to use the instanceof operator ~
How to get the length of an audio file in java
It should be the same as the sample when logging in to Twitter, but an error occurs ~ resolution
[WSL] Solution for the phenomenon that 404 is displayed when trying to insert Java with apt (personal memo)
[Java] Output the result of ffprobe -show_streams in JSON and map it to an object with Jackson
How to use classes in Java?
Update JAVA to the latest version to 1.8.0_144 (when downloading from the web and updating)
How to test a private method in Java and partially mock that method
I summarized the points to note when using resources and resources in combination