[DOCKER] Java container performance degradation in Menicoa environment

2018/04/08 postscript: Added a description about Java 10 to the summary.

2017/06/02 postscript: The contents of the survey have been summarized and the description has been significantly updated.

2017/06/01 postscript: Added "Efforts for this issue after JDK 8/9" at the end of the article. It seems that it has been fixed in 8u121.b34 and 8u131.b06 of OpenJDK8 (I will check separately if it is really fixed). Both workarounds have been released since 2017, so it's a good idea to check if you don't update your JDK / JRE frequently.

Summary (Java 10 migration)

Java10 officially now supports Docker containers. The Java program on the Docker container can now grasp the resource settings such as CPU and memory set in the Docker container, so the menicoa problem described in this article will be solved.

Regarding Docker support for Java10, the following Qiita article summarizes in detail including operation check.

https://qiita.com/c9katayama/items/b427a008623f0e228c50

Summary (Java 8/9)

If you limit the number of CPUs to a small number with Docker, taskset, etc. in the Menicoa environment, the performance of Java programs may drop significantly. There are two ways to deal with it.

If you only use the option to specify the CPU core ID, such as the --cpuset-cpus of the --docker command, you can handle it with a JDK update. (Java 8 8u121.b34 or later or 8u131.b06 or later) If you use an option that does not specify a CPU core ID, such as --cpus in the --docker command, you need to tune the appropriate JVM parameter settings when you start the Java program.

Occurrence event

In a menicore environment (a machine with a large number of CPU cores), performance will drop significantly when a Java program is executed on Docker with resource restrictions such as the number of CPU cores. The same applies when a Java program is executed using a container technology or command (ex. Tasket command) that uses resource restrictions by cgroups other than Docker.

Cause

In Java, the following JVM parameters are set based on the hardware resources of the machine.

--Number of GC (garbage collect) threads --Number of JIT compilation threads --Heap size

This is fine in itself. However, in Java, even if hardware resources are restricted by cgroups such as Docker, the JVM parameters are determined based on the hardware resources of the host machine, not the restricted hardware resources.

Therefore, for example, if you execute a Java program on a "Docker container with 20 CPU cores" on a host machine with 20 CPU cores, Java will determine the JVM parameters based on the 20 cores. .. However, the fact that only one CPU core can be used in the Docker container remains the same, so Java programs suffer from the overhead of a large number of GC threads and JIT compilation threads in addition to application processing, resulting in a significant decrease in performance. ..

Be careful of those who meet the following conditions.

--The host machine is a menicoa environment. (Equipped with dozens of CPU cores) --Java runs on the Docker container. --The above Docker container limits CPU resources. (With docker command --cpuset-cpus, --cpus, etc. Specify option)

Workaround

There are two main approaches to dealing with this.

Workaround 1: JDK update

As described in "(Reference) Addressing this issue after JDK8 / 9" in this article, cgroup-related issues have been addressed to some extent in the new version of JDK8 / 9. In the case of JDK8, it will be solved by updating to the following version or later.

--8u121.b34 or later --8u131.b06 or later

The confirmation results in the Docker container with the JDK before and after the action are listed below. I have confirmed it in an environment with 8 logical CPU cores, but in the JDK after dealing with it, the resource limit is properly reflected in the JVM parameter by the docker command --cpuset-cpus.

#--------------------------------------------------------------
#In the JDK before action--cpuset-The setting of the cpus option is not reflected in the JVM parameter
#--------------------------------------------------------------
$ docker run --rm --cpuset-cpus 0-1 openjdk:8u77-b03 java -XX:+PrintFlagsFinal -version | egrep "CICompilerCount |ParallelGCThreads"
     intx CICompilerCount                          := 4                                   {product}
    uintx ParallelGCThreads                         = 8                                   {product}
openjdk version "1.8.0_03-Ubuntu"
OpenJDK Runtime Environment (build 1.8.0_03-Ubuntu-8u77-b03-3ubuntu3-b03)
OpenJDK 64-Bit Server VM (build 25.03-b03, mixed mode)

#--------------------------------------------------------------
#After dealing with it, in JDK--cpuset-The setting of the cpus option is reflected in the JVM parameter
#--------------------------------------------------------------
$ docker run --rm --cpuset-cpus 0-1 openjdk:8u131-b11 java -XX:+PrintFlagsFinal -version | egrep "CICompilerCount |ParallelGCThreads"
     intx CICompilerCount                          := 2                                   {product}
    uintx ParallelGCThreads                         = 2                                   {product}
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-0ubuntu1.16.04.2-b11)
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)

However, in the case of an option that does not specify a specific CPU core ID like --cpus of the docker command, it seems that the problem cannot be avoided even with Java after handling. I haven't tried it, but it seems likely that this problem cannot be avoided for the --cpu-quota and --cpu-period options as well. In this case, it is necessary to tune the JVM parameter settings, which is another workaround.

#--------------------------------------------------------------
#After dealing with it, even in JDK--No effect with cpus option settings
#--------------------------------------------------------------
docker run --rm --cpus 2.0 openjdk:8u131-b11 java -XX:+PrintFlagsFinal -version | egrep "CICompilerCount |ParallelGCThreads"
     intx CICompilerCount                          := 4                                   {product}
    uintx ParallelGCThreads                         = 8                                   {product}
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-0ubuntu1.16.04.2-b11)
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)

Workaround 2: JVM parameter setting

It is solved by tuning by setting the JVM parameter when executing the Java program on the Docker container. It's more time-consuming than a JDK update, but it's the most reliable.

Change GC to serial GC

You may want to set the GC to serial GC if the following conditions are true:

--Limiting CPU resources to one or less CPU cores --The main processing content is batch processing (don't worry about Stop the World)

#Change GC to serial GC
java -XX:+UseSerialGC  ...

However, it is said that serial GC will be abolished in the future, so be careful if you plan to update Java.

Change the number of GC threads

If you want to use parallel GC, which is the default GC of Java8, you can handle it by reducing the number of threads of Parallel GC. The appropriate number of threads depends on the CPU resource limit specified in the docker command, so tuning is required.

#Set the number of Parallel GC threads to 1
java XX:ParallelGCThreads=1  ...

Since I have never set CSM, G1, etc., I will not describe it, but basically you can set the number of threads according to the CPU resource.

Reduce the number of JIT compilation threads.

JIT compilation also runs on multiple threads, and a reasonable number of threads will be generated in the Menikoa environment. For this reason, tuning is also required according to the CPU resource limit.

#Set the number of JIT compilation threads to 1
java -XX:CICompilerCount=1 ...

(Reference) For virtual machines

Determine the JVM parameters based on the Java program on the virtual machine and the hardware resources of the virtual machine. Therefore, even if the host machine is in a menicoa environment, the JVM parameters are set based on the hardware resources assigned to the virtual machine.

(Reference) Version information when a problem occurs

(Reference) Efforts for this problem after JDK8 / 9

This problem has been recognized since JDK9, and it seems that Issue is backported to JDK8 as far as the support status of Issue is seen. And it has been dealt with in JDK8 "8u121.b34" and "8u131.b06".

  1. JDK-8140793 - getAvailableProcessors may incorrectly report the number of cpus in Docker container --Issue about CPU issues covered in this article. --This problem has been resolved by the countermeasure of another case JDK-6515172. --Looking at the related issues, "8u121.b34" and "8u131.b06" have already been dealt with.

  2. JDK-8172318 - [linux] Experimental support for cgroup memory limits in container (ie Docker) environments --Since the same problem as this article occurs in memory, Issue to deal with so that the memory size of the Docker container controlled by cgroups on the JVM side can be grasped. --Introduced "experimentally" in JDK9. --A backport has been requested for JDK8. --Similar to the problem with JDK-6515172 / JDK-8140793, looking at the related issues, "8u121.b34" and "8u131.b06" have been addressed.

Recommended Posts

Java container performance degradation in Menicoa environment
JavaFX environment construction in Java 13
Java Spring environment in vs Code
Play Framework 2.6 (Java) environment construction in Eclipse
Java application development environment created in VM environment
Partization in Java
[Java] Environment construction
When there are environment variables in Java tests
Changes in Java 11
Solution for NetBeans 8.2 not working in Java 9 environment
Rock-paper-scissors in Java
Java environment construction
Java development environment
Pi in Java
Points stuck in building VSCode & Java development environment
[Beginner] Install java development tool in cloud9 development environment.
FizzBuzz in Java
Replacing system environment variables by reflection in java
[LeJOS] Let's program mindstorm-EV3 in Java [Environment construction part 2]
How to create a Java environment in just 3 seconds
Maven fails in Java7 environment (Received fatal alert: protocol_version)
[Java] Get the file in the jar regardless of the environment
[java] sort in list
Read JSON in Java
Interpreter implementation in Java
Make Blackjack in Java
Rock-paper-scissors app in Java
Constraint programming in Java
Put java8 in centos7
NVL-ish guy in Java
Combine arrays in Java
"Hello World" in Java
Java development environment memo
Callable Interface in Java
[Summary] Java environment preparation
Comments in Java source
Azure functions in java
Format XML in Java
Java app performance tuning
Simple htmlspecialchars in Java
Boyer-Moore implementation in Java
Hello World in Java
Use OpenCV in Java
webApi memorandum in java
Type determination in Java
Ping commands in Java
Various threads in java
Heapsort implementation (in java)
Zabbix API in Java
ASCII art in Java
Compare Lists in Java
POST JSON in Java
java development environment construction
Express failure in Java
Create JSON in Java
Date manipulation in Java 8
What's new in Java 8
Use PreparedStatement in Java
Java Performance Chapter 3 Java Performance Toolbox
What's new in Java 9,10,11
Parallel execution in Java