[JAVA] Performance analysis and failure diagnostic tools that can be used with OpenJDK

OpenJDK distributions (hereinafter referred to as Java or JDK) that include the Oracle JDK have various analysis tools / mechanisms, including JFR.

There is a part where the trend changed from JDK7 to JDK12, so I will write a little sloppy note.

Mechanism for acquiring metrics

Java has several ways to get performance metrics. The following three are typical.

• JMX --Log
• JPLIS(javaagent)

JMX

Java Management Extensions (JMX) is a protocol for monitoring and managing Java resources. Simply put, it's the Java version of SNMP. Imported from Java 1.5 as JSR-174.

It is also possible to use Managed Beans (MBeans) to get CPU and memory information, and to execute specific events (for example, forced GC).

It is possible to define MBeans by yourself, and because it is a technology that originally started from JavaEE, many application containers such as Weblogic and GlassFish are able to acquire various metrics such as the number of threads and the number of wait requests with JMX. I will. Therefore, it is the most basic choice when monitoring Java applications.

However, in recent years, Jolokia, an OSS that converts JMX to HTTP base, and Eclipse Microprofile Metrics, a new metric acquisition interface that replaces JMX, have appeared because it is difficult to link with the monitoring tool side of the original protocol. ..

Java log

In Java, detailed logs such as GC can be obtained from the JVM. Originally, Java has evolved to be able to collect various system logs with version upgrades, but it was inconsistent and it was necessary to learn each log format and setting method.

However, since JDK 9, the specification for system logging has been integrated as Unified JVM Logging. As a result, the convenience has improved, but the options etc. are different from those in Java 8, so be careful when viewing old documents.

For example, if it is a GC log, it will be output to the log file by specifying it in the JVM option as shown below.

-Xlog:gc*=debug:/path/to/gc_%p_%t.log:time,level,tags:filesize=100m,filecount=7

You may also want to have thread dumps in the log file on a regular basis, but this is not possible with the JVM's logging feature. Therefore, it was common to create a command to write to a file by periodically executing jstack, jcmd, or kill -3, which will be described later.

However, if you use JFR, the stack trace is included on the JFR side, so that will be enough.

JPLIS(javaagent)

Java Programming Language Instrumentation Services (JPLIS) is an interface for working with unfamiliar packages called java.lang.instrument. It is used by specifying javaagent as a JMV option.

This is an API where the agent rewrites the class information when the JVM loads the class. You can use this to get Java object information, or embed an aspect for profiling to get trace information.

The disadvantage is that there are few materials, but it is relatively easy to use when combined with AspectJ.

Many APMs are implemented using this Java agent as it can take almost any value in principle. However, due to the characteristics of rewriting bytecode, there is a risk of causing bugs that are difficult to identify the cause in a wide range, so be careful when testing / debugging.

JDK standard tools

There are third-party tools, but Java provides tools for analysis and diagnostics by default. VisualVM and JMC are no longer included in the Oracle JDK as they are opened, but this time they will be included in the standard tools.

VisualVM

HotSpot's flagship profiling tool until Java Mission Control was integrated into the JDK. The OSS version is developed below.

https://visualvm.github.io/index.html

One of the features is that it has a plugin structure because it is based on NetBeans Platform.

You can monitor JMX information and see various information such as CPU and memory / GC in real time and graphically. It is also a very sophisticated tool that outputs heap dumps and takes snapshots of analysis information.

It has been a little overshadowed since JMC was included in HotSpot, but it is also compatible with GraalVM, and it seems that the latest version is compatible with JFR even though it is a preview version, so it is a tool to be worried about again in the future. is.

JDK Mission Control(JMC)

It is the mainstream profiling tool in Java today. The OSS version is developed below.

https://github.com/JDKMissionControl/jmc

Originally called JRockit Mission Control, it is a tool that incorporates the profiling tool that was well-established in JRockit into HotSpot after the acquisition of Oracle's Sun.

It also works closely with JFR, which is also derived from JRockit. .. .. In fact, it is almost an option as a JFR visualization tool.

As a function, it is an integrated analysis tool similar to Visual VM, and real-time JMX monitoring, heap dump, and JFR acquisition / analysis are possible.

It is based on the Eclipse GUI framework as opposed to VisualVM, and also has flexible plug-in functionality. One of the features is that there are plug-ins specialized for products such as Weblogic.

The name has changed to "JRock it Mission Control"-> "Java Mission Control"-> "JDK Mission Control" due to the porting to HotSpot and OSS, but the abbreviation remains JMC.

In addition, the UI has changed significantly from the open JMC 7, and the automatic analysis function has been enhanced.

However, at the moment, the official version has not been released yet and the binary cannot be downloaded from the official, so it is still necessary to build it by yourself.

For the time being, it is better to use "Adopt OpenJDK Mission Control", "Zulu Mission Control", and "Liberica Mission Control" built by the OpenJDK distributor.

• AdoptOpenJDK Mission Control
• Zulu Mission Control
• Liberica Mission Control Version 7.0

jcmd

Like JFR, jcmd is a command originally used by JRockit. Many management tools were also provided for HotSpot developed by Sun, but since the commands are different, it has become possible to perform consistent management by introducing jcmd.

Mainly, you can perform JFR operation, get process, get thread dump, etc. The following are typical commands.

JFR tool

JFR Tool is a new command introduced from JDK12. A tool for converting JFR files to JSON and XML, and splitting / merging JFR files.

jfr print is useful when linking JFR to another tool instead of JMC. For example, when you get GC information, a lot of JSON like the following is spit out.

$ jfr print --json --categories GC --events jdk.GCPhaseParallel sample.jfr
{
    "recording": {
        "events": [{
            "type": "jdk.GCPhaseParallel"
            ,
            "values": {
                "startTime": "2019-09-04T09:43:30.056871672-08:00",
                "duration": "PT0.000000518S",
                "eventThread": {
                    "osName": "GC Thread#5",
                    "osThreadId": 38915,
                    "javaName": null,
                    "javaThreadId": 0,
...

You can also perform various processing by piped the JSON output result to the jq command.

$ jfr print --json \
    --categories GC,Profiling,Processsor,Heap,MyApp \
    --events jdk.GCPhaseParallel,jdk.ExecutionSample,jdk.CPULoad,jdk.GCHeapSummary  \
    chunk.jfr \
|jq '.recording.events[]' | jq -c '.|= .+ {"Key": "Value"}'

{"type":"jdk.CPULoad","value":{"startTime":"2019-09-08T16:13:01.980014338-
08:00","jvmUser":0.23814254,"jvmSystem":0.019405695,"machineTotal":0.5409429},"Key":"
Value"}
...

Splitting and merging with ʻassemble / disassemble` is a very useful feature during operation. Details will be described in Chapter 5, but by using this function, you can realize daily and hourly file rotation and backup of JFR Fal.

JDK standard tools (old version)

It is a group of tools that have not been used much recently because new alternative tools have been created.

JConsole

JMX is a profiling tool. Since it has been around for a long time, it is often introduced as a tool for performance testing in old books and websites.

JMX can acquire information such as CPU, heap, GC, and number of load classes with the specification called SNMP for Java. It's a good tool with a simple UI, but recently I feel like I've finished my role because I can do the same thing with VisualVM and JMC.

hprof

An old Java standard profiling tool. You can get / analyze CPU usage, heap information, heap dumps, etc.

However, it is not the type that can be applied to production with a considerable overhead. Also, during development and testing, the profiler that comes with the IDE seems to be easier to use now.

jps

This command displays the process ID of the JVM. In the case of Linux, you can of course identify it with the ps command, but it is convenient for filtering because processes other than Java are not displayed.

jstat

This command checks the information of the heap including GC. Java's memory structure is complex with various hierarchies, but you can monitor it.

jstack

This command gets a thread dump of the JVM.

It is often used when analyzing thread locks such as deadlocks. Rather than using it alone, pass the acquired thread dump to "Samurai", "Thread Logic", and "TDA" and use it.

Since the thread dump is important information in the event of a failure, create a script that executes it regularly, or tap it several times in the event of a failure to acquire thread information.

jmap

jmap is a tool for getting heap dumps.

You can get process information, heap information, class loader information, and various information such as heap summary, histogram, and statistical information.

The output is typically analyzed with jhat or a third-party tool. However, jmap can cause a full GC and should be used with caution in production.

Summary

Here is a brief summary of how to use the metrics acquisition / diagnostic tool in Java. I don't think it's a bad idea to use the old command, but since the operations are unified, it's basically safe to use the latest one.

In particular, JFR Tool is a tool that came in before I knew it, but it is quite convenient, so I would like to actively use it.

Then Happy Hacking!

reference

• JDK Tool Specifications
• Unified JVM Logging - SlideShare -Build your own profiling tool-IBM -Measure the bottleneck! How to trace only slow methods in AspectJ -JDK Flight Recorder Architecture Overview