HotSpot Compiler
At first, the Java VM executes and operates by executing the code to be executed as an interpreter, but it is executed by compiling the frequently executed part (-XX: CompileThreshold = 10000 or more on the Server VM) into the native machine language. Is speeding up. One of the causes of the Java VM crash is that a bug in this HotSpot compiler causes SIGSEGV + core dump to crash when trying to compile the target method.
hs_error_log
Along with core dump, hs error log files such as hs_error_log or hs_err_pid1234.log are usually output. This log file contains important information such as what method the HotSpot compiler tried to compile and resulted in an abend.
The standard output ends with an error as shown below.
# ulimit -c unlimited
# /usr/libexec/tomcat/server start
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f5073f646c0, pid=15045, tid=139983466055424
#
# JRE version: OpenJDK Runtime Environment (7.0_141-b02) (build 1.7.0_141-mockbuild_2017_06_13_12_10-b00)
# Java VM: OpenJDK 64-Bit Server VM (24.141-b02 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea 2.6.10
# Distribution: Red Hat Enterprise Linux Server release 7.3 (Maipo), package rhel-2.6.10.5.el7-x86_64 u141-b02
# Problematic frame:
# V [libjvm.so+0x8446c0]
#
# Core dump written. Default location: /usr/share/tomcat/core or core.15045
#
# An error report file with more information is saved as:
# /tmp/jvm-15045/hs_error.log
#
# If you would like to submit a bug report, please include
# instructions on how to reproduce the bug and visit:
# http://icedtea.classpath.org/bugzilla
#
Cancel(Core dump)
You can see what method the HotSpot compiler tried to compile and crashed from the following part of the hs error log file:
# vi /tmp/jvm-15045/hs_error.log
...Omission...
Current CompileTask:
C2: 205800 155 org.apache.tomcat.util.buf.ByteChunk::recycle (21 bytes)
...
Since we are forcing the core to be output for testing, this method usually does not lead to a core dump. From the above output result, the method in the CompileTask: column is the target method.
To avoid such problems with the HotSpot compiler, you can avoid it by excluding the method to be compiled from the HotSpot compiler. Create a .hotspot_compiler file in the current working directory when you start Java. The Java VM reads this file at startup, after which the specified method is never HotSpot compiled.
# vi .hotspot_compiler
exclude org/apache/tomcat/util/buf/ByteChunk recycle
Most of the reasons why the HotSpot compiler crashed on SIGSEGV are bugs in the Java VM. However, it is very difficult to reproduce the same abend event in another environment. Therefore, even if you contact the Java VM support desk, if it is a new problem that has never been seen before, the cause cannot be immediately clarified and it cannot be expected to be fixed. At that point, determine if it will reoccur in the latest Java VM, and if it occurs in the latest version, send the core file, linked library, hs error log, etc. to the support window with the expectation that it will be fixed in the future. I think it will be provided. In a production environment, you should set .hotspot_compiler first to avoid problems.
You may get into an unfortunate situation where you only have the java core file at hand, such as "I forgot to get the hs error file" or "I have already deleted it". Even in this case, you can check the target class name and method name from the core file.
Below, confirmed in RHEL 7.4 Java 1.7.0_141 environment
$ gdb /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.141-2.6.10.5.el7.x86_64/jre-abrt/bin/java /usr/share/tomcat/core.15045
...Omission...
(gdb) bt
#0 0x00007f50747b01f7 in raise () from /lib64/libc.so.6
#1 0x00007f50747b18e8 in abort () from /lib64/libc.so.6
#2 0x00007f5073f18de9 in os::abort (dump_core=<optimized out>) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.141-2.6.10.5.el7.x86_64/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:1635
#3 0x00007f50740a7bcf in VMError::report_and_die (this=this@entry=0x7f5070c49bd0)
at /usr/src/debug/java-1.7.0-openjdk-1.7.0.141-2.6.10.5.el7.x86_64/openjdk/hotspot/src/share/vm/utilities/vmError.cpp:1074
#4 0x00007f5073f21ff7 in JVM_handle_linux_signal (sig=11, info=0x7f5070c49e30, ucVoid=0x7f5070c49d00, abort_if_unrecognized=<optimized out>)
at /usr/src/debug/java-1.7.0-openjdk-1.7.0.141-2.6.10.5.el7.x86_64/openjdk/hotspot/src/os_cpu/linux_x86/vm/os_linux_x86.cpp:531
#5 <signal handler called>
#6 0x00007f5073f646c0 in Phase::Phase (this=0x0, pnum=Phase::Compiler)
at /usr/src/debug/java-1.7.0-openjdk-1.7.0.141-2.6.10.5.el7.x86_64/openjdk/hotspot/src/share/vm/opto/phase.cpp:83
#7 0x00007f5073b78711 in Compile::Compile (this=0x0, ci_env=0x7f5070c4ba40, compiler=0x7f506c072730, target=0x7f503c0ba4f0, osr_bci=-1, subsume_loads=<optimized out>,
do_escape_analysis=true, eliminate_boxing=true) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.141-2.6.10.5.el7.x86_64/openjdk/hotspot/src/share/vm/opto/compile.cpp:683
#8 0x00007f5073ae3d30 in C2Compiler::compile_method (this=0x7f506c072730, env=0x7f5070c4ba40, target=0x7f503c0ba4f0, entry_bci=-1)
at /usr/src/debug/java-1.7.0-openjdk-1.7.0.141-2.6.10.5.el7.x86_64/openjdk/hotspot/src/share/vm/opto/c2compiler.cpp:137
#9 0x00007f5073b7f788 in CompileBroker::invoke_compiler_on_method (task=task@entry=0x7f506c1595c0)
at /usr/src/debug/java-1.7.0-openjdk-1.7.0.141-2.6.10.5.el7.x86_64/openjdk/hotspot/src/share/vm/compiler/compileBroker.cpp:1761
#10 0x00007f5073b80b00 in CompileBroker::compiler_thread_loop ()
at /usr/src/debug/java-1.7.0-openjdk-1.7.0.141-2.6.10.5.el7.x86_64/openjdk/hotspot/src/share/vm/compiler/compileBroker.cpp:1597
#11 0x00007f5074059cfa in JavaThread::thread_main_inner (this=this@entry=0x7f506c0a8800)
at /usr/src/debug/java-1.7.0-openjdk-1.7.0.141-2.6.10.5.el7.x86_64/openjdk/hotspot/src/share/vm/runtime/thread.cpp:1687
#12 0x00007f507405a07f in JavaThread::run (this=0x7f506c0a8800)
at /usr/src/debug/java-1.7.0-openjdk-1.7.0.141-2.6.10.5.el7.x86_64/openjdk/hotspot/src/share/vm/runtime/thread.cpp:1667
#13 0x00007f5073f17de2 in java_start (thread=0x7f506c0a8800) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.141-2.6.10.5.el7.x86_64/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:910
#14 0x00007f5074f57e25 in start_thread () from /lib64/libpthread.so.0
#15 0x00007f507487334d in clone () from /lib64/libc.so.6
(gdb)
There should be a C2Compiler :: compile_method function like frame # 8 in the compile thread above. The method to be compiled is specified as target = 0x7f503c0ba4f0 in this third argument. From this target, check the class name and method name.
python
(gdb) frame 8
#8 0x00007f5073ae3d30 in C2Compiler::compile_method (this=0x7f506c072730, env=0x7f5070c4ba40, target=0x7f503c0ba4f0, entry_bci=-1)
at /usr/src/debug/java-1.7.0-openjdk-1.7.0.141-2.6.10.5.el7.x86_64/openjdk/hotspot/src/share/vm/opto/c2compiler.cpp:137
137 Compile C(env, this, target, entry_bci, subsume_loads, do_escape_analysis, eliminate_boxing);
(gdb)
(gdb) p ({ciMethod}0x7f503c0ba4f0)->_name
$1 = (ciSymbol *) 0x7f503c0ba590
(gdb) p ({ciSymbol}0x7f503c0ba590)->_symbol
$2 = (Symbol *) 0x7f506c4fdb10
(gdb) p &({Symbol}0x7f506c4fdb10)->_body
$3 = (signed char (*)[1]) 0x7f506c4fdb1a
(gdb) x/s 0x7f506c4fdb1a
0x7f506c4fdb1a: "recycle"
(gdb)
Method name"recycle"was.
(gdb) p ({ciMethod}0x7f503c0ba4f0)->_holder
$21 = (ciInstanceKlass *) 0x7f503c0ba5a0
(gdb) p ({ciInstanceKlass}0x7f503c0ba5a0)->_name
$22 = (ciSymbol *) 0x7f503c0ba630
(gdb) p ({ciSymbol}0x7f503c0ba630)->_symbol
$23 = (Symbol *) 0x7f506c529af0
(gdb) p &({Symbol}0x7f506c529af0)->_body
$24 = (signed char (*)[1]) 0x7f506c529afa
(gdb) x/s 0x7f506c529afa
0x7f506c529afa: "org/apache/tomcat/util/buf/ByteChunk"
(gdb)
name of the class"org/apache/tomcat/util/buf/ByteChunk"was.
It was a groping check, but it was a hassle, so I wrote a simple Python script. https://github.com/takimai39/gdb-java
(gdb) source target.py
(gdb) target 0x7f503c0ba4f0
exclude org/apache/tomcat/util/buf/ByteChunk recycle
(gdb)
I hope you find it useful.