[JAVA] Let's take a look at the Hotspot JVM startup procedure

Overview

Sometimes I need to read the JVM source code. The starting procedure is probably where to start. This article is a record of a little research on the JVM startup procedure and the libjvm.so that got there while experimenting.

Environment

In the working environment, use Vagrant 2.2.8 for Ubuntu 18.04. First, prepare this Vagrantfile.

Vagrantfile

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/bionic64"

  config.vbguest.auto_update = false

  config.vm.provider "virtualbox" do |vb|
    vb.memory = "4096"
  end
end

And register in the virtual machine

$ vagrant up
$ vagrant ssh

OpenJDK as a building

Install the required packages

$ apt-get update
$ DEBIAN_FRONTEND=noninteractive ln -fs /usr/share/zoneinfo/Asia/Tokyo /etc/localtime && apt-get install -y tzdata && dpkg-reconfigure --frontend noninteractive tzdata
$ apt-get install build-essential autoconf systemtap-sdt-dev libx11-dev libxext-dev libxrender-dev libxrandr-dev libxtst-dev libxt-dev libcups2-dev libfontconfig1-dev libasound2-dev unzip zip ccache gdb -y

Next, download the source code

$ cd /vagrant

$ mkdir jdk && cd jdk
$ git clone https://github.com/openjdk/jdk13u.git

$ wget https://download.java.net/java/GA/jdk12.0.2/e482c34c86bd4bf8b56c0b35558996b9/10/GPL/openjdk-12.0.2_linux-x64_bin.tar.gz
$ tar xvf openjdk-12.0.2_linux-x64_bin.tar.gz

Start with the building

$ cd jdk13u

$ bash configure --enable-debug --with-jvm-variants=server --enable-dtrace --with-boot-jdk=../jdk-12.0.2 --enable-ccache
$ make images

After waiting for about 2 hours, I finally succeeded. And let's examine the building and the results.

$ export JAVA_HOME=$PWD/build/linux-x86_64-server-fastdebug/jdk
$ $JAVA_HOME/bin/java -version
openjdk version "13.0.3-internal" 2020-04-14
OpenJDK Runtime Environment (fastdebug build 13.0.3-internal+0-adhoc.root.jdk13u)
OpenJDK 64-Bit Server VM (fastdebug build 13.0.3-internal+0-adhoc.root.jdk13u, mixed mode)

Root of startup procedure

More information with DEBUG variables

First, set the environment variable _JAVA_LAUNCHER_DEBUG to get more detailed startup information.

$ _JAVA_LAUNCHER_DEBUG=1 $JAVA_HOME/bin/java -version
----_JAVA_LAUNCHER_DEBUG----
Launcher state:
	First application arg index: -1
	debug:on
	javargs:off
	program name:java
	launcher name:openjdk
	javaw:off
	fullversion:13.0.3-internal+0-adhoc.root.jdk13u
Java args:
Command line args:
argv[0] = /vagrant/jdk/jdk13u/build/linux-x86_64-server-fastdebug/jdk/bin/java
argv[1] = -version
JRE path is /vagrant/jdk/jdk13u/build/linux-x86_64-server-fastdebug/jdk
jvm.cfg[0] = ->-server<-
jvm.cfg[1] = ->-client<-
1 micro seconds to parse jvm.cfg
Default VM: server
Does `/vagrant/jdk/jdk13u/build/linux-x86_64-server-fastdebug/jdk/lib/server/libjvm.so' exist ... yes.
mustsetenv: FALSE
JVM path is /vagrant/jdk/jdk13u/build/linux-x86_64-server-fastdebug/jdk/lib/server/libjvm.so
1 micro seconds to LoadJavaVM
JavaVM args:
    version 0x00010002, ignoreUnrecognized is JNI_FALSE, nOptions is 3
    option[ 0] = '-Dsun.java.launcher.diag=true'
    option[ 1] = '-Dsun.java.launcher=SUN_STANDARD'
    option[ 2] = '-Dsun.java.launcher.pid=11021'
openjdk version "13.0.3-internal" 2020-04-14
OpenJDK Runtime Environment (fastdebug build 13.0.3-internal+0-adhoc.root.jdk13u)
OpenJDK 64-Bit Server VM (fastdebug build 13.0.3-internal+0-adhoc.root.jdk13u, mixed mode)

Various functions are printed out, and the one that should be noted is here

JVM path is /vagrant/jdk/jdk13u/build/linux-x86_64-server-fastdebug/jdk/lib/server/libjvm.so

It seems that you are using a library called libjvm.so.

Find libjvm.so

If you look for libjvm.so in the source code, you will find java_md_solinux.c: 556. Dlopen (3) was found in /libjli/java_md_solinux.c#L556).

c:src/java.base/unix/native/libjli/java_md_solinux.c

...
    JLI_TraceLauncher("JVM path is %s\n", jvmpath);

    libjvm = dlopen(jvmpath, RTLD_NOW + RTLD_GLOBAL);
    if (libjvm == NULL) {
...

And then in that java_md_solinux.c: 609 JNI_CreateJavaVM I read the symbol .

c:src/java.base/unix/native/libjli/java_md_solinux.c

...
    ifn->CreateJavaVM = (CreateJavaVM_t)
        dlsym(libjvm, "JNI_CreateJavaVM");
...

This JNI_CreateJavaVM is probably the entrance to the JVM.

Until you get to JNI_CreateJavaVM

Actually [src / java.base / share / native / launcher / main.c](https://github.com/openjdk/jdk13u/blob/master/src/java.base/share/native/launcher/main If you read a little from .c), you can see the general procedure.

1. main.c::main
2. java.c::JLI_Launch
3. java_md_solinux.c::LoadJavaVM
4. java_md_solinux.c::JVMInit
5. java_md_solinux.c::CallJavaMainInNewThread
6. java.c::JavaMain
7. java.c::InitializeJVM
8. jni.cpp::JNI_CreateJavaVM

Let's verify with gdb.

$ gdb -q --args $JAVA_HOME/bin/java -version <<EOF
set print thread-events off
set print frame-arguments none
set breakpoint pending on
handle SIGSEGV nostop noprint pass

break JNI_CreateJavaVM
commands
silent
thread apply all backtrace
continue
end

run
EOF
Reading symbols from /vagrant/jdk/jdk13u/build/linux-x86_64-server-fastdebug/jdk/bin/java...(no debugging symbols found)...done.
(gdb) (gdb) (gdb) (gdb) Signal        Stop	Print	Pass to program	Description
SIGSEGV       No	No	Yes		Segmentation fault
(gdb) Signal        Stop	Print	Pass to program	Description
SIGSEGV       No	No	Yes		Segmentation fault
(gdb) Function "JNI_CreateJavaVM" not defined.
Breakpoint 1 (JNI_CreateJavaVM) pending.
(gdb) >>>>(gdb) (gdb) Starting program: /vagrant/jdk/jdk13u/build/linux-x86_64-server-fastdebug/jdk/bin/java -version
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Switching to Thread 0x7ffff7fea700 (LWP 11128)]

Thread 2 (Thread 0x7ffff7fea700 (LWP 11128)):
#0  JNI_CreateJavaVM (vm=..., penv=..., args=...) at ../../src/hotspot/share/prims/jni.cpp:4012
#1  0x00007ffff7bca1e6 in InitializeJVM (ifn=..., penv=..., pvm=...) at ../src/java.base/share/native/libjli/java.c:1539
#2  JavaMain (_args=...) at ../src/java.base/share/native/libjli/java.c:417
#3  0x00007ffff7bce239 in ThreadJavaMain (args=...) at ../src/java.base/unix/native/libjli/java_md_solinux.c:740
#4  0x00007ffff719c6db in start_thread (arg=...) at pthread_create.c:463
#5  0x00007ffff78f688f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7ffff7fec380 (LWP 11124)):
#0  0x00007ffff719dd2d in __GI___pthread_timedjoin_ex (threadid=..., thread_return=..., abstime=..., block=...) at pthread_join_common.c:89
#1  0x00007ffff719db5c in __pthread_join (threadid=..., thread_return=...) at pthread_join.c:24
#2  0x00007ffff7bcee2d in CallJavaMainInNewThread (stack_size=..., args=...) at ../src/java.base/unix/native/libjli/java_md_solinux.c:762
#3  0x00007ffff7bcb70d in ContinueInNewThread (ifn=..., threadStackSize=..., argc=..., argv=..., mode=..., what=..., ret=...) at ../src/java.base/share/native/libjli/java.c:2367
#4  0x00007ffff7bceefb in JVMInit (ifn=..., threadStackSize=..., argc=..., argv=..., mode=..., what=..., ret=...) at ../src/java.base/unix/native/libjli/java_md_solinux.c:809
#5  0x00007ffff7bccd1b in JLI_Launch (argc=..., argv=..., jargc=..., jargv=..., appclassc=..., appclassv=..., fullversion=..., dotversion=..., pname=..., lname=..., javaargs=..., cpwildcard=..., javaw=..., ergo=...) at ../src/java.base/share/native/libjli/java.c:344
#6  0x0000555555554ad3 in main ()
openjdk version "13.0.3-internal" 2020-04-14
OpenJDK Runtime Environment (fastdebug build 13.0.3-internal+0-adhoc.root.jdk13u)
OpenJDK 64-Bit Server VM (fastdebug build 13.0.3-internal+0-adhoc.root.jdk13u, mixed mode)
[Inferior 1 (process 11124) exited normally]
(gdb) quit

Take a peek at libjvm.so

Finally, let's take a look at libjvm.so. The format of Shared Library is elf, so use readelf and nm.

$ readelf -h $JAVA_HOME/lib/server/libjvm.so
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 03 00 00 00 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - GNU
  ABI Version:                       0
  Type:                              DYN (Shared object file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x303bf0
  Start of program headers:          64 (bytes into file)
  Start of section headers:          38283720 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         8
  Size of section headers:           64 (bytes)
  Number of section headers:         37
  Section header string table index: 36

There are 37 sections.

$ readelf -S $JAVA_HOME/lib/server/libjvm.so
There are 37 section headers, starting at offset 0x24829c8:

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .note.gnu.build-i NOTE             0000000000000200  00000200
       0000000000000024  0000000000000000   A       0     0     4
  [ 2] .hash             HASH             0000000000000228  00000228
       0000000000001028  0000000000000004   A       4     0     8
  [ 3] .gnu.hash         GNU_HASH         0000000000001250  00001250
       0000000000000a44  0000000000000000   A       4     0     8
  [ 4] .dynsym           DYNSYM           0000000000001c98  00001c98
       0000000000002c70  0000000000000018   A       5     1     8
  [ 5] .dynstr           STRTAB           0000000000004908  00004908
       0000000000001e60  0000000000000000   A       0     0     1
  [ 6] .gnu.version      VERSYM           0000000000006768  00006768
       00000000000003b4  0000000000000002   A       4     0     2
  [ 7] .gnu.version_d    VERDEF           0000000000006b20  00006b20
       0000000000000038  0000000000000000   A       5     2     8
  [ 8] .gnu.version_r    VERNEED          0000000000006b58  00006b58
       0000000000000130  0000000000000000   A       5     5     8
  [ 9] .rela.dyn         RELA             0000000000006c88  00006c88
       00000000002fa6d8  0000000000000018   A       4     0     8
  [10] .rela.plt         RELA             0000000000301360  00301360
       0000000000001830  0000000000000018  AI       4    28     8
  [11] .init             PROGBITS         0000000000302b90  00302b90
       0000000000000017  0000000000000000  AX       0     0     4
  [12] .plt              PROGBITS         0000000000302bb0  00302bb0
       0000000000001030  0000000000000010  AX       0     0     16
  [13] .plt.got          PROGBITS         0000000000303be0  00303be0
       0000000000000010  0000000000000008  AX       0     0     8
  [14] .text             PROGBITS         0000000000303bf0  00303bf0
       00000000016df6be  0000000000000000  AX       0     0     16
  [15] .fini             PROGBITS         00000000019e32b0  019e32b0
       0000000000000009  0000000000000000  AX       0     0     4
  [16] .rodata           PROGBITS         00000000019e32c0  019e32c0
       0000000000204358  0000000000000000   A       0     0     32
  [17] .stapsdt.base     PROGBITS         0000000001be7618  01be7618
       0000000000000001  0000000000000000   A       0     0     1
  [18] .eh_frame_hdr     PROGBITS         0000000001be761c  01be761c
       000000000006646c  0000000000000000   A       0     0     4
  [19] .eh_frame         PROGBITS         0000000001c4da88  01c4da88
       0000000000209730  0000000000000000   A       0     0     8
  [20] .gcc_except_table PROGBITS         0000000001e571b8  01e571b8
       00000000000000bc  0000000000000000   A       0     0     4
  [21] .tdata            PROGBITS         0000000002057f50  01e57f50
       0000000000000008  0000000000000000 WAT       0     0     8
  [22] .tbss             NOBITS           0000000002057f58  01e57f58
       0000000000000038  0000000000000000 WAT       0     0     8
  [23] .init_array       INIT_ARRAY       0000000002057f58  01e57f58
       0000000000001620  0000000000000008  WA       0     0     8
  [24] .fini_array       FINI_ARRAY       0000000002059578  01e59578
       0000000000000008  0000000000000008  WA       0     0     8
  [25] .data.rel.ro      PROGBITS         0000000002059580  01e59580
       00000000000ef5d8  0000000000000000  WA       0     0     32
  [26] .dynamic          DYNAMIC          0000000002148b58  01f48b58
       0000000000000240  0000000000000010  WA       5     0     8
  [27] .got              PROGBITS         0000000002148d98  01f48d98
       0000000000000268  0000000000000008  WA       0     0     8
  [28] .got.plt          PROGBITS         0000000002149000  01f49000
       0000000000000828  0000000000000008  WA       0     0     8
  [29] .data             PROGBITS         0000000002149840  01f49840
       000000000003d950  0000000000000000  WA       0     0     64
  [30] .bss              NOBITS           00000000021871c0  01f87190
       00000000000d1300  0000000000000000  WA       0     0     64
  [31] .comment          PROGBITS         0000000000000000  01f87190
       0000000000000029  0000000000000001  MS       0     0     1
  [32] .note.stapsdt     NOTE             0000000000000000  01f871bc
       000000000000d504  0000000000000000           0     0     4
  [33] .gnu_debuglink    PROGBITS         0000000000000000  01f946c0
       0000000000000018  0000000000000000           0     0     4
  [34] .symtab           SYMTAB           0000000000000000  01f946d8
       00000000001aede8  0000000000000018          35   73063     8
  [35] .strtab           STRTAB           0000000000000000  021434c0
       000000000033f3b1  0000000000000000           0     0     1
  [36] .shstrtab         STRTAB           0000000000000000  02482871
       0000000000000157  0000000000000000           0     0     1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  l (large), p (processor specific)

The center of interest is the .dynsym section, which is where the so-called dynamic symbols are stored. The previous JNI_CreateJavaVM is also here.

$ readelf --dyn-syms $JAVA_HOME/lib/server/libjvm.so | grep JNI_CreateJavaVM
   353: 0000000000f0b260  1396 FUNC    GLOBAL DEFAULT   14 JNI_CreateJavaVM@@SUNWprivate_1.1

With nm, you can also get the line number.

$ nm -gl --defined-only $JAVA_HOME/lib/server/libjvm.so | grep JNI_CreateJavaVM
0000000000f0b260 T JNI_CreateJavaVM	/vagrant/jdk/jdk13u/make/hotspot/../../src/hotspot/share/prims/jni.cpp:4012

Other requested libraries can be found using ldd.

$ ldd $JAVA_HOME/lib/server/libjvm.so
	linux-vdso.so.1 (0x00007fffeef52000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f92d1fab000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f92d1d8c000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f92d19ee000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f92d15fd000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f92d4408000)

Finally, when I built OpenJDK earlier, I put in a variable called --enable-dtrace, so the information on those DTrace Probes (USDT) is stored in the .note.stapsdt section.

$ readelf -n $JAVA_HOME/lib/server/libjvm.so | head -n 18

Displaying notes found in: .note.gnu.build-id
  Owner                 Data size	Description
  GNU                  0x00000014	NT_GNU_BUILD_ID (unique build ID bitstring)
    Build ID: 8fc95ac20704f218afd980ae1f303abfdf0b0586

Displaying notes found in: .note.stapsdt
  Owner                 Data size	Description
  stapsdt              0x00000050	NT_STAPSDT (SystemTap probe descriptors)
    Provider: hotspot
    Name: class__unloaded
    Location: 0x000000000092a226, Base: 0x0000000001be7618, Semaphore: 0x0000000000000000
    Arguments: 8@%rdx -4@%eax 8@152(%rdi) 1@$0
  stapsdt              0x00000050	NT_STAPSDT (SystemTap probe descriptors)
    Provider: hotspot
    Name: class__loaded
    Location: 0x000000000092a31b, Base: 0x0000000001be7618, Semaphore: 0x0000000000000000
    Arguments: 8@%rdx -4@%eax 8@152(%rdi) 1@%sil

Summary

What I learned so far

• java works like Launcher, after doing some initial processing go to libjvm.so.
• Variables for that Launcher can be viewed in detail with _JAVA_LAUNCHER_DEBUG.
• The entrance to libjvm.so is JNI_CreateJavaVM, and everything starts here.

Reference material

• HotSpot Runtime Overview