Status monitoring of java application using Elasticsearch

Introduction

With Java support for APM since Elastic Stack 6.3 (Beta), I took a look at application performance monitoring.

It also monitors Spring Pet Clinic, which is a sample application of Spring Boot.

Comparison of performance monitoring methods

There are generally two ways to monitor Java performance:

• JMX connection with Java Mission Control (jmc) etc.
• Java Fright Recoder(JFR)

When using Elastic Stack, the following methods can be considered to obtain Java-related information.

• Java Agent in APM
• Jolokia module of Metricbeat

Comparison of each method

Kibana + Elasticsearch + APM Server environment preparation

Prepare the minimum environment using the official image from Elastic.

https://www.docker.elastic.co/

docker-compose.yml

version: '3'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.5.1
    volumes:
      - ./esdata:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
  kibana:
    image: docker.elastic.co/kibana/kibana:6.5.1
    ports:
      - 5601:5601
  apm:
    image: docker.elastic.co/apm/apm-server:6.5.1
    ports:
     - 8200:8200

Build Spring Pet Clinic

This time, we will build using OpenJDK 11 of Ubuntu Server 18.04. However, it cannot be built as it is, so it takes some time.

terminal

sudo apt install default-jdk
git clone https://github.com/spring-projects/spring-petclinic.git
cd spring-petclinic

One hassle

pom.xml

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.22.1</version>
    <configuration>
        <useSystemClassLoader>false</useSystemClassLoader>
    </configuration>
</plugin>

Build

terminal

./mvnw package

Start-up

terminal

java -jar target/*.jar

OK if you access the browser and the screen is displayed

Monitoring using APM

Download APM

terminal

curl -O https://search.maven.org/remotecontent?filepath=co/elastic/apm/elastic-apm-agent/1.0.1/elastic-apm-agent-1.0.1.jar

Add APM Client to java runtime arguments. For detailed setting method, check Official and Kibana's Setup Instructions.

terminal(Additional example)

java -javaagent:elastic-apm-agent-1.0.1.jar -Delastic.apm.service_name=petclinic -Delastic.apm.application_packages=org.example -Delastic.apm.server_urls=http://localhost:8200 -jar target/*.jar

If PetClinic starts, APM is running correctly. If you can't start it, you should see an error message in the terminal, so check the error message.

Post-start settings

Select APM Setup Instructions from Kibana and press the Load Kibana objects button at the bottom. If successful, the monitored data will be read.

Information that can be obtained

Screenshot of the result of proper operation of the Pet Clinic screen

APM dashboard

Request processing status

You can check the list of average processing time of requests. For requests for which you want to check the details, you can move to the details screen by clicking the URL in the table at the bottom of the screen.

Request details

Details such as query issuance status and request parameters can be confirmed from the time required for each process.

Java application error

Monitoring using Metricbeat

There are two types of monitoring configurations using Metricbeat.

1. Make Java application coexist with Jolokia's JVM Agent service and war
2. Connect with JMX and access via Jolokia's proxy mode

This time, I will explain each of them. Since each installation method is different, please select according to your environment. For example, when building a new server, add apm jar and Jolokia jar to the startup parameters in advance. When adding to an existing service, connect to the existing application server using Jolokia's proxy mode, which connects to the existing application server with JMX, so as not to significantly affect the existing service.

The outline is as follows. For the former, use the JVM Anget mode. As for war, it is only deployed on the application server, so this time it is close to the latter, so I will omit it. Regarding the latter, we will use it by launching a lightweight Web application server and deploying war there. In both cases, metricbeat will be operated on the same server.

• If Jolokia and metricbeat are executed on different servers, please note that the date stored in Elasticsearch will be the server on which metricbeat is executed.

Also, please refer to Official for how to set Jolokia and what you can do.

Metricbeat initial settings

terminal

curl -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-6.5.1-linux-x86_64.tar.gz

metricbeat.yml settings

Set the URL of Elastic Search.

metricbeat.yml

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

Jolokia settings

1. When coexisting Jolokia's JVM Agent service in a Java application

Download Jolokia's JVM Agent.

terminal

curl -O http://search.maven.org/remotecontent?filepath=org/jolokia/jolokia-jvm/1.6.0/jolokia-jvm-1.6.0-agent.jar

Java option

terminal

java -javaagent:jolokia-jvm-1.6.0-agent.jar -jar target/spring-petclinic-2.1.0.BUILD-SNAPSHOT.jar

Confirmation of Jolokia startup

If you forget / at the end, JSON response will not be returned

terminal

curl http://localhost:8778/jolokia/ | jq

metricbeat settings for Jolokia jvm agent

Activate the Jolokia module. This time, for the sake of explanation, the information on the sytem module is unnecessary, so it is disabled.

terminal

./metricbeat modules disable system
./metricbeat modules enable jolokia

Jolokia.yml settings

This time, the startup time of the JVM and the usage rate of the Heap memory are acquired. Select the information you want to acquire by actually connecting with jmc etc. and checking.

yml:module.d/jolokia.yml

- module: jolokia
  metricsets: ["jmx"]
  period: 10s
  hosts: ["localhost:8778"]
  path: "/jolokia/?ignoreErrors=true&canonicalNaming=false"
  namespace: "metrics"
  jmx.mappings:
    - mbean: 'java.lang:type=Runtime'
      attributes:
        - attr: Uptime
          field: uptime
    - mbean: 'java.lang:type=Memory'
      attributes:
        - attr: HeapMemoryUsage
          field: memory.heap_usage
        - attr: NonHeapMemoryUsage
          field: memory.non_heap_usage

Operation check of Jolokia module

terminal

./metricbeat test modules

Start metricbeat and check its operation

I will omit the service of metricbeat and how to use Kibana.

terminal

./metricbeat

2. Connect with JMX and access via Jolokia proxy mode

Since there are many settings, perform in the following order.

• Enable JMX for application server
• Connecting to an application server using Jolokia's proxy mode

Enable JMX for application server

This time, the minimum startup parameters were added without setting security such as authentication. The Jolokia module of metricbeat supports authentication by user ID and password.

termainl

java -Dcom.sun.management.jmxremote.port=7091 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -jar target/spring-petclinic-2.1.0.BUILD-SNAPSHOT.jar

Check the connection using JMC.

Connecting to an application server using Jolokia's proxy mode

How to connect to the target Java application with JMX using Jolokia's proxy mode Click here for technical details and detailed settings (https://jolokia.org/reference/html/proxy.html)

jetty server settings

Download the jetty server and change the port number, etc. This time, PetClinic uses 8080 port, so change it to 8081 port. Select the jetty version from Official as needed.

terminal

curl -O https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-distribution/9.4.14.v20181114/jetty-distribution-9.4.14.v20181114.tar.gz
tar xf jetty-distribution-9.4.14.v20181114.tar.gz
cd jetty-distribution-9.4.14.v20181114/

start.ini

## Connector port to listen on
jetty.http.port=8081

Next, if you want to connect using Jolokia's proxy mode, you need to add an option. This time, add the necessary settings to web.xml. In addition, the authentication function cannot be enabled in this configuration, so it is disabled.

terminal

cd webapps/
curl -O http://search.maven.org/remotecontent?filepath=org/jolokia/jolokia-war/1.6.0/jolokia-war-1.6.0.war
mv jolokia-war-1.6.0.war jolokia.war
mkdir jolokia
cd jolokia
jar xf ../jolokia.war 

Add the following settings in the servlet tag.

WEB-INF/web.xml

<servlet>
  <init-param>
    <param-name>dispatcherClasses</param-name>
    <param-value>org.jolokia.jsr160.Jsr160RequestDispatcher</param-value>
  </init-param>
  ...
</servlet>

Disable the authentication function.

WEB-INF/web.xml

<web-app>
  ...
  <!--
  Security enabled by default. Please update to match you specific security setup (e.g. the auth-method)
  -->
  <!--
  <login-config>
    <auth-method>BASIC</auth-method>
    <realm-name>jolokia</realm-name>
  </login-config>

  <security-constraint>
    <web-resource-collection>
      <web-resource-name>Jolokia-Agent Access</web-resource-name>
      <url-pattern>/*</url-pattern>
    </web-resource-collection>
    <auth-constraint>
      <role-name>jolokia</role-name>
    </auth-constraint>
  </security-constraint>

  <security-role>
    <role-name>jolokia</role-name>
  </security-role>
  -->
</web-app>

Confirmation of Jolokia server startup

If you forget / at the end, JSON response will not be returned

tarminal

curl -O http://localhost:8081/jolokia/ | jq

Checking the connection by proxy mode of Jolokia server

tarminal(request)

curl -X POST -H 'Content-Type:application/json' -d '{"type":"read","mbean":"Tomcat:type=Server", "attribute" : "serverInfo", "target" : { "url": "service:jmx:rmi:///jndi/rmi://localhost:7091/jmxrmi" } }' http://localhost:8081/jolokia/ | jq

terminal(Result example)

{
  "request": {
    "mbean": "Tomcat:type=Server",
    "attribute": "serverInfo",
    "type": "read",
    "target": {
      "url": "service:jmx:rmi:///jndi/rmi://localhost:7091/jmxrmi"
    }
  },
  "value": "Apache Tomcat/9.0.12",
  "timestamp": 1543418665,
  "status": 200
}

If the settings are insufficient, an error message will be output, so check the contents and correct.

Setting Jolokia proxy mode for metricbeat

Activate the Jolokia module. This time, for the sake of explanation, the information on the sytem module is unnecessary, so it is disabled.

terminal

./metricbeat modules disable system
./metricbeat modules enable jolokia

jolokia.yml settings

This time, the startup time of the JVM and the usage rate of the Heap memory are acquired. Select the information you want to acquire by actually connecting with jmc etc. and checking. I've also added an MBean to get the version of Tomcat for testing to see if it's really getting it remotely.

yml:module.d/jolokia.yml

- module: jolokia
  metricsets: ["jmx"]
  period: 10s
  hosts: ["localhost:8081"]
  path: "/jolokia/?ignoreErrors=true&canonicalNaming=false"
  namespace: "metrics"
  jmx.mappings:
    - mbean: 'java.lang:type=Runtime'
      attributes:
        - attr: Uptime
          field: uptime
      target:
        url: 'service:jmx:rmi:///jndi/rmi://localhost:7091/jmxrmi'
    - mbean: 'java.lang:type=Memory'
      attributes:
        - attr: HeapMemoryUsage
          field: memory.heap_usage
        - attr: NonHeapMemoryUsage
          field: memory.non_heap_usage
      target:
        url: 'service:jmx:rmi:///jndi/rmi://localhost:7091/jmxrmi'
    - mbean: 'Tomcat:type=Server'
      attributes:
        - attr: serverInfo
          field: tomcat.serverInfo
      target:
        url: 'service:jmx:rmi:///jndi/rmi://localhost:7091/jmxrmi'

Operation check of Jolokia module

terminal

./metricbeat test modules

From here onward, the above operation check is the same, so it is omitted.

in conclusion

I was able to easily get the metrics of the web application by using the apm agent. I felt that it was very compatible with microservices. You should consider introducing a newly created Web application.

The Jolokia module of metricbeat is attractive for flexible data acquisition using MBeans. It can be applied to other than Web applications. You can also get metrics for existing services as long as you have JMX ready.