Status monitoring of java application using Elasticsearch

Introduction

With Java support for APM since Elastic Stack 6.3 (Beta), I took a look at application performance monitoring.

It also monitors Spring Pet Clinic, which is a sample application of Spring Boot.

Comparison of performance monitoring methods

There are generally two ways to monitor Java performance:

When using Elastic Stack, the following methods can be considered to obtain Java-related information.

Comparison of each method

Method Recording method Recorded content Addition to Java options Remarks
JMX PC connected by jmc The user sets the items that can be acquired by the MBean server JMX port opening and permissions Included with JDK
JFR PC with flight recorder started locally or on jmc Items recorded by JFR Boot option for JFR recording. Add the same settings as JMX when recording with jmc
Elastic APM Get it with APM Client and go to Elasticsearch via APM server Information for web applications APM jar or APM server address APM Client API
Metricbeat Obtained from Jolokia and from Metricbeat to Elasticsearch The user sets the items that can be acquired by the MBean server Add Jolokia jar, or open JMX port and permissions Jolokia

Kibana + Elasticsearch + APM Server environment preparation

Prepare the minimum environment using the official image from Elastic.

https://www.docker.elastic.co/

docker-compose.yml


version: '3'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.5.1
    volumes:
      - ./esdata:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
  kibana:
    image: docker.elastic.co/kibana/kibana:6.5.1
    ports:
      - 5601:5601
  apm:
    image: docker.elastic.co/apm/apm-server:6.5.1
    ports:
     - 8200:8200

Build Spring Pet Clinic

This time, we will build using OpenJDK 11 of Ubuntu Server 18.04. However, it cannot be built as it is, so it takes some time.

terminal


sudo apt install default-jdk
git clone https://github.com/spring-projects/spring-petclinic.git
cd spring-petclinic

One hassle

pom.xml


<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.22.1</version>
    <configuration>
        <useSystemClassLoader>false</useSystemClassLoader>
    </configuration>
</plugin>

Build

terminal


./mvnw package

Start-up

terminal


java -jar target/*.jar

OK if you access the browser and the screen is displayed

SpringPetClinic.png

Monitoring using APM

Download APM

terminal


curl -O https://search.maven.org/remotecontent?filepath=co/elastic/apm/elastic-apm-agent/1.0.1/elastic-apm-agent-1.0.1.jar

Add APM Client to java runtime arguments. For detailed setting method, check Official and Kibana's Setup Instructions.

terminal(Additional example)


java -javaagent:elastic-apm-agent-1.0.1.jar -Delastic.apm.service_name=petclinic -Delastic.apm.application_packages=org.example -Delastic.apm.server_urls=http://localhost:8200 -jar target/*.jar

If PetClinic starts, APM is running correctly. If you can't start it, you should see an error message in the terminal, so check the error message.

Post-start settings

Select APM Setup Instructions from Kibana and press the Load Kibana objects button at the bottom. If successful, the monitored data will be read.

apm_check.png

Information that can be obtained

Screenshot of the result of proper operation of the Pet Clinic screen

APM dashboard

apm_dash_1.png

Request processing status

You can check the list of average processing time of requests. For requests for which you want to check the details, you can move to the details screen by clicking the URL in the table at the bottom of the screen.

apm_request.png

Request details

Details such as query issuance status and request parameters can be confirmed from the time required for each process.

apm_transaction_1.png apm_transaction_2.png apm_transaction_3.png

Java application error

apm_error.png apm_error_detail.png

Monitoring using Metricbeat

There are two types of monitoring configurations using Metricbeat.

  1. Make Java application coexist with Jolokia's JVM Agent service and war
  2. Connect with JMX and access via Jolokia's proxy mode

This time, I will explain each of them. Since each installation method is different, please select according to your environment. For example, when building a new server, add apm jar and Jolokia jar to the startup parameters in advance. When adding to an existing service, connect to the existing application server using Jolokia's proxy mode, which connects to the existing application server with JMX, so as not to significantly affect the existing service.

The outline is as follows. For the former, use the JVM Anget mode. As for war, it is only deployed on the application server, so this time it is close to the latter, so I will omit it. Regarding the latter, we will use it by launching a lightweight Web application server and deploying war there. In both cases, metricbeat will be operated on the same server.

Also, please refer to Official for how to set Jolokia and what you can do.

Metricbeat initial settings

terminal


curl -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-6.5.1-linux-x86_64.tar.gz

metricbeat.yml settings

Set the URL of Elastic Search.

metricbeat.yml


output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

Jolokia settings

1. When coexisting Jolokia's JVM Agent service in a Java application

Download Jolokia's JVM Agent.

terminal


curl -O http://search.maven.org/remotecontent?filepath=org/jolokia/jolokia-jvm/1.6.0/jolokia-jvm-1.6.0-agent.jar

Java option

terminal


java -javaagent:jolokia-jvm-1.6.0-agent.jar -jar target/spring-petclinic-2.1.0.BUILD-SNAPSHOT.jar

Confirmation of Jolokia startup

If you forget / at the end, JSON response will not be returned

terminal


curl http://localhost:8778/jolokia/ | jq

metricbeat settings for Jolokia jvm agent

Activate the Jolokia module. This time, for the sake of explanation, the information on the sytem module is unnecessary, so it is disabled.

terminal


./metricbeat modules disable system
./metricbeat modules enable jolokia

Jolokia.yml settings

This time, the startup time of the JVM and the usage rate of the Heap memory are acquired. Select the information you want to acquire by actually connecting with jmc etc. and checking.

yml:module.d/jolokia.yml


- module: jolokia
  metricsets: ["jmx"]
  period: 10s
  hosts: ["localhost:8778"]
  path: "/jolokia/?ignoreErrors=true&canonicalNaming=false"
  namespace: "metrics"
  jmx.mappings:
    - mbean: 'java.lang:type=Runtime'
      attributes:
        - attr: Uptime
          field: uptime
    - mbean: 'java.lang:type=Memory'
      attributes:
        - attr: HeapMemoryUsage
          field: memory.heap_usage
        - attr: NonHeapMemoryUsage
          field: memory.non_heap_usage

Operation check of Jolokia module

terminal


./metricbeat test modules

Start metricbeat and check its operation

I will omit the service of metricbeat and how to use Kibana.

terminal


./metricbeat

heap_usage.png

2. Connect with JMX and access via Jolokia proxy mode

Since there are many settings, perform in the following order.

Enable JMX for application server

This time, the minimum startup parameters were added without setting security such as authentication. The Jolokia module of metricbeat supports authentication by user ID and password.

termainl


java -Dcom.sun.management.jmxremote.port=7091 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -jar target/spring-petclinic-2.1.0.BUILD-SNAPSHOT.jar

Check the connection using JMC.

jmc.png

Connecting to an application server using Jolokia's proxy mode

How to connect to the target Java application with JMX using Jolokia's proxy mode Click here for technical details and detailed settings (https://jolokia.org/reference/html/proxy.html)

jetty server settings

Download the jetty server and change the port number, etc. This time, PetClinic uses 8080 port, so change it to 8081 port. Select the jetty version from Official as needed.

terminal


curl -O https://repo1.maven.org/maven2/org/eclipse/jetty/jetty-distribution/9.4.14.v20181114/jetty-distribution-9.4.14.v20181114.tar.gz
tar xf jetty-distribution-9.4.14.v20181114.tar.gz
cd jetty-distribution-9.4.14.v20181114/

start.ini


## Connector port to listen on
jetty.http.port=8081

Next, if you want to connect using Jolokia's proxy mode, you need to add an option. This time, add the necessary settings to web.xml. In addition, the authentication function cannot be enabled in this configuration, so it is disabled.

terminal


cd webapps/
curl -O http://search.maven.org/remotecontent?filepath=org/jolokia/jolokia-war/1.6.0/jolokia-war-1.6.0.war
mv jolokia-war-1.6.0.war jolokia.war
mkdir jolokia
cd jolokia
jar xf ../jolokia.war 

Add the following settings in the servlet tag.

WEB-INF/web.xml


<servlet>
  <init-param>
    <param-name>dispatcherClasses</param-name>
    <param-value>org.jolokia.jsr160.Jsr160RequestDispatcher</param-value>
  </init-param>
  ...
</servlet>

Disable the authentication function.

WEB-INF/web.xml


<web-app>
  ...
  <!--
  Security enabled by default. Please update to match you specific security setup (e.g. the auth-method)
  -->
  <!--
  <login-config>
    <auth-method>BASIC</auth-method>
    <realm-name>jolokia</realm-name>
  </login-config>

  <security-constraint>
    <web-resource-collection>
      <web-resource-name>Jolokia-Agent Access</web-resource-name>
      <url-pattern>/*</url-pattern>
    </web-resource-collection>
    <auth-constraint>
      <role-name>jolokia</role-name>
    </auth-constraint>
  </security-constraint>

  <security-role>
    <role-name>jolokia</role-name>
  </security-role>
  -->
</web-app>

Confirmation of Jolokia server startup

If you forget / at the end, JSON response will not be returned

tarminal


curl -O http://localhost:8081/jolokia/ | jq

Checking the connection by proxy mode of Jolokia server

tarminal(request)


curl -X POST -H 'Content-Type:application/json' -d '{"type":"read","mbean":"Tomcat:type=Server", "attribute" : "serverInfo", "target" : { "url": "service:jmx:rmi:///jndi/rmi://localhost:7091/jmxrmi" } }' http://localhost:8081/jolokia/ | jq

terminal(Result example)


{
  "request": {
    "mbean": "Tomcat:type=Server",
    "attribute": "serverInfo",
    "type": "read",
    "target": {
      "url": "service:jmx:rmi:///jndi/rmi://localhost:7091/jmxrmi"
    }
  },
  "value": "Apache Tomcat/9.0.12",
  "timestamp": 1543418665,
  "status": 200
}

If the settings are insufficient, an error message will be output, so check the contents and correct.

Setting Jolokia proxy mode for metricbeat

Activate the Jolokia module. This time, for the sake of explanation, the information on the sytem module is unnecessary, so it is disabled.

terminal


./metricbeat modules disable system
./metricbeat modules enable jolokia

jolokia.yml settings

This time, the startup time of the JVM and the usage rate of the Heap memory are acquired. Select the information you want to acquire by actually connecting with jmc etc. and checking. I've also added an MBean to get the version of Tomcat for testing to see if it's really getting it remotely.

yml:module.d/jolokia.yml


- module: jolokia
  metricsets: ["jmx"]
  period: 10s
  hosts: ["localhost:8081"]
  path: "/jolokia/?ignoreErrors=true&canonicalNaming=false"
  namespace: "metrics"
  jmx.mappings:
    - mbean: 'java.lang:type=Runtime'
      attributes:
        - attr: Uptime
          field: uptime
      target:
        url: 'service:jmx:rmi:///jndi/rmi://localhost:7091/jmxrmi'
    - mbean: 'java.lang:type=Memory'
      attributes:
        - attr: HeapMemoryUsage
          field: memory.heap_usage
        - attr: NonHeapMemoryUsage
          field: memory.non_heap_usage
      target:
        url: 'service:jmx:rmi:///jndi/rmi://localhost:7091/jmxrmi'
    - mbean: 'Tomcat:type=Server'
      attributes:
        - attr: serverInfo
          field: tomcat.serverInfo
      target:
        url: 'service:jmx:rmi:///jndi/rmi://localhost:7091/jmxrmi'

Operation check of Jolokia module

terminal


./metricbeat test modules

From here onward, the above operation check is the same, so it is omitted.

in conclusion

I was able to easily get the metrics of the web application by using the apm agent. I felt that it was very compatible with microservices. You should consider introducing a newly created Web application.

The Jolokia module of metricbeat is attractive for flexible data acquisition using MBeans. It can be applied to other than Web applications. You can also get metrics for existing services as long as you have JMX ready.

Recommended Posts

Status monitoring of java application using Elasticsearch
Check the status of Java application without using monitoring tool
Summary of object-oriented programming using Java
I tried using GoogleHttpClient of Java
Is the version of Elasticsearch you are using compatible with Java 11?
Try using Java framework Nablarch [Web application]
I tried using Elasticsearch API in Java
ERRORCODE = -4471 occurs in Java application using Db2.
Memory measurement of Java application on Windows
Acquisition of input contents using Scanner (Java)
Role of JSP in Web application [Java]
Verification of performance impact when using Java volatile
Try similar search of Image Search using Java SDK [Search]
Example of using addition faster than using StringBuilder (Java)
Story of test automation using Appium [Android / java]
[Java] Overview of Java
Check the rate limit application status of docker pull
[Java10] Be careful of using var and generics together
Expired collection of java
Sorting using java comparator
Predicted Features of Java
[Java] Significance of serialVersionUID
Using Elasticsearch Java-API (Aggregation)
NIO.2 review of java
Review of java Shilber
java --Unification of comments
Scraping practice using Java ②
History of Java annotation
Scraping practice using Java ①
NIO review of java
[Java] Three features of Java
Summary of using DBFlow
Summary of Java support 2018
Story of making a task management application with swing, java
[Java] Comparison method of character strings and comparison method using regular expressions
Using multiple versions of Java with Brew on Mac + jEnv
Command to check the number and status of Java threads
[Java] Try editing the elements of the Json string using the library
Elasticsearch Operation via REST API using Apache HttpClient in Java
The story of pushing Java to Heroku using the BitBucket pipeline
Common misconceptions about Java application resource limitations when using Kubernetes