Thanks! An engineer in charge of the product inspection process in the production engineering department. Analyzing and visualizing csv logs with Excel Elastic Stack (docker-compose) --Elastic Stack is a continuation.
This article is intended for those who are new to Elastic Stack and who are thinking about trying it out.
When transferring a large number of files from filebeat to logstash, logstash may crash with the following error. I will show you how to deal with it.
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid1.hprof ...
Heap dump file created [3178685347 bytes in 34.188 secs]
warning: thread "[main]>worker11" terminated with exception (report_on_exception is true):
warning: thread "[main]>worker4" terminated with exception (report_on_exception is true):
java.lang.OutOfMemoryError: Java heap space
I have put a set of configuration files in GitLab, so please refer to it. Click here for repository-> elastic-stack
As the error content shows, the cause is insufficient heap size.
Create a jvm.options file and change the following heap size. Follow the instructions in JVM Setting to find out how much you should set. Please consult.
logstash/config/jvm.options
## JVM configuration
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
-Xms1g
-Xmx1g
You can limit the amount of processing by changing pipeline.batch.size or pipeline.workers. The default value for batch size is 125, and workers is the number of CPUs on the host PC. The official documentation is here.
logstash/config/pipelines.yml
- pipeline.id: filebeat-processing
pipeline.batch.size: 50
Regarding Elasticsearch, according to the Official Blog, the heap area usage of Elasticsearch from 7.7 Seems to go down significantly. If you are having trouble with Elasticsearch memory, please review the version.
Recommended Posts