Analyze and visualize csv logs with Excel Elastic Stack (docker-compose) --Set up with docker-compose

Introduction

Thanks! An engineer in charge of the product inspection process in the production engineering department. It is a continuation of Analyzing and visualizing csv logs with Excel Elastic Stack (docker-compose) --What is Elastic Stack.

Target audience

This article is intended for those who are new to Elastic Stack and who are thinking about trying it out.

Content of this article

With docker-compose, we have summarized what you need to do to launch Elastic Stack. I have put a set of configuration files in GitLab, so please refer to it.

Click here for repository-> elastic-stack

file organization

Kibana, Elasticsearch, Logstash, Filebeat 4 Docker will be launched together with Docker-compose. I created a config folder in each directory and prepared various settings.

docker-compose settings

• Elasticsearch
  Please refer to officially described docker-compose.
• Kibana
  Please refer to officially described docker-compose.
• Logstash
  Refer to logstash-docker on official GitHub. I also referred to this repository for the set of configuration files in config. I think I should have referred to the settings in the logstash repository.
• Filebeat
  For the configuration file, refer to Official GitHub.

docker-compose.yml

version: '3.7'
services:
  es01:
    build: ./elasticsearch
    container_name: es01
    environment:
      - node.name=es01
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      #Uncomment if you want to persist the data
      #- esdata01:/usr/share/elasticsearch/data 
      - ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - ./elasticsearch/config/log4j2.properties:/usr/share/elasticsearch/config/log4j2.properties
    #If you want to access the 9200, uncomment it
    #ports:
    #  - 9200:9200
    networks:
      - esnet

  kibana01:
    build: ./kibana
    container_name: kibana01
    links:
      - es01:elasticsearch
    ports:
      - 5601:5601
    networks:
      - esnet

  logstash01:
    build: ./logstash
    container_name: logstash01
    links:
      - es01:elasticsearch
    volumes:
      - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml
      - ./logstash/config/log4j2.properties:/usr/share/logstash/config/log4j2.properties
      - ./logstash/config/pipelines.yml:/usr/share/logstash/config/pipelines.yml
      - ./logstash/pipeline/logstash.conf:/usr/share/logstash/pipeline/logstash.conf
      - ./logstash/extra_patterns/date_jp:/opt/logstash/extra_patterns
    networks:
      - esnet

  filebeat01:
    build: ./beats/filebeat
    container_name: filebeat01
    links:
      - logstash01:logstash
    volumes:
      - ./beats/filebeat/config/filebeat.yml:/usr/share/filebeat/filebeat.yml
      - ./beats/filebeat/logs/:/var/log/
    networks:
      - esnet

#Uncomment if you want to persist the data
#volumes:
#  esdata01:
#    driver: local

networks:
  esnet:
    driver: bridge

Preparation before starting Elasticsearch

You need to add vm.mem_map_count settings before launching. If the kernel parameters are not reflected correctly, please execute the following.

$ sudo sysctl --system

If you start it without setting it, the following error will occur and it will not start.

max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

logstash settings

log4j2.properties I think that the log4j2.properties file should be set as necessary to set the running log at startup. If set correctly, I think the setup will be smooth. pipeline.yml logstash has input (input)-> processing (filter)-> output (output) as one pipeline, but when you want to input various logs, in the filter process, branch using if-else. Need to be processed. By adding the multipipeline setting, you can prepare multiple pipelines without using if-else. logstash.conf The specific processing is described here. Describe the content you want to implement in the three plugins input, filter, and output.

extra_patterns/date_jp Use this when you want to add a custom pattern when using the grok filter. Check Official custom_patterns. Since date_jp is an appropriately attached file name, you can use a descriptive name.

filebeat settings

filebeat.yml This can be achieved by setting enabled to true and setting it to * .csv in the settings when you want to read the csv file.

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/*.csv

filebeat normally transfers the contents of a file line by line, but by adding the multiline setting, multiple lines can be transferred together with \ n delimiters based on the set rules. Summary of Multiline settings that handle multiple lines with Filebeat I received it.

 ### Multiline options

  # Multiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  #multiline.pattern: ^\[
  multiline.pattern: (End)

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  #multiline.negate: false
  multiline.negate: true

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  #multiline.match: after
  multiline.match: before

logs folder

This is a folder for inputting sample logs for testing.

Start-up

You can check the startup of kibana by accessing http: // localhost: 5601 after starting docker for a while. If you look at the kibana startup log, you can see that it has started.

$ sudo docker-compose up -d

Data confirmation

Select Index Management-> Stack Management.

Add Index Pattern from Kibana-> Index Patterns.

The data captured by filebeat is filebeat-7.9.2, so you can add the data captured by Index Pattern by entering filebeat- * etc. in the index pattern name. You can use an asterisk to create an Index Pattern that contains multiple data sources.

Select the field to use for the time field. @timestamp is the default timestamp. If @timestamp is not processed by logstash etc., the time when the log was imported is set automatically.

Select Kibana-> Discover for the index and change the time range to match the log.

Finally, confirm that the data has been imported and you are done.

Finally

I explained the configuration file when launching with docker-compose. In the future, I would like to introduce how to analyze dates such as Japan time and how to handle csv files.