Run logstash with Docker and try uploading data to Elastic Cloud

environment

Environment: Elasticsearch uses Elastic Cloud logstash: 7.9.1 metricbeat: 7.9.1

Elastic Cloud is already set. Cloud Id and password are also noted.

The relationship between each function is like this

logstash image acquisition

Check the path of the latest image from the following

logstash | Docker @ Elastic

At the time of writing, the latest was 7.9.1.

docker pull docker.elastic.co/logstash/logstash:7.9.1

Start-up

docker run --rm -it -p 5044:5044 docker.elastic.co/logstash/logstash:7.9.1

Success if the following is displayed

[2020-09-22T02:58:20,563][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

Start confirmation

$ docker ps

CONTAINER ID        IMAGE                                       COMMAND                  CREATED             STATUS              PORTS                              NAMES
b8867b83444c        docker.elastic.co/logstash/logstash:7.9.1   "/usr/local/bin/dock…"   7 minutes ago       Up 7 minutes        0.0.0.0:5044->5044/tcp, 9600/tcp   heuristic_mestorf

Metricbeat

Upload data to Elastic Cloud Create a data source with Metricbeat and send it to Logstash

Download Metricbeat Download Metricbeat • Ship Metrics to Elasticsearch | Elastic

Edit metricbeat.yml

Comment out elasticsearch and enable logstash

#output.elasticsearch:
# Array of hosts to connect to.
# hosts: ["localhost:9200"]

output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

Run metricbeat.yml

./metricbeat -e

Check logstash

If the following is displayed on the logstash side, the data has been sent to metricbeat-> logstash.

{
    "@timestamp" => 2020-09-22T04:31:06.525Z,
         "event" => {
          "module" => "system",
        "duration" => 74705442,
         "dataset" => "system.process"
    },
       "service" => {
        "type" => "system"
    },
     "metricset" => {
          "name" => "process",
        "period" => 10000
    },
abridgement

Logstash setting change

By default, the received data is only output to the standard output, so change the setting to send the data to Elastic Cloud.

Check the Index name of the destination.

Throw Get like below to Endpoint

Get

https://xxxxxxxxxxx42b7b40a0773f8801e.us-east-1.aws.found.io:9243/_cat/indices?v

Add logstash.conf

Make the following contents in logstash.conf.

logstash.conf

input {
  beats {
    port => 5044
  }
}
output {
    elasticsearch {
        hosts => ["https://xxxxxxxxxxx42b7b40a0773f8801e.us-east-1.aws.found.io:9243"]
        user => "elastic"
        password => "xxxxxxxxxxxxxxxxxxxxx"
        index => "metricbeat-7.9.1-2020.09.22-000001"
    }
}

Restart logstash

docker run --rm -it -p 5044:5044 -v /Users/xxxxx/Documents/Logstash:/usr/share/logstash/pipeline/ docker.elastic.co/logstash/logstash:7.9.1 

Confirmed with Kibana

From Kibana's HOME screen, select Visualize and Explore Data → Metrics

Select Host Metrics

Success if displayed like this