Create external (URL) monitoring with Docker + Prometheus

The first Advent calendar writes about building a monitoring tool called Prometheus that I recently touched Is your website monitored for life and death? "Can I build it myself?" "It's not a big system to contract with SaaS ..." "It's hard to start a server from scratch", etc. I think it is one of the functions with high hurdles. Was However, by combining Docker and Prometheus, it is surprisingly easy to create an external monitoring mechanism, so I will write it.

What is external (URL) monitoring?

What is external monitoring? Is the website or API server operating normally? Is a monitoring method to check by HTTP access Check "Is the response normal?" "Is the response returned within the specified number of seconds?" The good thing is that it is accessed in the same way as the user, so it is possible to check the normality/abnormality of the entire system or API including the network. I think that it is often used in combination with the method of monitoring server resources (CPU, memory, etc.) from the inside. I'm using an AWS service, but it's a simple diagram

スクリーンショット 2020-12-18 8.07.35.png


What to do this time

Do not explain

I won't write it here, so check it out ...


This time I will do it on Mac, but since I am using Docker, I can boot on windows or linux

Docker image used

Folder / file structure

━ docker-compose.yml
┣ prometheus
┃  ┗ alert_roles.yml
┃  ┗ prometheus.yml
┣ blackbox_exporter
┃  ┗ config.yml
┗ alertmanager
  ┗ config.yml

Create docker-compose.yml


version: '3'
    image: prom/prometheus
    container_name: prometheus
      - ./prometheus:/etc/prometheus
    command: "--config.file=/etc/prometheus/prometheus.yaml"
      - 9090:9090
    restart: always
    image: prom/blackbox-exporter:latest
      - ./blackbox_exporter/config.yml:/etc/blackbox_exporter/config.yml
    image: prom/alertmanager
    container_name: alertmanager
      - ./alertmanager:/etc/alertmanager
    command: "--config.file=/etc/alertmanager/config.yaml"
      - 9093:9093
    restart: always


I will describe the URL monitoring settings in yml format In modules:, create settings for each protocol you want to check (tcp, pop3, ssh are also provided by default). This time, http_post_2xx: is added as the setting of the POST method. HTTP headers can be set in headers: Please refer to Configuration of blackbox_exporter for the contents that can be set.


    prober: http
    prober: http
      method: POST
        xxx: yyyy


Describe the external monitoring settings on the prometheus main unit Evaluation contents are defined in `` alert_roles.yml''


  #Interval when prometheus goes to exporter etc. to get information(This time blackbox_exporter)
  scrape_interval:     15s 
  #Interval to evaluate rule(This time rule_alert specified in files_roles.yml is the evaluation content)
  evaluation_interval: 15s 
      monitor: 'codelab-monitor'

  - /etc/prometheus/alert_roles.yml

    - scheme: http
      - targets:
        - alertmanager:9093

  - job_name: 'prometheus'
      - targets:
        - prometheus:9090
  #Handle one monitoring condition in units called jobs
  - job_name: 'blackbox_http'
    metrics_path: /probe
    # blackbox_exporter config.Specify the module defined in yml
      module: [http_post_2xx]
      #Specify the URL to be monitored
      - targets:
        - '[target_url]'
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox_exporter:9115


#Alert definition
- name: blackbox_exporter
  - alert: http_success
    #Metrics and conditions to evaluate
    # probe_Conditions for success"blackbox_http"Applying to job
    expr: probe_success{job='blackbox_http'} != 1
    #Duration until evaluation is NG(NG in the alert state for 10 seconds)
    for: 10s
      severity: critical
      summary: "{{ $labels.instance }}: http request not return 200"
      description: "{{ $labels.instance }} http request not return 200 for more than 10 seconds."


alertmaneager will notify you This time I'm setting a notification to slack See Official for slack webhooks


  #Specify slack webhook URL
  slack_api_url: '[slack webhook url]'
  smtp_smarthost: 'localhost:25'
  smtp_require_tls: false
  smtp_from: 'Alertmanager'

  receiver: 'test-route'
  #Grouping settings(Alert name)
  group_by: '[alertname]'
  #Time to first wait for alert group notifications to be sent
  group_wait: 10s
  #Minimum transmission interval in alert group(Time to wait for a new alert to be sent)
  group_interval: 5m
  #Time to wait before sending the notification again
  repeat_interval: 1h

- name: 'test-route'
  #slack channel name
  - channel: '#general'
  # email(* Not displayed in slack)
  - to: "[email protected]"

docker container start

Start docker container This time I will use docker-compose to raise multiple containers at once


$ docker-compose build
# "-d"Run in the background with an option
#Without options the container will stop when you close the terminal
$ docker-compose up -d

#Confirm container startup
$ docker ps
CONTAINER ID        IMAGE                           COMMAND                  CREATED             STATUS              PORTS                    NAMES
ab4455256cff        prom/prometheus                 "/bin/prometheus --c…"   5 days ago          Up 5 days >9090/tcp   prometheus
830bf6475888        prom/alertmanager               "/bin/alertmanager -…"   5 days ago          Up 5 days >9093/tcp   alertmanager
b6fcd4f26f57        prom/blackbox-exporter:latest   "/bin/blackbox_expor…"   5 days ago          Up 5 days           9115/tcp                 prometheus_sample_blackbox_exporter_1

Check Prometheus dashboard

Go to http: // localhost: 9090 In actual operation, it supports https, port forwarding, etc., but this time it is a development terminal, so leave it as it is Enter the evaluation metrics specified in alert_roles.yml and click execute to display the evaluation results in a graph. This time the http request evaluates success (200 OK) and if it is 1.0 it will be normal

スクリーンショット 2020-12-18 20.36.04.png

You can check the alert status by selecting Alert from the menu. The situation is color coded so you can see at a glance which rating is in error.

スクリーンショット 2020-12-18 20.53.45.png

Check slack notifications

Change the URL to be monitored to a URL that does not exist, and check the Alert so that the http response returns the 500 series. When the http request returns an error, the status of the Alert menu turns red.

スクリーンショット 2020-12-18 21.03.11.png

And slack will be notified that an error has occurred

スクリーンショット 2020-12-18 21.10.26.png

This time it was a chat service called Slack, but if there is a request from people who do not have a habit of responding to chat notifications, "It is hard to notice if you notify Slack! It is also possible to call the phone when Alert occurs I think it is important not to notify, but to design in consideration of the method and possibility that humans can recognize.


I think Prometheus looks honestly simple ... (it has all the necessary features, so it's enough), For those who want to make a dashboard with a cool design, I think it will be cool if it is linked with Grafana. Since there is a Docker image like Prometheus, Grafana will start by adding it to docker-compose.yml If you are interested, why don't you give it a try?

スクリーンショット 2020-12-18 21.23.08.png (The image is an official demo)

For those who say, "It's pretty hard to build a monitor ..." How about building a monitoring function with Docker + Prometheus, which can be easily built and code-managed settings, to celebrate Christmas?

Recommended Posts

Create external (URL) monitoring with Docker + Prometheus
Create a Vue3 environment with Docker!
Create SolrCloud verification environment with Docker
Create Rails 6 + MySQL environment with Docker compose
Create a MySQL environment with Docker from 0-> 1
Create Spring Boot-gradle-mysql development environment with Docker
[Docker] Create Node.js + express + webpack environment with Docker
[Memo] Create a CentOS 8 environment easily with Docker
Create jupyter notebook with Docker and run ruby
Create a Spring Boot development environment with docker
Create an E2E test environment with Docker x Cypress
[ARM64] Docker server monitoring with New Relic on Docker on RasPi4
[Note] Create a java environment from scratch with docker
Create Chisel development environment with Windows10 + WSL2 + VScode + Docker
Launch MariaDB with Docker
Rails deploy with Docker
Run Pico with docker
Explode Docker with WSL2
Use Puphpeteer with Docker
Operate Emby with Docker
Try WildFly with Docker
Use ngrok with Docker
Run Payara with Docker
[Docker] Connection with MySQL
Php settings with Docker
Disposable PHP with Docker
Install Composer with Docker
Create a Docker image with the Oracle JDK installed (yum
Get the URL issued by ngrok with the Docker expose plugin
Create Rails5 and postgresql environment with Docker and make pgadmin available
I tried to create a padrino development environment with Docker