Notify Slack of process state changes using Supervisor's EventListener

Introduction

Do you run a supervisor and rarely have a process down? I think there is a way to monitor it externally, but I tried to notify Slack of the state change of the process using the EventListener of Supervisor, which I had been interested in for a long time.

What i did

Creating a configuration file

Create a configuration file as shown below.

/etc/supervisord.d/process_state_event_listener.conf


[eventlistener:process_state_event_listener]
command=python /path/to/process_state_event_listener.py
events=PROCESS_STATE
redirect_stderr=false
stdout_logfile=/var/log/supervisor/event_listener_stdout.log
stdout_logfile_maxbytes=1MB
stdout_logfile_backups=5
stderr_logfile=/var/log/supervisor/event_listener_stderr.log
stderr_logfile_maxbytes=1MB
stderr_logfile_backups=5
environment=SLACK_WEB_HOOK_URL="https://hooks.slack.com/services/xxxxxxx/xxxxxxx/xxxxxxxxxx"

events We want to be notified of all process state changes, so specify PROCESS_STATE.

environment Set the URL for Incoming WebHook in your environment variable to notify Slack.

Reference URL

Creating an event listener

The final code looks like this:

/path/to/process_state_event_listener.py


import sys
import os
from supervisor import childutils
from datetime import datetime
import slackweb


def main():
    while True:
        run_forever()


def run_forever():

    headers, payload = childutils.listener.wait(sys.stdin, sys.stdout)
    eventname = headers['eventname']

    payload_header, data = childutils.eventdata(payload+'\n')

    childutils.listener.ok(sys.stdout)

    color = "good"
    if eventname in ['PROCESS_STATE_BACKOFF', 'PROCESS_STATE_EXITED', 'PROCESS_STATE_FATAL', 'PROCESS_STATE_UNKNOWN']:
        color = "danger"

    attachment = {
        "title": "event",
        "text": eventname,
        "color": color,
        "fields": [{
            "title": "datetime",
            "value": datetime.now().strftime("%Y/%m/%d %H:%M:%S")
        }, {
            "title": "hostname",
            "value": os.uname()[1]
        }]
    }

    if 'processname' in payload_header:
        attachment["fields"].append({
            "title": "process",
            "value": payload_header['processname']
        })

    if 'expected' in payload_header:
        attachment["fields"].append({
            "title": "expected",
            "value": payload_header['expected']
        })

    if 'SLACK_WEB_HOOK_URL' in os.environ:
        slack_url = os.environ['SLACK_WEB_HOOK_URL']
        slack = slackweb.Slack(url=slack_url)
        slack.notify(attachments=[attachment])


if __name__ == '__main__':
    main()

Notification image

supervisor_slack.png

Reference URL

--POST message from python to Slack via incoming webhook --Qiita

in conclusion

There is a plugin called superlance if you want to use email notification, but OP25B I didn't receive the email on gmail due to a problem, so I notified Slack. Since the use of Supervisor has increased, I would like to continue to apply it in various ways.

Recommended Posts

Notify Slack of process state changes using Supervisor's EventListener
Notify Slack of GitHub push
Allow Slack to notify you of the end of a time-consuming program process
Regularly notify Slack of missed Backlog issues
Notify Slack of how Keras is learning
I tried to notify slack of Redmine update