Email hipchat with postfix, fluentd and python on Azure

Overview

• Since there are many alert emails in infrastructure operation, I want to parse them properly and paste them in chat.
• I wanted to use it in various ways
• If you send an email this time, just send it to hipchat
• Processing flow

1. [Premise] Alert mail is sent from the monitoring system (this time it is troublesome, so use a command from localhost)
2. Receive mail with postfix
3. Kick python script 1 with / etc / aliases

• Python script 1 easily parses emails and delivers them to fluentd

4. Fluentd parses depending on the contents of the email and kicks python script 2 (this time it passes through without parse)

• python script 2 posts the data received from fluentd to hipchat
• Something that is awkward implementation will be described later (it is not necessary to go through fluentd)

Let's go in order

Build a server on Azure

Register from the following when you can get a free ticket for 20,000 yen for free in the first month. Microsoft Azure Free Trial: Try Azure | Azure Free Trial

The phone number, the first 0 is pulled out ... or something like that, it's usually registered. reference: [[Subscription] Personal authentication for new Windows Azure contracts --Microsoft Azure support team site --Site Home --MSDN Blogs](http://blogs.msdn.com/b/dsazurejp/archive/2013/09/12/ new-subscription-personal-indentification.aspx)

I made centos6.5 (what is OpenLogic?) With 1 core CPU and 1.75GB memory. The management screen is a resource, so it's Shareotsu.

I installed fluentd (for fluent-cat) with td-agent, ruby2.0.0 and gem.

reference: Install ruby using rbenv (CentOS edition) --Qiita

• I added openssl-devel and so on Install Fluentd from rpm package | Fluentd

Script execution from postfix

I haven't messed with the annoying virtual domain or anything around it. Please feel free to use it.

I messed with postfix in 3 places

/etc/postfix/main.cf

#Add the following
allow_mail_to_commands = alias,forward,include

/etc/aliases

#Add the following
hoge: :include:/home/hoge/.forward

/home/hoge/.forward

"|/usr/bin/python /home/hoge/mail_parser/script1.py"

It's usually sendmail, so you'll need to switch to postfix, but this one was annoying. The reason for including is to change the execution user of script1.py, isn't it? I got angry with a Permission Error, so ...

After changing the settings, gently

$ newaliases
$ /etc/init.d/postfix reload

And reflect the settings.

reference: [K-One Enterprise Engineer Memo (`・ ω ・ ´) Business !!: postfix Setting Part 5 Start the program triggered by receiving an email](http://k-1-ne-jp.blogspot.jp/ 2013/01/postfix_11.html) Skip mail with one-line mail command-Mountain walking programmer

Post fluentd from standard input

Script 1 is like the following, feeding raw mail to the parser from standard input.

/home/hoge/mail_parser/script1.py

#! /usr/local/bin/python
# -*- coding:utf-8 -*-

from fluent import sender
from fluent import event
import sys
import email

### get stdin
input_lines = sys.stdin.read()

### get mail text
mail_text = email.message_from_string(input_lines)

### send message to fluentd
sender.setup('out.test', host='localhost', port=24224)
event.Event('follow', {
    'from': mail_text["from"],
    'to': mail_text["to"],
    'date': mail_text["date"],
    'subject': mail_text["subject"],
    'body': mail_text.get_payload()
    })

fluent (td-agent) settings

Receive with tcp with in_forward and execute script 2 with out_exec.

/etc/td-agent/td-agent.conf

### built-in TCP input
<source>
  type forward
  port 24224
</source>

### file output
<match local.**>
  type file
  path /var/log/td-agent/access
</match>

### debug stdout
<match debug.**>
  type stdout
</match>

### out_exec
<match out.**>
  type exec
  command /usr/bin/python /var/td-agent/bin/hipchat_poster/script2.py
  time_key got_at
  time_format %Y-%m-%d %H:%M:%S
  format json
  flush_interval 5s
  buffer_path /var/td-agent/tmp/buffer
  buffer_chunk_limit 256m
  buffer_queue_limit 10
  retry_limit 3
  retry_wait 1s
</match>

If you change out.test to local.test or debug.test in script1.py above,

echo "postfix test mail" | mail -s "test mail" hoge@localhost

You can send a test email and check that the contents of the email are written to a file.

By the way, when passing data to out_exec, the path of the buffer file containing the data is passed to the argument of the execution command, so be careful there. Thankfully, the people around me told me about it, but I'm afraid of misunderstanding the specifications ... I was wondering if it would come with standard input ...

Also, if you don't control the fluentd buffer properly, it will get stuck, so be careful about that. I still have to understand a lot about it.

reference: exec Output Plugin | Fluentd Multi-stage fluentd + mongodb hamari place --stanaka's blog

Hipchat post from json in the argument file path

And script 2 reads json of the argument file path and posts a message to hipchat as follows.

It is necessary to create a hipchat account and issue tokens in advance, put hypcaht with pip, and open directories.

/var/td-agent/bin/hipchat_poster/script2.py

from hypchat import HypChat
import sys
import json

def main():
        ### api call rundown
        API_KEY = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
        ROOM_NUMBER = 000000
        DEFAULT_COLOR = "purple"

        ### get fluentd log
        f = open(sys.argv[1], 'r')
        data = json.load(f)
        input_lines = "[" + data["date"] + "] " + data["subject"] + ":" + data["body"]

        ### post message to hipchat
        hc = HypChat(API_KEY)
        room = hc.get_room(ROOM_NUMBER)
        message = input_lines
        room.notification(message, color=DEFAULT_COLOR)
        return 0

if __name__ == "__main__":
        main()

reference: Check the operation of Fluentd plugins in_exec and out_exec | OpenGroove

After that

echo "postfix test mail" | mail -s "test mail" hoge@localhost

When I posted it, I was able to post it to hipchat like the following. I'm happy.

Bonus: Reasons for implementation

• I've only used aws for azure, so I'm interested in it + I was about to use it for work + I talked a bit with pycon
• Because python is usually written
• Because hipchat is used in the company
• I wondered if postfix is awkward to set up with sendmail?
• Fluentd is a mess of parsing and plug-ins I'm sure you'll be happy if you use existing ones?

By the way, for the time being, I put monit or something in the monitoring and tried to fly the alert mail, but since it is for studying, I will make various explicit alert mails, and if I write a fluentd parser for it, I am very satisfied with this creation. I've done it, so next time.

that's all.