I managed to do it because the custom of attaching a zip with a password to an email and saying "I will send you the password separately" is troublesome.

That custom

There is a custom that a zip with a password is attached to an email and "the password will be sent separately". I don't do it myself, but it's annoying because I have to do it according to the other party.

The pros and cons of this approach do not matter here. No matter how much I preach, the situation of having this custom does not change.

And I don't think about breaking this practice. I'll leave that to something with enormous power.

The old idiot said. "Wrap it around a long one." However, I think it is better to think about how to wind it.

Think about how to wind smartly

There is only one thing I want to solve when it is rolled up. Don't be annoyed. If you make a Web system for this purpose and open a browser to do something like this, it will be overwhelming. I want to realize it as close to normal email transmission as possible.

So, after thinking about it, I tried to solve it with a serverless feeling using Amazon SES while allowing some restrictions.

specification

  1. Write an email normally (new, reply, forward)
  2. Throw the file as it is without zipping it
  3. Set the email address for SES to To and the person to whom you actually want to send the file to Reply-To.
  4. Believe in the system and press the submit button
  5. You and the other party will receive an email with a password-attached zip and a password notification email.

However, there are the following restrictions. Personally, it's acceptable.

--As a result, everyone reaches the other party by To. You can't Cc (I'm Bcc) --The name of the zip file will be the date and time (yymmddHHMMSS.zip) (the file name inside remains the same)

System configuration

flow_01.png

  1. Send an email to SES
  2. Email data is saved in S3
  3. Trigger it to start Lambda
  4. Lambda parses the email and generates a password and zip file
  5. Send a nice email (send it to yourself with Bcc just in case)

Implementation

Lambda It's the first time I wrote python seriously, but is it okay like this? It's about a battle between email, character encoding, and files.

# -*- coding: utf-8 -*-

import os
import sys
import string
import random
import json
import urllib.parse
import boto3
import re
import smtplib
import email
import base64
from email                import encoders
from email.header         import decode_header
from email.mime.base      import MIMEBase
from email.mime.multipart import MIMEMultipart
from email.mime.text      import MIMEText
from email.mime.image     import MIMEImage
from datetime             import datetime

sys.path.append(os.path.join(os.path.dirname(os.path.realpath(__file__)), 'vendored'))
import pyminizip

s3 = boto3.client('s3')

class MailParser(object):
    """
Email parsing class
    (reference) http://qiita.com/sayamada/items/a42d344fa343cd80cf86
    """

    def __init__(self, email_string):
        """
Initialization
        """
        self.email_message    = email.message_from_string(email_string)
        self.subject          = None
        self.from_address     = None
        self.reply_to_address = None
        self.body             = ""
        self.attach_file_list = []

        #Interpretation of eml
        self._parse()

    def get_attr_data(self):
        """
Get email data
        """
        attr = {
                "from":         self.from_address,
                "reply_to":     self.reply_to_address,
                "subject":      self.subject,
                "body":         self.body,
                "attach_files": self.attach_file_list
                }
        return attr


    def _parse(self):
        """
Parsing mail files
        """

        #Analysis of message header part
        self.subject          = self._get_decoded_header("Subject")
        self.from_address     = self._get_decoded_header("From")
        self.reply_to_address = self._get_decoded_header("Reply-To")

        #Extract only the character string of the email address
        from_list =  re.findall(r"<(.*@.*)>", self.from_address)
        if from_list:
            self.from_address = from_list[0]
        reply_to_list =  re.findall(r"<(.*@.*)>", self.reply_to_address)
        if reply_to_list:
            self.reply_to_address = ','.join(reply_to_list)

        #Analysis of message body part
        for part in self.email_message.walk():
            #If the ContentType is multipart, the actual content is even more
            #Since it is in the inside part, skip it
            if part.get_content_maintype() == 'multipart':
                continue
            #Get file name
            attach_fname = part.get_filename()
            #Should be the body if there is no file name
            if not attach_fname:
                charset = str(part.get_content_charset())
                if charset != None:
                    if charset == 'utf-8':
                        self.body += part.get_payload()
                    else:
                        self.body += part.get_payload(decode=True).decode(charset, errors="replace")
                else:
                    self.body += part.get_payload(decode=True)
            else:
                #If there is a file name, it's an attachment
                #Get the data
                self.attach_file_list.append({
                    "name": attach_fname,
                    "data": part.get_payload(decode=True)
                })

    def _get_decoded_header(self, key_name):
        """
Get the decoded result from the header object
        """
        ret = ""

        #A key that does not have a corresponding item returns an empty string
        raw_obj = self.email_message.get(key_name)
        if raw_obj is None:
            return ""
        #Make the decoded result unicode
        for fragment, encoding in decode_header(raw_obj):
            if not hasattr(fragment, "decode"):
                ret += fragment
                continue
            #UTF for the time being without encode-Decode with 8
            if encoding:
                ret += fragment.decode(encoding)
            else:
                ret += fragment.decode("UTF-8")
        return ret

class MailForwarder(object):

    def __init__(self, email_attr):
        """
Initialization
        """
        self.email_attr = email_attr
        self.encode     = 'utf-8'

    def send(self):
        """
Compress the attached file with a password, forward it, and send a password notification email
        """

        #Password generation
        password = self._generate_password()

        #zip data generation
        zip_name = datetime.now().strftime('%Y%m%d%H%M%S')
        zip_data = self._generate_zip(zip_name, password)

        #Send zip data
        self._forward_with_zip(zip_name, zip_data)

        #Send password
        self._send_password(zip_name, password)

    def _generate_password(self):
        """
Password generation
Shuffle by taking 4 letters each from symbols, letters and numbers
        """
        password_chars = ''.join(random.sample(string.punctuation, 4)) + \
                         ''.join(random.sample(string.ascii_letters, 4)) + \
                         ''.join(random.sample(string.digits, 4))

        return ''.join(random.sample(password_chars, len(password_chars)))

    def _generate_zip(self, zip_name, password):
        """
Generate data for password-protected Zip file
        """
        tmp_dir  = "/tmp/" + zip_name
        os.mkdir(tmp_dir)

        #Save the file locally
        for attach_file in self.email_attr['attach_files']:
            f = open(tmp_dir + "/" + attach_file['name'], 'wb')
            f.write(attach_file['data'])
            f.flush()
            f.close()

        #To zip with password
        dst_file_path = "/tmp/%s.zip" % zip_name
        src_file_names = ["%s/%s" % (tmp_dir, name) for name in os.listdir(tmp_dir)]

        pyminizip.compress_multiple(src_file_names, dst_file_path, password, 4)

        # #Read the generated zip file
        r = open(dst_file_path, 'rb')
        zip_data = r.read()
        r.close()

        return zip_data

    def _forward_with_zip(self, zip_name, zip_data):
        """
Generate data for password-protected Zip file
        """
        self._send_message(
                self.email_attr['subject'],
                self.email_attr["body"].encode(self.encode),
                zip_name,
                zip_data
                )
        return

    def _send_password(self, zip_name, password):
        """
Send password for zip file
        """

        subject = self.email_attr['subject']

        message = """
This is the password for the file you sent earlier.

[subject] {}
[file name] {}.zip
[password] {}
        """.format(subject, zip_name, password)

        self._send_message(
                '[password]%s' % subject,
                message,
                None,
                None
                )
        return

    def _send_message(self, subject, message, attach_name, attach_data):
        """
send e-mail
        """

        msg = MIMEMultipart()

        #header
        msg['Subject'] = subject
        msg['From']    = self.email_attr['from']
        msg['To']      = self.email_attr['reply_to']
        msg['Bcc']     = self.email_attr['from']

        #Text
        body = MIMEText(message, 'plain', self.encode)
        msg.attach(body)

        #Attachment
        if attach_data:
            file_name = "%s.zip" % attach_name
            attachment = MIMEBase('application', 'zip')
            attachment.set_param('name', file_name)
            attachment.set_payload(attach_data)
            encoders.encode_base64(attachment)
            attachment.add_header("Content-Dispositon", "attachment", filename=file_name)
            msg.attach(attachment)

        #Send
        smtp_server   = self._get_decrypted_environ("SMTP_SERVER")
        smtp_port     = self._get_decrypted_environ("SMTP_PORT")
        smtp_user     = self._get_decrypted_environ("SMTP_USER")
        smtp_password = self._get_decrypted_environ("SMTP_PASSWORD")
        smtp = smtplib.SMTP(smtp_server, smtp_port)
        smtp.ehlo()
        smtp.starttls()
        smtp.ehlo()
        smtp.login(smtp_user, smtp_password)
        smtp.send_message(msg)
        smtp.quit()
        print("Successfully sent email")

        return

    def _get_decrypted_environ(self, key):
        """
Decrypt encrypted environment variables
        """

        client = boto3.client('kms')
        encrypted_data = os.environ[key]
        return client.decrypt(CiphertextBlob=base64.b64decode(encrypted_data))['Plaintext'].decode('utf-8')

def lambda_handler(event, context):

    #Get bucket name and key name from event
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'])

    try:
        #Read the contents of the file from S3
        s3_object = s3.get_object(Bucket=bucket, Key=key)
        email_string = s3_object['Body'].read().decode('utf-8')

        #Analyze email
        parser = MailParser(email_string)

        #Email forwarding
        forwarder = MailForwarder(parser.get_attr_data())
        forwarder.send()
        return

    except Exception as e:
        print(e)
        raise e

pyminizip It seems that password-protected zip cannot be done with a standard library. So, I relied on an external library called pyminizip only here. However, this was a library that was built at the time of installation and made a binary, so I made a binary by setting up a Docker container of Amazon Linux locally to run it with Lambda. Is there any other good way? ..

AWS SAM By the way, I tested this locally using AWS SAM. It was good to write the SMTP server information directly and try it, but when I moved it to an environment variable, it didn't work and I was frustrated. It looks like it has been fixed but not released.

Introduction method

I will publish it because it is a big deal. Codename zaru. Please forgive me though the setting method remains muddy. .. https://github.com/Kta-M/zaru

I've only tried it in my environment (Mac, Thunderbird), so it may not work depending on the mailer or other environment. Please take responsibility for your actions.

SES SES is not yet available in the Tokyo region, so we will build it in the Oregon region (us-west-2).

Domain verification

First, we will verify the domain so that you can send an email to SES. There are various methods, so I will omit this area. For example, this may be helpful-> Send domain email using Amazon SES / Route53 with Rails

Rule creation

After verifying the domain, create a Rule.

From Rule Sets on the right side of the menu, click View Active Rule Set. ses_rule_01.png

Click Create Rule. ses_rule_02.png

Register the email address to receive. Enter the email address of the verified domain and click ʻAdd Recipient`. ses_rule_03.png

Register the action when receiving an email. Select S3 as the action type and specify the bucket to store the received mail data. At this time, if you create a bucket with Create S3 bucket, the required bucket policy will be registered automatically, which is convenient. A policy is set that allows file uploads from SES to the bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowSESPuts-XXXXXXXXXXXX",
            "Effect": "Allow",
            "Principal": {
                "Service": "ses.amazonaws.com"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::<ses-bucket-name>/*",
            "Condition": {
                "StringEquals": {
                    "aws:Referer": "XXXXXXXXXXXX"
                }
            }
        }
    ]
}

Also, the mail data saved in the bucket can be stored, so it may be better to set a life cycle so that it will be deleted after a certain period of time. ses_rule_04.png

Give the rule a name. The rest is by default. ses_rule_05.png

Check the registration details and register! ses_rule_06.png

Lambda

Deploy

Deploy to the Oregon region as well as SES. Since CloudFormation will be used, please create an S3 bucket to upload data.

# git clone [email protected]:Kta-M/zaru.git
# cd zaru
# aws cloudformation package --template-file template.yaml --s3-bucket <cfn-bucket-name> --output-template-file packaged.yaml
# aws cloudformation deploy --template-file packaged.yaml --stack-name zaru-stack --capabilities CAPABILITY_IAM --region us-west-2

If you go to the Lambda console, the function is created. You've also created the IAM role needed to execute this function. lambda_01.png

Trigger setting

Set Lambda to work by triggering mail data in the bucket.

Go to the Trigger tab on the function details screen. lambda_02.png

Click Add Trigger to create an S3 event. The bucket where the data comes from SES, the event type is Put. Other than that, it is the default. Bucket isLamb_03.png

Create an encryption key

In this Lambda function, we get the SMTP related information from the encrypted environment variables. Create a key to use for that encryption.

From the IAM console, click the encryption key at the bottom left. Change the region to Oregon and create a key. lambda_04.png

All you have to do is set an alias of your choice, and the rest is OK by default. lambda_05.png

Setting the number of environment variables

Go back to Lambda and set the environment variables you want to use in your function. At the bottom of the Code tab is a form for setting environment variables. Check Enable encryption helper and specify the encryption key you created earlier. For environment variables, enter the variable name and value (plain text) and press the `encrypt" button. Then, it will be encrypted with the specified encryption key. The following four environment variables are set.

Variable name Description Example
SMTP_SERVER smtp server smtp.example.com
SMTP_PORT smtp port 587
SMTP_USER Username to log in to smtp server [email protected]
SMTP_PASSWORD SMTP_USER password

lambda_06.png

Role settings

Finally, give the role that executes this Lambda function the required permissions. --Permission to retrieve data from S3 bucket to store mail data --Permission to decrypt environment variables using encryption key

First, go to the policy of the IAM console and go to Create Policy-> Create Your Own Policy to create the following two policies. lambda_07.png

** Policy: s3-get-object-zaru ** For <ses-bucket-name>, specify the bucket name to receive mail data from SES.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1505586008000",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::<ses-bucket-name>/*"
            ]
        }
    ]
}

** Policy; kms-decrypt-zaru ** For <kms-arn>, specify the ARN of the encryption key.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1448696327000",
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt"
            ],
            "Resource": [
                "<kms-arn>"
            ]
        }
    ]
}

Finally, attach these two policies to your Lambda function execution role. First, go to the role in the IAM console, select the role, and attach it fromAttach Policy. lambda_08.png

Operation check

It should now work. Please set the e-mail address set for SES in To and the e-mail address of the other party in Reply-To, and attach an appropriate file and send it. How is it?

Summary

Dontokoi zip attachment!

Recommended Posts

I managed to do it because the custom of attaching a zip with a password to an email and saying "I will send you the password separately" is troublesome.
If you guys in the scope kitchen can do it with a margin ~ ♪
Do you make something like a rocket?
When writing a test using DB with django, you may be able to do it faster by using `setUpTestData ()`
I managed to do it because the custom of attaching a zip with a password to an email and saying "I will send you the password separately" is troublesome.
It is a piggybacking story about the service that returns "Nyan" when you ping
It was a little difficult to do flask with the docker version of nginx-unit
Until you can install blender and run it with python for the time being
The sound of tic disorder at work is ... I managed to do it with the code
It is surprisingly troublesome to get a list of the last login date and time of Workspaces
Output the report as PDF from DB with Python and automatically attach it to an email and send it
What to do if you couldn't send an email to Yahoo with Python.
I want to send Gmail with Python, but I can't because of an error
I want to write an element to a file with numpy and check it.
It was a little difficult to do flask with the docker version of nginx-unit
I don't like to be frustrated with the release of Pokemon Go, so I made a script to detect the release and tweet it
A script that pings the registered server and sends an email with Gmail a certain number of times when it fails
I thought a little because the Trace Plot of the stan parameter is hard to see.
Make a note of what you want to do in the future with Raspberry Pi
What to do if you cat or tail a binary file and the terminal is garbled
[Python] What is a slice? An easy-to-understand explanation of how to use it with a concrete example.
I tried to make it possible to automatically send an email just by double-clicking the [Python] icon
Find the white Christmas rate by prefecture with Python and map it to a map of Japan
What to do when a part of the background image becomes transparent when the transparent image is combined with Pillow
[Python] Although it is a humanities, I will do my best to understand bit full search
It is troublesome to change the settings between the intranet and business trip / at home, so I was a little happy when I set up a forward proxy locally with Apache2.
When it is troublesome to copy what you built with vue
I made a POST script to create an issue on Github and register it in the Project
Is it possible to enter a venture before listing and make a lot of money with stock options?
[AWS lambda] Deploy including various libraries with lambda (generate a zip with a password and upload it to s3) @ Python
Since it is the 20th anniversary of the formation, I tried to visualize the lyrics of Perfume with Word Cloud
I tried to make it possible to automatically send an email just by double-clicking the [GAS / Python] icon
The story of making a tool to load an image with Python ⇒ save it as another name
I made an npm package to get the ID of the IC card with Raspberry Pi and PaSoRi
Read the data of the NFC reader connected to Raspberry Pi 3 with Python and send it to openFrameworks with OSC
Upload data to s3 of aws with a command and update it, and delete the used data (on the way)