Image processing with Lambda + OpenCV (gray image creation)

Image processing with Lambda + OpenCV (gray image creation)

This article is a hands-on article for reviewing and fixing the knowledge gained by developing Serverless Web App Mosaic. It is one of w2or3w / items / 87b57dfdbcf218de91e2).

It would be nice to read this article after looking at the following.

introduction

Implement Lambda function in Python 3.6 that performs image processing on the image triggered by uploading the image to S3. OpenCV is used for image processing.

content

Create a repository for Lambda function projects on Github

Create a repository for your project on GitHub. The repository name is sample_lambda_py_project.

Clone repository to Cloud9

$ git clone https://github.com/{YourGithubID}/sample_lambda_py_project.git

Creating a simple program

Create an appropriate folder (here, source) in sample_vue_project, and create a lambda_function.py file in it. For the time being, let's assume that the code is implemented inline in Previous article. (The content of the log message will be changed a little.)

lambda_function.py


from urllib.parse import unquote_plus
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')
    logger.info("Function Start (deploy from S3) : Bucket={0}, Key={1}" .format(bucket, key))

I think the Cloud9 environment looks like this. Screenshot 2019-12-31 at 15.04.28.png

Zip the function code and deploy it to Lambda via S3

It is not necessary for simple code as it is now, but if you use a library other than the standard library such as OpenCV, you need to upload the library as a package in a zip file.

So, first, let's zip it without using the library and upload it to Lambda via S3.

Zip compression

Compress under the directory where lambda_function.py is located with the name lambda-package.zip. Execute the following command in the directory where lambda_function.py is located.

$ zip -r ../lambda-package.zip *

Upload to S3

Upload the created lambda-package.zip to S3. Execute the following command while keeping the directory where lambda_function.py is located.

$ aws s3 cp ../lambda-package.zip s3://sample-vue-project-bucket-work/deploy/lambda-package.zip

Deploy in Lambda console

AWS Console> Lambda> Functions> S3Trigger ********-work To access. Set the function code as follows. Code entry type: File upload from Amazon S3 Runtime: Python 3.6 Handler: lambda_function.lambda_handler Amazon S3 link URL: The link URL of lambda-package.zip uploaded earlier Screenshot 2019-12-31 at 15.20.46.png Click the save button at the top right of the screen.

Operation check

After uploading the file from the sample_vue_project web app, AWS Console> CloudWatch> Log Group> aws / lambda / S3Trigger ********-work Please visit to see the latest log stream. Function Start (deploy from S3) : Bucket=sample-vue-project-bucket-work, Key=public/191231063457-16f5aaa24b1/faceA.jpeg It is OK if an INFO log like this is output. Screenshot 2019-12-31 at 15.35.36.png

OpenCV installation

In the source directory where lambda_function.py is located, run the following command.

$ pip --version
pip 9.0.3 from /usr/lib/python3.6/dist-packages (python 3.6)
$ pip install opencv-python -t .

The Lambda function needs to include the library in the zip package to upload, so -t . Specify this option and install it in the current directory.

Gray image converter

Update lambda_function.py.

lambda_function.py


# coding: UTF-8
import boto3
import os
from urllib.parse import unquote_plus
import numpy as np
import cv2
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
s3 = boto3.client("s3")

def lambda_handler(event, context):
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')
    logger.info("Function Start (deploy from S3) : Bucket={0}, Key={1}" .format(bucket, key))

    fileName = os.path.basename(key)
    orgFilePath = u'/tmp/' + fileName
    processedFilePath = u'/tmp/processed-' + fileName

    if (not key.startswith("public")):
        logger.info("not start with public")
        return

    keyOut = key.replace("public", "processed", 1)
    logger.info("Output local path = {0}".format(processedFilePath))

    try:
        s3.download_file(Bucket=bucket, Key=key, Filename=orgFilePath)

        orgImage = cv2.imread(orgFilePath)
        grayImage = cv2.cvtColor(orgImage, cv2.COLOR_RGB2GRAY)
        cv2.imwrite(processedFilePath, grayImage)

        s3.upload_file(Filename=processedFilePath, Bucket=bucket, Key=keyOut)
        logger.info("Function Completed : processed key = {0}".format(keyOut))

    except Exception as e:
        logger.exception(e)
        raise e
        
    finally:
        if os.path.exists(orgFilePath):
            os.remove(orgFilePath)
        if os.path.exists(processedFilePath):
            os.remove(processedFilePath)

(* I won't touch on the basic syntax of Python here. If you need it, learn somewhere.)

Uploaded Convert a file like public / 191231112635-16f5bb5247f / lenna.png to a gray image, Uploading to a path like processed / 191231112635-16f5bb5247f / lenna.png.

Lambda basic settings

You can specify the memory and timeout in the Lambda basic settings. By default, memory = 128MB, timeout 25 seconds. Photos taken with recent smartphones are large in size, so if you keep the default settings, you may run out of memory and an exception may occur. When you increase the processing performed by Lambda in the future, you may run out of time and time out. Adjust your Lambda preferences as needed. (* Regarding the size of the image, I think it is better to compress it on the client side before uploading.)

Deployment and operation check

Zip it up, upload it to S3, deploy it to Lambda, and then upload the image from your web app. It is OK if the monochrome image is uploaded below the S3 bucket. Check the CloudWatch log as well. lenna.png lenna (1).png (Lena is beautiful)

Afterword

When you operate from Lambda faction to S3 which is the trigger source like this sample program, the Lambda function is called by that operation. If you are not careful, you will end up with an infinite loop of Lambda, so be careful.

If you inadvertently fall into a Lambda infinite loop, you should notice it right away, but if you don't notice it right away, you don't want to think about it. The automatic billing addition machine will continue to do its job without hesitation.

The way to stop the infinite loop Lambda is to either remove the function or upload the function code that raises the exception. I think it's a good idea to upload the function code that raises the exception, because deleting a function can be cumbersome to create and configure again, but is there any other way? Please let me know.

In any case, you will have to operate it with your trembling hands. While sweating strangely.

Also, I haven't debugged the Python code called by Lambda yet. It seems that you can debug with Cloud9, so once you learn it, you will be able to proceed with development more efficiently. Once you've mastered it, try to write an article.

Recommended Posts

Image processing with Lambda + OpenCV (gray image creation)
Real-time image processing basics with opencv
Image processing with Python & OpenCV [Tone Curve]
Light image processing with Python x OpenCV
Image processing with MyHDL
Image Processing with PIL
XavierNX accelerates OpenCV image processing with GPU (CUDA)
Image processing with Python (Part 2)
Image processing with PIL (Pillow)
"Apple processing" with OpenCV3 + Python3
Image editing with python OpenCV
Image processing with Python (Part 1)
Image processing with Python (Part 3)
Get image features with OpenCV
Image recognition with Keras + OpenCV
[Python] Image processing with scikit-image
[Python] Using OpenCV with Python (Image Filtering)
[Python] Using OpenCV with Python (Image transformation)
Image processing with Python 100 knocks # 3 Binarization
Try blurring the image with opencv2
Image processing with Python 100 knocks # 2 Grayscale
Image processing with Python 100 knock # 10 median filter
Create an image processing viewer with PySimpleGUI
Image processing with Python 100 knocks # 8 Max pooling
Image processing with Python 100 knock # 12 motion filter
Image acquisition from camera with Python + OpenCV
Drawing with Matrix-Reinventor of Python Image Processing-
Easy image processing in Python with Pillow
Image processing with Python 100 knocks # 7 Average pooling
Image processing with Python 100 knocks # 9 Gaussian filter
[Image processing] Posterization
I tried "smoothing" the image with Python + OpenCV
python image processing
I tried "differentiating" the image with Python + OpenCV
Image cropping tool creation with OpenCV, precautions for projective conversion (only a little)
How to crop an image with Python + OpenCV
Image processing from scratch with python (4) Contour extraction
Image Processing with Python Environment Setup for Windows
[Small story] Test image generation with Python / OpenCV
Image processing 100 knocks ①
I tried simple image processing with Google Colaboratory.
[OpenCV / Python] I tried image analysis of cells with OpenCV
Image capture / OpenCV speed comparison with and without GPU
JPEG image generation by specifying quality with Python + OpenCV
Create miscellaneous Photoshop videos with Python + OpenCV ② Create still image Photoshop
Make Lambda Layers with Lambda
Processing datasets with pandas (1)
Flat Field image processing
Detect stoop with OpenCV
Read digital image processing
Rotate sprites with OpenCV
Parallel processing with multiprocessing
Data Augmentation with openCV
Easy TopView with OpenCV
Stumble with homebrew opencv3
Consider common pre-processing when processing DynamoDB Stream with Lambda (Python)
I tried "gamma correction" of the image with Python + OpenCV
"Garbage classification by image!" App creation diary day2-Fine-tuning with VGG16-
Paste png with alpha channel as transparent image with Python / OpenCV
[Let's play with Python] Image processing to monochrome and dots
[Python] Easy reading of serial number image files with OpenCV