Get Python scripts to run quickly in Cloud Run using responder

Summary

Cloud Run has been GA. I wanted to try a python script that used to run locally and take a little time to run in Cloud Run. Containers running in Cloud Run must be able to receive and run requests over HTTP. Cloud Run also has to take into account request timeouts. I will introduce the responder because it was easy to run the script quickly without considering the request timeout.

Premise story

Tried environment

responder

python's HTTP service framework. Since it implements ASGI (Asynchronous Server Gateway Interface), it can also perform asynchronous processing as the name Asynchronous implies. Asynchronous processing itself can be done with other frameworks. However, when I took a quick look at Flask and Django, which are famous for the python framework, I had to install additional packages and change the configuration file. Looking at the sample code of responder, it seemed easy to install and implement, so I tried using it.

What i did

About directory structure and each file

vaivailx@MacBook-Pro-2 responder_sample_for_cloudrun % tree -L 3 .
.
├── Dockerfile
├── requirements.txt
└── src
    ├── sample.py
    └── webapp.py

Each python file does the following:

This is the only requirements.txt.

responder==2.0.4

In Dockerfile, update the package with apt and install the package described in requirements.txt.

FROM python:3.7.5-buster

ENV APP_HOME /app
WORKDIR $APP_HOME
ADD ./requirements.txt /app
ADD ./src /app

# Avoid warnings by switching to noninteractive
ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update \
    && apt-get -y install --no-install-recommends apt-utils dialog 2>&1 \
    # Install pylint
    && pip --disable-pip-version-check --no-cache-dir install pylint \
    # Update Python environment based on requirements.txt
    && pip --disable-pip-version-check --no-cache-dir install -r requirements.txt \
    && rm -rf requirements.txt \
    # Clean up
    && apt-get autoremove -y \
    && apt-get clean -y \
    && rm -rf /var/lib/apt/lists/*

# Switch back to dialog for any ad-hoc use of apt-get
ENV DEBIAN_FRONTEND=

# run app
ENV PORT '8080'
CMD python3 webapp.py
EXPOSE 8080

Processing to accept HTTP request using responder. Simply prefix def with ʻasync` and the function will be asynchronous.

webapp.py

import logging

import responder

from sample import process_sample

logging.basicConfig(filename=f'/var/log/webapp.log', level=logging.DEBUG)

api = responder.API()

@api.route("/async")
async def asyncsample(req, resp):

    @api.background.task
    def process_param(params):
        t = int(params.get('time', 10))
        process_sample(t)

    process_param(req.params)
    resp.media = {'async success': True}

@api.route("/sync")
def syncsample(req, resp):

    t = int(req.params.get('time', 10))
    process_sample(t)
    resp.media = {'sync success': True}

if __name__ == '__main__':
    api.run()

The log is output because it can be confirmed at runtime with Croud Run. As the route name suggests, accessing the root path / async will execute asynchronous processing, and accessing the root path / sync will execute motivation processing. This time, I tried to sleep for the value of the query parameter.

sample.py

import time
import logging

def process_sample(secs):
    logging.debug('process starts.')
    time.sleep(secs)
    logging.debug('process ends.')

Confirmation of execution results in Cloud Run

Now, let's set the Cloud Run request timeout to 30 seconds and run it. (Since it is troublesome to wait for 5 minutes, the default value has been changed.)

** Synchronous execution: Execution time 10 seconds **

vaivailx@cloudshell:~ (cloudrun-test-259804)$ date;curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" https://responderasync-jpwo3ltkhq-an.a.run.app/sync?time=10
Fri Dec 13 09:52:59 +09 2019
{"sync success": true}vaivailx@cloudshell:~ (cloudrun-test-259804)$
sync10_result.png

→ Finished normally

** Synchronous execution: Execution time 40 seconds **

vaivailx@cloudshell:~ (cloudrun-test-259804)$ date;curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" https://responderasync-jpwo3ltkhq-an.a.run.
app/sync?time=40
Fri Dec 13 09:55:45 +09 2019
upstream request timeoutvaivailx@cloudshell:~ (cloudrun-test-259804)$
sync40_result.png

→ Time out

** Asynchronous execution: Execution time 10 seconds **

vaivailx@cloudshell:~ (cloudrun-test-259804)$ date;curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" https://responderasync-jpwo3ltkhq-an.a.run.app/async?time=10
Fri Dec 13 10:02:31 +09 2019
{"async success": true}vaivailx@cloudshell:~ (cloudrun-test-259804)$
async10_result.png

→ Finished normally

** Asynchronous execution: Execution time 40 seconds **

vaivailx@cloudshell:~ (cloudrun-test-259804)$ date;curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" https://responderasync-jpwo3ltkhq-an.a.run.app/async?time=40
Fri Dec 13 10:03:57 +09 2019
{"async success": true}vaivailx@cloudshell:~ (cloudrun-test-259804)$
async40_result.png

→ Finished normally

What I confirmed by the way

By the way, the HTTP request timeout was up to 900 seconds on the setting screen, but this is not the execution time It is the time from receiving the request to returning the response, and it was confirmed that it can be done for 900 seconds or more if it is asynchronous. The following is the result of asynchronous execution for 1200 seconds.

async1200_result.png

Finally

Recommended Posts

Get Python scripts to run quickly in Cloud Run using responder
Type Python scripts to run in QGIS Processing
How to install OpenCV on Cloud9 and run it in Python
Get Suica balance in Python (using libpafe)
Log in to Slack using requests in Python
How to get a stacktrace in python
Get Youtube data in Python using Youtube Data API
How to switch python versions in cloud9
Get Cloud Logging available in Python in 10 minutes
How to get a value from a parameter store in lambda (using python)
Get image URL using Flickr API in Python
How to get the files in the [Python] folder
How to run Leap Motion in non-Apple Python
What to do to get google spreadsheet in python
Use os.getenv to get environment variables in Python
What to do if you get "Python not configured." Using PyDev in Eclipse
Get date in Python
Cloud Run tutorial (python)
Quickly profile Python scripts
How to get the variable name itself in python
How to get the number of digits in Python
How to run python in virtual space (for MacOS)
How to run tests in bulk with Python unittest
Create a plugin to run Python Doctest in Vim (2)
Get LEAD data using Marketo's REST API in Python
A memorandum to run a python script in a bat file
How to run setUp only once in python unittest
How to exit when using Python in Terminal (Mac)
How to retrieve multiple arrays using slice in python.
How to execute a command using subprocess in Python
Get YouTube Comments in Python
Try to make it using GUI and PyQt in Python
To flush stdout in Python
Get last month in python
Run automatic jobs in python
Dynamically import scripts in Python
Run shell commands in python
Run Python unittests in parallel
Login to website in Python
Using verticalenv in shell scripts
Post to Twitter using Python
Developed a library to get Kindle collection list in Python
[OCI] Python script to get the IP address of a compute instance in Cloud Shell
Start to Selenium using python
[Question] How to get data of textarea data in real time using Python web framework bottle
I tried "How to get a method decorated in Python"
Run LINE Bot implemented in Python (Flask) "without using Heroku"
Get Terminal size in Python
To return char * in a callback function using ctypes in Python
Explicitly get EOF in python
How to get the last (last) value in a list in Python
How to get a list of built-in exceptions in python
How to run Notepad ++ Python
Hit REST in Python to get data from New Relic
Things to keep in mind when using Python with AtCoder
I tried to make a stopwatch using tkinter in python
Scripts that can be used when using bottle in Python
How to develop in Python
Get Evernote notes in Python
Things to keep in mind when using cgi with python.
I want to be able to run Python in VS Code