[Python] Asynchronous request with async / await

Use the asyncio module for asynchronous requests in Python. Let's create code that hits the API asynchronously using the requests package.

environment

API

I could have used time.sleep (), but this time I will assume a pattern to get the API rather than practice. I used JSON Placeholder as the mock server.

JSONPlaceholder

Write the code that requests the APIs to get the list (six below) synchronously and asynchronously, and returns the result of each result in one list.

https://jsonplaceholder.typicode.com/posts https://jsonplaceholder.typicode.com/comments https://jsonplaceholder.typicode.com/albums https://jsonplaceholder.typicode.com/photos https://jsonplaceholder.typicode.com/todos https://jsonplaceholder.typicode.com/users

Sync request

First, let's make a request in the synchronous process. If you write it according to the usual Python code, it will be as follows.

import time
import requests

BASE_URL = "https://jsonplaceholder.typicode.com/"


def calc_time(fn):
    """Decorator to measure function execution time"""
    def wrapper(*args, **kwargs):
        start = time.time()
        fn(*args, **kwargs)
        end = time.time()
        print(f"[{fn.__name__}] elapsed time: {end - start}")
        return
    return wrapper


def get_sync(path: str) -> dict:
    print(f"/{path} request")
    res = requests.get(BASE_URL + path)
    print(f"/{path} request done")
    return res.json()


@calc_time
def main_sync():
    data_ls = []
    paths = [
        "posts",
        "comments",
        "albums",
        "photos",
        "todos",
        "users",
    ]
    for path in paths:
        data_ls.append(get_sync(path))
    return data_ls

if __name__ == "__main__":
    main_sync()

When run, you will get output similar to the following:

Request 1-> Request 1 completed Request 2-> Request 2 completed Request 3-> Request 3 Completed ...

You can see that it is being executed.

/posts request
/posts request done
/comments request
/comments request done
/albums request
/albums request done
/photos request
/photos request done
/todos request
/todos request done
/users request
/users request done
[main_sync] elapsed time: 1.157785415649414

Asynchronous request

Then make an asynchronous request.

Asynchronous tasks run inside an event loop. To get the event loop, use ʻasyncio.get_event_loop () `.

loop.run_until_complete, as the name implies, is a method that executes an event loop until each task has been executed. The return value of this method is a list containing the return value of each asynchronous execution task. The order of execution is not guaranteed, but the return values are returned in the order passed to the argument, so it can be used even when the order is important.

The get_async declared with ʻasync def is called a coroutine function. The ʻawait expression inside a coroutine function pauses the execution of the coroutine function until the return value is returned.

import asyncio

#Coroutine function
async def get_async(path: str) -> dict:
    print(f"/{path} async request")
    url = BASE_URL + path
    loop = asyncio.get_event_loop()
    #Run in event loop
    res = await loop.run_in_executor(None, requests.get, url)
    print(f"/{path} async request done")
    return res.json()


@calc_time
def main_async():
    #Get event loop
    loop = asyncio.get_event_loop()
    #Asynchronous execution task in one Future object
    tasks = asyncio.gather(
        get_async("posts"),
        get_async("comments"),
        get_async("albums"),
        get_async("photos"),
        get_async("todos"),
        get_async("users"),
    )
    #Asynchronous execution, until each is finished
    results = loop.run_until_complete(tasks)
    return results


if __name__ == "__main__":
    main_async()

The output looks like this:

Request 1 Request 2 Request 3 ... Request 1 completed Request 2 completed Request 3 completed ...

You can see that it is processed. You can also see that the execution time has been significantly reduced.

/posts async request
/comments async request
/albums async request
/photos async request
/todos async request
/users async request
/users async request done
/todos async request done
/posts async request done
/albums async request done
/comments async request done
/photos async request done
[main_async] elapsed time: 0.17921733856201172

Recommended Posts

[Python] Asynchronous request with async / await
python async / await curio
Convert callback-style asynchronous API to async / await in Python
Python asynchronous processing ~ Full understanding of async and await ~
Python: How to use async with
Hide websockets async / await in Python3
Async / await with Kivy and tkinter
FizzBuzz with Python3
Scraping with Python
Statistics with python
Scraping with Python
Python with Go
Twilio with Python
Integrate with Python
Play Python async
Play with 2016-Python
AES256 with python
Tested with Python
python starts with ()
with syntax (Python)
Bingo with python
Zundokokiyoshi with python
Excel with Python
Microcomputer with Python
Cast with python
Serial communication with Python
Zip, unzip with python
Django 1.11 started with Python3.6
Primality test with Python
Socket communication with Python
Data analysis with python 2
Try scraping with Python.
Asynchronous programming with libev # 2
Learning Python with ChemTHEATER 03
Sequential search with Python
"Object-oriented" learning with python
Run Python with VBA
Handling yaml with python
Solve AtCoder 167 with python
Serial communication with python
[Python] Use JSON with Python
Learning Python with ChemTHEATER 05-1
Http request in python
Run prepDE.py with python3
1.1 Getting Started with Python
Collecting tweets with Python
Binarization with OpenCV / Python
Kernel Method with Python
Non-blocking with Python + uWSGI
Posting tweets with python
Drive WebDriver with python
Use mecab with Python3
[Python] Redirect with CGIHTTPServer
Voice analysis with python
Think yaml with python
Operate Kinesis with Python
Getting Started with Python
Use DynamoDB with Python
Zundko getter with python
Primality test with python
Run Blender with python