Since async / await was added in Python 3.5, I benchmarked it in terms of memory consumption and context switch cost.
This is the most advanced syntax in the asynchronous processing paradigm that allows you to write asynchronous processing and non-blocking I / O processing in a nice way. After being implemented in C #, it was implemented in C ++, VB, Node.js and finally came to Python! It's like that. The feature is that asynchronous processing, which has been written hard in Threading, can be written more concisely and more powerfully. Lightweight threads are also called microthreads or fibers, and are a solution to the problem that hardware performance cannot be fully utilized when the number of clients is large due to I / O waiting called "C10K problem" (problem of 10,000 clients). It is one. By switching the context at high speed when waiting for I / O and handling other clients, you will be able to write programs that bring out the performance of the hardware to the limit. In other words, if you master it, you will be able to write a server that operates at high speed and light weight like nginx.
Benchmark results show that coroutines using Python's async / await started up faster, consumed less memory, and context-switched faster. Isn't it an implementation that can be called a lightweight thread? According to the Wow Erlang book
, Erlamg's lightweight thread starts up in microseconds with 4KByte of memory consumption. The Python version consumes 3.44KByte of memory and starts up in 60 microseconds, which is comparable to Erlang.
item | value |
---|---|
Start and end times for 100,000 coroutines | 6.081sec |
Start and end times per coroutine | 0.0608ms |
Execution time of 1 million context switches | 51.435sec |
Execution time per context switch | 51.435μs |
Memory consumption when starting 100,000 coroutines | 146.86MByte |
Memory consumption per coroutine | 1.50KByte |
Memory consumption when 100,000 coroutines are started and context switch is executed 1 million times | 336.30MByte |
Memory consumption per context switch 1 million times | 3.44KByte |
Processing to perform multiple network I / O and file I / O asynchronously
Ordinary FizzBuzz
def fizzbuzz(i):
if i == 15:
return 'FizzBuzz'
if i % 5 == 0:
return 'Buzz'
if i % 3 == 0:
return 'Fizz'
return str(i)
for x in range(1, 10):
print(fizzbuzz(x))
The result is the same as it is only running in a coroutine.
1 coroutine
import asyncio
async def main():
await task_fizzbuzz()
async def task_fizzbuzz():
for x in range(1, 10):
print(fizzbuzz(x))
return None
loop = asyncio.get_event_loop()
#Call the main function in a coroutine
loop.run_until_complete(main())
loop.close()
Generate 100,000 coroutines and execute them at the same time. When one coroutine finishes executing, the context switches to execute the next coroutine.
Coroutine 100,000 simultaneous execution
# -*- coding: utf-8 -*-
# from asyncio import Queue
# from queue import Queue
import asyncio
async def task_fizzbuzz(prefix):
for x in range(1, 10):
# await asyncio.sleep(1)
print(prefix + "{}:".format(str(x)) + fizzbuzz(x))
return None
loop = asyncio.get_event_loop()
#Generate 100,000 coroutines
tasks = asyncio.wait([task_fizzbuzz(str(i) + ":") for i in range(1, 100000)])
loop.run_until_complete(tasks)
loop.close()
Execution result of 100,000 coroutines
....
71798:7:7
71798:8:8
71798:9:Fizz
84034:1:1
84034:2:2
84034:3:Fizz
84034:4:4
84034:5:Buzz
84034:6:Fizz
84034:7:7
84034:8:8
84034:9:Fizz
17235:1:1
17235:2:2
....
This is a test to see if context switching works well when sleeping during fizzbuzz logic.
100,000 simultaneous executions of coroutines with sleep
# -*- coding: utf-8 -*-
# from asyncio import Queue
# from queue import Queue
import asyncio
async def task_fizzbuzz(prefix):
for x in range(1, 10):
await asyncio.sleep(1) #Newly added code
print(prefix + "{}:".format(str(x)) + fizzbuzz(x))
return None
loop = asyncio.get_event_loop()
#Generate 100,000 coroutines
tasks = asyncio.wait([task_fizzbuzz(str(i) + ":") for i in range(1, 100000)])
#Parallel execution
loop.run_until_complete(tasks)
loop.close()
Execution result of 100,000 coroutines with sleep
....
75219:6:Fizz
8282:6:Fizz
57464:6:Fizz
75220:6:Fizz
8283:6:Fizz
57465:6:Fizz
75221:6:Fizz
8284:6:Fizz
57466:6:Fizz
75222:6:Fizz
8285:6:Fizz
57467:6:Fizz
75223:6:Fizz
....
When sleeping with ʻawait asyncio.sleep (1)` during coroutine execution, the result is that it is running at the same time because it switches to another coroutine.
Now that the verification is complete, we will benchmark with the following three programs.
# -*- coding: utf-8 -*-
import time
def fizzbuzz(i):
if i == 15:
return 'FizzBuzz'
if i % 5 == 0:
return 'Buzz'
if i % 3 == 0:
return 'Fizz'
return str(i)
COUNT = 100000
FIZZBUZZ_COUNT = 10
#Start measurement
ts = time.time()
#10 times FizzBuzz x 100,000 times
for x in range(COUNT):
for x in range(FIZZBUZZ_COUNT):
print(fizzbuzz(x))
#Wait 10 seconds to make the conditions the same as the others
for x in range(FIZZBUZZ_COUNT):
time.sleep(1)
#End of measurement
te = time.time()
#Result output
print("{}sec".format(te-ts))
# -*- coding: utf-8 -*-
import time
import asyncio
def fizzbuzz(i):
if i == 15:
return 'FizzBuzz'
if i % 5 == 0:
return 'Buzz'
if i % 3 == 0:
return 'Fizz'
return str(i)
COUNT = 100000
FIZZBUZZ_COUNT = 10
#Start measurement
ts = time.time()
#10 times FizzBuzz x 100,000 times
async def task_fizzbuzz(prefix):
for x in range(FIZZBUZZ_COUNT):
# await asyncio.sleep(1)
print(prefix + "{}:".format(str(x)) + fizzbuzz(x))
return None
loop = asyncio.get_event_loop()
tasks = asyncio.wait([task_fizzbuzz(str(i) + ":") for i in range(COUNT)])
loop.run_until_complete(tasks)
loop.close()
#Wait 10 seconds to make the conditions the same as the others
for x in range(FIZZBUZZ_COUNT):
time.sleep(1)
#End of measurement
te = time.time()
#Result output
print("{}sec".format(te-ts))
# -*- coding: utf-8 -*-
import time
import asyncio
def fizzbuzz(i):
if i == 15:
return 'FizzBuzz'
if i % 5 == 0:
return 'Buzz'
if i % 3 == 0:
return 'Fizz'
return str(i)
COUNT = 100000
FIZZBUZZ_COUNT = 10
#Start measurement
ts = time.time()
#10 times FizzBuzz x 100,000 times
async def task_fizzbuzz(prefix):
for x in range(FIZZBUZZ_COUNT):
await asyncio.sleep(1)
print(prefix + "{}:".format(str(x)) + fizzbuzz(x))
return None
loop = asyncio.get_event_loop()
tasks = asyncio.wait([task_fizzbuzz(str(i) + ":") for i in range(COUNT)])
loop.run_until_complete(tasks)
loop.close()
#End of measurement
te = time.time()
#Result output
print("{}sec".format(te-ts))
I only do it 3 times each. Context switch takes too much time (›´ω`‹) I haven't written it to keep the results simple, but Benchmark 2 seems to have 100,000 context switches.
PEP 0492 -- Coroutines with async and await syntax 18.5.3. Tasks and coroutines 18.5.9. Develop with asyncio What's new in Python 3.5 How to use async/await in Python 3.5? Asynchronous programming with Async and Await (C # and Visual Basic) (https://msdn.microsoft.com/en-us/library/hh191443.aspx) AsyncIO vs Gevent? : Python - Reddit)
Recommended Posts