python parallel / asynchronous execution memorandum

Trigger

Python's asynchronous execution material was often similar (official translation / re-editing?). It's tough for an old man who has a hard head, so I looked for an example, looked at a site in multiple languages, and finally moved by trial and error, so I tried to summarize only my liver.

Overview

All examples are external command execution patterns on Windows There is almost no commentary.

Check by PID

Easy and nice. This is what Ojisan came up with.

pid.py


import subprocess
from time import sleep
procs = {}
for h in servers:
    p = subprocess.Popen(["some", "command"], ...)
    # subprocess.run to stdout=It seems to work with PIPE, but it has not been investigated
    procs[p.pid] = p

while procs:
    for pid in list(procs):
        if procs[pid].poll in not None:
            del procs[pid]
    sleep(x)

procs can be a list of p from the end, My favorite is to empty it at the end with this method.

Parallel 1 (no STDOUT required)

This is OK if the target to be executed generates something. The run function is an asynchronous execution target. When I first made it, I called it with run_until_complete, so I was wondering if it would just wait with create_subprocess_shell.

async1.py


import asyncio
from asyncoio.subprocess import DEVNULL

async def run(param):
    #Either of the following two works
    p = await asyncio.create_subprocess_exec(*["some", "command", param],
           stdout=DEVNULL, ...)
    p = await asyncio.create_subprocess_shell("some command %s" % param,
           stdout=DEVNULL, ...)

    await p.wait()

if sys.platform.startswith('win'):
    loop = asyncio.ProactorEventLoop()
    asyncio.set_event_loop(loop)
else:
    loop = asyncio.get_event_loop()

funcs = asyncio.gather(*[run(p) for p in parameters])
loop.run_until_complete(funcs)

Parallel 2 (STDOUT required)

If you want to do something using the standard output of the execution target, this is the case. However, the results will be returned to the main ret in the order of end, so you have to endure it.

async_stdout.py


import asyncio
from asyncoio.subprocess import PIPE
import sys

async def run(param):
    p = await asyncio.create_subprocess_shell("some command %s"  param,
            stdout=PIPE, ...)
    return await (p.communicate())[0].decode('code').splitlines()

async def main():
    funcs = asyncio.gather(*[run(p) for p in parameters]*)
    for f in asyncio.as_completed(funcs):
        ret = await f

if sys.platform.startswith('win'):
    loop = asyncio.ProactorEventLoop()
    asyncio.set_event_loop(loop)
else:
    loop = asyncio.get_event_loop()

loop.run_until_complete(main())

Remarks

I feel like I'm writing just something similar. Will I write about openpyxl next?

Recommended Posts

python parallel / asynchronous execution memorandum
Python memorandum
Python Memorandum 2
Python memorandum
python memorandum
python memorandum
Python memorandum
python memorandum
Python memorandum
Easy parallel execution with python subprocess
Python basics memorandum
Python pathlib memorandum
Python memorandum (algorithm)
Python memorandum [links]
Parallel task execution using concurrent.futures in Python
[Python] Create an asynchronous task execution environment + monitoring environment
Python memorandum numbering variables
Function execution time (Python)
python memorandum (sequential update)
Parallel download in Python
Python memorandum (personal bookmark)
Python basic memorandum part 2
Output python execution time
[Python] Iterative processing_Personal memorandum
Memorandum @ Python OR Seminar
python memorandum super basic
Python Basic Course (3 Python Execution)
Effective Python Learning Memorandum Day 15 [15/100]
Cisco Memorandum _ Python config input
Python execution time measurement memo
Effective Python Learning Memorandum Day 6 [6/100]
Effective Python Learning Memorandum Day 12 [12/100]
Run Python unittests in parallel
Effective Python Learning Memorandum Day 9 [9/100]
Effective Python Learning Memorandum Day 8 [8/100]
ABC memorandum [ABC163 C --managementr] (Python)
About python beginner's memorandum function
Memorandum @ Python OR Seminar: matplotlib
A memorandum about correlation [Python]
Effective Python Learning Memorandum Day 14 [14/100]
Execution time measurement with Python With
Effective Python Learning Memorandum Day 1 [1/100]
Effective Python Learning Memorandum Day 13 [13/100]
Effective Python Learning Memorandum Day 3 [3/100]
Effective Python Learning Memorandum Day 5 [5/100]
Memorandum @ Python OR Seminar: Pandas
[python] Random number generation memorandum
Effective Python Learning Memorandum Day 4 [4/100]
Memorandum @ Python OR Seminar: scikit-learn
Effective Python Learning Memorandum Day 2 [2/100]
ABC memorandum [ABC159 C --Maximum Volume] (Python)
Python pywin32 (win32com) Excel operation memorandum
Python parallel processing (multiprocessing and Joblib)
[Python] A memorandum of beautiful soup4
python dict object memorandum (mysterious document)
Measure function execution time in Python
[Python] Asynchronous request with async / await
Git & Github & python & VScode Personal memorandum
Python (from first time to execution)
PIL (Python Imaging Library) installation memorandum
Python unittest module execution in vs2017