Execute function in parallel

There are many good articles on how to take an argument to one function and process it in parallel according to the value of the argument.

I thoroughly investigated the parallel processing and parallel processing of Python

However, there haven't been many methods for parallel processing multiple functions that do not take arguments, which have already been defined during development (although there may be few use cases), so I will summarize them.

The code is also available on Github

Since joblib is used for parallel processing this time, let's install it if you have not already installed it.

pip install joblib

First is the definition of a function that simply executes multiple functions in order.

def run_func(*funcs):
    [f() for f in funcs]

Next, let this function be processed in parallel.

from joblib import Parallel, delayed

def parallel_process_func(*target_funcs):
    Parallel(n_jobs=-1)([delayed(run_func)(func) for func in target_funcs])

n_jobs is the number of simultaneous processing, and when it is -1, parallel processing is performed using the core of the machine as much as possible. For example, if you set it to 3, it will be 3 parallel processing.

that's all. Let's actually move it.

def x():
    print('-- x() START --')
    [i for i in range(10000000)]
    print('-- x() END --')


def y():
    print('-- y() START --')
    [i for i in range(10000000)]
    print('-- y() END --')

parallel_process_func(x, y)

When executed, the output should look like the following.

-- x() START --
-- y() START --
-- x() END --
-- y() END --

It's being processed in parallel.

It seems that the usability is not bad because you only have to pass the function name to the defined parallel_process_func function.

Recommended Posts

Execute function in parallel
Duality in function
To execute a Python enumerate function in JavaScript
Generator function in JavaScript
Parallel download in Python
Covector to think in function
Create a function in Python
Use callback function in Python
Run Python unittests in parallel
ntile (decile) function in python
Implement timer function in pygame
Execute external command in python
Nonlinear function modeling in Python
Draw implicit function in python
Immediate function in python (lie)
Implement R's power.prop.test function in python
Create user authentication function in Airflow
sort warning in the pd.concat function
Function argument type definition in python
Included notation in Python function arguments
Implementation of login function in Django
Write AWS Lambda function in Python
Measure function execution time in Python
Function synthesis and application in Python
Read files in parallel with Python
Precautions when pickling a function in python
OR the List in Python (zip function)
How to execute commands in jupyter notebook
Define custom actions in JupyterHub (hook function)
Write and execute SQL directly in Elixir
Difference in output of even-length window function
Parallel task execution using concurrent.futures in Python