[Python] Easy parallel processing with Joblib

Python is [GIL (Global Interpreter Lock)](https://ja.wikipedia.org/wiki/%E3%82%B0%E3%83%AD%E3%83%BC%E3%83%90%E3% 83% AB% E3% 82% A4% E3% 83% B3% E3% 82% BF% E3% 83% 97% E3% 83% AA% E3% 82% BF% E3% 83% AD% E3% 83% 83% E3% 82% AF) is applied, and basically, if you have multiple CPU cores, you cannot use all the resources just by writing the code.

However, when processing a large amount of data, there are times when you want to use all the CPU resources of the machine to calculate as fast as possible.

With keywords such as Python parallel processing If you search, you can often see the explanation of multiprocessing module by the standard library, so I think many people are using this. ..

I would like to use this module when creating a system that incorporates parallel processing, but honestly it is a little troublesome when just writing abandoned code ...

Therefore, I would like to introduce a module called Joblib that can execute parallel processing more easily and quickly.

Not to mention the reduced amount of code, as another advantage

--It also displays the error that was thrown by the child process. --Child process will be terminated when it is terminated by Ctrl + c --There is an option to automatically display the overall progress

And so on. All of them are very useful for me because I have to implement them by myself in multiprocessing.

Installation

First is the installation. It is one shot with pip. The version at the time of writing this article is 0.9.3.

pip install joblib
Successfully installed joblib-0.9.3

The environment used for the test is a virtual 4-core, physical 2-core MacBook Pro.

Code used for testing

To simply compare the calculation times, test with code like the following. The calculation content itself has no particular meaning.

First, repeat a simple calculation, measure the calculation time with the time module, and display it.

# -*- coding: utf-8 -*-

from time import time

def process(n):
	return sum([i*n for i in range(100000)])

start = time()

#Iterative calculation
total = 0
for i in range(10000):
	total += process(i)
print(total)

print('{}It took a second'.format(time() - start))
249972500250000000
78.It took 2647480965 seconds

The answer to the calculation is 249972500250000000, which takes 78 seconds.

Parallelize with Joblib

Let's change only the iterative calculation part of the above code and parallelize it. Multi-process parallelization can be achieved by combining Parallel and delayed.

# -*- coding: utf-8 -*-
from joblib import Parallel, delayed
from time import time

def process(n):
	return sum([i*n for i in range(100000)])

start = time()

#Iterative calculation(Parallelization)
r = Parallel(n_jobs=-1)( [delayed(process)(i) for i in range(10000)] )
print(sum(r))

print('{}It took a second'.format(time() - start))
249972500250000000
37.5521140099 took

It has been shortened to 37 seconds! The answer to the calculation is also correct. I was able to shorten the calculation time quickly just by rewriting a part of the code.

The parallel argument n_jobs is the number of cores to use, and if you set this to -1, it will always execute with the maximum number of cores that can be used.

If you set it to 1, it will be the same as running without parallelization, so it is easy to restore it. It seems that there is no problem even if the iterative process is basically written in Parallel.

Also, if you specify a number from 0 to 10 in the argument verbose, the progress will be output according to the specified frequency. (0 means no output, 10 is the mode)

r = Parallel(n_jobs=-1, verbose=10)( [delayed(process)(i) for i in range(10000)] )

Bonus: Manipulate variables outside the method

In the above code example, it is possible to refer to variables existing in the external scope from within the process method that is executed in parallel, but it is not possible to assign a new value. This is because the memory area that can be referenced differs for each process.

The following code works normally, but parallelized code causes an error and cannot be executed.

#Referencing and manipulating the external variable number from within the process method
number = 0

def process(n):
	number = 3.14
	return sum([i*n for i in range(100000)])

To solve this, it is necessary to create a variable that shares a memory area between processes. It is provided in the multiprocessing module, so let's use it.

The above code can be realized as follows.

# -*- coding: utf-8 -*-
from joblib import Parallel, delayed
from multiprocessing import Value, Array

shared_int = Value('i', 1)

def process(n):
	shared_int.value = 3.14
	return sum([i*n for i in range(100000)])

#Iterative calculation(Parallelization)
Parallel(n_jobs=-1)( [delayed(process)(i) for i in range(10000)] )

print(shared_int.value)
3.14

By using the Value class, you can prepare numbers such as int and double as shared variables. Please note that it is necessary to specify the type with the first argument in advance.

It is also possible to prepare a list of specified types by using ʻArray ('d', [0.0, 0.0, 0.0])`.

It seems to be useful when you want to display your own custom progress!

Finally

I was surprised that I could implement it smoothly when I tried parallel processing with Python and unexpectedly.

The author posts various technical information on Twitter every day. I would be grateful if you could follow me.

https://twitter.com/YuhsakInoue

Recommended Posts

[Python] Easy parallel processing with Joblib
Python parallel processing (multiprocessing and Joblib)
Easy parallel execution with python subprocess
Easy image processing in Python with Pillow
Image processing with Python
Parallel processing with multiprocessing
How to do multi-core parallel processing with python
Parallel processing with no deep meaning in Python
Image processing with Python (Part 2)
100 Language Processing with Python Knock 2015
Parallel processing with local functions
"Apple processing" with OpenCV3 + Python3
[Co-occurrence analysis] Easy co-occurrence analysis with Python! [Python]
Acoustic signal processing with Python (2)
Acoustic signal processing with Python
Easy folder synchronization with Python
Parallel processing with Parallel of scikit-learn
Image processing with Python (Part 1)
Image processing with Python (Part 3)
Easy Python compilation with NUITKA-Utilities
Easy HTTP server with Python
[Python] Image processing with scikit-image
An introduction to Python distributed parallel processing with Ray
100 Language Processing Knock with Python (Chapter 1)
100 Language Processing Knock with Python (Chapter 3)
Image processing with Python 100 knocks # 3 Binarization
Easy Python + OpenCV programming with Canopy
Easy email sending with haste python3
Bayesian optimization very easy with Python
Python parallel / parallel processing sample code summary
Read files in parallel with Python
Easy data visualization with Python seaborn.
Image processing with Python 100 knocks # 2 Grayscale
Easy modeling with Blender and Python
Basics of binarized image processing with Python
Image processing with Python 100 knock # 10 median filter
Easy keyword extraction with TermExtract for Python
[Python] Super easy test with assert statement
[Python] Easy argument type check with dataclass
Image processing with Python 100 knocks # 8 Max pooling
Periodically perform arbitrary processing with Python Twisted
Let Heroku do background processing with Python
Easy introduction of speech recognition with Python
100 Language Processing Knock with Python (Chapter 2, Part 2)
Image processing with Python & OpenCV [Tone Curve]
3. Natural language processing with Python 2-1. Co-occurrence network
Image processing with Python 100 knock # 12 motion filter
3. Natural language processing with Python 1-1. Word N-gram
[Easy Python] Reading Excel files with openpyxl
100 Language Processing Knock with Python (Chapter 2, Part 1)
Easy web app with Python + Flask + Heroku
Drawing with Matrix-Reinventor of Python Image Processing-
Image processing with Python 100 knocks # 7 Average pooling
[Easy Python] Reading Excel files with pandas
Easy web scraping with Python and Ruby
[Python] Easy Reinforcement Learning (DQN) with Keras-RL
Light image processing with Python x OpenCV
Image processing with Python 100 knocks # 9 Gaussian filter
FizzBuzz with Python3
Scraping with Python
Python is easy