Parallel processing with Parallel of scikit-learn

2019/1016 Addendum: I wrote about the cache function of Joblib. [[Python] Use Joblib cache to omit the same calculation](https://tma15.github.io/blog/2019/10/06/python-Use joblib cache to omit the same calculation /)

Speaking of scikit-learn tutorials, the introduction of @Scaled_Wurm Man is very easy to understand. This time I happened to be reading the source code and note a niche that wasn't introduced in that blog entry. The conclusion is that if there is no particular reason, the part written in multiprocessing may be replaced with Parallel.

Parallel

• Class that performs parallel processing
• Basically, parallel processing is performed by multiprocessing, but it provides a help function that I wish I had.

Is multiprocessing useless?

Parallel is (the original is Notes in the source code)

• You don't have to create a list of function arguments
• Debug is easy
• You can eliminate the parallel function by setting n_jobs = 1 without changing the code.
• Traces back errors thrown by child processes
• You can see the progress
• You can abort a multiprocessing job with Ctrl-C without any hassle (Reference pool)))

argument

• n_jobs (int): Number of cpu to process in parallel
• verbose (int): The detail of the output message. Message output on all iterations with 10 or more.
• pre_dispatch: Number of cpu called before starting parallelism

Example

• It's faster to see the example
• delayed is a wrapper for the function you want to parallelize and returns the function and its arguments. Those functions are executed when Parallel call () is called.

Simple example

>>> from math import sqrt
>>> from sklearn.externals.joblib import Parallel, delayed
>>> Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
[0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]

Example of seeing progress

>>> from time import sleep
>>> from sklearn.externals.joblib import Parallel, delayed
>>> r = Parallel(n_jobs=2, verbose=5)(delayed(sleep)(.1) for _ in range(10)) #doctest: +SKIP
[Parallel(n_jobs=2)]: Done   1 out of  10 | elapsed:    0.1s remaining:    0.9s
[Parallel(n_jobs=2)]: Done   3 out of  10 | elapsed:    0.2s remaining:    0.5s
[Parallel(n_jobs=2)]: Done   6 out of  10 | elapsed:    0.3s remaining:    0.2s
[Parallel(n_jobs=2)]: Done   9 out of  10 | elapsed:    0.5s remaining:    0.1s
[Parallel(n_jobs=2)]: Done  10 out of  10 | elapsed:    0.5s finished

• The larger the value you put in verbose, the more messages you get
• Outputs the number of jobs, elapsed time, etc.

Specify pre_dispatch

• default is'all'
• If one job uses a lot of memory, I wonder if it will be used by reducing it.
• In the example below, 3 (1.5 * 2 jobs) jobs are called before parallel processing.

>>> from math import sqrt
>>> from sklearn.externals.joblib import Parallel, delayed

>>> def producer():
...     for i in range(6):
...         print('Produced %s' % i)
...         yield i

>>> out = Parallel(n_jobs=2, verbose=100, pre_dispatch='1.5*n_jobs')(
...                         delayed(sqrt)(i) for i in producer()) #doctest: +SKIP
Produced 0 ###The first one
Produced 1 ###Second
Produced 2 ###Third
[Parallel(n_jobs=2)]: Done   1 jobs       | elapsed:    0.0s
Produced 3
[Parallel(n_jobs=2)]: Done   2 jobs       | elapsed:    0.0s
Produced 4
[Parallel(n_jobs=2)]: Done   3 jobs       | elapsed:    0.0s
Produced 5
[Parallel(n_jobs=2)]: Done   4 jobs       | elapsed:    0.0s
[Parallel(n_jobs=2)]: Done   5 out of   6 | elapsed:    0.0s remaining:    0.0s
[Parallel(n_jobs=2)]: Done   6 out of   6 | elapsed:    0.0s finished