Parallel computing (pathos) when dealing with objects that cannot be pickled

Overview

We will introduce the solution when the following error is encountered when performing process parallel calculation using joblib and multiprocessing. The cause is that it cannot be pickled, which is used for interprocess communication.

error1}


TypeError: cannot pickle 'SwigPyObject' object

error2}


PicklingError: Could not pickle the task to send it to the workers.

Solution

Use pathos. You can install it with pip install pathos.

code}


# some_func, some_Please create a list appropriately
from pathos.multiprocessing import ProcessingPool
pool = ProcessingPool(nodes=4)
result_list = pool.map(lambda x: some_func(x), some_list)

bonus

If it is pickle at the time of saving, it can be solved by using dill.

reference

Recommended Posts

Parallel computing (pathos) when dealing with objects that cannot be pickled
Items that cannot be imported with sklearn
When Html cannot be output with Jupyter Notebook
Investigation when import cannot be done with python
Import libraries that cannot be pip installed with PyCharm
[android] When AVD cannot be used with permission denied
Dealing with the error that HTTP fetch error occurs in gpg and the key cannot be obtained
Solution when the image cannot be displayed with tkinter [python]
When Times New Roman cannot be used with matplotlib (Ubuntu)
Examples and solutions that cannot be optimized well with scipy.optimize.least_squares
Parallel computing with iPython notebook
When USB cannot be formatted
When pydub cannot be installed
Here's a summary of things that might be useful when dealing with complex numbers in Python
Measures that pip install cannot be done with pycharm or import ssl cannot be done
Address to the bug that node.surface cannot be obtained with python3 + mecab