I'm running in parallel using multiprocessing.pool.Pool.
In the cube_for_row function running in parallel, when I run it in parallel, I get" AssertionError: daemonic processes are not allowed to have children ".
main.py
from typing import List
import multiprocessing
def cube(value: float) -> float:
return value * value * value
def cube_for_row(row: List[float]) -> List[float]:
with multiprocessing.Pool() as pool:
return pool.map(cube, row)
def cube_for_matrix(matrix: List[List[float]], is_parallel:bool) -> List[List[float]]:
if is_parallel:
with multiprocessing.Pool() as pool:
return pool.map(cube_for_row, matrix)
else:
return [cube_for_row(row) for row in matrix]
def main():
matrix = [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]]
result1 = cube_for_matrix(matrix, False)
print(result1)
# OK
result2 = cube_for_matrix(matrix, True)
print(result2)
# NG
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/vagrant/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/vagrant/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/home/vagrant/Downloads/run.py", line 8, in cube_for_row
with multiprocessing.Pool() as p:
File "/home/vagrant/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/context.py", line 119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
File "/home/vagrant/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/pool.py", line 212, in __init__
self._repopulate_pool()
File "/home/vagrant/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/pool.py", line 303, in _repopulate_pool
return self._repopulate_pool_static(self._ctx, self.Process,
File "/home/vagrant/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/pool.py", line 326, in _repopulate_pool_static
w.start()
File "/home/vagrant/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/process.py", line 118, in start
assert not _current_process._config.get('daemon'), \
AssertionError: daemonic processes are not allowed to have children
"""
The error message says "Daemon process cannot have child process".
The function I'm passing to multiprocessing.pool.Pool seems to be running in a daemon process.
The multiprocessing.Process.daemon documentation states that "daemon processes cannot create child processes".
A Boolean flag that indicates whether it is a daemon process. This attribute must be set before start () is called. The initial value is inherited from the creating process. When a process terminates, it attempts to terminate all of its child daemon processes.
Note that the daemon process cannot create child processes. If it can be created, the child process of the daemon process may become an orphan when the parent process of the daemon process terminates. Furthermore, daemon processes are regular processes, not Unix daemons or services, and are terminated (and not joined) when a non-daemon process terminates.
Since it is an "Assertion Error", it can be ignored.
$ python -O main.py
[[1, 8, 27, 64], [125, 216, 343, 512], [729, 1000, 1331, 1728]]
If you just ignore it, it seems that a zombie process will be born.
For daemon processes, do not process in parallel.
def cube_for_row(row: List[float]) -> List[float]:
"""If it can be executed in parallel, execute it in parallel"""
process = multiprocessing.current_process()
if process.daemon:
return [cube(v) for v in row]
else:
with multiprocessing.Pool() as pool:
return pool.map(cube, row)