[Python] It seems that global variables cannot be referenced in Multiprocessing

I didn't have a report on this, so I'll drop a note.

In Python, the standard module Multiprocessing allows parallel computing. Hardware parallel computing like GPU is not supported, it is just multi-process processing on CPU.

The elements of the pooled argument list are passed to the arguments of the function you want to process in parallel.

If you write a script that refers to a global variable in a function that is processed in parallel, it will be as follows.

example.py


import multiprocessing as mp

a = 0

def init_global():
    global a
    a = 100

def func(proc):
    global a
    print("Inside MultiProcess: ", proc, a)
    return a

if __name__ == "__main__":

    print("Before Change", a)

    init_global()

    print("After Change", a)

    pool = mp.Pool(4)
    callback = pool.map(func, range(4))
    pool.terminate()
    pool.join()

    print(sum(callback))

result 2017-03-14_150753.jpg As you can see, you can refer to and assign in a normal function (init_global), but you cannot refer to a global variable in a function that is processed in parallel.


If you really want to refer to the value of a global variable in a parallel processing function, you have to specify the argument from the caller. (If you want to pass multiple arguments, you need to generate a tuple)

example.py


import multiprocessing as mp

a = 0
b = 1111

def init_global():
    global a
    a = 100

def func(proc):
    print("Inside MultiProcess: ", proc[0], proc[1])
    return proc[0]

if __name__ == "__main__":

    print("Before Change", a)

    init_global()

    print("After Change", a)

    pool = mp.Pool(4)
    args = [(a,b) for i in range(4)]
    callback = pool.map(func, args)
    pool.terminate()
    pool.join()

    print(sum(callback))

result 2017-03-14_151709.jpg I was able to hand it over.

There may be a proper method, but I couldn't find it, so please report if there is a better method.

Recommended Posts

[Python] It seems that global variables cannot be referenced in Multiprocessing
Processing of python3 that seems to be usable in paiza
Using global variables in python functions
[Python3] Dynamically define global variables in functions
Practice applying functions and global variables in Python
It seems that unidic-lite is required in mecab-python3
A solution to the problem that the Python version in Conda cannot be changed
Let's look at a differential equation that cannot be solved normally in Python
[Memorandum] Japanese keys cannot be used in python string.Template.substitute
list comprehension because operator.methodcaller cannot be used in python 2.5
Operators ++,-cannot be used in python (difference from php)
[Redash] Standard library cannot be used in python function
python> Is it possible to make in-line comments?> It seems that it has to be on multiple lines
Video cannot be loaded with Spyder in Python development environment
Error that Qt plugin "cocoa" cannot be found in python-opencv
Scripts that can be used when using bottle in Python
Handle environment variables in Python
Note that it supports Python 3
A record that GAMEBOY could not be done in Python. (PYBOY)
33 strings that should not be used as variable names in python
Python standard input summary that can be used in competition pro
Jupyter Notebook 6.0.2 cannot be installed in the Python 2.7 environment created in Anaconda
TensorFlow / python> // grammar> It seems to be python's integer division / In Python 2.X, describe from __future__ import division / floor division
There are no “private” instance variables in Python that can only be accessed from within an object.
Initializing global variables using Python decorators
Landmines hidden in Python class variables
I made a Discord bot in Python that translates when it reacts
Modules cannot be imported in Python on EC2 run from AWS Lambda
Address to the bug that node.surface cannot be obtained with python3 + mecab