To receive multiple return values ​​from a function executed using parallel processing (Pool) in Python

Introduction

When performing parallel processing with Python, I was confused about how to write in the following cases, so I summarized it as a memorandum.

usage environment

Implementation details

This time, I made the following sample code as an example.

  1. If you pass two values, prepare a function that finds the greatest common divisor and the least common multiple and returns each.
  2. Try calling each with and without parallel

Source code

No parallel

This simply calls the function for the length of the list NUMBERS.

python


import time

def calc(num_pair):
    x, y = num_pair
    low = min(x, y)
    gcd = 0

    # Greatest common divisor
    for i in range(low, 0, -1):
        if x % i == 0 and y % i == 0:
            gcd = i
            break
    
    # Least common multiple
    lcm = x * y // gcd

    return gcd, lcm


if __name__ == "__main__":

    result=[]
    NUMBERS = [
        (12345678,24680246),(91845010,35889830),
        (82163657,75546871),(46015383,43872681),
        (73739990,64003993),(26514146,33211514),
        (51395783,78597259),(99939535,88084461)
    ]
    start = time.time()

    gcd = []
    lcm = []
    for  i, pair in enumerate(NUMBERS):
        g, l = calc(NUMBERS[i])
        gcd.append(g)
        lcm.append(l)

    print(f'gcd = {gcd}')
    # gcd = [2, 10, 1, 3, 1, 2, 1, 1]
    print(f'lcm = {lcm}')
    # lcm = [152347185038394, 329630179524830, 6207207196267247, 672939406483941, 4719653803780070, 440287465538522, 4039567667958797, 8803120073065635]

    for i, pair in enumerate(NUMBERS):
        print(f'{pair[0]}When{pair[1]}The greatest common divisor of{gcd[i]},The least common multiple is{lcm[i]}is')
    #The output result is "with parallel"

    end = time.time()
    print(f'Time = {(end-start):.3f}')
    #Result of measuring 3 times
    # Time = 29.068
    # Time = 29.796
    # Time = 29.890

With parallel

The point is the content of the result received by p.map (). It's hard to know that the more the return value of a function, the more it fits into one tuple of result, but if you don't know it, you'll want to receive the value as written without parallel.

from multiprocessing import Pool
import time

def calc(num_pair):

    x, y = num_pair
    low = min(x, y)
    gcd = 0

    # Greatest common divisor
    for i in range(low, 0, -1):
        if x % i == 0 and y % i == 0:
            gcd = i
            break
    
    # Least common multiple
    lcm = x * y // gcd

    return gcd, lcm


if __name__ == "__main__":

    result=[]
    NUMBERS = [
        (12345678,24680246),(91845010,35889830),
        (82163657,75546871),(46015383,43872681),
        (73739990,64003993),(26514146,33211514),
        (51395783,78597259),(99939535,88084461)
    ]
    start = time.time()
    
    p = Pool(8)

    try:
        result = p.map(calc, NUMBERS)
    except Exception as e:
        print(e)

    print(f'result = {result}')
    # result = [(2, 152347185038394), (10, 329630179524830), (1, 6207207196267247), (3, 672939406483941), (1, 4719653803780070), (2, 440287465538522), (1, 4039567667958797), (1, 8803120073065635)]

    gcd = [i[0] for i in result]
    lcm = [j[1] for j in result]


    print(f'gcd = {gcd}')
    # gcd = [2, 10, 1, 3, 1, 2, 1, 1]
    print(f'lcm = {lcm}')
    # lcm = [152347185038394, 329630179524830, 6207207196267247, 672939406483941, 4719653803780070, 440287465538522, 4039567667958797, 8803120073065635]

    for i, pair in enumerate(NUMBERS):
        print(f'{pair[0]}When{pair[1]}The greatest common divisor of{gcd[i]},The least common multiple is{lcm[i]}is')
    #The greatest common divisor of 12345678 and 24680246 is 2.,The least common multiple is 152347185038394
    #The greatest common divisor of 91845010 and 35889830 is 10.,The least common multiple is 329630179524830
    #The greatest common divisor of 82163657 and 75546871 is 1.,The least common multiple is 6207207196267247
    #The greatest common divisor of 46015383 and 43872681 is 3,The least common multiple is 672939406483941
    #The greatest common divisor of 73739990 and 64003993 is 1.,The least common multiple is 4719653803780070
    #The greatest common divisor of 26514146 and 3321514 is 2,The least common multiple is 440287465538522
    #The greatest common divisor of 51395783 and 78597259 is 1.,The least common multiple is 4039567667958797
    #The greatest common divisor of 99939535 and 88084461 is 1.,The least common multiple is 8803120073065635

    end = time.time()
    print(f'Time = {(end-start):.3f}')
    #Result of measuring 3 times
    # Time = 18.861
    # Time = 16.983
    # Time = 18.362

Conclusion

If you call a function that returns multiple values ​​in Pool and map, it will be returned as a list of tuples stored for the number of returned values.

Supplement

This time, I made the above code intentionally to make it a time-consuming process for explanation, but if you want to find the greatest common divisor really quickly, it is better to use the existing library as follows.

import math
from multiprocessing import Pool
import time

def calc(num_pair):
    x, y = num_pair

    # Greatest common divisor
    gcd = math.gcd(x,y)
    
    # Least common multiple
    lcm = x * y // gcd

    return gcd, lcm


if __name__ == "__main__":
    result=[]
    NUMBERS = [
        (12345678,24680246),(91845010,35889830),
        (82163657,75546871),(46015383,43872681),
        (73739990,64003993),(26514146,33211514),
        (51395783,78597259),(99939535,88084461)
    ]
    start = time.time()

    gcd = []
    lcm = []
    for  i, pair in enumerate(NUMBERS):
        g, l = calc(NUMBERS[i])
        gcd.append(g)
        lcm.append(l)

    for i, pair in enumerate(NUMBERS):
        print(f'{pair[0]}When{pair[1]}The greatest common divisor of{gcd[i]},The least common multiple is{lcm[i]}is')

    end = time.time()
    print(f'Time = {(end-start):.5f}')
    # Time = 0.00013

It's much faster to use math.gcd (). When parallel processing is used here, for some reason Time = 0.17145 and no parallel processing finished faster.

Parallel processing is also quite profound.

Recommended Posts

To receive multiple return values ​​from a function executed using parallel processing (Pool) in Python
To return char * in a callback function using ctypes in Python
How to take multiple arguments when doing parallel processing using multiprocessing in python
How to slice a block multiple array from a multiple array in Python
How to get a value from a parameter store in lambda (using python)
Receive a list of the results of parallel processing in Python with starmap
To execute a Python enumerate function in JavaScript
A clever way to time processing in Python
Receive dictionary data from a Python program in AppleScript
Consider a conversion from a Python recursive function to a non-recursive function
How to retrieve multiple arrays using slice in python.
How to execute a command using subprocess in Python
[Python] How to call a c function from python (ctypes)
How to unit test a function containing the current time using freezegun in python
How to host web app backend processing in Python using a rental server subdomain
Create a function in Python
How to create an instance of a particular class from dict using __new__ () in python
A simple way to avoid multiple for loops in Python
How to define multiple variables in a python for statement
How to sample from any probability density function in Python
A memo of writing a basic function in Python using recursion
I tried to make a stopwatch using tkinter in python
Try running a function written in Python using Fn Project
Just try to receive a webhook in ngrok and python
Using Python mode in Processing
When you want to replace multiple characters in a string without using regular expressions in python3 series
Post a message from IBM Cloud Functions to Slack in Python
How to get a string from a command line argument in python
[Python] How to get & change rows / columns / values from a table.
I wrote a function to load a Git extension script in Python
A function that measures the processing time of a method in python
From installing Ansible to building a Python environment in Vagrant's virtual environment
How to delete multiple specified positions (indexes) in a Python list
Precautions when giving default values to arguments in Python function definitions
Call a Python function from p5.js.
How to generate a new loggroup in CloudWatch using python within Lambda
Create a tool to automatically furigana with html using Mecab from Python3
When running a Python shell from Electron, pass multiple arguments to run Python.
Connect to postgreSQL from Python and use stored procedures in a loop.
How to write a string when there are multiple lines in python
[Python] [Word] [python-docx] Try to create a template of a word sentence in Python using python-docx
Python: I want to measure the processing time of a function neatly
From Python to using MeCab (and CaboCha)
Precautions when pickling a function in python
Send a message from Python to Slack
Send email to multiple recipients in Python (Python 3)
Log in to Slack using requests in Python
How to get a stacktrace in python
Scraping a website using JavaScript in Python
Draw a tree in Python 3 using graphviz
Generate a class from a string in Python
Convert from Markdown to HTML in Python
Parallel task execution using concurrent.futures in Python
If you use Pandas' Plot function in Python, it is really seamless from data processing to graph creation
[Circuit x Python] How to find the transfer function of a circuit using Lcapy
How to plot multiple fits images side by side in galactic coordinates using python
If you want to display values using choices in a template in a Django model
[Python] Smasher tried to make the video loading process a function using a generator
A complete guidebook to using pyenv, pip and python in an offline environment
[Python] I tried to get the type name as a string from the type function
Create a setting in terraform to send a message from AWS Lambda Python3.8 to Slack