I wrote the following code to "download 500 files in parallel" in Python.
from time import sleep
from concurrent.futures import ThreadPoolExecutor
def func(i):
#Actually file download process
sleep(1)
print(f"{i}\n", flush=True, end="")
with ThreadPoolExecutor(max_workers=3) as executor:
for i in range(500):
# 0~Download file 499 in parallel
executor.submit(
lambda: func(i)
)
When I executed this, I got the behavior like "The process is executed many times with the same number" as shown below.
499
499
499
499
499
499
144
145
146
1
0
2
(Omitted below)
Apparently, "the value of ʻi is read when func is actually executed, so many values of ʻi
at the end of the loop are read."
By using functools.partial
as shown below, it will be executed as originally expected" to execute the process with all the values of ʻi` from 0 to 499 ".
from time import sleep
from functools import partial
from concurrent.futures import ThreadPoolExecutor
def func(i):
sleep(1)
print(f"{i}\n", flush=True, end="")
with ThreadPoolExecutor(max_workers=3) as executor:
for i in range(500):
executor.submit(
partial(func, i)
)
The output looks like this.
0
2
1
3
5
4
6
7
8
9
10
11
12
13
14
(Omitted below)
Also, as I learned from this blog, the default argument of Python There seems to be a way to bind with. I think it's easier to convey the intent of the program by using partial
, but it's certainly interesting.
from time import sleep
from concurrent.futures import ThreadPoolExecutor
def func(i):
sleep(1)
print(f"{i}\n", flush=True, end="")
with ThreadPoolExecutor(max_workers=3) as executor:
for i in range(500):
executor.submit(
lambda x=i: func(x)
)
From a friend, " submit
receives the argument of the function to execute But then I got a tsukkomi saying, "Is it useless?", But I think that's exactly the case. Here is the code for actual use.
with ThreadPoolExecutor(max_workers=3) as executor:
for i in range(500):
executor.submit(func, i)
Recommended Posts