Points that are easy to make mistakes when using lambda during Python loop processing

I wrote the following code to "download 500 files in parallel" in Python.

from time import sleep
from concurrent.futures import ThreadPoolExecutor

def func(i):
    #Actually file download process
    sleep(1)
    print(f"{i}\n", flush=True, end="")

with ThreadPoolExecutor(max_workers=3) as executor:
    for i in range(500):
        # 0~Download file 499 in parallel
        executor.submit(
            lambda: func(i)
        )

When I executed this, I got the behavior like "The process is executed many times with the same number" as shown below.

499
499
499
499
499
499
144
145
146
1
0
2
(Omitted below)

Apparently, "the value of ʻi is read when func is actually executed, so many values of ʻi at the end of the loop are read."

By using functools.partial as shown below, it will be executed as originally expected" to execute the process with all the values of ʻi` from 0 to 499 ".

from time import sleep
from functools import partial
from concurrent.futures import ThreadPoolExecutor

def func(i):
    sleep(1)
    print(f"{i}\n", flush=True, end="")

with ThreadPoolExecutor(max_workers=3) as executor:
    for i in range(500):
        executor.submit(
            partial(func, i)
        )

The output looks like this.

0
2
1
3
5
4
6
7
8
9
10
11
12
13
14
(Omitted below)

Also, as I learned from this blog, the default argument of Python There seems to be a way to bind with. I think it's easier to convey the intent of the program by using partial, but it's certainly interesting.

from time import sleep
from concurrent.futures import ThreadPoolExecutor

def func(i):
    sleep(1)
    print(f"{i}\n", flush=True, end="")

with ThreadPoolExecutor(max_workers=3) as executor:
    for i in range(500):
        executor.submit(
            lambda x=i: func(x)
        )

Postscript

From a friend, " submit receives the argument of the function to execute But then I got a tsukkomi saying, "Is it useless?", But I think that's exactly the case. Here is the code for actual use.

with ThreadPoolExecutor(max_workers=3) as executor:
    for i in range(500):
        executor.submit(func, i)

Recommended Posts

Points that are easy to make mistakes when using lambda during Python loop processing
Python list comprehensions that are easy to forget
Five useful Python data types that are easy to forget
Periodic execution processing when using tkinter [Python3]
10 Python errors that are common to beginners
How to take multiple arguments when doing parallel processing using multiprocessing in python
What are you using when testing with Python?
[LPIC 101] I tried to summarize the command options that are easy to make a mistake
A little trick to know when writing a Twilio application using Python on AWS Lambda
Error due to conflict between python when using gurobi
Easy way to scrape with python using Google Colab
Check types_map when using mimetypes on AWS Lambda (Python)
How to make a Python package using VS Code
[Introduction to Python] How to stop the loop using break?
Things to keep in mind when processing strings in Python2
What I was addicted to when using Python tornado
Things to keep in mind when processing strings in Python3
Python error messages are specific and easy to understand "ga" (before that, a colon (:) and a semicolon (;))
A story that makes it easy to estimate the living area using Elasticsearch and Python