Effective Python Memo Item 9 Consider generator expressions for large comprehensions

** Notes on Effective Python ** Item 9 Consider a generator formula for large comprehensions (p18 ~ 20)

Use generators when working with large lists

Processing comprehensions with large amounts of data has some problems.

  1. (Depending on the process) may generate a new list
  2. When calling a loop statement, etc., everything is expanded in memory, which may cause a memory crash.

The generator also solves these problems. The advantage of the generator is that it doesn't call the whole sequence every time. Since it is called one at a time and processed, there is almost no concern about memory overrun.

Generator expressions are similar in syntax to comprehensions and are written in () instead of [].

my_lists = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
it = (x for x in my_lists)
print(it)

>>>
<generator object <genexpr> at 0x103fe93a8>

※Caution It is the generator object that is returned by the generator expression, not the value. To generate a value, use the next () function to generate the value. Again, the generator doesn't generate any values!

print(next(it))
print(next(it))

>>>
1
2

You can also immediately assign the iterator returned by one generator to another generator.

my_lists = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
it = (x for x in my_lists)
roots = ((x, x ** 0.5) for x in it)
print(next(roots))
print(next(roots))

>>>
(1, 1.0)
(2, 1.4142135623730951)

Summary

We haven't planned the processing time this time, but due to the characteristics of the generator, it will not take much time. Therefore, it is a very convenient function for processing large-scale data.

Recommended Posts

Effective Python Memo Item 9 Consider generator expressions for large comprehensions
EP 9 Consider Generator Expressions for Large Comprehensions
Effective Python Memo Item 3
Effective Python Memo Item 8 Avoid three or more expressions in list comprehensions
Effective Python Note Item 16 Consider returning a generator without returning a list
Effective Python memo Item 10 Enumerate from range
Python memo (for myself): Array
Python code memo for yourself
Effective Python Memo Item 19 Give optional behavior to keyword arguments
Effective Python Memo Item 11 Use zip to process iterators in parallel
Effective Python Note Item 17 Respect for certainty when using iterators for arguments
Effective Python Memo Item 4 Write a helper function instead of a complicated expression
Effective Python memo Item 7 Use list comprehension instead of map and filter
Python comprehension (list and generator expressions) [additional]
Effective Python Memo Item 18 Use variable-length positional arguments to make the appearance cleaner
Effective Python Note Item 12 Avoid using else blocks after for and while loops