--Identify bottlenecks to execute code with sufficient speed and lean resource usage --Famous tool for python --Python and timeit
--Extended version of python interactive interpreter --Measure the average execution time by specifying the number of tests (-r) and the number of loops (-n) with the timeit module. --There are two patterns: direct execution of short code and import of code file.
pip install ipython
profiling.py
def fn_arr():
for i in range(1000): arr.append(i)
def fn_ls():
for i in range(1000): ls.append(i)
Single code execution
ipython
In [1]:from array import array
In [2]:arr = array('I', [])
In [3]:%timeit -r 10 -n 10 for i in range(1000): arr.insert(0, i)
10 loops, best of 10: 3.02 ms per loop
In [4]:%timeit -r 10 -n 10 for i in range(1000): ls.insert(0, i)
10 loops, best of 10: 3.8 ms per loop
Run file
ipython
In [1]: from profiling import fn_arr
In [2]: from profiling import fn_ls
In [3]: %timeit -r 10 -n 10 fn_arr()
10 loops, best of 10: 68.6 µs per loop
In [4]: %timeit -r 10 -n 10 fn_ls()
10 loops, best of 10: 8.92 µs per loop
Get the average execution time per loop when 10 loops are executed 10 times
cProfile --Standard profiling tool --Measure the execution time of all functions to be executed
profiling.py
def fn():
l = []
for _ in range(1000000):
l.append('hoge')
l.pop()
fn()
Specify a file and execute
python -m cProfile -s cumulative index.py
2000004 function calls in 23.897 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 23.897 23.897 {built-in method exec}
1 0.000 0.000 23.897 23.897 index.py:1(<module>)
1 12.231 12.231 23.897 23.897 index.py:3(fn)
1000000 5.833 0.000 5.833 0.000 {method 'append' of 'list' objects}
1000000 5.832 0.000 5.832 0.000 {method 'pop' of 'list' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
Output measurement results
python -m cProfile -o profile.stats index.py
Read and display the measurement result file
python
>>> import pstats
>>> p = pstats.Stats('profile.stats')
>>> p.sort_stats('cumulative')
<pstats.Stats object at 0x7f5cca3f0ef0>
>>> p.print_stats()
Sat Nov 28 06:58:28 2015 profile.stats
2000004 function calls in 8.477 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 8.477 8.477 {built-in method exec}
1 0.000 0.000 8.477 8.477 index.py:1(<module>)
1 4.375 4.375 8.477 8.477 index.py:3(fn)
1000000 2.088 0.000 2.088 0.000 {method 'pop' of 'list' objects}
1000000 2.015 0.000 2.015 0.000 {method 'append' of 'list' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
<pstats.Stats object at 0x7fadfcc20f28>
Besides
--print_callers () --Caller information
--print_callees () --Information when a function calls another function
Etc. can be displayed
line_profiler --A tool that measures the number of times processing is executed and the execution time in line units.
pip install line_profiler
git clone https://github.com/rkern/line_profiler.git
profiling
@profile
def fn():
l = []
for _ in range(1000000):
l.append('hoge')
l.pop()
fn()
> python line_profiler/kernprof.py -l -v index.py
Wrote profile results to index.py.lprof
Timer unit: 1e-06 s
Total time: 8.31965 s
File: index.py
Function: fn at line 2
Line # Hits Time Per Hit % Time Line Contents
==============================================================
2 @profile
3 def fn():
4 1 6 6.0 0.0 l = []
5 1000001 2641832 2.6 31.8 for _ in range(1000000):
6 1000000 2806484 2.8 33.7 l.append('hoge')
7 1000000 2871333 2.9 34.5 l.pop()
--His --Number of times called --Time --Run time --Per Hit --One execution time --% Time --Running time per line relative to total running time
memory_profiler --Memory version of line_profiler
pip install memory_profile
pip install psutil
Similar to line_profiler
> python -m memory_profiler index.py
Filename: index.py
Line # Mem usage Increment Line Contents
================================================
2 10.316 MiB 0.000 MiB @profile
3 def fn():
4 10.320 MiB 0.004 MiB l = []
5 10.320 MiB 0.000 MiB for _ in range(1000000):
6 10.320 MiB 0.000 MiB l.append('hoge')
7 10.320 MiB 0.000 MiB l.pop()
Recommended Posts