Python script profiling

Preparation

easy_install line_profiler

cProfile

--Profile the entire code --Clarify which function is taking the most time

python -m cProfile -s cumulative chi2.py > profile
less profile

cProfile results

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.002    0.002    7.516    7.516 chi2.py:3(<module>)
        1    0.000    0.000    6.099    6.099 chi2.py:101(chi2)
     4959    0.229    0.000    3.511    0.001 chi2.py:11(detect_step)
       29    0.001    0.000    3.291    0.113 chi2.py:84(calculate_border_bestfit)
      435    0.008    0.000    3.289    0.008 chi2.py:35(detect_step_all_ranges)
    76661    0.219    0.000    2.738    0.000 fromnumeric.py:2830(var)
        1    0.000    0.000    2.538    2.538 chi2.py:53(plot)
    76661    1.215    0.000    2.519    0.000 _methods.py:77(_var)
        1    0.000    0.000    2.421    2.421 pyplot.py:138(show)
        1    0.000    0.000    2.421    2.421 backend_bases.py:140(__call__)
        1    0.000    0.000    2.417    2.417 backend_macosx.py:28(mainloop)
        1    2.323    2.323    2.417    2.417 {matplotlib.backends._macosx.show}
        1    0.001    0.001    1.251    1.251 pyplot.py:17(<module>)

--Looking at the results, the chi2 function is taking the most time --Look closely at the chi2 function

Line profile

--Add @profile to chi2 function

chi2.py


@profile
def chi2(trajectory, t):
    s = []
    border_bestfit_best = []
    max_s = 0.0

    for nstep in range(1, 30):
        border_bestfit = calculate_border_bestfit(trajectory, nstep)
        border_counter = calculate_border_counter(trajectory, border_bestfit)
        s = calculate_chi2(trajectory, border_counter) / calculate_chi2(trajectory, border_bestfit)
        if max_s < s:
            max_s = s
            border_bestfit_best = border_bestfit

    # Calculate average trajectory
    average_trajectory = [np.average(trajectory[border_bestfit_best[i]:border_bestfit_best[i+1]])
                          for i in range(0, len(border_bestfit_best) - 1)]

    # Plot
    plot(trajectory, border_bestfit_best, average_trajectory)

    # Calculate step size
    step_size = [average_trajectory[i+1] - average_trajectory[i]
                 for i in range(0, len(average_trajectory)-1)]

    return step_size

--Create an lprof file using kernprof.py

kernprof.py -l chi2.py

Line profiler results

python -m line_profiler chi2.py.lprof
Timer unit: 1e-06 s

Total time: 6.75531 s
File: chi2.py
Function: chi2 at line 101

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   101                                           @profile
   102                                           def chi2(trajectory, t):
   103         1            3      3.0      0.0      s = []
   104         1            1      1.0      0.0      border_bestfit_best = []
   105         1            1      1.0      0.0      max_s = 0.0
   106
   107        30           23      0.8      0.0      for nstep in range(1, 30):
   108        29      4188417 144428.2     62.0          border_bestfit = calculate_border_bestfit(trajectory, nstep)
   109        29       310665  10712.6      4.6          border_counter = calculate_border_counter(trajectory, border_bestfit)
   110        29        46667   1609.2      0.7          s = calculate_chi2(trajectory, border_counter) / calculate_chi2(trajectory, border_bestfit)
   111        29           39      1.3      0.0          if max_s < s:
   112         7            5      0.7      0.0              max_s = s
   113         7            7      1.0      0.0              border_bestfit_best = border_bestfit
   114
   115                                               # Calculate average trajectory
   116         1            1      1.0      0.0      average_trajectory = [np.average(trajectory[border_bestfit_best[i]:border_bestfit_best[i+1]])
   117        12          289     24.1      0.0                            for i in range(0, len(border_bestfit_best) - 1)]
   118
   119                                               # Plot
   120         1      2209171 2209171.0     32.7      plot(trajectory, border_bestfit_best, average_trajectory)
   121
   122                                               # Calculate step size
   123         1            2      2.0      0.0      step_size = [average_trajectory[i+1] - average_trajectory[i]
   124        11           18      1.6      0.0                   for i in range(0, len(average_trajectory)-1)]
   125
   126         1            0      0.0      0.0      return step_size

--The time taken for each line is displayed --62% of the chi2 function's calculation time is used to calculate the calculate_border_bestfit function --This time, apply the calculate_border_bestfit function to the line profiler. ――By repeating this work, it becomes clear which part of the calculation is wasting time.

Recommended Posts

Python script profiling
Script python file
python script skeleton
Import python script
Python profiling basics
DynamoDB Script Memo (Python)
Python
POST json with Python3 script
Bitcoin price monitor python script
Run illustrator script from python
Python script for ldapsearch base64 decode
Get the script path in Python
[python] Copy script to generate copy log
Execute Python Script during CodeSys # RunTime
AWS Layer Creation Script for python
Launch a Python script as a service
Write a batch script with Python3.5 ~
[python] Script that (should) update pwsh
kafka python
Python basics ⑤
python + lottery 6
Python Summary
Built-in python
Python technique
Studying python
Python 2.7 Countdown
Python memorandum
python tips
python function ①
Python basics
Python memo
ufo-> python (3)
Execute Python script with cron of TS-220
install python
Python Singleton
Python basics ④
Python Memorandum 2
python memo
Python Jinja2
Python increment
Pin current directory to script directory in Python
atCoder 173 Python
[Python] function
Python installation
python tips
Sample script to trap signals in Python
Installing Python 3.4.3.
Try python
Python memo
Python iterative
Driver script for parametrically calculating Python scripts
Python algorithm
Python2 + word2vec
[Python] Variables
Python functions
Python sys.intern ()
Python tutorial
Python decimals
python underscore
Python summary
Start python