Building faster python code with line_profiler

The line_profiler or kernprof is a wonderful and simple tool to profile the timing of your python code.

Install:

pip install line_profiler, or from https://github.com/pyutils/line_profiler

Set up your code:

Put @profile above the functions you want to profile. Here’s a snippet of my code:

# the overall loop
for i, x in enumerate(X):
    splat.e_step(x) # just E step for now

# these functions defined elsewhere
@profile
def e_step(self, x):
    self.pf.predict()
    self.pf.observe(x, self.D, self.eps)
    if self.pf.check_resample():
        self.pf.resample()

The E step for this particular EM algorithm has three functions, and by profiling we can see which ones take the most time.

Run:

In terminal: kernprof -l -v ./<your_script>.py. This will run your script with profiling, the output looks like this:

Timer unit: 1e-06 s

Total time: 20.3182 s
File: /home/pgupta/code/SPLAT/./splat.py
Function: e_step at line 40

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    40                                               @profile
    41                                               def e_step(self, x):
    42      4000   15750909.0   3937.7     77.5          self.pf.predict()
    43      4000    1799683.0    449.9      8.9          self.pf.observe(x, self.D, self.eps)
    44      4000     128085.0     32.0      0.6          if self.pf.check_resample():
    45      1778    2639546.0   1484.6     13.0              self.pf.resample()

Hits is how many times the function was called, time is in microseconds, per hit is average time per hit, and % shows how much of the total time it took. The two important things are the per-hit time and the percent. Here the function self.pf.predict() takes the most in both.

Now what?

For me, it was to run the profiler again with @profile above the definition of pf.predict() to see what in that function is taking so long, and continue iterating until I get to more basic numpy kind of stuff. Then, try to restructure or rewrite the highest per-time or percentage functions there - hopefully this won’t be too difficult. Then run profiler again! The per-time should change and percentages will rebalance, suggesting the next place to optimize; this process should eventually get your code faster.

Happy coding!

🏃💨