Profiling with line_profiler in Data Analysis Python - Step-by-Step Execution

Concept Flow - Profiling with line_profiler

Write Python function

↓

Add @profile decorator

↓

Run script with line_profiler

↓

line_profiler measures time per line

↓

Output shows time spent on each line

↓

Analyze output to find slow lines

↓

Optimize slow lines

↓

Repeat profiling to check improvements

This flow shows how to add profiling to a Python function, run line_profiler to measure time per line, and use the results to optimize code.

Execution Sample

Data Analysis Python

from time import sleep

@profile
def slow_function():
    sleep(0.1)
    total = 0
    for i in range(3):
        total += i
    return total

This code defines a simple function with a sleep and a loop, decorated for line_profiler to measure time spent on each line.

Execution Table

Step	Line Number	Code Line	Time Spent (s)	Hits	Cumulative Time (s)
1	3	@profile	0.0000	1	0.0000
2	4	def slow_function():	0.0000	1	0.0000
3	5	sleep(0.1)	0.1001	1	0.1001
4	6	total = 0	0.0000	1	0.1001
5	7	for i in range(3):	0.0000	1	0.1001
6	8	total += i	0.0000	3	0.1001
7	9	return total	0.0000	1	0.1001
8	-	End of function	-	-	-

💡 Function completes after line 9; total time mostly spent on sleep(0.1) line.

Variable Tracker

Variable	Start	After sleep(0.1)	After loop iteration 1	After loop iteration 2	After loop iteration 3	Final
total	undefined	0	0	1	3	3
i	undefined	undefined	0	1	2	2

Key Moments - 3 Insights

Why does the sleep(0.1) line take most of the time?

Because sleep pauses the program for 0.1 seconds, line_profiler shows this as the biggest time consumer (see execution_table row 3).

Why does 'total += i' show zero time even though it runs 3 times?

Each addition is very fast, so total time is too small to show; line_profiler sums time but it rounds to zero here (see execution_table row 6).

What does 'Hits' mean in the profiling output?

'Hits' counts how many times a line runs; for example, the loop line runs once, but the addition line runs 3 times (see execution_table rows 5 and 6).

Visual Quiz - 3 Questions

Look at the execution table, which line uses the most time?

A Line 6: total = 0

B Line 5: sleep(0.1)

C Line 8: total += i

D Line 9: return total

💡 Hint

Check the 'Time Spent (s)' column in execution_table row 3.

At which step does the variable 'total' first get a value?

A After loop iteration 1

B After loop iteration 3

C After sleep(0.1)

D At start

💡 Hint

See variable_tracker row for 'total' after 'After sleep(0.1)'.

If the loop ran 5 times instead of 3, what changes in the execution table?

A 'Hits' for line 8 would be 5 instead of 3

B 'Time Spent' for sleep(0.1) would increase

C 'Hits' for line 7 would be 5

D No changes in the table

💡 Hint

Look at how 'Hits' counts executions for line 8 in execution_table row 6.

Concept Snapshot

Profiling with line_profiler:
- Add @profile decorator to functions
- Run script with 'kernprof -l script.py'
- Use 'python -m line_profiler script.py.lprof' to see line times
- Shows time spent per line and hit counts
- Helps find slow lines to optimize

Full Transcript

Profiling with line_profiler helps find which lines in a Python function take the most time. You add @profile above the function, run the script with kernprof, then view detailed timing per line. The example function sleeps for 0.1 seconds and sums numbers in a loop. The profiler shows most time is spent on sleep. Variables like 'total' change as the loop runs. Hits count how many times each line runs. This helps focus optimization on slow parts.

Practice

1. What is the main purpose of using line_profiler in Python data analysis?

easy

A. To measure the time each line of a function takes to execute

B. To visualize data distributions with graphs

C. To clean missing data automatically

D. To convert data types in a DataFrame

Solution

1. Step 1: Understand profiling purpose

   line_profiler is designed to measure execution time per line in functions.

2. Step 2: Compare options to profiling goal

   Options A, B, and D relate to data cleaning, visualization, and type conversion, not profiling.

3. Final Answer:

   To measure the time each line of a function takes to execute -> Option A

4. Quick Check:

   Profiling = measuring line execution time [OK]

Hint: Profiling finds slow lines by timing code line-by-line [OK]

Common Mistakes:

• Confusing profiling with data visualization
• Thinking profiling cleans or transforms data
• Assuming profiling measures overall script time only

2. Which of the following is the correct way to create a LineProfiler object and add a function my_func for profiling?

easy

A. profiler = LineProfiler(my_func); profiler.run()

B. profiler = LineProfiler(); profiler.add_function('my_func')

C. profiler = LineProfiler(); profiler.add(my_func)

D. profiler = LineProfiler(); profiler.add_function(my_func)

Solution

1. Step 1: Recall LineProfiler usage

   The correct method to add a function is add_function() with the function object.

2. Step 2: Check each option

   profiler = LineProfiler(); profiler.add_function(my_func) uses add_function(my_func) correctly. profiler = LineProfiler(my_func); profiler.run() wrongly passes function to constructor. profiler = LineProfiler(); profiler.add(my_func) uses non-existent add(). profiler = LineProfiler(); profiler.add_function('my_func') passes function name as string, which is incorrect.

3. Final Answer:

   profiler = LineProfiler(); profiler.add_function(my_func) -> Option D

4. Quick Check:

   Add function with add_function(func) [OK]

Hint: Use add_function(func) with function object, not string [OK]

Common Mistakes:

• Passing function name as string instead of function object
• Using wrong method like add() instead of add_function()
• Passing function to LineProfiler constructor

3. Given the code below, what will be printed by profiler.print_stats()?

from line_profiler import LineProfiler

def slow_func():
    total = 0
    for i in range(3):
        total += i
    return total

profiler = LineProfiler()
profiler.add_function(slow_func)
profiler.enable()
slow_func()
profiler.disable()
profiler.print_stats()

medium

A. No output because profiler.enable() was called before add_function()

B. SyntaxError because LineProfiler requires run() method

C. Detailed timing per line of slow_func showing time spent on the loop lines

D. Only total execution time without line-by-line details

Solution

1. Step 1: Understand profiling flow

   Profiler is created, function added, enabled, function called, then disabled and stats printed.

2. Step 2: Analyze output of print_stats()

   print_stats() shows detailed line-by-line timing for added functions, here slow_func, including loop lines.

3. Final Answer:

   Detailed timing per line of slow_func showing time spent on the loop lines -> Option C

4. Quick Check:

   print_stats() = detailed line timing [OK]

Hint: print_stats() shows line-by-line timing for added functions [OK]

Common Mistakes:

• Thinking print_stats() shows only total time
• Believing enable() must be after add_function() to work
• Confusing run() method requirement

4. Identify the error in the following code snippet that uses line_profiler:

from line_profiler import LineProfiler

def compute():
    s = 0
    for i in range(5):
        s += i
    return s

profiler = LineProfiler()
profiler.add_function('compute')
profiler.enable()
compute()
profiler.disable()
profiler.print_stats()

medium

A. Passing function name as string to add_function instead of function object

B. Missing call to profiler.run() to execute the function

C. Not importing line_profiler correctly

D. For loop range is invalid syntax

Solution

1. Step 1: Check add_function argument

   add_function requires the function object, not its name as a string.

2. Step 2: Identify error cause

   Passing 'compute' as string causes profiler not to track the function properly.

3. Final Answer:

   Passing function name as string to add_function instead of function object -> Option A

4. Quick Check:

   add_function(func) needs function object [OK]

Hint: Always pass function object, not string, to add_function() [OK]

Common Mistakes:

• Passing function name as string
• Thinking profiler.run() is mandatory
• Assuming import error without checking code

5. You want to profile two functions func_a and func_b in your data analysis script. You also want to compare which function is slower line-by-line. Which of the following code snippets correctly profiles both functions and prints their stats separately?

hard

A. profiler = LineProfiler()
profiler.add_function(func_a)
profiler.add_function(func_b)
profiler.enable()
func_a()
func_b()
profiler.disable()
profiler.print_stats()

B. profiler_a = LineProfiler()
profiler_a.add_function(func_a)
profiler_b = LineProfiler()
profiler_b.add_function(func_b)
profiler_a.enable()
func_a()
profiler_a.disable()
profiler_b.enable()
func_b()
profiler_b.disable()
profiler_a.print_stats()
profiler_b.print_stats()

C. profiler = LineProfiler(func_a, func_b)
profiler.run()
profiler.print_stats()

D. profiler = LineProfiler()
profiler.add_function('func_a')
profiler.add_function('func_b')
profiler.enable()
func_a()
func_b()
profiler.disable()
profiler.print_stats()

Solution

1. Step 1: Understand profiling multiple functions separately

   To compare functions separately, create separate LineProfiler instances for each.

2. Step 2: Analyze each option

   profiler = LineProfiler() profiler.add_function(func_a) profiler.add_function(func_b) profiler.enable() func_a() func_b() profiler.disable() profiler.print_stats() profiles both with one profiler but prints combined stats, not separate. profiler_a = LineProfiler() profiler_a.add_function(func_a) profiler_b = LineProfiler() profiler_b.add_function(func_b) profiler_a.enable() func_a() profiler_a.disable() profiler_b.enable() func_b() profiler_b.disable() profiler_a.print_stats() profiler_b.print_stats() profiles separately with two profilers and prints stats separately. profiler = LineProfiler(func_a, func_b) profiler.run() profiler.print_stats() wrongly passes functions to constructor and calls run() without arguments. profiler = LineProfiler() profiler.add_function('func_a') profiler.add_function('func_b') profiler.enable() func_a() func_b() profiler.disable() profiler.print_stats() passes function names as strings, which is incorrect.

3. Final Answer:

   profiler_a = LineProfiler() profiler_a.add_function(func_a) profiler_b = LineProfiler() profiler_b.add_function(func_b) profiler_a.enable() func_a() profiler_a.disable() profiler_b.enable() func_b() profiler_b.disable() profiler_a.print_stats() profiler_b.print_stats() -> Option B

4. Quick Check:

   Separate profilers for separate stats [OK]

Hint: Use separate LineProfiler instances to profile functions separately [OK]

Common Mistakes:

• Passing function names as strings
• Using one profiler for separate stats
• Misusing LineProfiler constructor arguments