Handling large files efficiently in Python - Mini Project: Build & Apply

Handling large files efficiently

📖 Scenario: Imagine you have a very large text file with millions of lines. You want to count how many lines contain the word "error" without loading the entire file into memory at once.

🎯 Goal: Build a Python program that reads a large file line by line, counts lines containing the word "error", and prints the total count.

📋 What You'll Learn

Create a variable for the file path with the exact name file_path and value 'large_log.txt'.

Create a variable called keyword and set it to the string 'error'.

Use a with open(file_path, 'r') block to read the file line by line.

Use a for loop with variable line to iterate over the file object.

Inside the loop, check if keyword is in line and count such lines in a variable called count.

Print the final count using print(count).

💡 Why This Matters

🌍 Real World

Large log files from servers or applications can be huge. Reading them line by line helps find errors or important info without crashing your computer.

💼 Career

Many jobs in data analysis, system administration, and software development require processing large files efficiently to troubleshoot or analyze data.

1

Set up the file path variable

Create a variable called file_path and set it to the string 'large_log.txt'.

Python

# Create a variable called file_path with value 'large_log.txt'
# Your code here

Need a hint?

Use = to assign the string 'large_log.txt' to the variable file_path.

2

Create the keyword variable

Create a variable called keyword and set it to the string 'error'.

Python

file_path = 'large_log.txt'
# Create a variable called keyword with value 'error'
# Your code here

Need a hint?

Assign the string 'error' to the variable keyword.

3

Read the file and count lines with the keyword

Create a variable called count and set it to 0. Then use with open(file_path, 'r') to open the file. Inside this block, use a for loop with variable line to read each line. Inside the loop, check if keyword is in line. If yes, increase count by 1.

Python

file_path = 'large_log.txt'
keyword = 'error'
# Initialize count to 0
# Open the file and read line by line
# For each line, check if keyword is in line and update count
# Your code here

Need a hint?

Remember to initialize count before the with block. Use for line in file to read lines one by one.

4

Print the total count

Write a print(count) statement to display the total number of lines containing the keyword.

Python

file_path = 'large_log.txt'
keyword = 'error'
count = 0
with open(file_path, 'r') as file:
    for line in file:
        if keyword in line:
            count += 1
# Print the count
# Your code here

Need a hint?

Use print(count) to show the final number.

Since the file is not provided, the count will be 0 when tested.