Network log capture in Selenium Python - Deep Dive

Overview - Network log capture

What is it?

Network log capture is the process of recording all the network requests and responses that happen when a web page loads or a web application runs. It helps testers see what data is sent and received between the browser and servers. This is useful to check if the right resources load, if APIs respond correctly, and if there are any errors in communication.

Why it matters

Without network log capture, testers would only see what appears on the screen, missing hidden problems like slow responses, failed requests, or wrong data sent behind the scenes. This could cause bugs to go unnoticed, leading to poor user experience or security issues. Capturing network logs helps find and fix these problems early.

Where it fits

Before learning network log capture, you should understand basic Selenium automation and how browsers work. After this, you can learn advanced debugging, performance testing, and security testing techniques that use network data.

Mental Model

Core Idea

Network log capture records every message between your browser and the internet so you can check what really happens behind the scenes.

Think of it like...

It's like listening to all the phone calls between a customer and a store to understand what was asked, what was promised, and if anything went wrong.

┌─────────────────────────────┐
│        Browser (Client)     │
│  ┌───────────────────────┐  │
│  │ Network Log Capture    │  │
│  └──────────┬────────────┘  │
└─────────────┼───────────────┘
              │ HTTP Requests
              ▼
       ┌───────────────┐
       │   Server/API   │
       └───────────────┘
              ▲
              │ HTTP Responses
              └───────────────

Build-Up - 6 Steps

1

Foundation

Understanding HTTP Requests and Responses

Concept: Learn what HTTP requests and responses are, as they are the core of network communication in web testing.

When you visit a website, your browser sends a request to a server asking for data like HTML, images, or API responses. The server replies with a response containing the requested data or an error message. These messages follow the HTTP protocol and include details like URLs, headers, and status codes.

Result

You understand that every web page load involves many small conversations between browser and server.

Knowing how HTTP works is essential because network logs are just records of these conversations.

2

Foundation

Basics of Selenium WebDriver Automation

Concept: Learn how Selenium controls a browser to automate web actions.

Selenium WebDriver lets you write code to open browsers, click buttons, fill forms, and check page content automatically. It interacts with the browser like a user would, but faster and repeatably.

Result

You can write simple scripts to open a page and interact with it.

Understanding Selenium basics is necessary before adding network log capture, which extends what Selenium can observe.

3

Intermediate

Capturing Network Logs with Browser DevTools Protocol

🤔Before reading on: do you think Selenium alone can capture network logs, or do you need extra tools? Commit to your answer.

Concept: Learn how Selenium can use the browser's built-in debugging tools to record network traffic.

Modern browsers like Chrome support the DevTools Protocol, which lets programs listen to network events. Selenium can connect to this protocol to capture logs. In Python, you set up a special driver option to enable logging, then read the captured data during or after tests.

Result

You can collect detailed network logs including URLs, headers, and response codes while running Selenium tests.

Knowing that Selenium uses browser debugging features explains why network capture is possible without external proxies.

4

Intermediate

Filtering and Analyzing Network Logs

🤔Before reading on: do you think all network logs are equally useful, or should you filter them? Commit to your answer.

Concept: Learn how to select relevant network entries and check their details for testing purposes.

Network logs include many requests like images, scripts, and API calls. You often want to focus on specific URLs or status codes. Using Python, you can filter logs by URL patterns or HTTP methods, then assert that responses have expected status codes or content.

Result

You can write tests that verify important network requests succeed and contain correct data.

Filtering logs helps focus on what matters and avoid noise, making tests more reliable and meaningful.

5

Advanced

Handling Asynchronous Network Requests in Tests

🤔Before reading on: do you think network requests always finish immediately, or can they happen after page load? Commit to your answer.

Concept: Learn how to wait for network activity that happens after initial page load, like AJAX calls.

Many web apps load data asynchronously after the page appears. To capture these, your test must wait for network events or specific responses before asserting. You can use Selenium waits combined with checking network logs repeatedly until expected requests appear.

Result

Your tests can reliably verify dynamic content loaded after page load.

Understanding asynchronous behavior prevents flaky tests that miss late network requests.

6

Expert

Integrating Network Log Capture with CI/CD Pipelines

🤔Before reading on: do you think network log capture is only useful during local testing, or also in automated pipelines? Commit to your answer.

Concept: Learn how to use network logs in continuous integration to catch regressions automatically.

In professional environments, tests run automatically on servers after code changes. Capturing network logs here helps detect backend failures or performance issues early. You can save logs as artifacts, analyze them with scripts, and fail builds if critical requests fail or slow down.

Result

Your team gains faster feedback on network-related bugs and improves software quality.

Using network logs in CI/CD turns manual debugging into automated quality checks, saving time and reducing errors.

Under the Hood

Network log capture works by connecting Selenium to the browser's debugging interface, which listens to all HTTP traffic. When the browser sends a request or receives a response, it emits events with details like URL, headers, and status. Selenium collects these events in real time or after test steps, storing them as structured data for analysis.

Why designed this way?

Browsers expose debugging protocols to help developers inspect and fix issues. Selenium leverages this existing interface instead of intercepting traffic externally, which avoids complex proxy setups and keeps tests faster and more stable.

┌───────────────┐
│ Selenium Test │
└──────┬────────┘
       │ Uses DevTools Protocol
       ▼
┌─────────────────────┐
│ Browser Debugging API│
│  (Network Events)    │
└─────────┬───────────┘
          │ Captures HTTP traffic
          ▼
    ┌─────────────┐
    │ Network Log │
    └─────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do you think network logs capture all browser activity including JavaScript errors? Commit yes or no.

Common Belief: Network log capture records everything happening in the browser, including JavaScript errors and console messages.

Reality: Network logs only capture HTTP requests and responses, not JavaScript errors or console logs. Those require separate capture methods.

Why it matters: Confusing network logs with full browser logs can cause testers to miss JavaScript errors, leading to incomplete debugging.

Quick: Do you think network logs can be captured by Selenium without any special setup? Commit yes or no.

Common Belief: Selenium automatically captures network logs without extra configuration.

Reality: Selenium needs explicit setup to enable network logging via browser options or DevTools commands.

Why it matters: Assuming automatic capture leads to empty logs and wasted debugging time.

Quick: Do you think capturing network logs slows down tests significantly? Commit yes or no.

Common Belief: Capturing network logs always makes tests much slower and less reliable.

Reality: While there is some overhead, modern browsers and Selenium handle network logging efficiently, and careful filtering minimizes impact.

Why it matters: Avoiding network capture due to fear of slowness can miss critical bugs.

Quick: Do you think network logs show encrypted HTTPS content in readable form? Commit yes or no.

Common Belief: Network logs show full content of HTTPS requests and responses in plain text.

Reality: Network logs show metadata and headers but not decrypted HTTPS body content unless the browser explicitly exposes it.

Why it matters: Expecting full HTTPS content can lead to confusion when sensitive data appears missing.

Expert Zone

1

Network logs can include timing details that help diagnose slow requests, but interpreting these requires understanding browser internals and network layers.

2

Some browsers limit the size or detail of network logs for performance reasons, so tests must handle partial data gracefully.

3

Combining network logs with browser console logs and performance metrics gives a fuller picture but requires careful synchronization.

When NOT to use

Network log capture is not suitable when testing purely UI layout or visual appearance where network data is irrelevant. For security testing of encrypted data, specialized tools like proxies with SSL interception are better.

Production Patterns

In real-world testing, network log capture is integrated with API contract tests, performance benchmarks, and error monitoring. Teams automate log analysis to detect regressions and use logs to reproduce hard-to-find bugs.

Connections

API Testing

Network log capture builds on API testing by verifying actual HTTP traffic during UI tests.

Understanding network logs helps testers confirm that UI actions trigger correct API calls, bridging UI and backend testing.

Performance Monitoring

Network logs provide raw data that performance monitoring tools analyze to find bottlenecks.

Knowing how to capture and interpret network logs aids in diagnosing slow page loads and optimizing user experience.

Telecommunications Protocol Analysis

Both involve capturing and analyzing message exchanges to diagnose problems.

Skills in network log capture transfer to understanding how communication protocols work in other fields like telecom or networking.

Common Pitfalls

#1 Trying to capture network logs without enabling browser logging options.

Wrong approach:

from selenium import webdriver
options = webdriver.ChromeOptions()
driver = webdriver.Chrome(options=options)
# No logging enabled

Correct approach:

from selenium import webdriver
options = webdriver.ChromeOptions()
options.set_capability('goog:loggingPrefs', {'performance': 'ALL'})
driver = webdriver.Chrome(options=options)

Root cause:

Not knowing that network logging must be explicitly enabled in browser options.

#2 Assuming all network requests finish before page load event.

Wrong approach:

driver.get('https://example.com')
logs = driver.get_log('performance')
# Immediately analyze logs without waiting

Correct approach:

driver.get('https://example.com')
from selenium.webdriver.support.ui import WebDriverWait
WebDriverWait(driver, 10).until(lambda d: check_network_condition())
logs = driver.get_log('performance')

Root cause:

Misunderstanding asynchronous network behavior and timing.

#3 Parsing raw network logs as plain JSON without filtering or decoding.

Wrong approach:

logs = driver.get_log('performance')
for entry in logs:
print(entry['message']) # Raw string, hard to read

Correct approach:

import json
logs = driver.get_log('performance')
for entry in logs:
message = json.loads(entry['message'])
# Extract useful fields from message

Root cause:

Not knowing the log format requires parsing before use.

Key Takeaways

Network log capture records all HTTP communication between browser and server during tests, revealing hidden issues.

It requires enabling browser debugging options and using Selenium's access to DevTools Protocol.

Filtering and waiting for asynchronous requests are key to meaningful and reliable network assertions.

Network logs complement UI tests by verifying backend interactions and performance.

Expert use includes integrating logs into automated pipelines for early bug detection and quality assurance.

Practice

1. What is the main purpose of capturing network logs in Selenium tests?

easy

A. To see the communication between browser and server during tests

B. To speed up the test execution time

C. To change the browser's UI elements

D. To generate random test data automatically

Solution

1. Step 1: Understand network logs in Selenium

   Network logs capture the details of requests and responses between the browser and server during test execution.

2. Step 2: Identify the main purpose

   This helps testers verify if the correct URLs are called and if responses are successful.

3. Final Answer:

   To see the communication between browser and server during tests -> Option A

4. Quick Check:

   Network logs show browser-server communication [OK]

Hint: Network logs = browser-server communication details [OK]

Common Mistakes:

• Thinking network logs speed up tests
• Confusing UI changes with network logs
• Assuming network logs generate test data

2. Which Chrome option enables performance logging to capture network events in Selenium?

easy

A. "args": ["--disable-extensions"]

B. "goog:loggingPrefs": {"performance": "ALL"}

C. "prefs": {"download.default_directory": "/tmp"}

D. "chromeOptions": {"headless": true}

Solution

1. Step 1: Identify the correct Chrome option for logging

   Performance logging is enabled by setting "goog:loggingPrefs" with "performance" set to "ALL".

2. Step 2: Understand other options

   Other options like headless mode or disabling extensions do not enable network log capture.

3. Final Answer:

   "goog:loggingPrefs": {"performance": "ALL"} -> Option B

4. Quick Check:

   Enable performance logs with goog:loggingPrefs [OK]

Hint: Use goog:loggingPrefs with performance ALL to capture network logs [OK]

Common Mistakes:

• Confusing headless mode with logging
• Using download preferences for network logs
• Disabling extensions instead of enabling logs

3. Given this code snippet, what will be printed?

logs = driver.get_log('performance')
for entry in logs:
    message = json.loads(entry['message'])['message']
    if message['method'] == 'Network.responseReceived':
        print(message['params']['response']['url'])

medium

A. All browser console error messages

B. Only URLs of failed network requests

C. All URLs of network responses received during the test

D. List of all clickable elements on the page

Solution

1. Step 1: Understand the log parsing code

   The code reads performance logs, filters for 'Network.responseReceived' events, and prints the response URLs.

2. Step 2: Identify what is printed

   It prints all URLs for network responses received, regardless of success or failure.

3. Final Answer:

   All URLs of network responses received during the test -> Option C

4. Quick Check:

   Code prints URLs from Network.responseReceived events [OK]

Hint: Network.responseReceived logs contain all response URLs [OK]

Common Mistakes:

• Assuming only failed URLs are printed
• Confusing network logs with console errors
• Thinking it lists page elements

4. What is wrong with this code snippet for capturing network logs?

options = webdriver.ChromeOptions()
options.set_capability('loggingPrefs', {'performance': 'ALL'})
driver = webdriver.Chrome(options=options)
logs = driver.get_log('performance')

medium

A. The capability key should be 'goog:loggingPrefs', not 'loggingPrefs'

B. The ChromeOptions object cannot be used with Chrome driver

C. The get_log method does not exist in Selenium

D. Performance logs cannot be captured in Chrome

Solution

1. Step 1: Check capability key correctness

   The correct key to enable performance logging is 'goog:loggingPrefs', not 'loggingPrefs'.

2. Step 2: Verify other statements

   ChromeOptions can be used with Chrome driver, get_log exists, and performance logs can be captured.

3. Final Answer:

   The capability key should be 'goog:loggingPrefs', not 'loggingPrefs' -> Option A

4. Quick Check:

   Use 'goog:loggingPrefs' to enable performance logs [OK]

Hint: Use 'goog:loggingPrefs' key for performance logging [OK]

Common Mistakes:

• Using wrong capability key
• Thinking ChromeOptions is incompatible
• Believing get_log method is missing

5. How can you verify in Selenium that a specific API call to 'https://api.example.com/data' returned status code 200 using network logs?

hard

A. Use driver.execute_script to intercept network calls directly

B. Use driver.get_log('browser') and search for the URL string

C. Check page source for the URL and assume status 200 if found

D. Parse performance logs, find 'Network.responseReceived' events, check if response URL matches and status is 200

Solution

1. Step 1: Capture and parse performance logs

   Enable performance logging and get logs with driver.get_log('performance').

2. Step 2: Filter 'Network.responseReceived' events and check URL and status

   Look for events where response URL matches 'https://api.example.com/data' and verify status code is 200.

3. Final Answer:

   Parse performance logs, find 'Network.responseReceived' events, check if response URL matches and status is 200 -> Option D

4. Quick Check:

   Check responseReceived events for URL and status 200 [OK]

Hint: Filter Network.responseReceived logs by URL and status code [OK]

Common Mistakes:

• Using browser logs instead of performance logs
• Assuming URL presence means status 200
• Trying to intercept network calls with execute_script