Network log capture in Selenium Python - Build an Automation Script

Capture network logs during page load

Preconditions (3)

Step 1: Open Chrome browser with logging enabled for performance

Step 2: Navigate to 'https://example.com'

Step 3: Wait until the page is fully loaded

Step 4: Capture the browser's network logs from performance logs

Step 5: Verify that at least one network request was made to 'https://example.com'

✅ Expected Result: Network logs contain at least one request to 'https://example.com' indicating page resources were loaded

Automation Requirements - Selenium with Python

Assertions Needed:

Assert that network logs are not empty

Assert that at least one network request URL contains 'https://example.com'

Best Practices:

Use ChromeOptions to enable performance logging

Use explicit waits to ensure page load completion

Parse performance logs carefully to extract network events

Avoid hardcoded sleeps; use WebDriverWait

Handle exceptions gracefully

Automated Solution

Selenium Python

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import json

# Setup Chrome options to enable performance logging
chrome_options = Options()
chrome_options.add_experimental_option('w3c', False)  # Disable W3C to access performance logs
chrome_options.set_capability('goog:loggingPrefs', {'performance': 'ALL'})

# Initialize WebDriver
service = Service()
driver = webdriver.Chrome(service=service, options=chrome_options)

try:
    driver.get('https://example.com')

    # Wait until the page's main element is loaded (example: <h1> tag)
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.TAG_NAME, 'h1'))
    )

    # Get performance logs
    logs = driver.get_log('performance')

    # Parse logs to find network requests
    network_requests = []
    for entry in logs:
        message = json.loads(entry['message'])['message']
        if message['method'] == 'Network.requestWillBeSent':
            url = message['params']['request']['url']
            network_requests.append(url)

    # Assertions
    assert len(network_requests) > 0, 'No network requests found in logs'
    assert any('https://example.com' in url for url in network_requests), 'No request to https://example.com found'

finally:
    driver.quit()

This script starts by setting Chrome options to enable performance logging, which is necessary to capture network logs.

We disable W3C mode because ChromeDriver currently requires this to access performance logs.

After launching the browser, it navigates to 'https://example.com' and waits explicitly for an <h1> element to appear, indicating the page loaded.

Then it retrieves the performance logs and parses each log entry to find network requests by checking for the 'Network.requestWillBeSent' event.

It collects all request URLs and asserts that there is at least one network request and that one of them contains 'https://example.com'.

Finally, it closes the browser to clean up.

Common Mistakes - 4 Pitfalls

{'mistake': 'Not enabling performance logging in Chrome options', 'why_bad': 'Without enabling performance logging, network logs cannot be captured, so the test will fail or have no data.', 'correct_approach': "Always set ChromeOptions with 'goog:loggingPrefs' for 'performance' to 'ALL' before starting the driver."}

Using hardcoded sleep instead of explicit waits

{'mistake': 'Parsing logs without checking event types', 'why_bad': 'Performance logs contain many event types; parsing all without filtering leads to incorrect data and false assertions.', 'correct_approach': "Filter logs by 'Network.requestWillBeSent' to get only network request events."}

Not quitting the driver in a finally block

Bonus Challenge

Now add data-driven testing with 3 different URLs: 'https://example.com', 'https://www.wikipedia.org', and 'https://www.python.org'. For each URL, capture network logs and verify requests.

Show Hint

Practice

1. What is the main purpose of capturing network logs in Selenium tests?

easy

A. To see the communication between browser and server during tests

B. To speed up the test execution time

C. To change the browser's UI elements

D. To generate random test data automatically

Solution

1. Step 1: Understand network logs in Selenium

   Network logs capture the details of requests and responses between the browser and server during test execution.

2. Step 2: Identify the main purpose

   This helps testers verify if the correct URLs are called and if responses are successful.

3. Final Answer:

   To see the communication between browser and server during tests -> Option A

4. Quick Check:

   Network logs show browser-server communication [OK]

Hint: Network logs = browser-server communication details [OK]

Common Mistakes:

• Thinking network logs speed up tests
• Confusing UI changes with network logs
• Assuming network logs generate test data

2. Which Chrome option enables performance logging to capture network events in Selenium?

easy

A. "args": ["--disable-extensions"]

B. "goog:loggingPrefs": {"performance": "ALL"}

C. "prefs": {"download.default_directory": "/tmp"}

D. "chromeOptions": {"headless": true}

Solution

1. Step 1: Identify the correct Chrome option for logging

   Performance logging is enabled by setting "goog:loggingPrefs" with "performance" set to "ALL".

2. Step 2: Understand other options

   Other options like headless mode or disabling extensions do not enable network log capture.

3. Final Answer:

   "goog:loggingPrefs": {"performance": "ALL"} -> Option B

4. Quick Check:

   Enable performance logs with goog:loggingPrefs [OK]

Hint: Use goog:loggingPrefs with performance ALL to capture network logs [OK]

Common Mistakes:

• Confusing headless mode with logging
• Using download preferences for network logs
• Disabling extensions instead of enabling logs

3. Given this code snippet, what will be printed?

logs = driver.get_log('performance')
for entry in logs:
    message = json.loads(entry['message'])['message']
    if message['method'] == 'Network.responseReceived':
        print(message['params']['response']['url'])

medium

A. All browser console error messages

B. Only URLs of failed network requests

C. All URLs of network responses received during the test

D. List of all clickable elements on the page

Solution

1. Step 1: Understand the log parsing code

   The code reads performance logs, filters for 'Network.responseReceived' events, and prints the response URLs.

2. Step 2: Identify what is printed

   It prints all URLs for network responses received, regardless of success or failure.

3. Final Answer:

   All URLs of network responses received during the test -> Option C

4. Quick Check:

   Code prints URLs from Network.responseReceived events [OK]

Hint: Network.responseReceived logs contain all response URLs [OK]

Common Mistakes:

• Assuming only failed URLs are printed
• Confusing network logs with console errors
• Thinking it lists page elements

4. What is wrong with this code snippet for capturing network logs?

options = webdriver.ChromeOptions()
options.set_capability('loggingPrefs', {'performance': 'ALL'})
driver = webdriver.Chrome(options=options)
logs = driver.get_log('performance')

medium

A. The capability key should be 'goog:loggingPrefs', not 'loggingPrefs'

B. The ChromeOptions object cannot be used with Chrome driver

C. The get_log method does not exist in Selenium

D. Performance logs cannot be captured in Chrome

Solution

1. Step 1: Check capability key correctness

   The correct key to enable performance logging is 'goog:loggingPrefs', not 'loggingPrefs'.

2. Step 2: Verify other statements

   ChromeOptions can be used with Chrome driver, get_log exists, and performance logs can be captured.

3. Final Answer:

   The capability key should be 'goog:loggingPrefs', not 'loggingPrefs' -> Option A

4. Quick Check:

   Use 'goog:loggingPrefs' to enable performance logs [OK]

Hint: Use 'goog:loggingPrefs' key for performance logging [OK]

Common Mistakes:

• Using wrong capability key
• Thinking ChromeOptions is incompatible
• Believing get_log method is missing

5. How can you verify in Selenium that a specific API call to 'https://api.example.com/data' returned status code 200 using network logs?

hard

A. Use driver.execute_script to intercept network calls directly

B. Use driver.get_log('browser') and search for the URL string

C. Check page source for the URL and assume status 200 if found

D. Parse performance logs, find 'Network.responseReceived' events, check if response URL matches and status is 200

Solution

1. Step 1: Capture and parse performance logs

   Enable performance logging and get logs with driver.get_log('performance').

2. Step 2: Filter 'Network.responseReceived' events and check URL and status

   Look for events where response URL matches 'https://api.example.com/data' and verify status code is 200.

3. Final Answer:

   Parse performance logs, find 'Network.responseReceived' events, check if response URL matches and status is 200 -> Option D

4. Quick Check:

   Check responseReceived events for URL and status 200 [OK]

Hint: Filter Network.responseReceived logs by URL and status code [OK]

Common Mistakes:

• Using browser logs instead of performance logs
• Assuming URL presence means status 200
• Trying to intercept network calls with execute_script