Network log capture in Selenium Python - Framework Patterns

Framework Mode - Network log capture

Folder Structure

network_log_capture_project/
├── tests/
│   ├── test_login.py
│   └── test_api_calls.py
├── pages/
│   ├── login_page.py
│   └── dashboard_page.py
├── utils/
│   ├── network_logger.py
│   └── helpers.py
├── config/
│   ├── config.yaml
│   └── credentials.yaml
├── conftest.py
└── requirements.txt

Test Framework Layers

• Driver Layer: Manages Selenium WebDriver setup and teardown, including enabling network logging capabilities.
• Page Objects: Encapsulate UI elements and actions for each page (e.g., LoginPage, DashboardPage).
• Tests: Test cases that use page objects and utilities to perform actions and assertions.
• Utilities: Helper modules like network_logger.py to capture and process network logs.
• Configuration: YAML files for environment settings, browser options, and credentials.
• Fixtures (conftest.py): Pytest fixtures to initialize WebDriver with network logging enabled and provide reusable setup.

Configuration Patterns

• Environment Settings: Use config/config.yaml to define URLs, timeouts, and environment-specific variables.
• Browser Options: Configure Chrome or Firefox to enable performance logging for network capture via Selenium capabilities.
• Credentials: Store sensitive data like usernames and passwords securely in config/credentials.yaml, avoid hardcoding.
• Dynamic Loading: Use conftest.py to load config files and pass settings to tests and utilities.

Test Reporting and CI/CD Integration

• Test Reports: Use Pytest with plugins like pytest-html to generate readable reports including test results and captured network logs summaries.
• Network Logs Storage: Save network logs as JSON files per test run for debugging and audit purposes.
• CI/CD Integration: Integrate tests in pipelines (e.g., GitHub Actions, Jenkins) to run on pull requests and merges, collecting logs and reports as artifacts.
• Alerts: Configure notifications on test failures with links to logs and reports for quick triage.

Best Practices

1. Use Browser Performance Logging: Enable browser performance logs via Selenium capabilities to capture network traffic reliably.
2. Separate Concerns: Keep network log capture logic in utility modules, not mixed with test or page object code.
3. Manage Sensitive Data Securely: Never hardcode credentials; use config files with proper access control.
4. Explicit Waits: Use explicit waits to ensure network requests complete before capturing logs.
5. Clean Up Resources: Properly close WebDriver sessions to avoid resource leaks and stale logs.

Self Check

Where in this folder structure would you add a new utility to parse and filter network logs for specific API endpoints?

Key Result

Enable browser performance logging via Selenium capabilities and organize network log capture in utility modules for clean, maintainable test automation.

Practice

1. What is the main purpose of capturing network logs in Selenium tests?

easy

A. To see the communication between browser and server during tests

B. To speed up the test execution time

C. To change the browser's UI elements

D. To generate random test data automatically

Solution

1. Step 1: Understand network logs in Selenium

   Network logs capture the details of requests and responses between the browser and server during test execution.

2. Step 2: Identify the main purpose

   This helps testers verify if the correct URLs are called and if responses are successful.

3. Final Answer:

   To see the communication between browser and server during tests -> Option A

4. Quick Check:

   Network logs show browser-server communication [OK]

Hint: Network logs = browser-server communication details [OK]

Common Mistakes:

• Thinking network logs speed up tests
• Confusing UI changes with network logs
• Assuming network logs generate test data

2. Which Chrome option enables performance logging to capture network events in Selenium?

easy

A. "args": ["--disable-extensions"]

B. "goog:loggingPrefs": {"performance": "ALL"}

C. "prefs": {"download.default_directory": "/tmp"}

D. "chromeOptions": {"headless": true}

Solution

1. Step 1: Identify the correct Chrome option for logging

   Performance logging is enabled by setting "goog:loggingPrefs" with "performance" set to "ALL".

2. Step 2: Understand other options

   Other options like headless mode or disabling extensions do not enable network log capture.

3. Final Answer:

   "goog:loggingPrefs": {"performance": "ALL"} -> Option B

4. Quick Check:

   Enable performance logs with goog:loggingPrefs [OK]

Hint: Use goog:loggingPrefs with performance ALL to capture network logs [OK]

Common Mistakes:

• Confusing headless mode with logging
• Using download preferences for network logs
• Disabling extensions instead of enabling logs

3. Given this code snippet, what will be printed?

logs = driver.get_log('performance')
for entry in logs:
    message = json.loads(entry['message'])['message']
    if message['method'] == 'Network.responseReceived':
        print(message['params']['response']['url'])

medium

A. All browser console error messages

B. Only URLs of failed network requests

C. All URLs of network responses received during the test

D. List of all clickable elements on the page

Solution

1. Step 1: Understand the log parsing code

   The code reads performance logs, filters for 'Network.responseReceived' events, and prints the response URLs.

2. Step 2: Identify what is printed

   It prints all URLs for network responses received, regardless of success or failure.

3. Final Answer:

   All URLs of network responses received during the test -> Option C

4. Quick Check:

   Code prints URLs from Network.responseReceived events [OK]

Hint: Network.responseReceived logs contain all response URLs [OK]

Common Mistakes:

• Assuming only failed URLs are printed
• Confusing network logs with console errors
• Thinking it lists page elements

4. What is wrong with this code snippet for capturing network logs?

options = webdriver.ChromeOptions()
options.set_capability('loggingPrefs', {'performance': 'ALL'})
driver = webdriver.Chrome(options=options)
logs = driver.get_log('performance')

medium

A. The capability key should be 'goog:loggingPrefs', not 'loggingPrefs'

B. The ChromeOptions object cannot be used with Chrome driver

C. The get_log method does not exist in Selenium

D. Performance logs cannot be captured in Chrome

Solution

1. Step 1: Check capability key correctness

   The correct key to enable performance logging is 'goog:loggingPrefs', not 'loggingPrefs'.

2. Step 2: Verify other statements

   ChromeOptions can be used with Chrome driver, get_log exists, and performance logs can be captured.

3. Final Answer:

   The capability key should be 'goog:loggingPrefs', not 'loggingPrefs' -> Option A

4. Quick Check:

   Use 'goog:loggingPrefs' to enable performance logs [OK]

Hint: Use 'goog:loggingPrefs' key for performance logging [OK]

Common Mistakes:

• Using wrong capability key
• Thinking ChromeOptions is incompatible
• Believing get_log method is missing

5. How can you verify in Selenium that a specific API call to 'https://api.example.com/data' returned status code 200 using network logs?

hard

A. Use driver.execute_script to intercept network calls directly

B. Use driver.get_log('browser') and search for the URL string

C. Check page source for the URL and assume status 200 if found

D. Parse performance logs, find 'Network.responseReceived' events, check if response URL matches and status is 200

Solution

1. Step 1: Capture and parse performance logs

   Enable performance logging and get logs with driver.get_log('performance').

2. Step 2: Filter 'Network.responseReceived' events and check URL and status

   Look for events where response URL matches 'https://api.example.com/data' and verify status code is 200.

3. Final Answer:

   Parse performance logs, find 'Network.responseReceived' events, check if response URL matches and status is 200 -> Option D

4. Quick Check:

   Check responseReceived events for URL and status 200 [OK]

Hint: Filter Network.responseReceived logs by URL and status code [OK]

Common Mistakes:

• Using browser logs instead of performance logs
• Assuming URL presence means status 200
• Trying to intercept network calls with execute_script