Continuous Integration testing in Testing Fundamentals - Build an Automation Script

Verify that the CI pipeline runs automated tests on each code commit

Preconditions (3)

Step 1: Commit a small code change to the repository

Step 2: Push the commit to the remote repository branch monitored by the CI tool

Step 3: Observe the CI pipeline execution triggered by the commit

Step 4: Wait for the automated tests to run as part of the pipeline

Step 5: Check the test results reported by the CI tool

✅ Expected Result: The CI pipeline triggers automatically on the commit, runs all automated tests, and reports a pass status if all tests succeed or fail status if any test fails

Automation Requirements - No specific code framework; focus on verifying CI pipeline behavior via logs or API

Assertions Needed:

CI pipeline triggered after commit

Automated tests executed

Test results reported as pass or fail

Best Practices:

Use CI tool APIs or logs to verify pipeline execution

Automate commit and push steps using scripts if possible

Check test result summaries rather than raw logs for clarity

Automated Solution

Testing Fundamentals

import requests
import time

def trigger_commit_and_check_ci(repo_api_url, ci_status_api_url, commit_payload, headers):
    # Step 1: Commit code via API (simulate commit)
    commit_response = requests.post(repo_api_url, json=commit_payload, headers=headers)
    if commit_response.status_code != 201:
        raise Exception(f"Commit failed with status {commit_response.status_code}")

    commit_id = commit_response.json().get('commit_id')

    # Step 2: Poll CI status API until pipeline completes
    for _ in range(30):  # wait up to ~3 minutes
        ci_response = requests.get(f"{ci_status_api_url}/{commit_id}", headers=headers)
        if ci_response.status_code == 200:
            status = ci_response.json().get('status')
            if status in ['passed', 'failed']:
                return status
        time.sleep(6)

    raise TimeoutError("CI pipeline did not complete in expected time")

# Example usage (replace URLs and payload with real values)
# repo_api_url = 'https://api.example.com/repos/myproject/commits'
# ci_status_api_url = 'https://ci.example.com/api/status'
# commit_payload = {"message": "Test commit", "changes": [{"file": "test.txt", "content": "Hello"}]}
# headers = {"Authorization": "Bearer token"}

# result = trigger_commit_and_check_ci(repo_api_url, ci_status_api_url, commit_payload, headers)
# assert result == 'passed', f"CI tests failed with status {result}"

This script simulates a code commit by sending a POST request to the repository API. It then polls the CI tool's status API to check if the pipeline triggered by that commit has completed. The polling waits up to about 3 minutes, checking every 6 seconds. Once the pipeline finishes, it returns the status, which should be either 'passed' or 'failed'. The assertion at the end ensures the tests passed.

This approach automates the manual test case steps by programmatically committing code and verifying the CI pipeline result via APIs, which is a common practice in Continuous Integration testing.

Common Mistakes - 3 Pitfalls

Not waiting for the CI pipeline to finish before checking results

Hardcoding commit data without simulating real changes

Parsing raw CI logs instead of using status APIs

Bonus Challenge

Now add data-driven testing by committing three different code changes and verifying the CI pipeline for each

Show Hint

Practice

1. What is the main purpose of Continuous Integration (CI) testing in software development?

easy

A. To automatically run tests on every code change to find bugs early

B. To manually test software after all development is complete

C. To deploy software directly to users without testing

D. To write code without any testing involved

Solution

1. Step 1: Understand CI testing purpose

   Continuous Integration testing runs tests automatically whenever code changes happen.

2. Step 2: Compare options with CI goals

   Only To automatically run tests on every code change to find bugs early describes automatic testing on code changes to catch bugs early.

3. Final Answer:

   To automatically run tests on every code change to find bugs early -> Option A

4. Quick Check:

   CI testing = automatic tests on code changes [OK]

Hint: CI testing runs tests automatically on code updates [OK]

Common Mistakes:

• Thinking CI testing is manual
• Confusing CI with deployment
• Believing testing happens after development only

2. Which of the following is the correct syntax to trigger a test run in a CI pipeline configuration file?

easy

A. run_tests: true

B. test: run

C. execute tests

D. script: pytest

Solution

1. Step 1: Identify common CI syntax

   CI pipelines often use a script key to run commands like pytest for tests.

2. Step 2: Check options for valid syntax

   Only script: pytest uses correct syntax: script: pytest runs tests properly.

3. Final Answer:

   script: pytest -> Option D

4. Quick Check:

   CI test command syntax = script: pytest [OK]

Hint: Look for 'script:' followed by test command [OK]

Common Mistakes:

• Using invalid keys like 'run_tests' or 'test: run'
• Writing commands without proper keys
• Confusing natural language with config syntax

3. Consider this simplified CI test script snippet:

steps:
  - script: echo "Run tests"
  - script: pytest tests/
  - script: echo "Tests complete"

What will be the output when this CI pipeline runs?

medium

A. Run tests\nTests complete

B. Only pytest tests output

C. Run tests\n(pytest runs tests output)\nTests complete

D. No output because script commands are ignored

Solution

1. Step 1: Analyze each script step output

   The first script prints "Run tests", the second runs pytest showing test results, the third prints "Tests complete".

2. Step 2: Combine outputs in order

   All outputs appear in sequence: first message, pytest output, then final message.

3. Final Answer:

   Run tests\n(pytest runs tests output)\nTests complete -> Option C

4. Quick Check:

   CI scripts run sequentially showing all outputs [OK]

Hint: CI scripts run in order showing all outputs [OK]

Common Mistakes:

• Ignoring pytest output
• Assuming only first or last script runs
• Thinking scripts are skipped

4. A CI pipeline fails to run tests because the test command is written as script: pytes instead of pytest. What is the best way to fix this error?

medium

A. Change pytes to pytest in the script command

B. Remove the script command entirely

C. Add a new script command with pytest but keep the old one

D. Ignore the error and rerun the pipeline

Solution

1. Step 1: Identify the typo in the test command

   The command pytes is a misspelling of pytest, causing the failure.

2. Step 2: Correct the command to fix the pipeline

   Fixing the typo to pytest ensures the tests run properly.

3. Final Answer:

   Change pytes to pytest in the script command -> Option A

4. Quick Check:

   Fix typos in commands to fix CI errors [OK]

Hint: Check spelling of commands in CI scripts [OK]

Common Mistakes:

• Removing test commands instead of fixing
• Adding duplicate commands without removing errors
• Ignoring errors and rerunning

5. You want to ensure your CI pipeline only runs tests on files that changed in the last commit to save time. Which approach best applies this in a CI configuration?

hard

A. Run all tests every time regardless of changes

B. Use a script step that detects changed files and runs tests only on those

C. Disable tests in the CI pipeline

D. Manually run tests after deployment

Solution

1. Step 1: Understand selective testing in CI

   To save time, CI can detect changed files and run only related tests.

2. Step 2: Identify the correct approach

   Use a script step that detects changed files and runs tests only on those describes using a script to detect changes and run targeted tests, which is efficient.

3. Final Answer:

   Use a script step that detects changed files and runs tests only on those -> Option B

4. Quick Check:

   Selective tests on changed files save CI time [OK]

Hint: Run tests only on changed files to save CI time [OK]

Common Mistakes:

• Running all tests every time wasting resources
• Disabling tests in CI
• Relying on manual testing after deployment