Continuous Integration testing in Testing Fundamentals - Framework Patterns

Framework Mode - Continuous Integration testing

Folder Structure for Continuous Integration Testing

ci-testing-project/
├── src/
│   └── main/                  # Application source code
├── tests/                     # Automated test scripts
│   ├── unit/                  # Unit tests
│   ├── integration/           # Integration tests
│   └── e2e/                   # End-to-end tests
├── ci/                       # CI pipeline scripts and configs
│   ├── Jenkinsfile            # Jenkins pipeline definition
│   ├── github-actions.yml     # GitHub Actions workflow
│   └── scripts/               # Helper scripts for CI
├── config/                   # Configuration files
│   ├── test-config.yaml       # Test environment configs
│   └── credentials.env        # Secure credentials (not committed)
├── reports/                  # Test reports generated by CI
└── README.md                 # Project documentation
    

Test Framework Layers in Continuous Integration

• Source Code Layer: The main application code to be tested.
• Test Layer: Automated tests organized by type (unit, integration, e2e).
• CI Pipeline Layer: Scripts and configuration files that define how tests run automatically on code changes.
• Configuration Layer: Environment settings, credentials, and parameters for tests and CI.
• Reporting Layer: Collects and displays test results after each CI run.
• Utilities Layer: Helper functions and scripts used by tests or CI pipelines.

Configuration Patterns for Continuous Integration Testing

• Environment Variables: Use environment variables to store sensitive data like credentials, avoiding hardcoding.
• Config Files: Use YAML or JSON files to define test environments (dev, staging, prod) with parameters like URLs and timeouts.
• Parameterization: Allow tests to accept parameters so they can run in different environments without code changes.
• CI Pipeline Config: Define pipeline steps (build, test, report) in files like Jenkinsfile or GitHub Actions YAML.
• Secrets Management: Use CI platform secret stores to securely inject credentials during pipeline runs.

Test Reporting and CI/CD Integration

• Automated Test Execution: CI runs tests automatically on every code push or pull request.
• Test Reports: Generate readable reports (HTML, XML) showing pass/fail status, errors, and logs.
• Notifications: Configure CI to send notifications (email, Slack) on test failures or successes.
• Build Status Badges: Display build and test status badges in project README or repository to show current health.
• Fail Fast: Stop pipeline on critical test failures to save resources and alert developers quickly.

Best Practices for Continuous Integration Testing Frameworks

1. Keep Tests Fast and Reliable: Fast tests help CI pipelines finish quickly and reliably catch issues.
2. Isolate Tests: Tests should not depend on each other or external state to avoid flaky results.
3. Use Version Control for All CI Configs: Keep pipeline scripts and configs in the same repository for traceability.
4. Run Tests on Every Change: Automate tests to run on every commit or pull request to catch bugs early.
5. Secure Secrets: Never store passwords or keys in code; use CI secret management features.

Self Check Question

Where in this folder structure would you add a new automated end-to-end test for the login feature?

Key Result

Continuous Integration testing frameworks automate running tests on every code change using organized layers and secure configurations.

Practice

1. What is the main purpose of Continuous Integration (CI) testing in software development?

easy

A. To automatically run tests on every code change to find bugs early

B. To manually test software after all development is complete

C. To deploy software directly to users without testing

D. To write code without any testing involved

Solution

1. Step 1: Understand CI testing purpose

   Continuous Integration testing runs tests automatically whenever code changes happen.

2. Step 2: Compare options with CI goals

   Only To automatically run tests on every code change to find bugs early describes automatic testing on code changes to catch bugs early.

3. Final Answer:

   To automatically run tests on every code change to find bugs early -> Option A

4. Quick Check:

   CI testing = automatic tests on code changes [OK]

Hint: CI testing runs tests automatically on code updates [OK]

Common Mistakes:

• Thinking CI testing is manual
• Confusing CI with deployment
• Believing testing happens after development only

2. Which of the following is the correct syntax to trigger a test run in a CI pipeline configuration file?

easy

A. run_tests: true

B. test: run

C. execute tests

D. script: pytest

Solution

1. Step 1: Identify common CI syntax

   CI pipelines often use a script key to run commands like pytest for tests.

2. Step 2: Check options for valid syntax

   Only script: pytest uses correct syntax: script: pytest runs tests properly.

3. Final Answer:

   script: pytest -> Option D

4. Quick Check:

   CI test command syntax = script: pytest [OK]

Hint: Look for 'script:' followed by test command [OK]

Common Mistakes:

• Using invalid keys like 'run_tests' or 'test: run'
• Writing commands without proper keys
• Confusing natural language with config syntax

3. Consider this simplified CI test script snippet:

steps:
  - script: echo "Run tests"
  - script: pytest tests/
  - script: echo "Tests complete"

What will be the output when this CI pipeline runs?

medium

A. Run tests\nTests complete

B. Only pytest tests output

C. Run tests\n(pytest runs tests output)\nTests complete

D. No output because script commands are ignored

Solution

1. Step 1: Analyze each script step output

   The first script prints "Run tests", the second runs pytest showing test results, the third prints "Tests complete".

2. Step 2: Combine outputs in order

   All outputs appear in sequence: first message, pytest output, then final message.

3. Final Answer:

   Run tests\n(pytest runs tests output)\nTests complete -> Option C

4. Quick Check:

   CI scripts run sequentially showing all outputs [OK]

Hint: CI scripts run in order showing all outputs [OK]

Common Mistakes:

• Ignoring pytest output
• Assuming only first or last script runs
• Thinking scripts are skipped

4. A CI pipeline fails to run tests because the test command is written as script: pytes instead of pytest. What is the best way to fix this error?

medium

A. Change pytes to pytest in the script command

B. Remove the script command entirely

C. Add a new script command with pytest but keep the old one

D. Ignore the error and rerun the pipeline

Solution

1. Step 1: Identify the typo in the test command

   The command pytes is a misspelling of pytest, causing the failure.

2. Step 2: Correct the command to fix the pipeline

   Fixing the typo to pytest ensures the tests run properly.

3. Final Answer:

   Change pytes to pytest in the script command -> Option A

4. Quick Check:

   Fix typos in commands to fix CI errors [OK]

Hint: Check spelling of commands in CI scripts [OK]

Common Mistakes:

• Removing test commands instead of fixing
• Adding duplicate commands without removing errors
• Ignoring errors and rerunning

5. You want to ensure your CI pipeline only runs tests on files that changed in the last commit to save time. Which approach best applies this in a CI configuration?

hard

A. Run all tests every time regardless of changes

B. Use a script step that detects changed files and runs tests only on those

C. Disable tests in the CI pipeline

D. Manually run tests after deployment

Solution

1. Step 1: Understand selective testing in CI

   To save time, CI can detect changed files and run only related tests.

2. Step 2: Identify the correct approach

   Use a script step that detects changed files and runs tests only on those describes using a script to detect changes and run targeted tests, which is efficient.

3. Final Answer:

   Use a script step that detects changed files and runs tests only on those -> Option B

4. Quick Check:

   Selective tests on changed files save CI time [OK]

Hint: Run tests only on changed files to save CI time [OK]

Common Mistakes:

• Running all tests every time wasting resources
• Disabling tests in CI
• Relying on manual testing after deployment