Continuous testing in CI/CD in Testing Fundamentals - Framework Patterns

Framework Mode - Continuous testing in CI/CD

Folder Structure

  project-root/
  ├── src/                  # Application source code
  ├── tests/                # Automated tests
  │   ├── unit/             # Unit tests
  │   ├── integration/      # Integration tests
  │   ├── e2e/              # End-to-end tests
  │   └── fixtures/         # Test data and mocks
  ├── ci/                   # CI/CD pipeline scripts and configs
  ├── config/               # Environment and test configurations
  │   ├── dev.env           # Development environment variables
  │   ├── staging.env       # Staging environment variables
  │   └── prod.env          # Production environment variables
  ├── reports/              # Test reports and logs
  ├── utils/                # Helper functions and utilities
  ├── package.json          # Project dependencies and scripts (example for JS)
  └── README.md             # Project documentation
  

Test Framework Layers

• Test Scripts Layer: Contains automated tests (unit, integration, e2e) that verify application behavior.
• Test Data & Fixtures Layer: Provides mock data and setup needed for tests to run consistently.
• Utilities Layer: Helper functions for common tasks like API calls, data generation, or custom assertions.
• Configuration Layer: Manages environment variables, test settings, and credentials for different deployment stages.
• CI/CD Pipeline Layer: Automates test execution on code changes, integrates with version control and deployment tools.
• Reporting Layer: Collects and presents test results clearly for developers and stakeholders.

Configuration Patterns

• Environment Files: Use separate files for dev, staging, and production to store URLs, credentials, and flags.
• Parameterization: Pass environment and browser info as parameters or environment variables to tests.
• Secrets Management: Store sensitive data securely outside the codebase, e.g., CI secrets or vaults.
• Config Loader Utility: Centralize reading and applying configuration to avoid duplication.
• Version Control: Keep config files (except secrets) under version control for traceability.

Test Reporting and CI/CD Integration

• Automated Test Runs: Trigger tests automatically on code commits or pull requests via CI tools (e.g., Jenkins, GitHub Actions).
• Clear Reports: Generate readable reports (HTML, JUnit XML) showing pass/fail status and error details.
• Notifications: Send alerts on test failures to developers via email or chat tools.
• Test Coverage: Include coverage reports to measure how much code is tested.
• Gatekeeping: Use test results to block deployments if critical tests fail, ensuring quality before release.

Framework Design Principles

1. Automate Everything: Tests should run automatically on every code change without manual steps.
2. Keep Tests Fast and Reliable: Fast tests give quick feedback; reliable tests avoid false failures.
3. Isolate Tests: Tests should not depend on each other or external state to avoid flaky results.
4. Use Clear Naming: Name tests and folders clearly to understand their purpose at a glance.
5. Integrate Early: Add tests to CI/CD pipelines early to catch issues sooner and improve confidence.

Self Check

Where in this folder structure would you add a new end-to-end test for the user login feature?

Key Result

Continuous testing in CI/CD means automating tests to run on every code change, integrated tightly with deployment pipelines for fast feedback and quality assurance.

Practice

1. What is the main purpose of continuous testing in a CI/CD pipeline?

easy

A. To automatically run tests on every code change

B. To manually test software after deployment

C. To delay testing until the final release

D. To only test code on the developer's machine

Solution

1. Step 1: Understand continuous testing concept

   Continuous testing means running tests automatically during the CI/CD process.

2. Step 2: Identify the purpose in CI/CD

   Its goal is to catch bugs early by testing every code change automatically.

3. Final Answer:

   To automatically run tests on every code change -> Option A

4. Quick Check:

   Continuous testing = automatic tests on code changes [OK]

Hint: Continuous testing means tests run automatically every change [OK]

Common Mistakes:

• Thinking testing is manual only
• Believing tests run after deployment
• Assuming testing happens only once at release

2. Which of the following is the correct syntax to run tests in a simple CI pipeline using a shell command?

easy

A. sh tests.sh

B. run tests.sh

C. execute tests.sh

D. start tests.sh

Solution

1. Step 1: Recall shell command syntax

   To run a shell script, the correct command is 'sh scriptname.sh'.

2. Step 2: Check each option

   sh tests.sh uses 'sh tests.sh' which correctly runs the script. Others are invalid shell commands.

3. Final Answer:

   sh tests.sh -> Option A

4. Quick Check:

   Shell script run = sh script.sh [OK]

Hint: Use 'sh script.sh' to run shell scripts in pipelines [OK]

Common Mistakes:

• Using 'run' or 'start' which are not shell commands
• Forgetting to use 'sh' before script name
• Confusing shell commands with other languages

3. Given this CI pipeline snippet:

steps:
  - name: Run tests
    run: |
      echo "Starting tests"
      pytest tests/
      echo "Tests done"

What will be the output if all tests pass?

medium

A. Starting tests\nTests failed

B. Starting tests\npytest: command not found

C. Starting tests\nTests done

D. Tests done\nStarting tests

Solution

1. Step 1: Analyze the commands in order

   The pipeline runs 'echo "Starting tests"', then 'pytest tests/', then 'echo "Tests done"'.

2. Step 2: Understand output if tests pass

   If pytest runs successfully, it outputs test results but here we focus on echo commands output. So output includes 'Starting tests' then 'Tests done'.

3. Final Answer:

   Starting tests\nTests done -> Option C

4. Quick Check:

   Echo commands print start and done messages [OK]

Hint: Echo commands print messages before and after tests [OK]

Common Mistakes:

• Assuming pytest output is shown instead of echo
• Mixing order of printed lines
• Thinking tests failed without error

4. In a CI pipeline, the test step fails with error: sh: pytest: command not found. What is the most likely cause?

medium

A. The test script has a syntax error

B. The pipeline skipped the test step

C. The tests directory is missing

D. pytest is not installed in the pipeline environment

Solution

1. Step 1: Interpret the error message

   The error 'sh: pytest: command not found' means the shell cannot find the pytest command.

2. Step 2: Identify cause in CI environment

   This usually means pytest is not installed or not in PATH in the pipeline environment.

3. Final Answer:

   pytest is not installed in the pipeline environment -> Option D

4. Quick Check:

   Command not found = missing installation [OK]

Hint: Command not found means tool missing or not installed [OK]

Common Mistakes:

• Assuming syntax error causes command not found
• Thinking missing tests directory causes this error
• Believing pipeline skipped the step without error

5. You want to add continuous testing to your CI/CD pipeline so tests run only if code changes affect the src/ directory. Which approach best achieves this?

hard

A. Run tests on every pipeline run regardless of changes

B. Configure the pipeline to run tests only when files in src/ change

C. Run tests only after deployment to production

D. Manually trigger tests after code review

Solution

1. Step 1: Understand conditional testing in CI/CD

   To save time, tests should run only when relevant code changes occur.

2. Step 2: Apply file path filters in pipeline configuration

   Configuring the pipeline to detect changes in src/ and run tests only then is the best approach.

3. Final Answer:

   Configure the pipeline to run tests only when files in src/ change -> Option B

4. Quick Check:

   Run tests conditionally based on changed files [OK]

Hint: Run tests only if relevant files change to save time [OK]

Common Mistakes:

• Running tests every time wastes resources
• Testing only after deployment delays bug detection
• Manual triggers reduce automation benefits