Continuous testing in CI/CD in Testing Fundamentals - Full Explanation

Introduction

Imagine building a large puzzle where each piece must fit perfectly before adding the next. Continuous testing in CI/CD helps catch problems early by checking software automatically as it is built, so errors don't pile up and cause bigger issues later.

Explanation

Continuous Integration (CI)

Continuous Integration means developers frequently add their code changes to a shared project. Each addition triggers automated tests to check if the new code works well with the existing code. This helps find problems quickly before they grow.

CI ensures code changes are tested often to catch errors early.

Continuous Delivery/Deployment (CD)

Continuous Delivery means the software is always ready to be released after passing tests. Continuous Deployment goes further by automatically releasing the software to users once tests pass. Both rely on testing to keep software reliable and up-to-date.

CD uses testing to keep software ready or automatically release it safely.

Automated Testing

Automated tests run by computers check the software without needing people to do it manually. These tests can include checking if features work, if the software is fast, or if it handles errors well. Automation makes testing faster and more consistent.

Automated testing speeds up and standardizes checking software quality.

Feedback Loop

When tests find problems, the system quickly tells developers so they can fix issues right away. This fast feedback helps keep the project healthy and avoids delays caused by late bug discovery.

Fast feedback from tests helps developers fix problems quickly.

Real World Analogy

Think of a bakery where each cake layer is checked for quality before adding the next. If a layer is uneven, the baker fixes it immediately instead of waiting until the whole cake is done. This way, the final cake is perfect and ready to serve without surprises.

Continuous Integration (CI) → Adding each cake layer carefully and checking it before continuing

Continuous Delivery/Deployment (CD) → Having the cake ready to serve anytime or automatically sending it to customers

Automated Testing → Using tools like a level or thermometer to check cake layers quickly and reliably

Feedback Loop → The baker immediately knowing if a layer is bad and fixing it right away

Diagram

Diagram

┌───────────────┐      ┌───────────────┐      ┌───────────────┐
│ Code Changes  │─────▶│ Automated     │─────▶│ Feedback to   │
│ by Developer │      │ Tests Run     │      │ Developer     │
└───────────────┘      └───────────────┘      └───────────────┘
                                   │
                                   ▼
                         ┌───────────────────┐
                         │ Continuous Delivery│
                         │ or Deployment     │
                         └───────────────────┘

This diagram shows how code changes trigger automated tests, which provide feedback and lead to delivery or deployment.

Key Facts

Continuous Integration → A practice where developers frequently merge code changes into a shared repository.

Continuous Delivery → Keeping software in a deployable state after passing automated tests.

Continuous Deployment → Automatically releasing software to users after successful tests.

Automated Testing → Using software tools to run tests without manual effort.

Feedback Loop → The process of quickly informing developers about test results.

Code Example

Testing Fundamentals

import unittest

def add(a, b):
    return a + b

class TestAddFunction(unittest.TestCase):
    def test_add_positive(self):
        self.assertEqual(add(2, 3), 5)
    def test_add_negative(self):
        self.assertEqual(add(-1, -1), -2)

if __name__ == '__main__':
    unittest.main()

Common Confusions

Believing continuous testing means only running tests once at the end of development.

Believing continuous testing means only running tests once at the end of development. Continuous testing means running automated tests frequently throughout development, not just once at the end.

Thinking continuous delivery and continuous deployment are the same.

Thinking continuous delivery and continuous deployment are the same. Continuous delivery means software is ready to release; continuous deployment means software is released automatically after tests pass.

Summary

Continuous testing in CI/CD helps catch software problems early by running automated tests frequently.

It supports continuous integration by testing code changes and continuous delivery/deployment by ensuring software is always ready or automatically released.

Fast feedback from automated tests keeps developers informed and helps maintain software quality.

Practice

1. What is the main purpose of continuous testing in a CI/CD pipeline?

easy

A. To automatically run tests on every code change

B. To manually test software after deployment

C. To delay testing until the final release

D. To only test code on the developer's machine

Solution

1. Step 1: Understand continuous testing concept

   Continuous testing means running tests automatically during the CI/CD process.

2. Step 2: Identify the purpose in CI/CD

   Its goal is to catch bugs early by testing every code change automatically.

3. Final Answer:

   To automatically run tests on every code change -> Option A

4. Quick Check:

   Continuous testing = automatic tests on code changes [OK]

Hint: Continuous testing means tests run automatically every change [OK]

Common Mistakes:

• Thinking testing is manual only
• Believing tests run after deployment
• Assuming testing happens only once at release

2. Which of the following is the correct syntax to run tests in a simple CI pipeline using a shell command?

easy

A. sh tests.sh

B. run tests.sh

C. execute tests.sh

D. start tests.sh

Solution

1. Step 1: Recall shell command syntax

   To run a shell script, the correct command is 'sh scriptname.sh'.

2. Step 2: Check each option

   sh tests.sh uses 'sh tests.sh' which correctly runs the script. Others are invalid shell commands.

3. Final Answer:

   sh tests.sh -> Option A

4. Quick Check:

   Shell script run = sh script.sh [OK]

Hint: Use 'sh script.sh' to run shell scripts in pipelines [OK]

Common Mistakes:

• Using 'run' or 'start' which are not shell commands
• Forgetting to use 'sh' before script name
• Confusing shell commands with other languages

3. Given this CI pipeline snippet:

steps:
  - name: Run tests
    run: |
      echo "Starting tests"
      pytest tests/
      echo "Tests done"

What will be the output if all tests pass?

medium

A. Starting tests\nTests failed

B. Starting tests\npytest: command not found

C. Starting tests\nTests done

D. Tests done\nStarting tests

Solution

1. Step 1: Analyze the commands in order

   The pipeline runs 'echo "Starting tests"', then 'pytest tests/', then 'echo "Tests done"'.

2. Step 2: Understand output if tests pass

   If pytest runs successfully, it outputs test results but here we focus on echo commands output. So output includes 'Starting tests' then 'Tests done'.

3. Final Answer:

   Starting tests\nTests done -> Option C

4. Quick Check:

   Echo commands print start and done messages [OK]

Hint: Echo commands print messages before and after tests [OK]

Common Mistakes:

• Assuming pytest output is shown instead of echo
• Mixing order of printed lines
• Thinking tests failed without error

4. In a CI pipeline, the test step fails with error: sh: pytest: command not found. What is the most likely cause?

medium

A. The test script has a syntax error

B. The pipeline skipped the test step

C. The tests directory is missing

D. pytest is not installed in the pipeline environment

Solution

1. Step 1: Interpret the error message

   The error 'sh: pytest: command not found' means the shell cannot find the pytest command.

2. Step 2: Identify cause in CI environment

   This usually means pytest is not installed or not in PATH in the pipeline environment.

3. Final Answer:

   pytest is not installed in the pipeline environment -> Option D

4. Quick Check:

   Command not found = missing installation [OK]

Hint: Command not found means tool missing or not installed [OK]

Common Mistakes:

• Assuming syntax error causes command not found
• Thinking missing tests directory causes this error
• Believing pipeline skipped the step without error

5. You want to add continuous testing to your CI/CD pipeline so tests run only if code changes affect the src/ directory. Which approach best achieves this?

hard

A. Run tests on every pipeline run regardless of changes

B. Configure the pipeline to run tests only when files in src/ change

C. Run tests only after deployment to production

D. Manually trigger tests after code review

Solution

1. Step 1: Understand conditional testing in CI/CD

   To save time, tests should run only when relevant code changes occur.

2. Step 2: Apply file path filters in pipeline configuration

   Configuring the pipeline to detect changes in src/ and run tests only then is the best approach.

3. Final Answer:

   Configure the pipeline to run tests only when files in src/ change -> Option B

4. Quick Check:

   Run tests conditionally based on changed files [OK]

Hint: Run tests only if relevant files change to save time [OK]

Common Mistakes:

• Running tests every time wastes resources
• Testing only after deployment delays bug detection
• Manual triggers reduce automation benefits