Smoke testing and sanity testing in Testing Fundamentals - Framework Patterns

Framework Mode - Smoke testing and sanity testing

Folder Structure

  smoke_sanity_tests_project/
  ├── tests/
  │   ├── smoke/
  │   │   ├── test_login_smoke.py
  │   │   └── test_checkout_smoke.py
  │   ├── sanity/
  │   │   ├── test_payment_sanity.py
  │   │   └── test_profile_update_sanity.py
  │   └── __init__.py
  ├── pages/
  │   ├── login_page.py
  │   ├── checkout_page.py
  │   ├── payment_page.py
  │   └── profile_page.py
  ├── utils/
  │   ├── helpers.py
  │   └── data_loader.py
  ├── config/
  │   ├── config.yaml
  │   └── env_variables.py
  ├── reports/
  │   └── latest_report.html
  ├── conftest.py
  └── pytest.ini
  

Test Framework Layers

• Test Layer: Contains smoke and sanity test scripts organized by test type. These tests quickly check critical features (smoke) or verify specific bug fixes or functionality (sanity).
• Page Object Layer: Encapsulates UI elements and actions for pages like login, checkout, payment, and profile. This keeps tests clean and reusable.
• Utility Layer: Helper functions for common tasks like data loading, waiting, or logging.
• Configuration Layer: Holds environment settings, browser options, and credentials to run tests in different setups.
• Reporting Layer: Generates test execution reports and stores them for review.

Configuration Patterns

Use a config.yaml file to store environment URLs, browser types, and credentials. For example:

  environments:
    dev:
      url: "https://dev.example.com"
      browser: "chrome"
    staging:
      url: "https://staging.example.com"
      browser: "firefox"
  credentials:
    username: "testuser"
    password: "password123"
  

Load these settings in env_variables.py and use fixtures in conftest.py to pass them to tests. This allows running smoke or sanity tests on different environments easily.

Test Reporting and CI/CD Integration

• Use pytest's built-in reporting or plugins like pytest-html to generate clear HTML reports showing which smoke and sanity tests passed or failed.
• Store reports in the reports/ folder for easy access.
• Integrate tests into CI/CD pipelines (e.g., GitHub Actions, Jenkins) to run smoke tests automatically on every build and sanity tests after bug fixes.
• Fail the build if smoke tests fail to prevent broken code from progressing.

Best Practices

1. Keep smoke tests broad but shallow: Quickly check main features to catch major issues early.
2. Keep sanity tests narrow and deep: Focus on verifying specific fixes or functionality after changes.
3. Organize tests by type: Separate smoke and sanity tests in folders for clarity and easy execution.
4. Use page objects: Avoid duplicating UI code and make maintenance easier.
5. Automate test runs in CI/CD: Ensure tests run regularly without manual effort.

Self Check

Where in this folder structure would you add a new smoke test for the "Search" feature?

Key Result

Organize smoke and sanity tests separately with clear layers for pages, utilities, config, and reporting to enable fast, reliable checks of critical and specific features.

Practice

1. What is the main purpose of smoke testing in software testing?

easy

A. To perform security testing on the application

B. To quickly check if the main features of a new build work

C. To verify the user interface design

D. To test detailed functionality of all features

Solution

1. Step 1: Understand smoke testing purpose

   Smoke testing is done to quickly verify that the main features of a new software build are working and the build is stable enough for further testing.

2. Step 2: Compare options with smoke testing definition

   To quickly check if the main features of a new build work matches this purpose exactly, while others describe different testing types.

3. Final Answer:

   To quickly check if the main features of a new build work -> Option B

4. Quick Check:

   Smoke testing = Quick main feature check [OK]

Hint: Smoke test = quick main feature check after build [OK]

Common Mistakes:

• Confusing smoke testing with detailed testing
• Thinking smoke testing checks UI design
• Mixing smoke testing with security testing

2. Which of the following is the correct description of sanity testing?

easy

A. Checking specific functionality after small changes or bug fixes

B. Testing all features in detail after a major release

C. Performing load testing to check performance

D. Running automated tests for regression

Solution

1. Step 1: Define sanity testing

   Sanity testing is a quick check of specific functionality or bug fixes after small changes to ensure they work as expected.

2. Step 2: Match options to sanity testing

   Checking specific functionality after small changes or bug fixes correctly describes this focused testing, while others describe different testing types.

3. Final Answer:

   Checking specific functionality after small changes or bug fixes -> Option A

4. Quick Check:

   Sanity testing = Focused check after small changes [OK]

Hint: Sanity test = check specific fixes after small changes [OK]

Common Mistakes:

• Confusing sanity testing with full regression testing
• Thinking sanity testing covers all features
• Mixing sanity testing with performance testing

3. Consider this scenario: A new build is deployed with multiple new features and bug fixes. Which testing should be done first to decide if detailed testing can proceed?

medium

A. Sanity testing on the fixed bugs only

B. Full regression testing on all features

C. Smoke testing on main features of the build

D. Performance testing under load

Solution

1. Step 1: Identify testing purpose after new build

   After a new build with many changes, smoke testing is done first to quickly check main features and decide if the build is stable for further testing.

2. Step 2: Evaluate options for initial testing

   Smoke testing on main features of the build matches this initial check. Sanity testing (Sanity testing on the fixed bugs only) is for small changes, not full builds. Full regression (Full regression testing on all features) and performance (Performance testing under load) come later.

3. Final Answer:

   Smoke testing on main features of the build -> Option C

4. Quick Check:

   New build? Start with smoke testing [OK]

Hint: New build? Start with smoke testing main features [OK]

Common Mistakes:

• Starting with sanity testing on big builds
• Skipping smoke testing and doing full regression immediately
• Confusing performance testing as first step

4. A tester runs sanity testing after a small bug fix but finds unrelated features failing. What is the likely cause?

medium

A. Sanity testing scope was too narrow, missing regression tests

B. Smoke testing was done incorrectly

C. The bug fix introduced new unrelated bugs

D. Performance testing was skipped

Solution

1. Step 1: Understand sanity testing scope

   Sanity testing focuses on verifying specific fixes or features, not the entire application.

2. Step 2: Analyze failure of unrelated features

   If unrelated features fail, it means sanity testing missed broader checks. Full regression testing is needed to catch such issues.

3. Final Answer:

   Sanity testing scope was too narrow, missing regression tests -> Option A

4. Quick Check:

   Unrelated failures after sanity test? Need regression [OK]

Hint: Sanity test checks fixes only; unrelated fails need regression [OK]

Common Mistakes:

• Blaming smoke testing for unrelated failures
• Assuming bug fix always causes unrelated failures
• Ignoring need for regression testing

5. A software team wants to improve testing efficiency after frequent small updates. Which approach best combines smoke and sanity testing?

hard

A. Perform performance testing before smoke or sanity testing

B. Skip smoke testing and run full regression after each update

C. Only run sanity testing on all features after every update

D. Run smoke testing on main features after every build, then sanity testing on specific fixes

Solution

1. Step 1: Understand testing goals for frequent small updates

   To improve efficiency, quick checks are needed to catch major issues early, then focused checks on fixes.

2. Step 2: Evaluate combined approach of smoke and sanity testing

   Running smoke testing on main features ensures build stability, followed by sanity testing on specific fixes ensures correctness of changes.

3. Step 3: Eliminate other options

   Skipping smoke testing or running full regression every time is inefficient. Performance testing is unrelated to this goal.

4. Final Answer:

   Run smoke testing on main features after every build, then sanity testing on specific fixes -> Option D

5. Quick Check:

   Smoke then sanity for efficient small update testing [OK]

Hint: Smoke test main features, then sanity test fixes for efficiency [OK]

Common Mistakes:

• Skipping smoke testing for faster results
• Running full regression after every small update
• Confusing performance testing with smoke/sanity