Run static analysis on source code and verify reported issues

Preconditions (2)

Step 1: Open the terminal or command line interface

Step 2: Navigate to the project directory containing the source code

Step 3: Run the static analysis tool command to scan the source code

Step 4: Wait for the tool to complete the scan

Step 5: Review the output report generated by the tool

Step 6: Identify any warnings or errors reported by the tool

✅ Expected Result: The static analysis tool completes the scan without crashing and outputs a report listing any code issues such as syntax errors, code smells, or potential bugs.

Automation Requirements - Python unittest

Assertions Needed:

Verify the static analysis tool command runs successfully

Verify the output report file is created

Verify the report contains expected warning or error messages

Best Practices:

Use subprocess module to run external commands

Capture and check command exit codes

Parse output files safely

Use assertions to validate expected results

Automated Solution

Testing Fundamentals

import unittest
import subprocess
import os

class TestStaticAnalysis(unittest.TestCase):
    def test_static_analysis_run(self):
        # Define the command to run the static analysis tool
        # Example: pylint scanning a Python file named example.py
        command = ['pylint', 'example.py', '--output-format=text']

        # Run the command and capture output
        result = subprocess.run(command, capture_output=True, text=True)

        # Assert the command ran successfully (exit code 0 or 1 for pylint warnings)
        self.assertIn(result.returncode, [0, 1], f"Static analysis failed with code {result.returncode}")

        # Assert output contains pylint warning or error messages
        output = result.stdout
        self.assertTrue(len(output) > 0, "Static analysis output is empty")
        self.assertIn('warning', output.lower() + 'error', output.lower(), "No warnings or errors found in output")

        # Optionally, write output to a report file
        report_path = 'static_analysis_report.txt'
        with open(report_path, 'w') as f:
            f.write(output)

        # Assert report file is created and not empty
        self.assertTrue(os.path.exists(report_path), "Report file was not created")
        self.assertTrue(os.path.getsize(report_path) > 0, "Report file is empty")

if __name__ == '__main__':
    unittest.main()

This test script uses Python's unittest framework to automate running a static analysis tool (like pylint) on a source code file named example.py.

We use the subprocess.run function to execute the tool command and capture its output and exit code.

The test asserts that the tool runs successfully by checking the exit code is either 0 (no issues) or 1 (issues found but no crash).

It then checks that the output contains some warnings or errors by looking for keywords in the output text.

The output is saved to a report file, and the test verifies that this file exists and is not empty.

This approach ensures the static analysis tool runs correctly and produces a meaningful report, automating the manual test steps.

Common Mistakes - 4 Pitfalls

{'mistake': 'Not checking the exit code of the static analysis tool', 'why_bad': 'The tool might fail silently or crash, and the test would not detect it.', 'correct_approach': "Always check the command's exit code to confirm it ran successfully."}

Hardcoding file paths without verifying they exist

Parsing output with fragile string matching

{'mistake': "Not capturing or checking the tool's output", 'why_bad': 'You miss verifying if the tool found any issues.', 'correct_approach': 'Capture stdout/stderr and assert expected warnings or errors.'}

Bonus Challenge

Now add data-driven testing to run static analysis on three different source files

Show Hint