Defect metrics in Testing Fundamentals - Build an Automation Script

Calculate and verify defect metrics from a defect log

Preconditions (2)

Step 1: Open the defect log file

Step 2: Count total number of defects

Step 3: Count number of open defects

Step 4: Count number of closed defects

Step 5: Calculate defect density given total defects and total lines of code (LOC)

Step 6: Calculate defect removal efficiency (DRE) using defects found before release and total defects

Step 7: Verify that the calculated metrics match expected values

✅ Expected Result: The defect metrics (total defects, open defects, closed defects, defect density, DRE) are correctly calculated and match the expected values

Automation Requirements - Python unittest

Assertions Needed:

Assert total defects count equals expected

Assert open defects count equals expected

Assert closed defects count equals expected

Assert defect density calculation is correct

Assert defect removal efficiency calculation is correct

Best Practices:

Use setup method to load defect data

Use clear variable names for metrics

Use assertEqual for exact matches

Keep test focused on metric calculations

Avoid hardcoding values inside test methods

Automated Solution

Testing Fundamentals

import unittest

class DefectMetricsTest(unittest.TestCase):
    def setUp(self):
        # Sample defect log data: list of dicts
        self.defects = [
            {'id': 'D1', 'severity': 'High', 'status': 'Open', 'reported_date': '2024-01-01', 'found_before_release': True},
            {'id': 'D2', 'severity': 'Medium', 'status': 'Closed', 'reported_date': '2024-01-02', 'found_before_release': True},
            {'id': 'D3', 'severity': 'Low', 'status': 'Closed', 'reported_date': '2024-01-03', 'found_before_release': False},
            {'id': 'D4', 'severity': 'High', 'status': 'Open', 'reported_date': '2024-01-04', 'found_before_release': True},
            {'id': 'D5', 'severity': 'Medium', 'status': 'Closed', 'reported_date': '2024-01-05', 'found_before_release': False}
        ]
        self.total_loc = 1000  # total lines of code

    def test_defect_metrics(self):
        total_defects = len(self.defects)
        open_defects = sum(1 for d in self.defects if d['status'] == 'Open')
        closed_defects = sum(1 for d in self.defects if d['status'] == 'Closed')

        defect_density = total_defects / self.total_loc  # defects per LOC

        defects_found_before_release = sum(1 for d in self.defects if d['found_before_release'])
        # Defect Removal Efficiency (DRE) = defects found before release / total defects
        dre = defects_found_before_release / total_defects

        # Expected values based on sample data
        expected_total_defects = 5
        expected_open_defects = 2
        expected_closed_defects = 3
        expected_defect_density = 5 / 1000
        expected_dre = 3 / 5

        self.assertEqual(total_defects, expected_total_defects, "Total defects count mismatch")
        self.assertEqual(open_defects, expected_open_defects, "Open defects count mismatch")
        self.assertEqual(closed_defects, expected_closed_defects, "Closed defects count mismatch")
        self.assertAlmostEqual(defect_density, expected_defect_density, places=5, msg="Defect density mismatch")
        self.assertAlmostEqual(dre, expected_dre, places=5, msg="Defect Removal Efficiency mismatch")

if __name__ == '__main__':
    unittest.main()

This test uses Python's unittest framework to automate defect metrics calculation.

The setUp method prepares sample defect data and total lines of code.

The test method calculates total defects, open and closed defects counts, defect density, and defect removal efficiency (DRE).

Assertions check that these calculated values match expected values derived from the sample data.

Using assertEqual and assertAlmostEqual ensures exact and approximate matches respectively.

This approach keeps the test clear, focused, and easy to maintain.

Common Mistakes - 4 Pitfalls

Hardcoding expected values inside the test method without explanation

Not using setup method to prepare test data

Using print statements instead of assertions

Calculating metrics incorrectly, e.g., dividing by zero or wrong denominator

Bonus Challenge

Now add data-driven testing with 3 different defect logs having different defect counts and LOC values

Show Hint

Practice

1. What does defect density measure in software testing?

easy

A. Time taken to fix a defect

B. Total number of test cases executed

C. Number of defects per size of the software

D. Number of users reporting defects

Solution

1. Step 1: Understand defect density meaning

   Defect density is a metric that counts defects relative to software size, like per thousand lines of code.

2. Step 2: Compare options with definition

   Only Number of defects per size of the software matches this definition; others relate to different metrics.

3. Final Answer:

   Number of defects per size of the software -> Option C

4. Quick Check:

   Defect density = defects / software size [OK]

Hint: Defect density = defects divided by software size [OK]

Common Mistakes:

• Confusing defect density with total defects
• Mixing defect density with test execution count
• Thinking defect density measures fix time

2. Which of the following is the correct formula for calculating defect density?

easy

A. Defect Density = Total Defects / Software Size

B. Defect Density = Software Size / Total Defects

C. Defect Density = Total Defects x Software Size

D. Defect Density = Total Defects + Software Size

Solution

1. Step 1: Recall defect density formula

   Defect density is calculated by dividing total defects by software size.

2. Step 2: Match formula with options

   Defect Density = Total Defects / Software Size correctly shows division of defects by size; others use incorrect operations.

3. Final Answer:

   Defect Density = Total Defects / Software Size -> Option A

4. Quick Check:

   Defect density = defects ÷ size [OK]

Hint: Defect density divides defects by size, not multiply [OK]

Common Mistakes:

• Multiplying defects by size instead of dividing
• Reversing numerator and denominator
• Adding defects and size instead of dividing

3. A software module has 1200 lines of code and 24 reported defects. What is the defect density?

medium

A. 20 defects per 1000 lines

B. 50 defects per 1000 lines

C. 0.02 defects per line

D. 0.02 defects per 1000 lines

Solution

1. Step 1: Calculate defects per 1000 lines

   Defect density = (24 defects / 1200 lines) x 1000 = 20 defects per 1000 lines.

2. Step 2: Compare with options

   20 defects per 1000 lines matches 20 defects per 1000 lines; others are incorrect calculations or units.

3. Final Answer:

   20 defects per 1000 lines -> Option A

4. Quick Check:

   (24/1200)*1000 = 20 [OK]

Hint: Multiply defects/lines by 1000 for defect density [OK]

Common Mistakes:

• Not scaling defects per 1000 lines
• Confusing defects per line with per 1000 lines
• Incorrect division order

4. A defect report shows severity as 'High' but the status is marked 'Closed' without fixing. What is the likely issue?

medium

A. Defect density calculation error

B. Defect was fixed but not retested

C. Test cases are incomplete

D. Defect severity and status are inconsistent

Solution

1. Step 1: Analyze severity and status meaning

   High severity means serious defect; status 'Closed' usually means fixed or resolved.

2. Step 2: Identify inconsistency

   Closing a high severity defect without fixing is inconsistent and indicates a reporting or process error.

3. Final Answer:

   Defect severity and status are inconsistent -> Option D

4. Quick Check:

   High severity + Closed without fix = inconsistency [OK]

Hint: High severity defects should not be closed without fix [OK]

Common Mistakes:

• Assuming defect density relates to status
• Confusing test case completeness with defect status
• Thinking defect was fixed without evidence

5. You have two modules: Module A with 2000 lines and 40 defects, Module B with 1000 lines and 30 defects. Which module has better quality based on defect density?

hard

A. Module A has better quality because it has fewer defects

B. Module A has better quality because its defect density is lower

C. Both modules have the same quality

D. Module B has better quality because it has fewer lines of code

Solution

1. Step 1: Calculate defect density for Module A

   Module A defect density = 40 / 2000 = 0.02 defects per line.

2. Step 2: Calculate defect density for Module B

   Module B defect density = 30 / 1000 = 0.03 defects per line.

3. Step 3: Compare defect densities

   Lower defect density means better quality; Module A's 0.02 is less than Module B's 0.03.

4. Final Answer:

   Module A has better quality because its defect density is lower -> Option B

5. Quick Check:

   0.02 < 0.03 means Module A better [OK]

Hint: Lower defect density means better quality [OK]

Common Mistakes:

• Comparing only defect counts without size
• Assuming fewer lines means better quality
• Saying both have same quality without calculation