Flaky test management in Testing Fundamentals - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to mark a test as flaky in pytest.

Testing Fundamentals

@pytest.mark.[1]
def test_example():
    assert True

Drag options to blanks, or click blank then click option'

Aparametrize

Bskip

Cflaky

Dslow

Attempts:

3 left

💡 Hint

Common Mistakes

Using @pytest.mark.skip will skip the test instead of marking it flaky.

Using @pytest.mark.parametrize is for running tests with multiple inputs.

Using @pytest.mark.flaky marks the test as flaky so it can be retried or handled specially.

2fill in blank

medium

Complete the code to retry a flaky test up to 3 times using pytest-rerunfailures.

Testing Fundamentals

@pytest.mark.flaky(reruns=[1])
def test_flaky():
    assert random.choice([True, False])

Drag options to blanks, or click blank then click option'

A3

B1

C5

D0

Attempts:

3 left

💡 Hint

Common Mistakes

Setting reruns=0 disables retries.

Setting reruns=1 retries only once, which may be insufficient.

Setting reruns=3 retries the flaky test up to 3 times before failing.

3fill in blank

hard

Fix the error in the flaky test decorator syntax.

Testing Fundamentals

@pytest.mark.flaky(reruns=[1])
def test_network():
    assert ping_server()

Drag options to blanks, or click blank then click option'

A'3'

BNone

CTrue

D3

Attempts:

3 left

💡 Hint

Common Mistakes

Using a string '3' causes a type error.

Using True or None is invalid for reruns parameter.

reruns expects an integer, so 3 without quotes is correct.

4fill in blank

hard

Fill both blanks to create a dictionary of test names and their flaky status.

Testing Fundamentals

flaky_tests = {test_name: [1] for test_name in tests if tests[test_name] [2] True}

Drag options to blanks, or click blank then click option'

Atests[test_name]

Btest_name

C==

D!=

Attempts:

3 left

💡 Hint

Common Mistakes

Using test_name as value instead of flaky status.

Using '!=' instead of '==' causes wrong filtering.

We use tests[test_name] to get the flaky status and check if it equals True.

5fill in blank

hard

Fill both blanks to filter flaky tests and print their names.

Testing Fundamentals

for test, is_flaky in tests.items():
    if is_flaky [1] True:
        print(test,'Flaky test detected: '[2] test)

Drag options to blanks, or click blank then click option'

A==

B!=

C+

D,

Attempts:

3 left

💡 Hint

Common Mistakes

Using '!=' reverses the condition.

Using '+' instead of ',' in print causes syntax errors.

Using ',' instead of '+' for string concatenation inside print.

We check if is_flaky == True, then print with comma to separate, and use + to concatenate strings.

Practice

1. What is a flaky test in software testing?

easy

A. A test that always passes successfully

B. A test that sometimes passes and sometimes fails without code changes

C. A test that is never executed

D. A test that always fails due to a bug

Solution

1. Step 1: Understand flaky test behavior

   A flaky test is one that shows inconsistent results, passing or failing randomly without changes in the code.

2. Step 2: Compare options to definition

   Only A test that sometimes passes and sometimes fails without code changes describes this behavior correctly; others describe consistent outcomes or unrelated cases.

3. Final Answer:

   A test that sometimes passes and sometimes fails without code changes -> Option B

4. Quick Check:

   Flaky test = random pass/fail [OK]

Hint: Flaky tests fail unpredictably without code changes [OK]

Common Mistakes:

• Thinking flaky tests always fail
• Confusing flaky tests with skipped tests
• Assuming flaky tests are due to code bugs only

2. Which of the following is the correct way to mark a flaky test in Python using pytest?

easy

A. @pytest.mark.flaky

B. @pytest.flaky()

C. @flaky_test

D. @pytest.mark.flaky(reruns=3)

Solution

1. Step 1: Recall pytest flaky test marking syntax

   Pytest uses decorators with @pytest.mark.flaky and parameters like reruns to mark flaky tests.

2. Step 2: Identify correct syntax among options

   @pytest.mark.flaky(reruns=3) correctly uses @pytest.mark.flaky(reruns=3) to rerun flaky tests up to 3 times.

3. Final Answer:

   @pytest.mark.flaky(reruns=3) -> Option D

4. Quick Check:

   pytest flaky mark = @pytest.mark.flaky(reruns=3) [OK]

Hint: Look for decorator with reruns parameter in pytest [OK]

Common Mistakes:

• Using incorrect decorator syntax
• Missing reruns parameter for flaky tests
• Confusing pytest.mark with other decorators

3. Consider this pseudocode for a flaky test management system:

tests = ['testA', 'testB', 'testC']
flaky_tests = set()

for test in tests:
    result = run_test(test)
    if result == 'fail':
        flaky_tests.add(test)

print(sorted(flaky_tests))

If run_test('testA') returns 'pass', run_test('testB') returns 'fail', and run_test('testC') returns 'fail', what will be printed?

medium

A. ['testB', 'testC']

B. ['testA', 'testB', 'testC']

C. ['testA']

D. []

Solution

1. Step 1: Analyze test results and flaky test additions

   Only tests with 'fail' results are added to flaky_tests set: testB and testC.

2. Step 2: Sort and print flaky_tests

   Sorted flaky_tests will be ['testB', 'testC'].

3. Final Answer:

   ['testB', 'testC'] -> Option A

4. Quick Check:

   Failed tests = ['testB', 'testC'] [OK]

Hint: Add only failed tests to flaky set, then sort [OK]

Common Mistakes:

• Including passed tests in flaky list
• Not sorting the output
• Confusing set behavior with list

4. You have a flaky test marked with a retry decorator, but it still fails intermittently. What is the most likely cause?

medium

A. The retry count is too low to cover all flaky failures

B. The test code has a syntax error

C. The test is not marked as flaky

D. The test always passes and never fails

Solution

1. Step 1: Understand retry decorator purpose

   Retry decorators rerun flaky tests a set number of times to reduce false failures.

2. Step 2: Identify cause of intermittent failure despite retries

   If failures persist, likely the retry count is too low to catch all flaky cases.

3. Final Answer:

   The retry count is too low to cover all flaky failures -> Option A

4. Quick Check:

   Low retry count causes flaky failures [OK]

Hint: Check retry count if flaky test still fails [OK]

Common Mistakes:

• Assuming syntax error causes flaky behavior
• Forgetting to mark test as flaky
• Ignoring retry count limits

5. A team wants to manage flaky tests without blocking releases. Which approach best balances fixing flaky tests and maintaining release flow?

hard

A. Run flaky tests only once before release

B. Remove flaky tests permanently to avoid failures

C. Temporarily mark flaky tests to skip during release, then fix them later

D. Ignore flaky tests and release as usual

Solution

1. Step 1: Understand impact of flaky tests on releases

   Flaky tests can block releases if they fail randomly, so managing them is critical.

2. Step 2: Evaluate options for balancing fixes and release flow

   Temporarily marking flaky tests to skip during release avoids blocking while allowing later fixes.

3. Final Answer:

   Temporarily mark flaky tests to skip during release, then fix them later -> Option C

4. Quick Check:

   Skip flaky tests temporarily to keep releases smooth [OK]

Hint: Skip flaky tests temporarily, fix later to keep releases smooth [OK]

Common Mistakes:

• Deleting flaky tests instead of fixing
• Ignoring flaky tests causing release failures
• Running flaky tests only once risking missed failures