Flaky test detection and retry in PyTest - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is a flaky test?

A flaky test is a test that sometimes passes and sometimes fails without any changes in the code. It behaves unpredictably.

beginner

Why is flaky test detection important?

Detecting flaky tests helps keep the test suite reliable. It prevents false failures that waste time and hide real problems.

intermediate

How can pytest help retry flaky tests?

Pytest can use the 'pytest-rerunfailures' plugin to automatically rerun failed tests a set number of times before marking them as failed.

intermediate

Show a simple pytest example to retry a flaky test up to 3 times.

Use the decorator @pytest.mark.flaky(reruns=3) on the test function. This reruns the test up to 3 times if it fails.

advanced

What is a good practice to reduce flaky tests besides retrying?

Fix the root cause of flakiness by improving test isolation, avoiding timing issues, and using mocks or stubs for unstable dependencies.

What does a flaky test do?

A Always fails

B Sometimes passes and sometimes fails without code changes

C Always passes

D Runs faster than other tests

A flaky test is unpredictable and can pass or fail randomly.

Which pytest plugin helps retry failed tests?

A pytest-html

B pytest-cov

C pytest-mock

D pytest-rerunfailures

pytest-rerunfailures reruns failed tests automatically.

How do you specify retry count for flaky tests in pytest?

A @pytest.mark.flaky(reruns=3)

B pytest.retry(3)

C pytest.flaky(3)

D pytest.rerun(3)

The @pytest.mark.flaky decorator with reruns sets retry count.

What is NOT a good way to reduce flaky tests?

A Fix timing issues

B Use mocks for unstable parts

C Ignore flaky tests forever

D Improve test isolation

Ignoring flaky tests does not solve the problem.

If a test fails once but passes on retry, what does it indicate?

A Possible flakiness

B Test is perfect

C Code is broken

D Test is slow

Passing on retry suggests the test might be flaky.

Explain what a flaky test is and why it is a problem in software testing.

Think about tests that don't behave the same every time.

Describe how you can use pytest to detect and retry flaky tests.

Focus on the tools and settings in pytest.