Performance metrics (response time, throughput) in Testing Fundamentals - Framework Patterns

Framework Mode - Performance metrics (response time, throughput)

Folder Structure

performance-testing-project/
├── tests/
│   ├── load_tests/
│   │   ├── user_login_load_test.py
│   │   └── api_throughput_test.py
│   └── stress_tests/
│       └── high_traffic_stress_test.py
├── reports/
│   └── performance_reports/
├── utils/
│   ├── metrics_collector.py
│   └── data_generator.py
├── config/
│   ├── environments.yaml
│   └── test_settings.yaml
└── conftest.py
  

Test Framework Layers

• Test Scripts Layer: Contains performance test cases like load tests and stress tests that simulate user behavior and measure metrics.
• Metrics Collection Layer: Utilities to collect and calculate performance metrics such as response time and throughput.
• Configuration Layer: Holds environment settings, test parameters, and thresholds for performance goals.
• Reporting Layer: Generates readable reports and dashboards showing test results and metric trends.
• Data Generation Layer: Provides test data or user scenarios to simulate realistic load.

Configuration Patterns

• Environment Files: Use YAML or JSON files to define different environments (dev, staging, production) with URLs and credentials.
• Test Settings: Define parameters like number of virtual users, ramp-up time, test duration, and thresholds for acceptable response times.
• Browser or Protocol Settings: Configure HTTP headers, authentication tokens, or browser types if needed for web performance tests.
• Centralized Config Access: Use a config loader utility to read and provide config values to tests and utilities consistently.

Test Reporting and CI/CD Integration

• Automated Reports: Generate HTML or JSON reports showing key metrics like average response time, max response time, throughput, and error rates.
• Trend Analysis: Store historical results to compare performance over time and detect regressions.
• CI/CD Integration: Integrate performance tests into pipelines to run on each build or release, with pass/fail criteria based on thresholds.
• Alerts: Configure notifications if performance degrades beyond acceptable limits.

Framework Design Principles

• Separate Metrics Collection from Test Logic: Keep metric calculation in utilities to reuse and maintain easily.
• Use Realistic Load Patterns: Simulate real user behavior to get meaningful performance data.
• Parameterize Tests: Allow easy adjustment of load size, duration, and environment without code changes.
• Clear Thresholds for Pass/Fail: Define acceptable limits for response time and throughput to automate result evaluation.
• Maintain Historical Data: Keep past results to track performance trends and catch regressions early.

Self Check

Where in this framework structure would you add a new utility to calculate the 95th percentile response time from raw test data?

Key Result

Organize performance tests with clear layers for tests, metrics collection, configuration, and reporting to measure response time and throughput effectively.

Practice

1. What does response time measure in performance testing?

easy

A. The total number of requests per second

B. How fast a single request is completed

C. The amount of data transferred in a request

D. The number of users accessing the system

Solution

1. Step 1: Understand response time meaning

   Response time measures the duration taken to complete one request from start to finish.

2. Step 2: Differentiate from throughput

   Throughput measures how many requests happen over time, not the speed of one request.

3. Final Answer:

   How fast a single request is completed -> Option B

4. Quick Check:

   Response time = speed of one request [OK]

Hint: Response time = speed of one request [OK]

Common Mistakes:

• Confusing response time with throughput
• Thinking response time counts total requests
• Mixing data size with response time

2. Which of the following is the correct unit commonly used to measure throughput?

easy

A. Milliseconds (ms)

B. Bytes

C. Requests per second (RPS)

D. Percentage (%)

Solution

1. Step 1: Identify throughput unit

   Throughput measures how many requests are processed per unit time, commonly requests per second (RPS).

2. Step 2: Exclude other units

   Milliseconds measure time, bytes measure data size, and percentage is not a throughput unit.

3. Final Answer:

   Requests per second (RPS) -> Option C

4. Quick Check:

   Throughput = requests per second [OK]

Hint: Throughput counts requests per second [OK]

Common Mistakes:

• Using time units for throughput
• Confusing data size with throughput
• Selecting percentage as throughput unit

3. Given a system that processes 120 requests in 10 seconds, what is the throughput?

medium

A. 10 requests per second

B. 0.12 requests per second

C. 1200 requests per second

D. 12 requests per second

Solution

1. Step 1: Calculate throughput formula

   Throughput = Total requests / Total time = 120 / 10 seconds.

2. Step 2: Perform division

   120 divided by 10 equals 12 requests per second.

3. Final Answer:

   12 requests per second -> Option D

4. Quick Check:

   120 ÷ 10 = 12 [OK]

Hint: Divide total requests by total time [OK]

Common Mistakes:

• Dividing time by requests instead of requests by time
• Multiplying requests and time
• Confusing throughput with response time

4. A tester recorded the following response times (in ms): 100, 200, 150, 250. The average response time was incorrectly calculated as 700 ms. What is the correct average response time?

medium

A. 175 ms

B. 700 ms

C. 150 ms

D. 200 ms

Solution

1. Step 1: Sum all response times

   Add 100 + 200 + 150 + 250 = 700 ms total.

2. Step 2: Calculate average response time

   Divide total 700 ms by 4 requests = 175 ms average.

3. Final Answer:

   175 ms -> Option A

4. Quick Check:

   700 ÷ 4 = 175 [OK]

Hint: Sum times, then divide by count [OK]

Common Mistakes:

• Using total sum as average
• Dividing by wrong number of requests
• Mixing units in calculation

5. A web server has a throughput of 50 requests per second and an average response time of 200 ms. If the server's throughput doubles but response time remains the same, what happens to the server's capacity and user experience?

hard

A. Capacity doubles; user experience stays consistent

B. Capacity stays same; user experience worsens

C. Capacity doubles; user experience worsens

D. Capacity halves; user experience improves

Solution

1. Step 1: Understand throughput doubling

   Doubling throughput means the server can handle twice as many requests per second, so capacity doubles.

2. Step 2: Analyze response time unchanged

   Since response time stays at 200 ms, the speed per request remains consistent, so user experience does not worsen.

3. Final Answer:

   Capacity doubles; user experience stays consistent -> Option A

4. Quick Check:

   Throughput ↑, response time same = better capacity, same experience [OK]

Hint: More throughput + same response time = better capacity, same experience [OK]

Common Mistakes:

• Assuming response time worsens with throughput increase
• Confusing capacity with response time
• Thinking capacity decreases when throughput increases