Integration testing pipelines in Apache Spark - Step-by-Step Execution

Concept Flow - Integration testing pipelines

Start: Define pipeline components

↓

Write integration tests

↓

Set up test data

↓

Run pipeline on test data

↓

Check outputs vs expected

↓

Pass?

No→Log errors & debug

Yes↓

Pipeline integration verified

↓

End

Integration testing pipelines means running the whole data flow with test data to check if all parts work together correctly.

Execution Sample

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
input_df = spark.createDataFrame([(1, 'a'), (2, 'b')], ['id', 'value'])
result_df = pipeline_function(input_df)
expected_output = [(1, 'a_processed'), (2, 'b_processed')]
assert result_df.collect() == expected_output

This code runs a pipeline function on test data and checks if the output matches what we expect.

Execution Table

Step	Action	Input Data	Output Data	Check Result
1	Create test input DataFrame	[{id:1,value:'a'},{id:2,value:'b'}]	DataFrame with 2 rows	N/A
2	Run pipeline_function on input_df	DataFrame with 2 rows	Processed DataFrame	N/A
3	Collect output to list	Processed DataFrame	[Expected rows]	N/A
4	Compare output with expected_output	[Expected rows]	[Expected rows]	Pass if equal
5	Test ends	N/A	N/A	Pass - pipeline integration works

💡 Test ends after output matches expected results, confirming pipeline integration success.

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
input_df	None	DataFrame with 2 rows	DataFrame with 2 rows	DataFrame with 2 rows	DataFrame with 2 rows
result_df	None	None	Processed DataFrame	Processed DataFrame	Processed DataFrame
output_list	None	None	None	[Expected rows]	[Expected rows]

Key Moments - 2 Insights

Why do we collect the DataFrame to a list before comparing?

Because Spark DataFrames are lazy and distributed, collecting converts them to local data for easy comparison, as shown in step 3 of the execution_table.

What if the output does not match expected_output?

The test fails and you log errors to debug the pipeline, as indicated by the 'No' branch after 'Pass?' in the concept_flow.

Visual Quiz - 3 Questions

Look at the execution_table, what is the output after step 2?

A List of expected rows

B Test input DataFrame

C Processed DataFrame

D None

💡 Hint

Check the 'Output Data' column in row 2 of the execution_table.

At which step do we verify if the pipeline output matches the expected output?

A Step 4

B Step 3

C Step 1

D Step 5

💡 Hint

Look at the 'Check Result' column in the execution_table for the step that says 'Pass if equal'.

If the test input had 3 rows instead of 2, how would the variable_tracker change after Step 1?

A result_df would have 3 rows

B input_df would have 3 rows

C output_list would be empty

D No change in input_df

💡 Hint

Variable_tracker shows input_df after Step 1 as the test input DataFrame size.

Concept Snapshot

Integration testing pipelines:
- Run full pipeline on test data
- Collect output locally
- Compare output with expected
- Pass means all parts work together
- Fail means debug pipeline steps

Full Transcript

Integration testing pipelines means running the entire data pipeline with prepared test data to check if all parts work together correctly. We start by defining the pipeline components and writing integration tests. Then we set up test data and run the pipeline on it. After running, we collect the output data locally and compare it with the expected results. If they match, the test passes and confirms the pipeline integration works. If not, we log errors and debug. The execution table shows step-by-step actions: creating test input, running the pipeline, collecting output, comparing results, and ending the test. Variables like input_df and result_df change as the pipeline runs. Collecting the DataFrame to a list is important because Spark DataFrames are lazy and distributed, so we need local data to compare easily. If output does not match expected, the test fails and debugging is needed. This process ensures the pipeline works end-to-end as expected.