Medallion architecture (bronze, silver, gold) in Apache Spark - Step-by-Step Execution

Concept Flow - Medallion architecture (bronze, silver, gold)

Raw Data Ingested

↓

Bronze Layer: Raw Storage

↓

Silver Layer: Cleaned & Enriched

↓

Gold Layer: Aggregated & Business Ready

↓

Data Used for Analytics & Reporting

Data flows from raw ingestion (bronze) to cleaning/enrichment (silver), then to aggregation/business-ready form (gold) for analysis.

Execution Sample

Apache Spark

bronzeDF = spark.read.format('json').load('/data/raw')
silverDF = bronzeDF.filter('age > 18').dropDuplicates()
goldDF = silverDF.groupBy('country').count()

This code loads raw data into bronze, cleans it in silver, and aggregates counts by country in gold.

Execution Table

Step	Action	DataFrame	Rows Count	Description
1	Load raw JSON data	bronzeDF	1000	Raw data loaded with all records
2	Filter age > 18	silverDF	700	Removed records with age <= 18
3	Drop duplicates	silverDF	680	Removed duplicate records
4	Group by country and count	goldDF	10	Aggregated counts per country
5	End	-	-	Data ready for analytics

💡 Data processing stops after gold layer aggregation for analytics use.

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4
bronzeDF	empty	1000 rows raw data	1000 rows raw data	1000 rows raw data	1000 rows raw data
silverDF	undefined	undefined	700 rows filtered	680 rows deduplicated	680 rows deduplicated
goldDF	undefined	undefined	undefined	undefined	10 rows aggregated

Key Moments - 2 Insights

Why do we keep the bronze layer data even after cleaning in silver?

The bronze layer stores raw data as-is for traceability and reprocessing if needed, as shown in execution_table step 1 where bronzeDF remains unchanged after silver processing.

What happens if we skip dropping duplicates in silver?

Skipping duplicates removal would cause gold layer aggregations to count repeated records, leading to inaccurate results, as seen in execution_table step 3 where duplicates are removed.

Visual Quiz - 3 Questions

Look at the execution table, how many rows does silverDF have after filtering age > 18?

A 700

B 680

C 1000

D 10

💡 Hint

Check execution_table row 2 under Rows Count for silverDF after filtering.

At which step does the data get aggregated by country?

A Step 2

B Step 3

C Step 4

D Step 1

💡 Hint

Look at execution_table step 4 description about grouping and counting.

If duplicates were not dropped in silver, what would likely happen to goldDF row count?

A It would stay the same

B It would increase

C It would decrease

D It would become zero

💡 Hint

Refer to variable_tracker silverDF after step 3 and goldDF after step 4 for row counts.

Concept Snapshot

Medallion architecture organizes data in 3 layers:
- Bronze: raw data storage
- Silver: cleaned and enriched data
- Gold: aggregated, business-ready data
Each layer builds on the previous for better quality and usability.

Full Transcript

The Medallion architecture organizes data processing into three layers: bronze, silver, and gold. First, raw data is ingested into the bronze layer without changes. Next, the silver layer cleans and enriches this data, for example by filtering and removing duplicates. Finally, the gold layer aggregates the cleaned data to produce business-ready datasets for analysis. This stepwise approach helps keep raw data safe, improves data quality, and prepares data for reporting. The example code loads raw JSON data into bronze, filters and deduplicates in silver, then groups by country in gold. The execution table shows how row counts change at each step. Key points include why raw data is kept in bronze and the importance of removing duplicates before aggregation. The visual quiz tests understanding of these steps and their effects on data.

Practice

1. What is the primary purpose of the bronze layer in the Medallion architecture?

easy

A. Store metadata about data quality

B. Store cleaned and enriched data

C. Store raw, unprocessed data as it arrives

D. Store aggregated business-level data

Solution

1. Step 1: Understand the role of the bronze layer

   The bronze layer is the first stage where raw data is ingested without transformation.

2. Step 2: Differentiate from silver and gold layers

   Silver contains cleaned data, and gold contains aggregated data, so they are not raw.

3. Final Answer:

   Store raw, unprocessed data as it arrives -> Option C

4. Quick Check:

   Bronze = Raw data [OK]

Hint: Bronze = raw data layer, first step [OK]

Common Mistakes:

• Confusing bronze with silver or gold layers
• Thinking bronze stores aggregated data
• Assuming bronze is cleaned data

2. Which of the following is the correct order of layers in the Medallion architecture from raw to business-ready data?

easy

A. Bronze -> Silver -> Gold

B. Silver -> Bronze -> Gold

C. Gold -> Silver -> Bronze

D. Silver -> Gold -> Bronze

Solution

1. Step 1: Recall the Medallion architecture flow

   Data flows from raw to cleaned to aggregated: bronze, then silver, then gold.

2. Step 2: Match the order to options

   Bronze -> Silver -> Gold correctly lists bronze first, then silver, then gold.

3. Final Answer:

   Bronze -> Silver -> Gold -> Option A

4. Quick Check:

   Order = Bronze, Silver, Gold [OK]

Hint: Remember: Bronze first, then Silver, then Gold [OK]

Common Mistakes:

• Reversing the order of layers
• Mixing silver and bronze positions
• Assuming gold comes before silver

3. Given the following PySpark code snippet, which layer does the resulting DataFrame most likely represent?

bronzeDF = spark.read.format('json').load('/data/raw/events')
silverDF = bronzeDF.filter('event_type IS NOT NULL').dropDuplicates()
goldDF = silverDF.groupBy('event_type').count()

medium

A. goldDF is gold layer

B. silverDF is bronze layer

C. bronzeDF is gold layer

D. bronzeDF is silver layer

Solution

1. Step 1: Analyze each DataFrame's transformation

   bronzeDF loads raw JSON data (raw data = bronze). silverDF filters and removes duplicates (cleaned data = silver). goldDF groups and counts (aggregated data = gold).

2. Step 2: Identify the layer of goldDF

   goldDF performs aggregation, which matches the gold layer's purpose.

3. Final Answer:

   goldDF is gold layer -> Option A

4. Quick Check:

   Aggregation = Gold layer [OK]

Hint: Aggregation means gold layer [OK]

Common Mistakes:

• Confusing raw load as silver
• Thinking filtering is aggregation
• Mixing up silver and gold transformations

4. You have a silver layer DataFrame but notice it still contains duplicate records. Which change will best fix this issue in your Spark code?

medium

A. Add .dropDuplicates() after loading raw data in bronze layer

B. Add .dropDuplicates() after cleaning transformations in silver layer

C. Add .groupBy() in bronze layer to aggregate data

D. Add .filter() in gold layer to remove duplicates

Solution

1. Step 1: Understand where duplicates should be removed

   Duplicates should be removed during cleaning, which happens in the silver layer.

2. Step 2: Evaluate options for removing duplicates

   Adding .dropDuplicates() in silver layer after cleaning transformations is the correct fix.

3. Final Answer:

   Add .dropDuplicates() after cleaning transformations in silver layer -> Option B

4. Quick Check:

   Remove duplicates in silver layer [OK]

Hint: Clean data (silver) removes duplicates [OK]

Common Mistakes:

• Removing duplicates too early in bronze layer
• Using groupBy instead of dropDuplicates
• Filtering duplicates only in gold layer

5. You want to build a dashboard showing total sales by region updated daily. Which Medallion architecture layer should you query for the most accurate and business-ready data?

hard

A. Query the bronze layer for raw sales data

B. Query the bronze layer after filtering duplicates

C. Query the silver layer for cleaned sales data

D. Query the gold layer for aggregated sales data

Solution

1. Step 1: Identify the dashboard data needs

   The dashboard requires aggregated, business-ready data (total sales by region).

2. Step 2: Match data needs to Medallion layers

   Gold layer contains aggregated and business-level data, ideal for dashboards.

3. Final Answer:

   Query the gold layer for aggregated sales data -> Option D

4. Quick Check:

   Dashboard needs gold layer data [OK]

Hint: Dashboards use gold layer data [OK]

Common Mistakes:

• Using raw or cleaned data instead of aggregated
• Querying bronze layer directly for reports
• Filtering bronze layer instead of using gold