Medallion architecture (bronze, silver, gold) in Apache Spark - Deep Dive

Overview - Medallion architecture (bronze, silver, gold)

What is it?

Medallion architecture is a way to organize data in layers to make it easier to clean, improve, and use. It divides data into three stages: bronze, silver, and gold. Bronze holds raw data as it arrives, silver contains cleaned and enriched data, and gold stores data ready for analysis or reporting. This structure helps teams work with data step-by-step.

Why it matters

Without this layered approach, data can be messy and hard to trust, making analysis slow and error-prone. Medallion architecture solves this by organizing data so each step improves quality and usability. This means faster insights, better decisions, and less time fixing problems. It helps companies turn raw data into valuable knowledge efficiently.

Where it fits

Learners should know basic data storage and processing concepts before this. Understanding data cleaning and transformation helps. After this, learners can explore advanced data pipelines, machine learning workflows, and real-time analytics that build on clean, organized data.

Mental Model

Core Idea

Medallion architecture organizes data into three layers—bronze, silver, and gold—each adding more quality and value to prepare data for analysis.

Think of it like...

It's like making a meal: bronze is gathering raw ingredients, silver is prepping and cooking them, and gold is plating the dish ready to serve.

┌─────────────┐
│   Bronze    │  Raw data, unprocessed
├─────────────┤
│   Silver    │  Cleaned and enriched data
├─────────────┤
│    Gold     │  Final, trusted data for use
└─────────────┘

Build-Up - 6 Steps

1

Foundation

Understanding raw data ingestion

Concept: Introduce the first layer, bronze, which stores raw data exactly as it arrives.

Bronze layer is where data lands first. It keeps data in its original form without changes. This helps preserve the source data for auditing or reprocessing. For example, logs or sensor data are saved here as files or tables.

Result

You have a safe copy of all incoming data, even if it is messy or incomplete.

Knowing that raw data is preserved prevents accidental data loss and supports traceability.

2

Foundation

Basics of data cleaning and enrichment

Concept: Introduce the silver layer where data is cleaned and improved for better quality.

Silver layer takes bronze data and fixes errors, fills missing values, and adds useful info. For example, converting timestamps to a standard format or joining with reference data. This makes data easier to analyze.

Result

Data becomes more reliable and consistent, ready for deeper processing.

Understanding cleaning as a separate step helps isolate errors and improves trust in data.

3

Intermediate

Creating trusted datasets in gold layer

🤔Before reading on: do you think gold layer data is raw or fully processed? Commit to your answer.

Concept: Gold layer holds final, high-quality data ready for business use and reporting.

Gold data is aggregated, filtered, and formatted for specific uses like dashboards or machine learning. It often combines multiple silver datasets. This layer is what analysts and apps consume.

Result

You get clean, fast, and reliable data tailored for decision-making.

Knowing gold is the final product clarifies why it must be accurate and performant.

4

Intermediate

Data flow and transformations between layers

🤔Before reading on: do you think data flows one-way or can it move back between layers? Commit to your answer.

Concept: Data moves from bronze to silver to gold through transformations that add value at each step.

Bronze data is read and processed to create silver data, which is then further refined into gold. Each step uses Spark jobs or pipelines to transform data. Sometimes feedback loops fix earlier layers if needed.

Result

A clear, repeatable process ensures data quality improves progressively.

Understanding the flow helps design reliable pipelines and troubleshoot issues.

5

Advanced

Implementing medallion architecture with Apache Spark

🤔Before reading on: do you think Spark handles all layers the same way or differently? Commit to your answer.

Concept: Use Spark to build scalable pipelines that read, transform, and write data at each medallion layer.

Spark reads raw data into DataFrames for bronze, applies cleaning functions for silver, and runs aggregations for gold. Data is saved in Delta Lake tables for reliability and versioning. This setup supports incremental updates and schema enforcement.

Result

You get efficient, fault-tolerant data pipelines that scale with data size.

Knowing Spark's role in each layer reveals how to optimize performance and maintain data integrity.

6

Expert

Handling schema evolution and data quality surprises

🤔Before reading on: do you think schema changes break all layers or can they be managed smoothly? Commit to your answer.

Concept: Medallion architecture must handle changes in data structure and unexpected data issues gracefully.

Delta Lake supports schema evolution allowing silver and gold layers to adapt to new fields without breaking. Quality checks and monitoring catch anomalies early. Automated tests and alerts help maintain pipeline health.

Result

Data pipelines remain stable and trustworthy even as data changes over time.

Understanding schema evolution and monitoring is key to running medallion pipelines in production.

Under the Hood

Medallion architecture uses layered storage and processing. Raw data is ingested into immutable bronze tables. Spark jobs read bronze data, apply transformations, and write cleaned silver tables. Further aggregation and filtering produce gold tables. Delta Lake manages data versions and schema changes, enabling incremental updates and rollback. This layered approach isolates errors and improves data quality stepwise.

Why designed this way?

It was designed to solve the problem of messy, untrustworthy data slowing analytics. Early data lakes stored raw data but lacked structure, causing confusion. Medallion architecture adds order and quality control by layering data and using Delta Lake for reliability. Alternatives like single-layer lakes or direct ETL to final tables were less flexible and harder to maintain.

┌─────────────┐       ┌─────────────┐       ┌─────────────┐
│   Bronze    │──────▶│   Silver    │──────▶│    Gold     │
│ (Raw data) │       │(Cleaned data)│       │(Trusted data)│
└─────────────┘       └─────────────┘       └─────────────┘
       │                    │                     │
       ▼                    ▼                     ▼
  Delta Lake           Delta Lake            Delta Lake
  (Versioned)          (Versioned)           (Versioned)

Myth Busters - 4 Common Misconceptions

Quick: Is bronze data always clean and ready for analysis? Commit yes or no.

Common Belief: Bronze data is clean enough to use directly for reports.

Reality: Bronze data is raw and often messy; it needs cleaning in silver before use.

Why it matters: Using bronze data directly can cause wrong insights due to errors or missing values.

Quick: Does gold layer contain raw data or processed summaries? Commit your answer.

Common Belief: Gold layer stores raw data just like bronze but in a different place.

Reality: Gold layer contains processed, aggregated, and trusted data ready for business use.

Why it matters: Confusing gold with raw data leads to poor performance and unreliable analytics.

Quick: Can data flow backward from gold to bronze layers? Commit yes or no.

Common Belief: Data flows only forward from bronze to gold without feedback.

Reality: Sometimes data issues require going back to fix bronze or silver layers.

Why it matters: Ignoring backward fixes can cause persistent errors and pipeline failures.

Quick: Does schema evolution automatically break all medallion layers? Commit yes or no.

Common Belief: Any schema change breaks the entire pipeline and requires manual fixes everywhere.

Reality: Delta Lake supports schema evolution allowing smooth updates without full breaks.

Why it matters: Believing schema changes always break pipelines causes unnecessary fear and delays.

Expert Zone

1

Silver layer often contains multiple versions of cleaned data to support different use cases and rollback.

2

Gold layer design balances between denormalized tables for speed and normalized tables for flexibility.

3

Incremental data processing in medallion pipelines requires careful watermarking and state management to avoid duplicates.

When NOT to use

Medallion architecture is less suitable for very small datasets or simple one-step transformations where layering adds overhead. Alternatives include direct ETL pipelines or simple data marts without multiple layers.

Production Patterns

In production, medallion pipelines use Delta Lake with Spark Structured Streaming for near real-time updates. Automated quality checks run after each layer. Data catalogs and lineage tools track data flow. Teams use CI/CD to deploy pipeline changes safely.

Connections

ETL (Extract, Transform, Load)

Medallion architecture builds on ETL by structuring transformations into clear layers.

Understanding ETL helps grasp how medallion layers separate extraction, cleaning, and loading into manageable steps.

Software Development Staging Environments

Medallion layers resemble staging environments: bronze is development, silver is testing, gold is production.

Knowing staging environments clarifies why data quality improves stepwise before final use.

Manufacturing Assembly Line

Medallion architecture parallels assembly lines where raw materials are processed in stages to finished products.

Seeing data as a product moving through stages helps design efficient, quality-focused pipelines.

Common Pitfalls

#1 Skipping the silver layer and trying to clean data directly in gold.

Wrong approach:

spark.read.format('delta').load('bronze_path').write.format('delta').save('gold_path')

Correct approach:

val silverDF = spark.read.format('delta').load('bronze_path').transform(cleaningFunctions)
silverDF.write.format('delta').save('silver_path')
val goldDF = spark.read.format('delta').load('silver_path').transform(aggregationFunctions)
goldDF.write.format('delta').save('gold_path')

Root cause:

Misunderstanding the purpose of each layer leads to mixing raw and cleaned data, causing unreliable results.

#2 Not using Delta Lake and storing data as plain files without versioning.

Wrong approach:

spark.read.json('bronze_path').write.json('silver_path')

Correct approach:

spark.read.format('delta').load('bronze_path').write.format('delta').save('silver_path')

Root cause:

Ignoring versioning and schema enforcement causes data corruption and difficult debugging.

#3 Hardcoding schema and ignoring schema evolution.

Wrong approach:

spark.read.schema(fixedSchema).json('bronze_path')

Correct approach:

spark.read.format('delta').option('mergeSchema', 'true').load('bronze_path')

Root cause:

Assuming schema never changes leads to pipeline failures when data evolves.

Key Takeaways

Medallion architecture organizes data into bronze, silver, and gold layers to improve quality step-by-step.

Bronze stores raw data, silver cleans and enriches it, and gold prepares trusted data for analysis.

Using Apache Spark and Delta Lake enables scalable, reliable pipelines with versioning and schema evolution.

Understanding data flow and transformations between layers helps build maintainable and efficient data systems.

Handling schema changes and monitoring data quality are essential for production-ready medallion pipelines.

Practice

1. What is the primary purpose of the bronze layer in the Medallion architecture?

easy

A. Store metadata about data quality

B. Store cleaned and enriched data

C. Store raw, unprocessed data as it arrives

D. Store aggregated business-level data

Solution

1. Step 1: Understand the role of the bronze layer

   The bronze layer is the first stage where raw data is ingested without transformation.

2. Step 2: Differentiate from silver and gold layers

   Silver contains cleaned data, and gold contains aggregated data, so they are not raw.

3. Final Answer:

   Store raw, unprocessed data as it arrives -> Option C

4. Quick Check:

   Bronze = Raw data [OK]

Hint: Bronze = raw data layer, first step [OK]

Common Mistakes:

• Confusing bronze with silver or gold layers
• Thinking bronze stores aggregated data
• Assuming bronze is cleaned data

2. Which of the following is the correct order of layers in the Medallion architecture from raw to business-ready data?

easy

A. Bronze -> Silver -> Gold

B. Silver -> Bronze -> Gold

C. Gold -> Silver -> Bronze

D. Silver -> Gold -> Bronze

Solution

1. Step 1: Recall the Medallion architecture flow

   Data flows from raw to cleaned to aggregated: bronze, then silver, then gold.

2. Step 2: Match the order to options

   Bronze -> Silver -> Gold correctly lists bronze first, then silver, then gold.

3. Final Answer:

   Bronze -> Silver -> Gold -> Option A

4. Quick Check:

   Order = Bronze, Silver, Gold [OK]

Hint: Remember: Bronze first, then Silver, then Gold [OK]

Common Mistakes:

• Reversing the order of layers
• Mixing silver and bronze positions
• Assuming gold comes before silver

3. Given the following PySpark code snippet, which layer does the resulting DataFrame most likely represent?

bronzeDF = spark.read.format('json').load('/data/raw/events')
silverDF = bronzeDF.filter('event_type IS NOT NULL').dropDuplicates()
goldDF = silverDF.groupBy('event_type').count()

medium

A. goldDF is gold layer

B. silverDF is bronze layer

C. bronzeDF is gold layer

D. bronzeDF is silver layer

Solution

1. Step 1: Analyze each DataFrame's transformation

   bronzeDF loads raw JSON data (raw data = bronze). silverDF filters and removes duplicates (cleaned data = silver). goldDF groups and counts (aggregated data = gold).

2. Step 2: Identify the layer of goldDF

   goldDF performs aggregation, which matches the gold layer's purpose.

3. Final Answer:

   goldDF is gold layer -> Option A

4. Quick Check:

   Aggregation = Gold layer [OK]

Hint: Aggregation means gold layer [OK]

Common Mistakes:

• Confusing raw load as silver
• Thinking filtering is aggregation
• Mixing up silver and gold transformations

4. You have a silver layer DataFrame but notice it still contains duplicate records. Which change will best fix this issue in your Spark code?

medium

A. Add .dropDuplicates() after loading raw data in bronze layer

B. Add .dropDuplicates() after cleaning transformations in silver layer

C. Add .groupBy() in bronze layer to aggregate data

D. Add .filter() in gold layer to remove duplicates

Solution

1. Step 1: Understand where duplicates should be removed

   Duplicates should be removed during cleaning, which happens in the silver layer.

2. Step 2: Evaluate options for removing duplicates

   Adding .dropDuplicates() in silver layer after cleaning transformations is the correct fix.

3. Final Answer:

   Add .dropDuplicates() after cleaning transformations in silver layer -> Option B

4. Quick Check:

   Remove duplicates in silver layer [OK]

Hint: Clean data (silver) removes duplicates [OK]

Common Mistakes:

• Removing duplicates too early in bronze layer
• Using groupBy instead of dropDuplicates
• Filtering duplicates only in gold layer

5. You want to build a dashboard showing total sales by region updated daily. Which Medallion architecture layer should you query for the most accurate and business-ready data?

hard

A. Query the bronze layer for raw sales data

B. Query the bronze layer after filtering duplicates

C. Query the silver layer for cleaned sales data

D. Query the gold layer for aggregated sales data

Solution

1. Step 1: Identify the dashboard data needs

   The dashboard requires aggregated, business-ready data (total sales by region).

2. Step 2: Match data needs to Medallion layers

   Gold layer contains aggregated and business-level data, ideal for dashboards.

3. Final Answer:

   Query the gold layer for aggregated sales data -> Option D

4. Quick Check:

   Dashboard needs gold layer data [OK]

Hint: Dashboards use gold layer data [OK]

Common Mistakes:

• Using raw or cleaned data instead of aggregated
• Querying bronze layer directly for reports
• Filtering bronze layer instead of using gold