Medallion architecture (bronze, silver, gold) in Apache Spark - Mini Project: Build & Apply

Medallion Architecture with Apache Spark

📖 Scenario: You work at a company that collects raw sales data every day. The data is messy and needs cleaning before it can be used for reports. You will organize the data using the Medallion architecture: Bronze (raw data), Silver (cleaned data), and Gold (aggregated data).

🎯 Goal: Build a simple Medallion architecture pipeline in Apache Spark. You will create a raw data DataFrame (Bronze), clean it to remove bad records (Silver), and then aggregate sales by product (Gold).

📋 What You'll Learn

Create a raw sales DataFrame called bronze_df with given data

Create a threshold variable min_quantity to filter records

Filter bronze_df to create silver_df with only records where quantity is at least min_quantity

Aggregate silver_df by product to get total quantity sold as gold_df

Print gold_df to show the final aggregated sales

💡 Why This Matters

🌍 Real World

Many companies collect raw data that is messy and incomplete. The Medallion architecture helps organize data in layers for easier cleaning and analysis.

💼 Career

Data engineers and data scientists use Medallion architecture to build reliable data pipelines that prepare data for business insights and reporting.

1

Create the Bronze DataFrame with raw sales data

Create a Spark DataFrame called bronze_df with these exact rows: {'product': 'apple', 'quantity': 10}, {'product': 'banana', 'quantity': 5}, {'product': 'orange', 'quantity': 0}, {'product': 'apple', 'quantity': 3}, {'product': 'banana', 'quantity': 8}.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('MedallionProject').getOrCreate()
# Create bronze_df DataFrame with raw sales data
# Your code here

Hint

Use spark.createDataFrame() with a list of dictionaries for the raw data.

2

Set the minimum quantity threshold

Create a variable called min_quantity and set it to 5. This will be used to filter out low quantity sales.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('MedallionProject').getOrCreate()

bronze_data = [
    {'product': 'apple', 'quantity': 10},
    {'product': 'banana', 'quantity': 5},
    {'product': 'orange', 'quantity': 0},
    {'product': 'apple', 'quantity': 3},
    {'product': 'banana', 'quantity': 8}
]

bronze_df = spark.createDataFrame(bronze_data)

# Create min_quantity variable and set to 5
# Your code here

Hint

Just create a variable min_quantity and assign the number 5.

3

Filter Bronze data to create Silver data

Create a new DataFrame called silver_df by filtering bronze_df to keep only rows where quantity is greater than or equal to min_quantity.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('MedallionProject').getOrCreate()

bronze_data = [
    {'product': 'apple', 'quantity': 10},
    {'product': 'banana', 'quantity': 5},
    {'product': 'orange', 'quantity': 0},
    {'product': 'apple', 'quantity': 3},
    {'product': 'banana', 'quantity': 8}
]

bronze_df = spark.createDataFrame(bronze_data)

min_quantity = 5

# Filter bronze_df to create silver_df with quantity >= min_quantity
# Your code here

Hint

Use filter() on bronze_df with the condition bronze_df.quantity >= min_quantity.

4

Aggregate Silver data to create Gold data and print

Create a DataFrame called gold_df by grouping silver_df by product and summing the quantity as total_quantity. Then print gold_df to show the final aggregated sales.

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import sum

spark = SparkSession.builder.appName('MedallionProject').getOrCreate()

bronze_data = [
    {'product': 'apple', 'quantity': 10},
    {'product': 'banana', 'quantity': 5},
    {'product': 'orange', 'quantity': 0},
    {'product': 'apple', 'quantity': 3},
    {'product': 'banana', 'quantity': 8}
]

bronze_df = spark.createDataFrame(bronze_data)

min_quantity = 5

silver_df = bronze_df.filter(bronze_df.quantity >= min_quantity)

# Group silver_df by product and sum quantity as total_quantity into gold_df
# Print gold_df
# Your code here

Hint

Use groupBy('product').agg(sum('quantity').alias('total_quantity')) on silver_df. Then use show() to print.

Practice

1. What is the primary purpose of the bronze layer in the Medallion architecture?

easy

A. Store metadata about data quality

B. Store cleaned and enriched data

C. Store raw, unprocessed data as it arrives

D. Store aggregated business-level data

Solution

1. Step 1: Understand the role of the bronze layer

   The bronze layer is the first stage where raw data is ingested without transformation.

2. Step 2: Differentiate from silver and gold layers

   Silver contains cleaned data, and gold contains aggregated data, so they are not raw.

3. Final Answer:

   Store raw, unprocessed data as it arrives -> Option C

4. Quick Check:

   Bronze = Raw data [OK]

Hint: Bronze = raw data layer, first step [OK]

Common Mistakes:

• Confusing bronze with silver or gold layers
• Thinking bronze stores aggregated data
• Assuming bronze is cleaned data

2. Which of the following is the correct order of layers in the Medallion architecture from raw to business-ready data?

easy

A. Bronze -> Silver -> Gold

B. Silver -> Bronze -> Gold

C. Gold -> Silver -> Bronze

D. Silver -> Gold -> Bronze

Solution

1. Step 1: Recall the Medallion architecture flow

   Data flows from raw to cleaned to aggregated: bronze, then silver, then gold.

2. Step 2: Match the order to options

   Bronze -> Silver -> Gold correctly lists bronze first, then silver, then gold.

3. Final Answer:

   Bronze -> Silver -> Gold -> Option A

4. Quick Check:

   Order = Bronze, Silver, Gold [OK]

Hint: Remember: Bronze first, then Silver, then Gold [OK]

Common Mistakes:

• Reversing the order of layers
• Mixing silver and bronze positions
• Assuming gold comes before silver

3. Given the following PySpark code snippet, which layer does the resulting DataFrame most likely represent?

bronzeDF = spark.read.format('json').load('/data/raw/events')
silverDF = bronzeDF.filter('event_type IS NOT NULL').dropDuplicates()
goldDF = silverDF.groupBy('event_type').count()

medium

A. goldDF is gold layer

B. silverDF is bronze layer

C. bronzeDF is gold layer

D. bronzeDF is silver layer

Solution

1. Step 1: Analyze each DataFrame's transformation

   bronzeDF loads raw JSON data (raw data = bronze). silverDF filters and removes duplicates (cleaned data = silver). goldDF groups and counts (aggregated data = gold).

2. Step 2: Identify the layer of goldDF

   goldDF performs aggregation, which matches the gold layer's purpose.

3. Final Answer:

   goldDF is gold layer -> Option A

4. Quick Check:

   Aggregation = Gold layer [OK]

Hint: Aggregation means gold layer [OK]

Common Mistakes:

• Confusing raw load as silver
• Thinking filtering is aggregation
• Mixing up silver and gold transformations

4. You have a silver layer DataFrame but notice it still contains duplicate records. Which change will best fix this issue in your Spark code?

medium

A. Add .dropDuplicates() after loading raw data in bronze layer

B. Add .dropDuplicates() after cleaning transformations in silver layer

C. Add .groupBy() in bronze layer to aggregate data

D. Add .filter() in gold layer to remove duplicates

Solution

1. Step 1: Understand where duplicates should be removed

   Duplicates should be removed during cleaning, which happens in the silver layer.

2. Step 2: Evaluate options for removing duplicates

   Adding .dropDuplicates() in silver layer after cleaning transformations is the correct fix.

3. Final Answer:

   Add .dropDuplicates() after cleaning transformations in silver layer -> Option B

4. Quick Check:

   Remove duplicates in silver layer [OK]

Hint: Clean data (silver) removes duplicates [OK]

Common Mistakes:

• Removing duplicates too early in bronze layer
• Using groupBy instead of dropDuplicates
• Filtering duplicates only in gold layer

5. You want to build a dashboard showing total sales by region updated daily. Which Medallion architecture layer should you query for the most accurate and business-ready data?

hard

A. Query the bronze layer for raw sales data

B. Query the bronze layer after filtering duplicates

C. Query the silver layer for cleaned sales data

D. Query the gold layer for aggregated sales data

Solution

1. Step 1: Identify the dashboard data needs

   The dashboard requires aggregated, business-ready data (total sales by region).

2. Step 2: Match data needs to Medallion layers

   Gold layer contains aggregated and business-level data, ideal for dashboards.

3. Final Answer:

   Query the gold layer for aggregated sales data -> Option D

4. Quick Check:

   Dashboard needs gold layer data [OK]

Hint: Dashboards use gold layer data [OK]

Common Mistakes:

• Using raw or cleaned data instead of aggregated
• Querying bronze layer directly for reports
• Filtering bronze layer instead of using gold