Medallion architecture (bronze, silver, gold) in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Purpose of the Bronze Layer in Medallion Architecture

What is the main purpose of the Bronze layer in the Medallion architecture?

A Aggregate data into summarized reports for dashboards.

B Store fully cleaned and enriched data ready for business use.

C Store raw, unprocessed data as it arrives from source systems.

D Archive old data that is no longer needed for analysis.

💡 Hint

Think about where data lands first before any cleaning or transformation.

The Bronze layer is where raw data is stored exactly as it arrives. It acts as the foundation for further processing.

❓ data_output

intermediate

Data Count After Silver Layer Transformation

Given the following Spark code that reads from the Bronze layer and applies a filter to create the Silver layer, what is the count of records in the Silver DataFrame?

Apache Spark

bronze_df = spark.createDataFrame([
  (1, 'Alice', 25, 'NY'),
  (2, 'Bob', 30, 'CA'),
  (3, 'Charlie', 35, 'NY'),
  (4, 'David', 40, 'TX')
], ['id', 'name', 'age', 'state'])
silver_df = bronze_df.filter(bronze_df.state == 'NY')

A 2

B 3

C 1

D 4

💡 Hint

Count how many rows have state equal to 'NY'.

Only Alice and Charlie have state 'NY', so the count is 2.

❓ visualization

advanced

Visualizing Data Quality Improvements Across Layers

You have data quality scores for each Medallion layer: Bronze (60%), Silver (85%), Gold (95%). Which visualization best shows the improvement in data quality from Bronze to Gold?

A A pie chart showing the percentage of data quality for the Gold layer only.

B A scatter plot showing age vs. income for Gold layer data.

C A line chart with time on x-axis and data volume on y-axis.

D A bar chart with three bars labeled Bronze, Silver, Gold showing scores 60, 85, 95 respectively.

💡 Hint

Think about how to compare data quality scores across layers clearly.

A bar chart with three bars clearly shows the increasing data quality from Bronze to Gold layers.

🔧 Debug

advanced

Identify the Error in Gold Layer Aggregation Code

What error will this Spark code produce when aggregating data in the Gold layer?

Apache Spark

gold_df = silver_df.groupBy('state').agg({'age': 'avg', 'salary': 'sum'})

A AnalysisException: cannot resolve 'salary' given input columns

B SyntaxError: invalid syntax in agg function

C TypeError: unsupported operand type(s) for +: 'int' and 'str'

D No error, code runs successfully

💡 Hint

Check if all columns used in aggregation exist in silver_df.

If 'salary' column does not exist in silver_df, Spark raises AnalysisException for unresolved column.

🚀 Application

expert

Choosing the Correct Layer for Real-Time Dashboard Data

You need to build a real-time dashboard that shows the latest sales data with minimal delay. Which Medallion layer should you query to get the freshest data with basic cleaning applied?

A Archive layer, because it stores historical data for analysis.

B Silver layer, because it has cleaned data and is updated frequently.

C Gold layer, because it has aggregated and business-ready data.

D Bronze layer, because it has raw data with no delay.

💡 Hint

Consider freshness and data quality balance for dashboards.

The Silver layer provides cleaned data with frequent updates, ideal for near real-time dashboards.

Practice

1. What is the primary purpose of the bronze layer in the Medallion architecture?

easy

A. Store metadata about data quality

B. Store cleaned and enriched data

C. Store raw, unprocessed data as it arrives

D. Store aggregated business-level data

Solution

1. Step 1: Understand the role of the bronze layer

   The bronze layer is the first stage where raw data is ingested without transformation.

2. Step 2: Differentiate from silver and gold layers

   Silver contains cleaned data, and gold contains aggregated data, so they are not raw.

3. Final Answer:

   Store raw, unprocessed data as it arrives -> Option C

4. Quick Check:

   Bronze = Raw data [OK]

Hint: Bronze = raw data layer, first step [OK]

Common Mistakes:

• Confusing bronze with silver or gold layers
• Thinking bronze stores aggregated data
• Assuming bronze is cleaned data

2. Which of the following is the correct order of layers in the Medallion architecture from raw to business-ready data?

easy

A. Bronze -> Silver -> Gold

B. Silver -> Bronze -> Gold

C. Gold -> Silver -> Bronze

D. Silver -> Gold -> Bronze

Solution

1. Step 1: Recall the Medallion architecture flow

   Data flows from raw to cleaned to aggregated: bronze, then silver, then gold.

2. Step 2: Match the order to options

   Bronze -> Silver -> Gold correctly lists bronze first, then silver, then gold.

3. Final Answer:

   Bronze -> Silver -> Gold -> Option A

4. Quick Check:

   Order = Bronze, Silver, Gold [OK]

Hint: Remember: Bronze first, then Silver, then Gold [OK]

Common Mistakes:

• Reversing the order of layers
• Mixing silver and bronze positions
• Assuming gold comes before silver

3. Given the following PySpark code snippet, which layer does the resulting DataFrame most likely represent?

bronzeDF = spark.read.format('json').load('/data/raw/events')
silverDF = bronzeDF.filter('event_type IS NOT NULL').dropDuplicates()
goldDF = silverDF.groupBy('event_type').count()

medium

A. goldDF is gold layer

B. silverDF is bronze layer

C. bronzeDF is gold layer

D. bronzeDF is silver layer

Solution

1. Step 1: Analyze each DataFrame's transformation

   bronzeDF loads raw JSON data (raw data = bronze). silverDF filters and removes duplicates (cleaned data = silver). goldDF groups and counts (aggregated data = gold).

2. Step 2: Identify the layer of goldDF

   goldDF performs aggregation, which matches the gold layer's purpose.

3. Final Answer:

   goldDF is gold layer -> Option A

4. Quick Check:

   Aggregation = Gold layer [OK]

Hint: Aggregation means gold layer [OK]

Common Mistakes:

• Confusing raw load as silver
• Thinking filtering is aggregation
• Mixing up silver and gold transformations

4. You have a silver layer DataFrame but notice it still contains duplicate records. Which change will best fix this issue in your Spark code?

medium

A. Add .dropDuplicates() after loading raw data in bronze layer

B. Add .dropDuplicates() after cleaning transformations in silver layer

C. Add .groupBy() in bronze layer to aggregate data

D. Add .filter() in gold layer to remove duplicates

Solution

1. Step 1: Understand where duplicates should be removed

   Duplicates should be removed during cleaning, which happens in the silver layer.

2. Step 2: Evaluate options for removing duplicates

   Adding .dropDuplicates() in silver layer after cleaning transformations is the correct fix.

3. Final Answer:

   Add .dropDuplicates() after cleaning transformations in silver layer -> Option B

4. Quick Check:

   Remove duplicates in silver layer [OK]

Hint: Clean data (silver) removes duplicates [OK]

Common Mistakes:

• Removing duplicates too early in bronze layer
• Using groupBy instead of dropDuplicates
• Filtering duplicates only in gold layer

5. You want to build a dashboard showing total sales by region updated daily. Which Medallion architecture layer should you query for the most accurate and business-ready data?

hard

A. Query the bronze layer for raw sales data

B. Query the bronze layer after filtering duplicates

C. Query the silver layer for cleaned sales data

D. Query the gold layer for aggregated sales data

Solution

1. Step 1: Identify the dashboard data needs

   The dashboard requires aggregated, business-ready data (total sales by region).

2. Step 2: Match data needs to Medallion layers

   Gold layer contains aggregated and business-level data, ideal for dashboards.

3. Final Answer:

   Query the gold layer for aggregated sales data -> Option D

4. Quick Check:

   Dashboard needs gold layer data [OK]

Hint: Dashboards use gold layer data [OK]

Common Mistakes:

• Using raw or cleaned data instead of aggregated
• Querying bronze layer directly for reports
• Filtering bronze layer instead of using gold