Broadcast variables in Apache Spark - Mini Project: Build & Apply

Using Broadcast Variables in Apache Spark

📖 Scenario: You work at a retail company analyzing sales data. You want to efficiently share a small lookup table of product categories across all worker nodes in a Spark cluster.

🎯 Goal: Learn how to create and use a broadcast variable in Apache Spark to share a small dictionary of product categories with all tasks, then use it to enrich sales data.

📋 What You'll Learn

Create a dictionary of product categories

Broadcast the dictionary using SparkContext.broadcast()

Use the broadcast variable inside an RDD map transformation

Print the enriched sales data with product categories

💡 Why This Matters

🌍 Real World

Broadcast variables help share small read-only data efficiently across many worker nodes in distributed computing, avoiding costly data transfer.

💼 Career

Understanding broadcast variables is important for data engineers and data scientists working with Apache Spark to optimize distributed data processing.

1

Create the product categories dictionary

Create a dictionary called product_categories with these exact entries: 101: 'Electronics', 102: 'Clothing', 103: 'Groceries'.

Apache Spark

# Create the product_categories dictionary
# Your code here

Need a hint?

Use curly braces {} to create a dictionary with keys and values.

2

Broadcast the product categories dictionary

Create a broadcast variable called broadcast_categories by broadcasting the product_categories dictionary using sc.broadcast(product_categories).

Apache Spark

product_categories = {101: 'Electronics', 102: 'Clothing', 103: 'Groceries'}
# Broadcast the product_categories dictionary
# Your code here

Need a hint?

Use sc.broadcast() to share the dictionary efficiently across nodes.

3

Use the broadcast variable to enrich sales data

Create an RDD called sales_rdd from the list [(1, 101, 5), (2, 102, 3), (3, 103, 10)]. Then use sales_rdd.map() with a lambda function that uses broadcast_categories.value to add the product category name to each record. Store the result in enriched_sales.

Apache Spark

product_categories = {101: 'Electronics', 102: 'Clothing', 103: 'Groceries'}
broadcast_categories = sc.broadcast(product_categories)
# Create sales_rdd and enrich it with product categories
# Your code here

Need a hint?

Use sc.parallelize() to create an RDD from a list. Use broadcast_categories.value inside the lambda to access the dictionary.

4

Print the enriched sales data

Collect the enriched_sales RDD and print the list of enriched records.

Apache Spark

product_categories = {101: 'Electronics', 102: 'Clothing', 103: 'Groceries'}
broadcast_categories = sc.broadcast(product_categories)
sales_rdd = sc.parallelize([(1, 101, 5), (2, 102, 3), (3, 103, 10)])
enriched_sales = sales_rdd.map(lambda record: (record[0], record[1], record[2], broadcast_categories.value.get(record[1], 'Unknown')))
# Print the collected enriched sales data
# Your code here

Need a hint?

Use enriched_sales.collect() to get the list and print() to display it.