Multi-column joins in Apache Spark - Mini Project: Build & Apply

Multi-column joins

📖 Scenario: You work at a retail company. You have two tables: one with customer orders and another with customer details. You want to combine these tables to see the orders along with customer information. Both tables share customer_id and region columns. Joining on both columns ensures you match the right customer in the right region.

🎯 Goal: Build a Spark DataFrame join using two columns (customer_id and region) to combine orders and customer details.

📋 What You'll Learn

Create a Spark DataFrame called orders with columns order_id, customer_id, region, and amount.

Create a Spark DataFrame called customers with columns customer_id, region, and customer_name.

Create a variable called join_columns that holds the list of columns ["customer_id", "region"].

Use join_columns to join orders and customers DataFrames on these columns.

Print the resulting joined DataFrame.

💡 Why This Matters

🌍 Real World

Multi-column joins are common when combining data from different sources that share multiple keys, like customer ID and region, to ensure accurate matching.

💼 Career

Data scientists and data engineers often join datasets on multiple columns to prepare clean, combined data for analysis or machine learning.

1

Create the orders DataFrame

Create a Spark DataFrame called orders with these exact rows and columns: order_id, customer_id, region, and amount. Use these rows: (1, 101, 'East', 250), (2, 102, 'West', 450), (3, 103, 'East', 300), (4, 104, 'North', 150).

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("MultiColumnJoin").getOrCreate()

# Create the orders DataFrame with the specified rows and columns
# Your code here

Hint

Use spark.createDataFrame with a list of tuples and specify the column names as a list.

2

Create the customers DataFrame

Create a Spark DataFrame called customers with these exact rows and columns: customer_id, region, and customer_name. Use these rows: (101, 'East', 'Alice'), (102, 'West', 'Bob'), (103, 'East', 'Charlie'), (105, 'South', 'Diana').

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("MultiColumnJoin").getOrCreate()

orders = spark.createDataFrame([
    (1, 101, 'East', 250),
    (2, 102, 'West', 450),
    (3, 103, 'East', 300),
    (4, 104, 'North', 150)
], ["order_id", "customer_id", "region", "amount"])

# Create the customers DataFrame with the specified rows and columns
# Your code here

Hint

Use spark.createDataFrame with a list of tuples and specify the column names as a list.

3

Define the join columns and join DataFrames

Create a variable called join_columns that holds the list ["customer_id", "region"]. Then join the orders DataFrame with the customers DataFrame using join_columns as the join keys. Store the result in a variable called joined_df.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("MultiColumnJoin").getOrCreate()

orders = spark.createDataFrame([
    (1, 101, 'East', 250),
    (2, 102, 'West', 450),
    (3, 103, 'East', 300),
    (4, 104, 'North', 150)
], ["order_id", "customer_id", "region", "amount"])

customers = spark.createDataFrame([
    (101, 'East', 'Alice'),
    (102, 'West', 'Bob'),
    (103, 'East', 'Charlie'),
    (105, 'South', 'Diana')
], ["customer_id", "region", "customer_name"])

# Create join_columns list and join orders and customers DataFrames on these columns
# Your code here

Hint

Define join_columns as a list of column names. Use orders.join(customers, on=join_columns) to join.

4

Show the joined DataFrame

Print the joined_df DataFrame using the show() method to display the joined data.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("MultiColumnJoin").getOrCreate()

orders = spark.createDataFrame([
    (1, 101, 'East', 250),
    (2, 102, 'West', 450),
    (3, 103, 'East', 300),
    (4, 104, 'North', 150)
], ["order_id", "customer_id", "region", "amount"])

customers = spark.createDataFrame([
    (101, 'East', 'Alice'),
    (102, 'West', 'Bob'),
    (103, 'East', 'Charlie'),
    (105, 'South', 'Diana')
], ["customer_id", "region", "customer_name"])

join_columns = ["customer_id", "region"]

joined_df = orders.join(customers, on=join_columns, how="inner")

# Show the joined DataFrame
# Your code here

Hint

Use joined_df.show() to print the DataFrame.

Practice

1.

What does a multi-column join in Apache Spark do?

Choose the best description.

easy

A. It sorts a DataFrame by multiple columns.

B. It merges two DataFrames by adding columns without any condition.

C. It filters rows from a single DataFrame based on multiple columns.

D. It joins two DataFrames using more than one column to match rows.

Solution

1. Step 1: Understand join basics

   A join combines rows from two tables based on matching column values.

2. Step 2: Recognize multi-column join

   Multi-column join uses more than one column to find matching rows, improving accuracy.

3. Final Answer:

   It joins two DataFrames using more than one column to match rows. -> Option D

4. Quick Check:

   Multi-column join = match on multiple columns [OK]

Hint: Multi-column join matches rows on several columns at once [OK]

Common Mistakes:

• Confusing join with filtering
• Thinking join adds columns without condition
• Mixing join with sorting

2.

Which of the following is the correct syntax to perform a multi-column inner join between df1 and df2 on columns "id" and "date"?

df1.join(df2, _____, how="inner")

easy

A. ["id", "date"]

B. "id, date"

C. df1["id"] == df2["id"] and df1["date"] == df2["date"]

D. on=("id", "date")

Solution

1. Step 1: Check join parameter for columns

   The on parameter accepts a list of column names for multi-column join.

2. Step 2: Identify correct list syntax

   Using ["id", "date"] correctly passes both columns as a list.

3. Final Answer:

   ["id", "date"] -> Option A

4. Quick Check:

   List of columns = ["id", "date"] [OK]

Hint: Use a list of column names for multi-column join [OK]

Common Mistakes:

• Passing columns as a single string
• Using boolean expressions instead of list
• Incorrect tuple syntax for columns

3.

Given the following two DataFrames:

df1 = spark.createDataFrame([
  (1, "2023-01-01", "A"),
  (2, "2023-01-02", "B"),
  (3, "2023-01-03", "C")
], ["id", "date", "value"])

df2 = spark.createDataFrame([
  (1, "2023-01-01", "X"),
  (2, "2023-01-04", "Y"),
  (3, "2023-01-03", "Z")
], ["id", "date", "value2"])

result = df1.join(df2, ["id", "date"], how="inner").select("id", "date", "value", "value2")
result.show()

What will be the output?

medium

A. +---+----------+-----+-------+
|id |date |value|value2 |
+---+----------+-----+-------+
|1 |2023-01-01|A |X |
|2 |2023-01-04|B |Y |
|3 |2023-01-03|C |Z |
+---+----------+-----+-------+

B. +---+----------+-----+-------+
|id |date |value|value2 |
+---+----------+-----+-------+
|2 |2023-01-02|B |Y |
+---+----------+-----+-------+

C. +---+----------+-----+-------+
|id |date |value|value2 |
+---+----------+-----+-------+
|1 |2023-01-01|A |X |
|3 |2023-01-03|C |Z |
+---+----------+-----+-------+

D. +---+----------+-----+-------+
|id |date |value|value2 |
+---+----------+-----+-------+
|1 |2023-01-01|A |null |
|2 |2023-01-02|B |null |
|3 |2023-01-03|C |Z |
+---+----------+-----+-------+

Solution

1. Step 1: Understand inner join on multiple columns

   Inner join keeps rows where both id and date match in both DataFrames.

2. Step 2: Identify matching rows

   Rows with (1, "2023-01-01") and (3, "2023-01-03") match in both DataFrames; (2, "2023-01-02") and (2, "2023-01-04") do not.

3. Final Answer:

   Rows with id 1 and 3 with matching dates and values -> Option C

4. Quick Check:

   Inner join = only matching rows on all columns [OK]

Hint: Inner join keeps only rows matching all join columns [OK]

Common Mistakes:

• Including rows with partial matches
• Confusing inner join with outer join
• Ignoring the multi-column condition

4.

Consider this code snippet:

result = df1.join(df2, on=["id", "date"], how="inner")

When running this, you get an error: TypeError: join() got an unexpected keyword argument 'on'. What is the likely cause?

medium

A. The how parameter must be a list, not a string.

B. The on parameter should be passed as the second positional argument, not as a keyword.

C. The join method does not support multi-column joins.

D. The DataFrames must have the same schema before joining.

Solution

1. Step 1: Check join method signature

   In Spark, join() expects the join columns as the second positional argument, not as a keyword on.

2. Step 2: Correct usage

   Use df1.join(df2, ["id", "date"], how="inner") without on=.

3. Final Answer:

   The on parameter should be passed as the second positional argument, not as a keyword. -> Option B

4. Quick Check:

   Join columns = positional argument [OK]

Hint: Pass join columns as positional argument, not 'on=' keyword [OK]

Common Mistakes:

• Using 'on=' keyword in join
• Passing 'how' as a list
• Assuming join needs identical schemas

5.

You have two DataFrames, orders and customers. You want to join them on customer_id and region, but some customers have missing region values in one DataFrame. You want to keep all orders and add customer info where possible. Which join type and approach should you use?

hard

A. Use a left join on customer_id only, then filter or fill missing region values separately.

B. Use an inner join on both columns; only rows with matching customer_id and region appear.

C. Use a left join on both columns; missing region values will exclude some matches.

D. Use a full outer join on both columns to keep all rows from both DataFrames.

Solution

1. Step 1: Understand join requirements

   You want all orders (left DataFrame) and customer info where available, but region may be missing in one DataFrame.

2. Step 2: Choose join columns carefully

   Joining on both columns will exclude rows with missing region. Joining only on customer_id keeps all orders.

3. Step 3: Handle missing region values separately

   After join, fill or filter missing region values as needed.

4. Final Answer:

   Use a left join on customer_id only, then handle missing region values separately. -> Option A

5. Quick Check:

   Left join on key column + handle missing data [OK]

Hint: Join on reliable keys first, handle missing columns after [OK]

Common Mistakes:

• Joining on columns with missing values directly
• Using inner join and losing rows
• Using full outer join when not needed