Adding and renaming columns in Apache Spark - Step-by-Step Execution

Concept Flow - Adding and renaming columns

Start with DataFrame

↓

Add new column with value or expression

↓

Rename existing column

↓

Result: DataFrame with updated columns

↓

End

Start with a DataFrame, add new columns using expressions, rename columns, and get the updated DataFrame.

Execution Sample

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame([(1, "Alice"), (2, "Bob")], ["id", "name"])
df2 = df.withColumn("age", df.id + 20).withColumnRenamed("name", "first_name")
df2.show()

Create a DataFrame, add a new column 'age' by adding 20 to 'id', rename 'name' to 'first_name', then show the result.

Execution Table

Step	Action	DataFrame Columns	Notes
1	Create DataFrame	[id, name]	Initial DataFrame with 2 columns
2	Add column 'age' = id + 20	[id, name, age]	New column 'age' added
3	Rename column 'name' to 'first_name'	[id, first_name, age]	Column 'name' renamed
4	Show DataFrame	[id, first_name, age]	Displays updated DataFrame
5	End	[id, first_name, age]	No more changes

💡 All columns added and renamed as requested, execution ends.

Variable Tracker

Variable	Start	After Step 2	After Step 3	Final
df.columns	['id', 'name']	['id', 'name']	['id', 'name']	['id', 'name']
df2.columns	N/A	['id', 'name', 'age']	['id', 'first_name', 'age']	['id', 'first_name', 'age']

Key Moments - 2 Insights

Why does df.columns not change after adding or renaming columns?

Because df is immutable in Spark. Adding or renaming columns creates a new DataFrame (df2), leaving df unchanged, as shown in execution_table steps 2 and 3.

What happens if you rename a column that does not exist?

Spark will throw an error. The execution_table assumes the column exists before renaming, so renaming must be done on existing columns only.

Visual Quiz - 3 Questions

Look at the execution_table, what columns does df2 have after step 2?

A [id, name]

B [id, first_name, age]

C [id, name, age]

D [id, age]

💡 Hint

Check the 'DataFrame Columns' column for step 2 in execution_table.

At which step does the column 'name' get renamed?

A Step 3

B Step 2

C Step 1

D Step 4

💡 Hint

Look for the action mentioning renaming in execution_table.

If you add a column 'age' as df.id + 10 instead of +20, how would df2.columns after step 2 change?

A Columns would be [id, first_name, age]

B Columns would be [id, name, age] but age values differ

C Columns would be [id, name]

D Columns would be [id, name, age, age2]

💡 Hint

Adding a column changes columns list but expression affects values, not column names.

Concept Snapshot

Adding and renaming columns in Spark:
- Use withColumn('new_col', expr) to add or replace columns.
- Use withColumnRenamed('old', 'new') to rename columns.
- DataFrames are immutable; these return new DataFrames.
- Chain methods to add and rename in one step.
- Show() displays the updated DataFrame.

Full Transcript

We start with a Spark DataFrame with columns 'id' and 'name'. We add a new column 'age' by adding 20 to 'id' using withColumn. Then we rename the 'name' column to 'first_name' using withColumnRenamed. Each operation returns a new DataFrame, so the original stays unchanged. Finally, we show the updated DataFrame with columns 'id', 'first_name', and 'age'.

Practice

1. What does the withColumn method do in Apache Spark DataFrame?

easy

A. Adds a new column or updates an existing column in the DataFrame

B. Deletes a column from the DataFrame

C. Renames a column in the DataFrame

D. Filters rows based on a condition

Solution

1. Step 1: Understand the purpose of withColumn

   The withColumn method is used to add a new column or replace an existing column in a Spark DataFrame.

2. Step 2: Compare with other methods

   Unlike withColumnRenamed which renames columns, withColumn changes or adds columns with new values or expressions.

3. Final Answer:

   Adds a new column or updates an existing column in the DataFrame -> Option A

4. Quick Check:

   withColumn = Add or update column [OK]

Hint: Use withColumn to add or change columns fast [OK]

Common Mistakes:

• Confusing withColumn with withColumnRenamed
• Thinking it deletes columns
• Assuming it filters rows

2. Which of the following is the correct syntax to rename a column named oldName to newName in a Spark DataFrame df?

easy

A. df.rename("oldName", "newName")

B. df.withColumn("oldName", "newName")

C. df.renameColumn("oldName", "newName")

D. df.withColumnRenamed("oldName", "newName")

Solution

1. Step 1: Identify the correct method for renaming

   The method to rename a column in Spark DataFrame is withColumnRenamed.

2. Step 2: Check syntax correctness

   The syntax is df.withColumnRenamed("oldName", "newName"). Other options are invalid methods or misuse of withColumn.

3. Final Answer:

   df.withColumnRenamed("oldName", "newName") -> Option D

4. Quick Check:

   Rename column = withColumnRenamed [OK]

Hint: Rename columns only with withColumnRenamed method [OK]

Common Mistakes:

• Using withColumn instead of withColumnRenamed
• Trying non-existent methods like renameColumn
• Passing arguments in wrong order

3. Given the following Spark code:

from pyspark.sql.functions import lit

df = spark.createDataFrame([(1, "Alice")], ["id", "name"])
df2 = df.withColumn("age", lit(30)).withColumnRenamed("name", "first_name")
df2.show()

What will be the output?

medium

A. +---+----------+---+
| id|first_name|age|
+---+----------+---+
| 1| Alice| 30|
+---+----------+---+

B. +---+-----+---+
| id| name|age|
+---+-----+---+
| 1|Alice| 30|
+---+-----+---+

C. +---+----------+
| id|first_name|
+---+----------+
| 1| Alice|
+---+----------+

D. Error: withColumnRenamed cannot be chained after withColumn

Solution

1. Step 1: Add new column 'age' with value 30

   The withColumn("age", lit(30)) adds a new column 'age' with constant value 30.

2. Step 2: Rename column 'name' to 'first_name'

   The withColumnRenamed("name", "first_name") renames the 'name' column to 'first_name'.

3. Step 3: Show the resulting DataFrame

   The DataFrame now has columns: 'id', 'first_name', and 'age' with one row (1, Alice, 30).

4. Final Answer:

   +---+----------+---+ | id|first_name|age| +---+----------+---+ | 1| Alice| 30| +---+----------+---+ -> Option A

5. Quick Check:

   withColumn + withColumnRenamed = added column + renamed column [OK]

Hint: Chain withColumn then withColumnRenamed to add and rename [OK]

Common Mistakes:

• Assuming withColumnRenamed cannot be chained
• Forgetting to import lit for constant values
• Expecting original column names unchanged

4. Consider this code snippet:

df = spark.createDataFrame([(1, "John")], ["id", "name"])
df2 = df.withColumnRenamed("Name", "first_name")
df2.show()

What is the issue and how to fix it?

medium

A. withColumnRenamed cannot rename columns; use withColumn instead

B. Column 'Name' does not exist; use correct case 'name' in withColumnRenamed

C. Syntax error: missing parentheses in withColumnRenamed

D. No issue; code runs correctly and renames column

Solution

1. Step 1: Check column names in DataFrame

   The DataFrame has columns 'id' and 'name' (lowercase). The code tries to rename 'Name' (uppercase N) which does not exist.

2. Step 2: Fix the column name case

   Change withColumnRenamed("Name", "first_name") to withColumnRenamed("name", "first_name") to match exact column name.

3. Final Answer:

   Column 'Name' does not exist; use correct case 'name' in withColumnRenamed -> Option B

4. Quick Check:

   Column names are case-sensitive in Spark [OK]

Hint: Match exact column name case when renaming [OK]

Common Mistakes:

• Ignoring case sensitivity of column names
• Using withColumn instead of withColumnRenamed to rename
• Assuming code runs without error despite wrong column name

5. You have a DataFrame df with columns ['id', 'score']. You want to add a new column grade based on score: if score ≥ 90 then 'A', else 'B'. Then rename score to exam_score. Which code correctly does this?

hard

A. df2 = df.withColumn('grade', when(df.score >= 90, 'A').otherwise('B')).renameColumn('score', 'exam_score')

B. df2 = df.withColumnRenamed('score', 'exam_score').withColumn('grade', when(df.score >= 90, 'A').otherwise('B'))

C. from pyspark.sql.functions import when

df2 = df.withColumn('grade', when(df.score >= 90, 'A').otherwise('B')).withColumnRenamed('score', 'exam_score')

D. df2 = df.withColumn('grade', when(df.score > 90, 'A').otherwise('B')).withColumnRenamed('score', 'exam_score')

Solution

1. Step 1: Add 'grade' column with condition on 'score'

   Use when to assign 'A' if score ≥ 90, else 'B'. This is done with withColumn('grade', when(df.score >= 90, 'A').otherwise('B')).

2. Step 2: Rename 'score' column to 'exam_score'

   Use withColumnRenamed('score', 'exam_score') to rename the column after adding 'grade'. The order matters to keep references correct.

3. Step 3: Check other options

   df2 = df.withColumnRenamed('score', 'exam_score').withColumn('grade', when(df.score >= 90, 'A').otherwise('B')) uses old column name after renaming, causing error. df2 = df.withColumn('grade', when(df.score >= 90, 'A').otherwise('B')).renameColumn('score', 'exam_score') uses non-existent renameColumn. df2 = df.withColumn('grade', when(df.score > 90, 'A').otherwise('B')).withColumnRenamed('score', 'exam_score') uses > 90 instead of ≥ 90, which changes logic.

4. Final Answer:

   from pyspark.sql.functions import when df2 = df.withColumn('grade', when(df.score >= 90, 'A').otherwise('B')).withColumnRenamed('score', 'exam_score') -> Option C

5. Quick Check:

   Add column with when, then rename column [OK]

Hint: Add conditional column first, then rename original column [OK]

Common Mistakes:

• Renaming column before adding new column using old name
• Using wrong comparison operator (> instead of >=)
• Using non-existent renameColumn method