Adding and renaming columns in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of adding a new column with a constant value

What is the output DataFrame after running this Spark code that adds a new column with a constant value?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import lit

spark = SparkSession.builder.getOrCreate()
data = [(1, "Alice"), (2, "Bob")]
df = spark.createDataFrame(data, ["id", "name"])
df2 = df.withColumn("country", lit("USA"))
df2.show()

A [Row(id=1, name='Alice', country='USA'), Row(id=2, name='Bob', country='USA')]

B [Row(id=1, name='Alice'), Row(id=2, name='Bob')]

C [Row(id=1, name='Alice', country=None), Row(id=2, name='Bob', country=None)]

D SyntaxError

💡 Hint

Adding a column with lit() sets the same value for all rows.

The withColumn method adds a new column 'country' with the constant value 'USA' for every row.

❓ data_output

intermediate

Result of renaming a column in a Spark DataFrame

After renaming the column 'name' to 'first_name' in this DataFrame, what are the column names?

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
data = [(1, "Alice"), (2, "Bob")]
df = spark.createDataFrame(data, ["id", "name"])
df_renamed = df.withColumnRenamed("name", "first_name")
print(df_renamed.columns)

A ['id', 'name', 'first_name']

B ['id', 'first_name']

C ['first_name', 'id']

D ['id', 'name']

💡 Hint

withColumnRenamed changes the name of one column only.

The column 'name' is renamed to 'first_name', so the DataFrame columns are ['id', 'first_name'].

🔧 Debug

advanced

Identify the error when adding a column with an expression

What error does this code produce when trying to add a new column 'age_plus_ten' by adding 10 to the 'age' column?

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
data = [(1, 20), (2, 30)]
df = spark.createDataFrame(data, ["id", "age"])
df2 = df.withColumn("age_plus_ten", df["age"] + 10)
df2.show()

A TypeError: unsupported operand type(s) for +: 'Column' and 'int'

B SyntaxError

C NameError: name 'df' is not defined

D No error, outputs DataFrame with 'age_plus_ten' column

💡 Hint

Spark supports arithmetic operations on Column objects.

Adding an integer to a Spark Column object works correctly and creates a new column with the sum.

❓ visualization

advanced

Visualize the effect of renaming multiple columns

Given this DataFrame, what is the output of df_renamed.columns after renaming 'name' to 'first_name' and 'age' to 'years'?

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
data = [(1, "Alice", 20), (2, "Bob", 30)]
df = spark.createDataFrame(data, ["id", "name", "age"])
df_renamed = df.withColumnRenamed("name", "first_name").withColumnRenamed("age", "years")
print(df_renamed.columns)

A ['id', 'first_name', 'years']

B ['id', 'name', 'age']

C ['first_name', 'years', 'id']

D ['id', 'first_name', 'age']

💡 Hint

Each withColumnRenamed changes one column name.

Both 'name' and 'age' columns are renamed, so the columns are ['id', 'first_name', 'years'].

🚀 Application

expert

Add a new column based on condition and rename existing column

You have a DataFrame with columns 'id' and 'score'. You want to add a new column 'passed' that is True if score >= 50, else False. Then rename 'score' to 'exam_score'. Which code produces the correct final DataFrame?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import when

spark = SparkSession.builder.getOrCreate()
data = [(1, 45), (2, 75), (3, 50)]
df = spark.createDataFrame(data, ["id", "score"])

# Choose the correct option below

A

)(wohs.2fd
)'erocs_maxe' ,'erocs'(demaneRnmuloChtiw.))eslaF(esiwrehto.)eurT ,05 => ]'erocs'[fd(nehw ,'dessap'(nmuloChtiw.fd = 2fd

B

df2 = df.withColumnRenamed('score', 'exam_score').withColumn('passed', when(df['score'] >= 50, True).otherwise(False))
df2.show()

C

df2 = df.withColumn('passed', when(df['score'] >= 50, True).otherwise(False)).withColumnRenamed('score', 'exam_score')
df2.show()

D

df2 = df.withColumn('passed', when(df['score'] >= 50, 'Yes').otherwise('No')).withColumnRenamed('score', 'exam_score')
df2.show()

💡 Hint

Order matters: use original column name in condition before renaming.

Option C correctly adds 'passed' based on 'score' >= 50, then renames 'score' to 'exam_score'. Option C uses 'score' after renaming, causing error. Option C uses > 50 instead of >= 50. Option C uses strings instead of booleans.

Practice

1. What does the withColumn method do in Apache Spark DataFrame?

easy

A. Adds a new column or updates an existing column in the DataFrame

B. Deletes a column from the DataFrame

C. Renames a column in the DataFrame

D. Filters rows based on a condition

Solution

1. Step 1: Understand the purpose of withColumn

   The withColumn method is used to add a new column or replace an existing column in a Spark DataFrame.

2. Step 2: Compare with other methods

   Unlike withColumnRenamed which renames columns, withColumn changes or adds columns with new values or expressions.

3. Final Answer:

   Adds a new column or updates an existing column in the DataFrame -> Option A

4. Quick Check:

   withColumn = Add or update column [OK]

Hint: Use withColumn to add or change columns fast [OK]

Common Mistakes:

• Confusing withColumn with withColumnRenamed
• Thinking it deletes columns
• Assuming it filters rows

2. Which of the following is the correct syntax to rename a column named oldName to newName in a Spark DataFrame df?

easy

A. df.rename("oldName", "newName")

B. df.withColumn("oldName", "newName")

C. df.renameColumn("oldName", "newName")

D. df.withColumnRenamed("oldName", "newName")

Solution

1. Step 1: Identify the correct method for renaming

   The method to rename a column in Spark DataFrame is withColumnRenamed.

2. Step 2: Check syntax correctness

   The syntax is df.withColumnRenamed("oldName", "newName"). Other options are invalid methods or misuse of withColumn.

3. Final Answer:

   df.withColumnRenamed("oldName", "newName") -> Option D

4. Quick Check:

   Rename column = withColumnRenamed [OK]

Hint: Rename columns only with withColumnRenamed method [OK]

Common Mistakes:

• Using withColumn instead of withColumnRenamed
• Trying non-existent methods like renameColumn
• Passing arguments in wrong order

3. Given the following Spark code:

from pyspark.sql.functions import lit

df = spark.createDataFrame([(1, "Alice")], ["id", "name"])
df2 = df.withColumn("age", lit(30)).withColumnRenamed("name", "first_name")
df2.show()

What will be the output?

medium

A. +---+----------+---+
| id|first_name|age|
+---+----------+---+
| 1| Alice| 30|
+---+----------+---+

B. +---+-----+---+
| id| name|age|
+---+-----+---+
| 1|Alice| 30|
+---+-----+---+

C. +---+----------+
| id|first_name|
+---+----------+
| 1| Alice|
+---+----------+

D. Error: withColumnRenamed cannot be chained after withColumn

Solution

1. Step 1: Add new column 'age' with value 30

   The withColumn("age", lit(30)) adds a new column 'age' with constant value 30.

2. Step 2: Rename column 'name' to 'first_name'

   The withColumnRenamed("name", "first_name") renames the 'name' column to 'first_name'.

3. Step 3: Show the resulting DataFrame

   The DataFrame now has columns: 'id', 'first_name', and 'age' with one row (1, Alice, 30).

4. Final Answer:

   +---+----------+---+ | id|first_name|age| +---+----------+---+ | 1| Alice| 30| +---+----------+---+ -> Option A

5. Quick Check:

   withColumn + withColumnRenamed = added column + renamed column [OK]

Hint: Chain withColumn then withColumnRenamed to add and rename [OK]

Common Mistakes:

• Assuming withColumnRenamed cannot be chained
• Forgetting to import lit for constant values
• Expecting original column names unchanged

4. Consider this code snippet:

df = spark.createDataFrame([(1, "John")], ["id", "name"])
df2 = df.withColumnRenamed("Name", "first_name")
df2.show()

What is the issue and how to fix it?

medium

A. withColumnRenamed cannot rename columns; use withColumn instead

B. Column 'Name' does not exist; use correct case 'name' in withColumnRenamed

C. Syntax error: missing parentheses in withColumnRenamed

D. No issue; code runs correctly and renames column

Solution

1. Step 1: Check column names in DataFrame

   The DataFrame has columns 'id' and 'name' (lowercase). The code tries to rename 'Name' (uppercase N) which does not exist.

2. Step 2: Fix the column name case

   Change withColumnRenamed("Name", "first_name") to withColumnRenamed("name", "first_name") to match exact column name.

3. Final Answer:

   Column 'Name' does not exist; use correct case 'name' in withColumnRenamed -> Option B

4. Quick Check:

   Column names are case-sensitive in Spark [OK]

Hint: Match exact column name case when renaming [OK]

Common Mistakes:

• Ignoring case sensitivity of column names
• Using withColumn instead of withColumnRenamed to rename
• Assuming code runs without error despite wrong column name

5. You have a DataFrame df with columns ['id', 'score']. You want to add a new column grade based on score: if score ≥ 90 then 'A', else 'B'. Then rename score to exam_score. Which code correctly does this?

hard

A. df2 = df.withColumn('grade', when(df.score >= 90, 'A').otherwise('B')).renameColumn('score', 'exam_score')

B. df2 = df.withColumnRenamed('score', 'exam_score').withColumn('grade', when(df.score >= 90, 'A').otherwise('B'))

C. from pyspark.sql.functions import when

df2 = df.withColumn('grade', when(df.score >= 90, 'A').otherwise('B')).withColumnRenamed('score', 'exam_score')

D. df2 = df.withColumn('grade', when(df.score > 90, 'A').otherwise('B')).withColumnRenamed('score', 'exam_score')

Solution

1. Step 1: Add 'grade' column with condition on 'score'

   Use when to assign 'A' if score ≥ 90, else 'B'. This is done with withColumn('grade', when(df.score >= 90, 'A').otherwise('B')).

2. Step 2: Rename 'score' column to 'exam_score'

   Use withColumnRenamed('score', 'exam_score') to rename the column after adding 'grade'. The order matters to keep references correct.

3. Step 3: Check other options

   df2 = df.withColumnRenamed('score', 'exam_score').withColumn('grade', when(df.score >= 90, 'A').otherwise('B')) uses old column name after renaming, causing error. df2 = df.withColumn('grade', when(df.score >= 90, 'A').otherwise('B')).renameColumn('score', 'exam_score') uses non-existent renameColumn. df2 = df.withColumn('grade', when(df.score > 90, 'A').otherwise('B')).withColumnRenamed('score', 'exam_score') uses > 90 instead of ≥ 90, which changes logic.

4. Final Answer:

   from pyspark.sql.functions import when df2 = df.withColumn('grade', when(df.score >= 90, 'A').otherwise('B')).withColumnRenamed('score', 'exam_score') -> Option C

5. Quick Check:

   Add column with when, then rename column [OK]

Hint: Add conditional column first, then rename original column [OK]

Common Mistakes:

• Renaming column before adding new column using old name
• Using wrong comparison operator (> instead of >=)
• Using non-existent renameColumn method