Type casting and null handling in Apache Spark - Mini Project: Build & Apply

Type Casting and Null Handling in Apache Spark

📖 Scenario: You work as a data analyst for a retail company. You receive sales data where some numbers are stored as strings and some values are missing (null). You need to clean this data by converting the strings to numbers and handling the missing values properly.

🎯 Goal: Build a Spark DataFrame with sales data, convert the sales amount from string to integer, replace null sales with zero, and display the cleaned data.

📋 What You'll Learn

Create a Spark DataFrame with specific sales data including null values

Create a variable for the replacement value for nulls

Cast the sales column from string to integer and replace nulls with zero

Print the final cleaned DataFrame

💡 Why This Matters

🌍 Real World

Data often comes with missing values and wrong data types. Cleaning data by converting types and handling nulls is a key step before analysis.

💼 Career

Data scientists and analysts frequently clean and prepare data using Spark for big data projects, ensuring accurate and reliable results.

1

Create the initial Spark DataFrame

Create a Spark DataFrame called sales_df with the following data: [('Store1', '100'), ('Store2', null), ('Store3', '250'), ('Store4', null)]. The columns should be named 'store' and 'sales'. Use spark.createDataFrame() and Row from pyspark.sql.

Apache Spark

# Your code here to create sales_df with the specified data and columns

Hint

Use Row to create each row with store and sales fields. Use None for null values.

2

Set the replacement value for null sales

Create a variable called replacement_value and set it to 0. This will be used to replace null sales values.

Apache Spark

from pyspark.sql import SparkSession, Row

spark = SparkSession.builder.appName('TypeCastingNullHandling').getOrCreate()
data = [Row(store='Store1', sales='100'), Row(store='Store2', sales=None), Row(store='Store3', sales='250'), Row(store='Store4', sales=None)]
sales_df = spark.createDataFrame(data)

# Create replacement_value variable and set it to 0
# Your code here

Hint

Just create a variable named replacement_value and assign 0 to it.

3

Cast sales to integer and replace nulls

Create a new DataFrame called cleaned_df by casting the sales column of sales_df from string to integer using cast('int'). Then replace null values in sales with replacement_value using fillna().

Apache Spark

from pyspark.sql import SparkSession, Row

spark = SparkSession.builder.appName('TypeCastingNullHandling').getOrCreate()
data = [Row(store='Store1', sales='100'), Row(store='Store2', sales=None), Row(store='Store3', sales='250'), Row(store='Store4', sales=None)]
sales_df = spark.createDataFrame(data)

replacement_value = 0

# Cast sales to int and replace nulls with replacement_value
# Your code here

Hint

Use withColumn with col('sales').cast('int') to convert types. Then use fillna({'sales': replacement_value}) to replace nulls.

4

Display the cleaned DataFrame

Print the contents of cleaned_df using the show() method to display the cleaned sales data.

Apache Spark

from pyspark.sql import SparkSession, Row
from pyspark.sql.functions import col

spark = SparkSession.builder.appName('TypeCastingNullHandling').getOrCreate()
data = [Row(store='Store1', sales='100'), Row(store='Store2', sales=None), Row(store='Store3', sales='250'), Row(store='Store4', sales=None)]
sales_df = spark.createDataFrame(data)

replacement_value = 0

cleaned_df = sales_df.withColumn('sales', col('sales').cast('int')).fillna({'sales': replacement_value})

# Print cleaned_df to show the final data
# Your code here

Hint

Use cleaned_df.show() to display the DataFrame.

Practice

1. What does the cast() function do in Apache Spark DataFrames?

easy

A. It sorts the DataFrame by a column.

B. It removes null values from a DataFrame.

C. It filters rows based on a condition.

D. It changes the data type of a column to another specified type.

Solution

1. Step 1: Understand the purpose of cast()

   The cast() function is used to convert the data type of a column in a Spark DataFrame to another type, such as from string to integer.

2. Step 2: Compare with other options

   Other options describe different functions: removing nulls, filtering, or sorting, which are not related to type casting.

3. Final Answer:

   It changes the data type of a column to another specified type. -> Option D

4. Quick Check:

   cast() changes type = A [OK]

Hint: Remember: cast() changes column types, not data values [OK]

Common Mistakes:

• Confusing cast() with filtering or sorting functions
• Thinking cast() removes null values
• Assuming cast() modifies data content instead of type

2. Which of the following is the correct syntax to cast a column named age to integer type in Spark DataFrame?

easy

A. df.select(df.age.convert('int'))

B. df.select(df.age.cast('int'))

C. df.select(df.age.toInt())

D. df.select(cast(df.age, 'int'))

Solution

1. Step 1: Recall the correct cast syntax in Spark

   In Spark, to cast a column, you use column.cast('type'). So df.age.cast('int') is correct.

2. Step 2: Check other options for syntax errors

   df.select(cast(df.age, 'int')) uses cast as a function incorrectly. df.select(df.age.toInt()) uses a method toInt() which does not exist in Spark DataFrame columns. df.select(df.age.convert('int')) uses convert() which is not a Spark method.

3. Final Answer:

   df.select(df.age.cast('int')) -> Option B

4. Quick Check:

   Use column.cast('type') syntax = D [OK]

Hint: Use column.cast('type') to cast columns [OK]

Common Mistakes:

• Using Python native methods like toInt() on Spark columns
• Trying to use cast as a standalone function
• Using non-existent methods like convert()

3. Given the following Spark code:

from pyspark.sql.functions import col

df = spark.createDataFrame([("25",), ("30",), (None,)], ["age"])
df2 = df.select(col("age").cast("int").alias("age_int"))
df2.show()

What will be the output?

medium

A. +-------+
|age_int|
+-------+
| 25|
| 30|
| null|
+-------+

B. +-------+
|age_int|
+-------+
| 25|
| 30|
| 0|
+-------+

C. RuntimeError due to casting None to int

D. SyntaxError in code

Solution

1. Step 1: Understand casting behavior with nulls

   Casting a string column with None values to integer results in integer values for valid strings and null for None values, not zero or error.

2. Step 2: Check the output of show()

   The DataFrame will display 25 and 30 as integers, and null for the None entry.

3. Final Answer:

   +-------+ |age_int| +-------+ | 25| | 30| | null| +-------+ -> Option A

4. Quick Check:

   Nulls remain null after cast = B [OK]

Hint: Casting None keeps null, does not convert to zero [OK]

Common Mistakes:

• Expecting None to become 0 after cast
• Thinking casting None causes error
• Confusing syntax errors with runtime behavior

4. Identify the error in this Spark code snippet that tries to replace nulls in column salary with 0:

df = df.withColumn("salary", df["salary"].cast("int").fillna(0))

medium

A. withColumn() requires a function, not a column.

B. cast() cannot convert salary to int.

C. fillna() cannot be called on a Column object; it should be called on DataFrame.

D. The syntax is correct; no error.

Solution

1. Step 1: Understand method chaining on Column vs DataFrame

   In Spark, fillna() is a DataFrame method, not a Column method. Calling fillna() on a Column object causes an error.

2. Step 2: Correct usage

   To replace nulls after casting, use df = df.withColumn('salary', df['salary'].cast('int')) then df = df.fillna({'salary': 0}).

3. Final Answer:

   fillna() cannot be called on a Column object; it should be called on DataFrame. -> Option C

4. Quick Check:

   fillna() on DataFrame, not Column = C [OK]

Hint: Remember: fillna() is for DataFrame, not Column [OK]

Common Mistakes:

• Calling fillna() on a Column instead of DataFrame
• Assuming cast() changes null values automatically
• Misusing withColumn() syntax

5. You have a DataFrame with a string column score containing numeric strings and some nulls. You want to create a new integer column score_int where nulls become -1 after casting. Which code correctly does this?

hard

A. df.withColumn('score_int', when(col('score').isNull(), -1).otherwise(col('score').cast('int')))

B. df.withColumn('score_int', col('score').cast('int').fillna(-1))

C. df.withColumn('score_int', col('score').cast('int').fillna({'score_int': -1}))

D. df.withColumn('score_int', col('score').cast('int').na.fill(-1))

Solution

1. Step 1: Understand null handling with casting

   Casting null strings to int results in null integers. To replace nulls with -1 in the new column, use when() to check nulls before casting.

2. Step 2: Analyze options

   df.withColumn('score_int', when(col('score').isNull(), -1).otherwise(col('score').cast('int'))) uses when() to assign -1 if null, else cast. Options B, C, and D try to fill nulls after casting, but fillna() or na.fill() on columns or DataFrames after casting may not replace nulls correctly or require separate steps.

3. Final Answer:

   df.withColumn('score_int', when(col('score').isNull(), -1).otherwise(col('score').cast('int'))) -> Option A

4. Quick Check:

   Use when() for conditional null replacement before cast = A [OK]

Hint: Use when() to replace nulls before casting [OK]

Common Mistakes:

• Using fillna() on columns instead of DataFrame
• Expecting fillna() to replace nulls after cast automatically
• Not handling nulls before casting