Handling missing values (drop_na, fill) in R Programming - Step-by-Step Execution

Concept Flow - Handling missing values (drop_na, fill)

Start with data frame

↓

Check for missing values

↓

drop_na()

↓

Remove rows

↓

Cleaned data frame

Start with a data frame, check for missing values, then either remove rows with missing data using drop_na() or fill missing values using fill(), resulting in a cleaned data frame.

Execution Sample

R Programming

library(tidyr)
data <- data.frame(x = c(1, NA, 3), y = c(NA, 2, 3))
data_clean <- drop_na(data)
data_filled <- fill(data, x, .direction = "down")

This code creates a data frame with missing values, then removes rows with any NA using drop_na(), and fills missing x values downward using fill().

Execution Table

Step	Data Frame State	Action	Resulting Data Frame
1	x: 1, NA, 3; y: NA, 2, 3	Initial data frame with missing values	x: 1, NA, 3; y: NA, 2, 3
2	x: 1, NA, 3; y: NA, 2, 3	Apply drop_na() to remove rows with any NA	x: 3; y: 3 (only row 3 remains)
3	x: 1, NA, 3; y: NA, 2, 3	Apply fill(x, .direction = "down") to fill NA in x	x: 1, 1, 3; y: NA, 2, 3
4	x: 1, 1, 3; y: NA, 2, 3	No further action	Final cleaned data frames ready

💡 All missing values handled either by removal or filling; process complete.

Variable Tracker

Variable	Start	After drop_na	After fill	Final
data	x: 1, NA, 3; y: NA, 2, 3	Unchanged	Unchanged	Unchanged
data_clean	NA	x: 3; y: 3	NA	x: 3; y: 3
data_filled	NA	NA	x: 1, 1, 3; y: NA, 2, 3	x: 1, 1, 3; y: NA, 2, 3

Key Moments - 2 Insights

Why does drop_na() remove the second row but keep the third row even though the third row has no missing x value?

drop_na() removes rows with any NA in any column. The second row has NA in x, so it is removed. The third row has no NA in any column, so it stays (see execution_table step 2).

How does fill() decide what value to use to replace NA in the x column?

fill() replaces NA with the last non-NA value above it in the column (downward direction). So the NA in row 2 is replaced by 1 from row 1 (see execution_table step 3).

Visual Quiz - 3 Questions

Look at the execution_table at step 2. What rows remain after drop_na() is applied?

A First and third rows

B Only the third row

C Only the first row

D All rows

💡 Hint

Check the 'Resulting Data Frame' column at step 2 in execution_table.

According to variable_tracker, what is the value of x in the second row after fill() is applied?

A NA

B 3

C 1

D 2

💡 Hint

Look at 'data_filled' column in variable_tracker for the x variable.

If we changed fill() direction to 'up', what would happen to the NA in x at row 2?

A It would be replaced by 3 from row 3

B It would remain NA

C It would be replaced by 1 from row 1

D All rows would be removed

💡 Hint

fill() with direction 'up' fills NA with next non-NA value below (see fill() behavior).

Concept Snapshot

Handling missing values in R with tidyr:
- drop_na(data): removes rows with any NA
- fill(data, column, .direction): fills NA with nearby values
- drop_na removes rows; fill replaces NA
- Use fill direction 'down' or 'up' to control filling
- Clean data frames ready for analysis

Full Transcript

We start with a data frame containing missing values (NA). We use drop_na() to remove any rows that have missing values in any column, leaving only complete rows. Alternatively, we use fill() to replace missing values in a specific column by carrying forward or backward the last known value. In the example, drop_na() removes the second row because it has NA, and fill() replaces the NA in the second row's x column with the value from the first row. This way, we clean the data by either removing or filling missing values, preparing it for further use.

Practice

1. What does the drop_na() function do in R's tidyr package?

easy

A. Fills missing values with the mean of the column.

B. Replaces missing values with zeros.

C. Removes rows that contain missing values in specified columns.

D. Sorts the data frame by missing values.

Solution

1. Step 1: Understand the purpose of drop_na()

   The function is designed to remove rows that have missing (NA) values in specified columns or in the entire data frame if no columns are specified.

2. Step 2: Compare with other options

   Options A and C describe filling missing values, which is done by fill() or other functions, not drop_na(). Sorts the data frame by missing values. sorts the data frame by missing values, which is unrelated.

3. Final Answer:

   Removes rows that contain missing values in specified columns. -> Option C

4. Quick Check:

   drop_na() removes rows with NA [OK]

Hint: Remember: drop_na drops rows with NA, fill replaces NA [OK]

Common Mistakes:

• Confusing drop_na() with fill()
• Thinking drop_na() fills missing values
• Assuming drop_na() sorts data

2. Which of the following is the correct syntax to fill missing values in a data frame df using the fill() function from tidyr for the column score?

easy

A. df.fill(score, direction = "down")

B. fill(df, score, direction = "down")

C. fill(score, df, .direction = "down")

D. df %>% fill(score, .direction = "down")

Solution

1. Step 1: Recall the pipe operator usage

   The fill() function is typically used with the pipe operator %>% to pass the data frame as the first argument.

2. Step 2: Check the argument order and syntax

   df %>% fill(score, .direction = "down") correctly uses df %>% fill(score, .direction = "down"). Options B and C have wrong syntax or argument order. df.fill(score, direction = "down") uses incorrect syntax not valid in R.

3. Final Answer:

   df %>% fill(score, .direction = "down") -> Option D

4. Quick Check:

   Use pipe %>% before fill() with column name [OK]

Hint: Use pipe %>% before fill() and specify column [OK]

Common Mistakes:

• Passing data frame as second argument
• Using dot direction without dot prefix
• Using Python-like method syntax

3. Given the data frame df below:

df <- data.frame(id = 1:5, score = c(10, NA, 15, NA, 20))
df_filled <- df %>% fill(score, .direction = "down")
print(df_filled)

What will be the output of print(df_filled)?

medium

A. id score
1 1 10
2 2 10
3 3 15
4 4 15
5 5 20

B. id score
1 1 10
2 2 NA
3 3 15
4 4 NA
5 5 20

C. id score
1 1 10
2 2 15
3 3 15
4 4 20
5 5 20

D. id score
1 1 NA
2 2 NA
3 3 15
4 4 15
5 5 20

Solution

1. Step 1: Understand fill() with .direction = "down"

   The function fills missing values by copying the last known non-NA value downward to replace NA values.

2. Step 2: Apply fill() to the data

   Row 2's NA is replaced by 10 (from row 1), row 4's NA is replaced by 15 (from row 3). Other values remain unchanged.

3. Final Answer:

   id score 1 1 10 2 2 10 3 3 15 4 4 15 5 5 20 -> Option A

4. Quick Check:

   fill() copies last known value down [OK]

Hint: fill() with down copies last known value to NA below [OK]

Common Mistakes:

• Assuming fill() replaces NA with next value
• Thinking fill() leaves NA unchanged
• Confusing fill() direction

4. Consider the following code snippet:

library(tidyr)
df <- data.frame(x = c(1, NA, 3), y = c(NA, 2, 3))
df_clean <- drop_na(df, x)
print(df_clean)

What is the output of print(df_clean)?

medium

A. x y
1 1 NA
3 3 3

B. x y
2 NA 2
3 3 3

C. x y
1 1 NA
2 NA 2
3 3 3

D. x y
3 3 3

Solution

1. Step 1: Understand drop_na(df, x)

   This removes rows where column x has NA values. Row 2 has NA in x, so it is removed.

2. Step 2: Identify remaining rows

   Rows 1 and 3 remain. Row 1 has x=1, y=NA. Row 3 has x=3, y=3.

3. Final Answer:

   x y 1 1 NA 3 3 3 -> Option A

4. Quick Check:

   drop_na(df, x) removes rows with NA in x [OK]

Hint: drop_na(df, col) removes rows with NA in that column [OK]

Common Mistakes:

• Removing rows with NA in all columns instead of specified
• Confusing drop_na() with fill()
• Expecting NA values to be replaced

5. You have a data frame df with columns time and temperature, where some temperature values are missing (NA). You want to fill missing temperatures by carrying the last known temperature forward, but only within each day (grouped by time). Which code correctly achieves this?

hard

A. df %>% fill(temperature, .direction = "up") %>% group_by(time)

B. df %>% group_by(time) %>% fill(temperature, .direction = "down")

C. df %>% fill(temperature, .direction = "down") %>% group_by(time)

D. df %>% group_by(time) %>% drop_na(temperature)

Solution

1. Step 1: Group data by time

   Grouping by time ensures fill() works within each day separately.

2. Step 2: Use fill() with .direction = "down"

   Filling downward copies last known temperature forward within each group.

3. Step 3: Check other options

   df %>% fill(temperature, .direction = "down") %>% group_by(time) fills before grouping, so fill applies to whole data ignoring groups. df %>% fill(temperature, .direction = "up") %>% group_by(time) fills upward, which is opposite of requirement. df %>% group_by(time) %>% drop_na(temperature) drops rows with NA instead of filling.

4. Final Answer:

   df %>% group_by(time) %>% fill(temperature, .direction = "down") -> Option B

5. Quick Check:

   Group then fill down to fill within groups [OK]

Hint: Group first, then fill down to fill missing within groups [OK]

Common Mistakes:

• Filling before grouping
• Using fill direction up instead of down
• Dropping rows instead of filling