Handling encoding issues in Pandas - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this code when reading a CSV with encoding errors?

Consider a CSV file with some invalid UTF-8 bytes. What will be the output of the following code snippet?

Pandas

import pandas as pd
from io import BytesIO

bytes_data = b'name,age\nAlice,30\nBob,25\nJos\xe9,22'

try:
    df = pd.read_csv(BytesIO(bytes_data), encoding='utf-8')
    result = df.to_dict()
except Exception as e:
    result = str(type(e))

A {'name': {0: 'Alice', 1: 'Bob', 2: 'José'}, 'age': {0: 30, 1: 25, 2: 22}}

B {'name': {0: 'Alice', 1: 'Bob', 2: 'Jos\xE9'}, 'age': {0: 30, 1: 25, 2: 22}}

C <class 'ParserError'>

D <class 'UnicodeDecodeError'>

💡 Hint

Think about what happens when pandas tries to decode bytes that are not valid UTF-8.

The string contains an invalid UTF-8 byte sequence (\xE9 alone is not valid UTF-8). When pandas tries to read it with encoding='utf-8', it raises a UnicodeDecodeError.

❓ data_output

intermediate

What is the content of the DataFrame after reading with errors='replace'?

Given the same CSV data with invalid UTF-8 bytes, what will be the DataFrame content if we use errors='replace' in read_csv?

Pandas

import pandas as pd
from io import BytesIO

bytes_data = b'name,age\nAlice,30\nBob,25\nJos\xe9,22'

df = pd.read_csv(BytesIO(bytes_data), encoding='utf-8', encoding_errors='replace')
result = df.to_dict()

A {'name': {0: 'Alice', 1: 'Bob', 2: 'Jos�'}, 'age': {0: 30, 1: 25, 2: 22}}

B {'name': {0: 'Alice', 1: 'Bob', 2: 'Jos\xE9'}, 'age': {0: 30, 1: 25, 2: 22}}

C <class 'UnicodeDecodeError'>

D {'name': {0: 'Alice', 1: 'Bob', 2: 'José'}, 'age': {0: 30, 1: 25, 2: 22}}

💡 Hint

Using errors='replace' replaces invalid bytes with a special character.

The invalid byte is replaced by the Unicode replacement character � (U+FFFD), so the name becomes 'Jos�'.

🔧 Debug

advanced

Why does this code raise a UnicodeDecodeError?

Examine the code below. Why does it raise a UnicodeDecodeError?

Pandas

import pandas as pd
from io import BytesIO

bytes_data = b'name,age\nAlice,30\nBob,25\nJos\xe9,22'

try:
    df = pd.read_csv(BytesIO(bytes_data), encoding='utf-8')
    result = df.to_dict()
except Exception as e:
    result = str(type(e))

A Because the BytesIO object is not supported by pandas

B Because the byte \xe9 is not valid UTF-8 on its own

C Because the CSV header is missing

D Because the data contains a missing value

💡 Hint

Check the encoding of the byte \xe9 in UTF-8.

The byte \xe9 alone is not a valid UTF-8 character. UTF-8 encodes 'é' as two bytes: \xc3\xa9. Hence, decoding fails.

🚀 Application

advanced

How to correctly read a CSV with Latin-1 encoding containing special characters?

You have a CSV file encoded in Latin-1 with names containing accented characters. Which code snippet correctly reads it into a DataFrame preserving the characters?

A pd.read_csv('file.csv', encoding='latin-1')

B pd.read_csv('file.csv', encoding='utf-8')

C pd.read_csv('file.csv', encoding='ascii')

D pd.read_csv('file.csv', encoding='utf-16')

💡 Hint

Use the encoding that matches the file's actual encoding.

Since the file is encoded in Latin-1, specifying encoding='latin-1' correctly decodes special characters.

🧠 Conceptual

expert

What is the effect of using encoding='utf-8-sig' when reading a CSV file?

When reading a CSV file that starts with a Byte Order Mark (BOM), what does specifying encoding='utf-8-sig' do?

A It causes a UnicodeDecodeError because BOM is not supported

B It treats the file as ASCII encoding ignoring UTF-8 characters

C It removes the BOM from the start of the file so it does not appear in the data

D It converts all characters to uppercase

💡 Hint

Think about what BOM means and how utf-8-sig handles it.

The utf-8-sig encoding detects and removes the BOM if present, preventing it from appearing as part of the data.

Practice

1. What does the encoding parameter in pandas read_csv function do?

easy

A. It sorts the data alphabetically.

B. It tells pandas how to read text characters correctly from the file.

C. It compresses the file to save space.

D. It changes the file format from CSV to Excel.

Solution

1. Step 1: Understand the role of encoding

   Encoding defines how text characters are stored and read in files.

2. Step 2: Apply encoding in pandas

   The encoding parameter in read_csv tells pandas how to interpret the text characters correctly.

3. Final Answer:

   It tells pandas how to read text characters correctly from the file. -> Option B

4. Quick Check:

   Encoding parameter = read text correctly [OK]

Hint: Encoding fixes text reading issues in pandas [OK]

Common Mistakes:

• Thinking encoding changes file format
• Confusing encoding with sorting or compression
• Ignoring encoding when reading files with special characters

2. Which of the following is the correct way to read a CSV file with UTF-8 encoding in pandas?

easy

A. pd.read_csv('file.csv', encoding='utf-8')

B. pd.read_csv('file.csv', encode='utf8')

C. pd.read_csv('file.csv', encoding=utf8)

D. pd.read_csv('file.csv', encoding=UTF8)

Solution

1. Step 1: Check correct parameter name and value

   The parameter is encoding and the value should be a string like 'utf-8'.

2. Step 2: Validate syntax correctness

   pd.read_csv('file.csv', encoding='utf-8') uses correct parameter and string value with quotes. pd.read_csv('file.csv', encode='utf8') uses wrong parameter name. pd.read_csv('file.csv', encoding=utf8) misses quotes around encoding value. pd.read_csv('file.csv', encoding=UTF8) misses quotes around encoding value.

3. Final Answer:

   pd.read_csv('file.csv', encoding='utf-8') -> Option A

4. Quick Check:

   Correct syntax = pd.read_csv('file.csv', encoding='utf-8') [OK]

Hint: Use encoding='utf-8' with quotes in read_csv [OK]

Common Mistakes:

• Using wrong parameter name like 'encode'
• Missing quotes around encoding value
• Using uppercase or incorrect encoding string

3. What will be the output of this code if the file contains special characters and is encoded in 'latin1' but read with default encoding?

import pandas as pd
df = pd.read_csv('data_latin1.csv')
print(df.head())

medium

A. DataFrame with correct special characters displayed

B. SyntaxError

C. Empty DataFrame

D. UnicodeDecodeError or garbled characters in DataFrame

Solution

1. Step 1: Understand file encoding mismatch

   The file is encoded in 'latin1' but pandas tries to read with default 'utf-8'. This causes decoding errors or wrong characters.

2. Step 2: Predict pandas behavior on encoding error

   Without specifying encoding, pandas may raise UnicodeDecodeError or show garbled text in the DataFrame.

3. Final Answer:

   UnicodeDecodeError or garbled characters in DataFrame -> Option D

4. Quick Check:

   Encoding mismatch = decoding error [OK]

Hint: Match file encoding with encoding parameter to avoid errors [OK]

Common Mistakes:

• Assuming default encoding always works
• Expecting empty DataFrame on encoding error
• Confusing syntax errors with encoding errors

4. You try to read a CSV file with special characters using pd.read_csv('file.csv', encoding='utf-8') but get a UnicodeDecodeError. What is the best way to fix this?

medium

A. Change encoding to 'latin1' or the correct file encoding

B. Remove the encoding parameter completely

C. Rename the file extension to .txt

D. Use pd.read_csv without pandas

Solution

1. Step 1: Identify cause of UnicodeDecodeError

   This error means the file encoding does not match 'utf-8'.

2. Step 2: Fix by specifying correct encoding

   Changing encoding to 'latin1' or the actual file encoding fixes the error by matching how characters are read.

3. Final Answer:

   Change encoding to 'latin1' or the correct file encoding -> Option A

4. Quick Check:

   Fix encoding mismatch = Change encoding to 'latin1' or the correct file encoding [OK]

Hint: Try 'latin1' if 'utf-8' causes decode errors [OK]

Common Mistakes:

• Removing encoding parameter and expecting no error
• Changing file extension instead of encoding
• Trying to read CSV without pandas

5. You have a CSV file with mixed encodings: some rows are UTF-8, others are Latin1. How can you read this file correctly in pandas?

hard

A. Read the file in binary mode and decode each line separately

B. Use encoding='latin1' to read all rows without errors

C. Use a library like chardet to detect encoding per line and decode accordingly

D. Read the file with encoding='utf-8' and ignore errors, then fix rows manually

Solution

1. Step 1: Understand mixed encoding challenge

   Standard pandas read_csv expects one encoding for whole file. Mixed encodings cause errors or wrong characters.

2. Step 2: Use encoding detection per line

   Using a library like chardet allows detecting encoding for each line, then decoding properly before loading into pandas.

3. Final Answer:

   Use a library like chardet to detect encoding per line and decode accordingly -> Option C

4. Quick Check:

   Mixed encoding needs per-line detection [OK]

Hint: Use chardet for mixed encoding files [OK]

Common Mistakes:

• Assuming one encoding fits all lines
• Ignoring errors and losing data
• Trying to fix encoding manually without detection