Handling encoding issues in Pandas - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this code when reading a CSV with encoding errors?

Consider a CSV file with some invalid UTF-8 bytes. What will be the output of the following code snippet?

Pandas

import pandas as pd
from io import BytesIO

bytes_data = b'name,age\nAlice,30\nBob,25\nJos\xe9,22'

try:
    df = pd.read_csv(BytesIO(bytes_data), encoding='utf-8')
    result = df.to_dict()
except Exception as e:
    result = str(type(e))

A {'name': {0: 'Alice', 1: 'Bob', 2: 'José'}, 'age': {0: 30, 1: 25, 2: 22}}

B {'name': {0: 'Alice', 1: 'Bob', 2: 'Jos\xE9'}, 'age': {0: 30, 1: 25, 2: 22}}

C <class 'ParserError'>

D <class 'UnicodeDecodeError'>

💡 Hint

Think about what happens when pandas tries to decode bytes that are not valid UTF-8.

The string contains an invalid UTF-8 byte sequence (\xE9 alone is not valid UTF-8). When pandas tries to read it with encoding='utf-8', it raises a UnicodeDecodeError.

❓ data_output

intermediate

What is the content of the DataFrame after reading with errors='replace'?

Given the same CSV data with invalid UTF-8 bytes, what will be the DataFrame content if we use errors='replace' in read_csv?

Pandas

import pandas as pd
from io import BytesIO

bytes_data = b'name,age\nAlice,30\nBob,25\nJos\xe9,22'

df = pd.read_csv(BytesIO(bytes_data), encoding='utf-8', encoding_errors='replace')
result = df.to_dict()

A {'name': {0: 'Alice', 1: 'Bob', 2: 'Jos�'}, 'age': {0: 30, 1: 25, 2: 22}}

B {'name': {0: 'Alice', 1: 'Bob', 2: 'Jos\xE9'}, 'age': {0: 30, 1: 25, 2: 22}}

C <class 'UnicodeDecodeError'>

D {'name': {0: 'Alice', 1: 'Bob', 2: 'José'}, 'age': {0: 30, 1: 25, 2: 22}}

💡 Hint

Using errors='replace' replaces invalid bytes with a special character.

The invalid byte is replaced by the Unicode replacement character � (U+FFFD), so the name becomes 'Jos�'.

🔧 Debug

advanced

Why does this code raise a UnicodeDecodeError?

Examine the code below. Why does it raise a UnicodeDecodeError?

Pandas

import pandas as pd
from io import BytesIO

bytes_data = b'name,age\nAlice,30\nBob,25\nJos\xe9,22'

try:
    df = pd.read_csv(BytesIO(bytes_data), encoding='utf-8')
    result = df.to_dict()
except Exception as e:
    result = str(type(e))

A Because the BytesIO object is not supported by pandas

B Because the byte \xe9 is not valid UTF-8 on its own

C Because the CSV header is missing

D Because the data contains a missing value

💡 Hint

Check the encoding of the byte \xe9 in UTF-8.

The byte \xe9 alone is not a valid UTF-8 character. UTF-8 encodes 'é' as two bytes: \xc3\xa9. Hence, decoding fails.

🚀 Application

advanced

How to correctly read a CSV with Latin-1 encoding containing special characters?

You have a CSV file encoded in Latin-1 with names containing accented characters. Which code snippet correctly reads it into a DataFrame preserving the characters?

A pd.read_csv('file.csv', encoding='latin-1')

B pd.read_csv('file.csv', encoding='utf-8')

C pd.read_csv('file.csv', encoding='ascii')

D pd.read_csv('file.csv', encoding='utf-16')

💡 Hint

Use the encoding that matches the file's actual encoding.

Since the file is encoded in Latin-1, specifying encoding='latin-1' correctly decodes special characters.

🧠 Conceptual

expert

What is the effect of using encoding='utf-8-sig' when reading a CSV file?

When reading a CSV file that starts with a Byte Order Mark (BOM), what does specifying encoding='utf-8-sig' do?

A It causes a UnicodeDecodeError because BOM is not supported

B It treats the file as ASCII encoding ignoring UTF-8 characters

C It removes the BOM from the start of the file so it does not appear in the data

D It converts all characters to uppercase

💡 Hint

Think about what BOM means and how utf-8-sig handles it.

The utf-8-sig encoding detects and removes the BOM if present, preventing it from appearing as part of the data.