How to Use textFile to Create RDD in PySpark

In PySpark, you can create an RDD from a text file using the textFile method of a SparkContext object. This method reads the file line by line and returns an RDD where each element is a line of the file.

Syntax

The textFile method is called on a SparkContext object and takes the file path as input. It returns an RDD where each element is a line from the file.

• sc: SparkContext object
• path: Path to the text file (local or distributed storage)
• Returns: RDD of strings (each string is a line)

python

rdd = sc.textFile(path)

Example

This example shows how to create an RDD from a local text file and print its contents line by line.

python

from pyspark import SparkContext

# Initialize SparkContext
sc = SparkContext('local[*]', 'TextFileExample')

# Path to a sample text file
file_path = 'sample.txt'

# Create RDD from text file
rdd = sc.textFile(file_path)

# Collect and print each line
lines = rdd.collect()
for line in lines:
    print(line)

# Stop SparkContext
sc.stop()

Output

Hello world
This is a sample text file
PySpark RDD creation example

Common Pitfalls

Common mistakes when using textFile include:

• Using an incorrect file path or missing file causes errors.
• Not initializing SparkContext before calling textFile.
• Forgetting to stop SparkContext after use.
• Assuming textFile reads the entire file as one string instead of line by line.

python

from pyspark import SparkContext

# Wrong: Not initializing SparkContext
# rdd = sc.textFile('sample.txt')  # This will fail because sc is not defined

# Correct way:
sc = SparkContext('local[*]', 'Example')
rdd = sc.textFile('sample.txt')
sc.stop()

Quick Reference

Summary tips for using textFile in PySpark:

• Always create a SparkContext before calling textFile.
• Use the correct file path accessible by Spark.
• textFile returns an RDD of lines, not the whole file as one string.
• Call collect() or other actions to retrieve data from the RDD.
• Stop SparkContext when done to free resources.

Key Takeaways

Use SparkContext's textFile method to create an RDD from a text file.

Each element of the RDD represents one line of the file.

Ensure SparkContext is initialized before calling textFile.

Provide the correct file path accessible to Spark.

Stop SparkContext after completing your operations.