Select, filter, and where operations in Apache Spark - Mini Project: Build & Apply

Select, filter, and where operations

📖 Scenario: You work as a data analyst for a small online bookstore. You have a dataset of books with details like title, author, genre, and price. Your manager wants you to find books that are affordable and belong to a specific genre.

🎯 Goal: Build a Spark DataFrame with book data, set a price limit, filter books by genre and price using select, filter, and where operations, and display the filtered results.

📋 What You'll Learn

Create a Spark DataFrame with exact book data

Create a variable for the maximum price allowed

Use select to choose specific columns

Use filter or where to select books by genre and price

Print the final filtered DataFrame

💡 Why This Matters

🌍 Real World

Filtering and selecting data is a common task in data analysis to focus on relevant information, such as affordable books in a specific genre.

💼 Career

Data analysts and data scientists often use select, filter, and where operations in Spark to prepare and explore large datasets efficiently.

1

Create the books DataFrame

Create a Spark DataFrame called books_df with these exact entries: {'title': 'The Alchemist', 'author': 'Paulo Coelho', 'genre': 'Fiction', 'price': 10}, {'title': 'Deep Learning', 'author': 'Ian Goodfellow', 'genre': 'Education', 'price': 50}, {'title': 'Clean Code', 'author': 'Robert C. Martin', 'genre': 'Education', 'price': 40}, {'title': 'The Hobbit', 'author': 'J.R.R. Tolkien', 'genre': 'Fiction', 'price': 15}, {'title': 'Cooking 101', 'author': 'Jamie Oliver', 'genre': 'Cooking', 'price': 20}. Use Spark's createDataFrame method with a list of dictionaries.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('BookStore').getOrCreate()
# Create the books_df DataFrame with the exact book entries
# Your code here

Need a hint?

Use spark.createDataFrame with a list of dictionaries containing the exact book details.

2

Set the maximum price limit

Create a variable called max_price and set it to 20 to represent the maximum price allowed for books.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('BookStore').getOrCreate()

books_data = [
    {'title': 'The Alchemist', 'author': 'Paulo Coelho', 'genre': 'Fiction', 'price': 10},
    {'title': 'Deep Learning', 'author': 'Ian Goodfellow', 'genre': 'Education', 'price': 50},
    {'title': 'Clean Code', 'author': 'Robert C. Martin', 'genre': 'Education', 'price': 40},
    {'title': 'The Hobbit', 'author': 'J.R.R. Tolkien', 'genre': 'Fiction', 'price': 15},
    {'title': 'Cooking 101', 'author': 'Jamie Oliver', 'genre': 'Cooking', 'price': 20}
]

books_df = spark.createDataFrame(books_data)
# Create a variable max_price and set it to 20
# Your code here

Need a hint?

Just create a variable named max_price and assign it the value 20.

3

Filter books by genre and price

Use select to choose the columns 'title', 'author', and 'price' from books_df. Then use filter or where to keep only books where genre is 'Fiction' and price is less than or equal to max_price. Save the result in a new DataFrame called filtered_books.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('BookStore').getOrCreate()

books_data = [
    {'title': 'The Alchemist', 'author': 'Paulo Coelho', 'genre': 'Fiction', 'price': 10},
    {'title': 'Deep Learning', 'author': 'Ian Goodfellow', 'genre': 'Education', 'price': 50},
    {'title': 'Clean Code', 'author': 'Robert C. Martin', 'genre': 'Education', 'price': 40},
    {'title': 'The Hobbit', 'author': 'J.R.R. Tolkien', 'genre': 'Fiction', 'price': 15},
    {'title': 'Cooking 101', 'author': 'Jamie Oliver', 'genre': 'Cooking', 'price': 20}
]

books_df = spark.createDataFrame(books_data)

max_price = 20
# Use select and filter or where to create filtered_books with only 'title', 'author', 'price' columns
# and where genre is 'Fiction' and price <= max_price
# Your code here

Need a hint?

Use select('title', 'author', 'price') and then where with conditions joined by &.

4

Display the filtered books

Use print and filtered_books.show() to display the filtered books DataFrame.

Apache Spark

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('BookStore').getOrCreate()

books_data = [
    {'title': 'The Alchemist', 'author': 'Paulo Coelho', 'genre': 'Fiction', 'price': 10},
    {'title': 'Deep Learning', 'author': 'Ian Goodfellow', 'genre': 'Education', 'price': 50},
    {'title': 'Clean Code', 'author': 'Robert C. Martin', 'genre': 'Education', 'price': 40},
    {'title': 'The Hobbit', 'author': 'J.R.R. Tolkien', 'genre': 'Fiction', 'price': 15},
    {'title': 'Cooking 101', 'author': 'Jamie Oliver', 'genre': 'Cooking', 'price': 20}
]

books_df = spark.createDataFrame(books_data)

max_price = 20

filtered_books = books_df.select('title', 'author', 'price').where((books_df.genre == 'Fiction') & (books_df.price <= max_price))
# Print the filtered_books DataFrame
# Your code here

Need a hint?

Use print('Filtered books:') and then filtered_books.show() to display the data.