Why streaming enables real-time analytics in Apache Spark - See It in Action

Why Streaming Enables Real-Time Analytics

📖 Scenario: Imagine you work for a company that wants to monitor website visits as they happen. Instead of waiting for a whole day to analyze the data, the company wants to see visitor counts every few seconds to react quickly.

🎯 Goal: You will build a simple streaming data process using Apache Spark that counts website visits in real-time. This will show how streaming helps get instant insights.

📋 What You'll Learn

Create a static list of website visit events as initial data

Set a batch interval configuration for streaming

Use Spark Structured Streaming to count visits per page in real-time

Print the streaming counts to the console

💡 Why This Matters

🌍 Real World

Companies use streaming to monitor website traffic instantly and react quickly to user behavior.

💼 Career

Data scientists and engineers use streaming analytics to provide real-time insights for business decisions.

1

Create initial website visit data

Create a list called visit_data with these dictionaries representing website visits: {'page': 'home', 'user': 'Alice'}, {'page': 'about', 'user': 'Bob'}, {'page': 'home', 'user': 'Charlie'}, {'page': 'contact', 'user': 'David'}, and {'page': 'home', 'user': 'Eve'}.

Apache Spark

# Create a list called visit_data with the given visit dictionaries
# Your code here

Need a hint?

Use a Python list with dictionaries for each visit.

2

Set batch interval for streaming

Create a variable called batch_interval and set it to 5 to represent 5 seconds between streaming batches.

Apache Spark

visit_data = [
    {'page': 'home', 'user': 'Alice'},
    {'page': 'about', 'user': 'Bob'},
    {'page': 'home', 'user': 'Charlie'},
    {'page': 'contact', 'user': 'David'},
    {'page': 'home', 'user': 'Eve'}
]
# Create a variable batch_interval and set it to 5
# Your code here

Need a hint?

Just assign the number 5 to the variable batch_interval.

3

Create streaming DataFrame and count visits per page

Use Spark to create a streaming DataFrame from visit_data. Then group by the page column and count visits. Store the result in a variable called visit_counts.

Apache Spark

visit_data = [
    {'page': 'home', 'user': 'Alice'},
    {'page': 'about', 'user': 'Bob'},
    {'page': 'home', 'user': 'Charlie'},
    {'page': 'contact', 'user': 'David'},
    {'page': 'home', 'user': 'Eve'}
]
batch_interval = 5

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('RealTimeVisits').getOrCreate()

# Create streaming DataFrame and count visits per page
# Your code here

Need a hint?

Use spark.createDataFrame(visit_data) to make a DataFrame, then groupBy('page').count() to count visits.

4

Print the visit counts

Print the visit_counts DataFrame to show the number of visits per page.

Apache Spark

visit_data = [
    {'page': 'home', 'user': 'Alice'},
    {'page': 'about', 'user': 'Bob'},
    {'page': 'home', 'user': 'Charlie'},
    {'page': 'contact', 'user': 'David'},
    {'page': 'home', 'user': 'Eve'}
]
batch_interval = 5

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('RealTimeVisits').getOrCreate()

from pyspark.sql.functions import col

static_df = spark.createDataFrame(visit_data)

visit_counts = static_df.groupBy('page').count()

# Print the visit_counts DataFrame
# Your code here

Need a hint?

Use visit_counts.show() to print the counts.