Batch vs real-time ingestion in Hadoop - Hands-On Comparison

Batch vs Real-time Data Ingestion in Hadoop

📖 Scenario: You work at a company that collects sales data from stores. Some data comes in big chunks once a day (batch), and some data comes in quickly as sales happen (real-time). You want to understand how to handle both types of data ingestion using Hadoop tools.

🎯 Goal: Build a simple example to show how batch and real-time data ingestion can be set up and processed in Hadoop. You will create data samples, configure ingestion types, process the data accordingly, and display the results.

📋 What You'll Learn

Create sample sales data for batch and real-time ingestion

Set a configuration variable to select ingestion type

Write core logic to process data differently based on ingestion type

Print the processed sales summary

💡 Why This Matters

🌍 Real World

Companies collect data in batches (like daily reports) and in real-time (like live sales). Understanding how to handle both helps in timely decision making.

💼 Career

Data engineers and analysts often work with batch and real-time data ingestion pipelines using Hadoop and related tools.

1

Create sample sales data for batch and real-time ingestion

Create two lists called batch_sales and realtime_sales. batch_sales should have these dictionaries: {'store': 'StoreA', 'sales': 100}, {'store': 'StoreB', 'sales': 150}. realtime_sales should have these dictionaries: {'store': 'StoreA', 'sales': 10}, {'store': 'StoreB', 'sales': 15}.

Hadoop

# Create batch_sales and realtime_sales lists with given dictionaries
# Your code here

Hint

Use list of dictionaries to represent sales data for each store.

2

Set ingestion type configuration

Create a variable called ingestion_type and set it to the string 'batch' or 'realtime' to select which data to process.

Hadoop

batch_sales = [{'store': 'StoreA', 'sales': 100}, {'store': 'StoreB', 'sales': 150}]
realtime_sales = [{'store': 'StoreA', 'sales': 10}, {'store': 'StoreB', 'sales': 15}]
# Set ingestion_type to 'batch' or 'realtime'
# Your code here

Hint

Set ingestion_type to either 'batch' or 'realtime' as a string.

3

Process sales data based on ingestion type

Create a variable called sales_data that is batch_sales if ingestion_type is 'batch', else realtime_sales. Then create a dictionary called total_sales that sums sales per store from sales_data using a for loop with variables record.

Hadoop

batch_sales = [{'store': 'StoreA', 'sales': 100}, {'store': 'StoreB', 'sales': 150}]
realtime_sales = [{'store': 'StoreA', 'sales': 10}, {'store': 'StoreB', 'sales': 15}]
ingestion_type = 'batch'
# Select sales_data based on ingestion_type
# Sum sales per store into total_sales dictionary
# Your code here

Hint

Use a conditional expression to pick sales_data. Use a for loop to add sales per store.

4

Print the total sales summary

Write a print statement to display the total_sales dictionary.

Hadoop

batch_sales = [{'store': 'StoreA', 'sales': 100}, {'store': 'StoreB', 'sales': 150}]
realtime_sales = [{'store': 'StoreA', 'sales': 10}, {'store': 'StoreB', 'sales': 15}]
ingestion_type = 'batch'
sales_data = batch_sales if ingestion_type == 'batch' else realtime_sales
total_sales = {}
for record in sales_data:
    store = record['store']
    sales = record['sales']
    total_sales[store] = total_sales.get(store, 0) + sales
# Print the total_sales dictionary
# Your code here

Hint

Use print(total_sales) to show the sales summary.

Practice

1. Which statement best describes batch ingestion in Hadoop?

easy

A. It only works with streaming data sources.

B. It processes data instantly as it arrives.

C. It requires real-time user interaction to start.

D. It collects and processes data in groups at scheduled times.

Solution

1. Step 1: Understand batch ingestion concept

   Batch ingestion collects data in chunks or groups rather than continuously.

2. Step 2: Compare with real-time ingestion

   Real-time ingestion processes data immediately, unlike batch which waits for scheduled times.

3. Final Answer:

   It collects and processes data in groups at scheduled times. -> Option D

4. Quick Check:

   Batch ingestion = scheduled group processing [OK]

Hint: Batch = grouped data at set times, not instant [OK]

Common Mistakes:

• Confusing batch with real-time ingestion
• Thinking batch requires instant processing
• Assuming batch needs user interaction

2. Which of the following is the correct syntax to start a real-time ingestion job in Hadoop using Apache Flume?

easy

A. hadoop fs -put data.txt /input

B. flume-ng agent -n agent1 -c conf -f conf/flume.conf

C. hadoop jar batchjob.jar input output

D. flume-ng batch -start agent1

Solution

1. Step 1: Identify real-time ingestion tool syntax

   Apache Flume uses the command flume-ng agent to start an agent for real-time ingestion.

2. Step 2: Check other options for correctness

   Options B and C relate to batch or file operations, and D is not a valid Flume command.

3. Final Answer:

   flume-ng agent -n agent1 -c conf -f conf/flume.conf -> Option B

4. Quick Check:

   Flume agent command = real-time start [OK]

Hint: Flume agent command starts real-time ingestion [OK]

Common Mistakes:

• Using batch job commands for real-time ingestion
• Confusing file upload with ingestion start
• Incorrect Flume command syntax

3. Consider this simplified code snippet for batch ingestion in Hadoop MapReduce:

hadoop jar batchjob.jar input_dir output_dir

What will happen when this command runs?

medium

A. It processes all files in input_dir at once and writes results to output_dir.

B. It continuously processes new files as they arrive in input_dir.

C. It deletes input_dir after processing.

D. It streams data from input_dir to output_dir in real-time.

Solution

1. Step 1: Analyze the batch job command

   The command runs a MapReduce job on all data present in input_dir at the time of execution.

2. Step 2: Understand batch vs real-time behavior

   Batch jobs process data in bulk, not continuously or in real-time.

3. Final Answer:

   It processes all files in input_dir at once and writes results to output_dir. -> Option A

4. Quick Check:

   Batch job = bulk processing [OK]

Hint: Batch job runs once on all current input data [OK]

Common Mistakes:

• Thinking batch jobs run continuously
• Assuming input directory is deleted
• Confusing batch with streaming

4. You wrote this Flume configuration to ingest data in real-time:

agent1.sources = source1
agent1.sinks = sink1
agent1.channels = channel1
agent1.sources.source1.type = exec
agent1.sources.source1.command = tail -F /var/log/syslog
agent1.channels.channel1.type = memory
agent1.sinks.sink1.type = hdfs
agent1.sinks.sink1.hdfs.path = /data/logs/

But no data appears in HDFS. What is the likely error?

medium

A. Flume agent was not started with the correct configuration file.

B. The source command 'tail -F' is incorrect; should be 'tail -f'.

C. The channel is not connected to the sink and source.

D. The sink path '/data/logs/' does not exist in HDFS.

Solution

1. Step 1: Check Flume configuration connections

   Flume requires explicit binding of sources, channels, and sinks. Missing bindings cause no data flow.

2. Step 2: Verify bindings in config

   The config lacks lines like agent1.sources.source1.channels = channel1 and agent1.sinks.sink1.channel = channel1.

3. Final Answer:

   The channel is not connected to the sink and source. -> Option C

4. Quick Check:

   Flume needs source-channel-sink bindings [OK]

Hint: Always bind source, channel, and sink in Flume config [OK]

Common Mistakes:

• Ignoring missing bindings in Flume config
• Assuming wrong tail command causes failure
• Thinking sink path must pre-exist

5. You have a large dataset arriving daily and also need to monitor user activity instantly. Which ingestion strategy should you use in Hadoop for each case?

hard

A. Use batch ingestion for daily data and real-time ingestion for user activity.

B. Use real-time ingestion for both daily data and user activity.

C. Use batch ingestion for user activity and real-time ingestion for daily data.

D. Use batch ingestion only for both cases.

Solution

1. Step 1: Analyze data arrival and processing needs

   Large daily data fits batch ingestion since it processes data in groups periodically.

2. Step 2: Consider immediacy of user activity monitoring

   User activity requires instant processing, so real-time ingestion is suitable.

3. Final Answer:

   Use batch ingestion for daily data and real-time ingestion for user activity. -> Option A

4. Quick Check:

   Batch = large periodic, Real-time = instant [OK]

Hint: Batch for big sets, real-time for instant needs [OK]

Common Mistakes:

• Mixing batch and real-time roles
• Using real-time for large daily data unnecessarily
• Ignoring immediacy of user activity data