Batch vs real-time ingestion in Hadoop

Introduction

We need to bring data into our system to analyze it. Batch and real-time ingestion are two ways to do this, each with different speed and use cases.

When you collect daily sales data from stores and process it once a day.

When you want to monitor live sensor data from machines to detect problems immediately.

When you gather website logs every hour to analyze user behavior.

When you track live social media feeds to respond quickly to trends.

When you update customer records in bulk at night versus updating them instantly when a change happens.

Syntax

Hadoop

Batch ingestion:
  - Collect data in groups (batches).
  - Process batches at scheduled times.

Real-time ingestion:
  - Collect data continuously.
  - Process data immediately as it arrives.

Batch ingestion handles large amounts of data at once.

Real-time ingestion handles data instantly but usually in smaller pieces.

Examples

This uploads a file to Hadoop HDFS for batch processing later.

Hadoop

Batch ingestion example:
hadoop fs -put /local/data/file.csv /hdfs/data/batch/
Process batch data once a day with a scheduled job.

Kafka sends data continuously to Hadoop for immediate processing.

Hadoop

Real-time ingestion example:
Use Apache Kafka to stream data into Hadoop in real-time.

Sample Program

This code shows two ways to bring data into Hadoop. First, it uploads a batch file to HDFS. Then, it listens to a Kafka topic to simulate real-time data ingestion.

Hadoop

# This is a conceptual example showing batch vs real-time ingestion in Hadoop environment

# Batch ingestion simulation: upload a file to HDFS
import subprocess

batch_file = '/local/data/sales.csv'
hdfs_path = '/hdfs/data/batch/'

# Upload batch file to HDFS
subprocess.run(['hadoop', 'fs', '-put', batch_file, hdfs_path])

print('Batch ingestion: File uploaded to HDFS for later processing.')

# Real-time ingestion simulation: consume messages from Kafka and write to HDFS
from kafka import KafkaConsumer

consumer = KafkaConsumer('sensor-data', bootstrap_servers=['localhost:9092'])

print('Real-time ingestion: Listening to Kafka topic and writing to HDFS...')

count = 0
for message in consumer:
    # Here we would write message.value to HDFS or process immediately
    count += 1
    if count >= 3:  # Limit to 3 messages for demo
        break

print(f'Real-time ingestion: Processed {count} messages from Kafka.')

Important Notes

Batch ingestion is simpler but slower; good for large, less urgent data.

Real-time ingestion is faster but more complex; good for immediate insights.

Choosing depends on how quickly you need the data and system resources.

Summary

Batch ingestion collects and processes data in groups at set times.

Real-time ingestion processes data instantly as it arrives.

Use batch for large, periodic data and real-time for immediate needs.