Kafka integration with Hadoop

Introduction

Kafka integration with Hadoop helps move data from real-time streams into big storage for analysis. It connects fast data sources with Hadoop's storage and processing power.

You want to collect live data from sensors and store it in Hadoop for later analysis.

You have user activity logs streaming in and want to save them in Hadoop for batch processing.

You need to combine real-time data with historical data stored in Hadoop.

You want to build a data pipeline that moves messages from Kafka topics into Hadoop storage automatically.

Syntax

Hadoop

kafka-console-consumer --bootstrap-server <kafka-broker> --topic <topic-name> --from-beginning | hadoop fs -appendToFile - <hdfs-path>

This command reads messages from Kafka and writes them into Hadoop's HDFS.

You can also use tools like Apache Flume or Apache NiFi for more complex integration.

Examples

This reads all messages from the 'sensor-data' Kafka topic and appends them to a file in HDFS under the specified path.

Hadoop

kafka-console-consumer --bootstrap-server localhost:9092 --topic sensor-data --from-beginning | hadoop fs -appendToFile - /user/hadoop/sensor-data/data.txt

Starts a Flume agent that reads from Kafka and writes to HDFS using a configuration file.

Hadoop

flume-ng agent -n agent1 -c conf -f flume-kafka-hdfs.conf

Sample Program

This script reads all messages from Kafka topic 'test-topic' and appends them as a file in HDFS. Then it lists and shows the saved data.

Hadoop

# This example uses Kafka console consumer and Hadoop fs commands
# Step 1: Start Kafka consumer to read from topic 'test-topic'
# Step 2: Pipe output to HDFS file '/user/hadoop/test-topic-data/data.txt'

kafka-console-consumer --bootstrap-server localhost:9092 --topic test-topic --from-beginning | hadoop fs -appendToFile - /user/hadoop/test-topic-data/data.txt

# After running, check data in HDFS
hadoop fs -ls /user/hadoop/test-topic-data
hadoop fs -cat /user/hadoop/test-topic-data/data.txt

Important Notes

Make sure Kafka and Hadoop services are running before integration.

Data formats should be compatible between Kafka messages and Hadoop storage.

For large-scale or continuous data, use tools like Apache Flume or NiFi instead of simple console commands.

Summary

Kafka integration with Hadoop moves streaming data into big storage for analysis.

Use simple commands for small data or tools like Flume for production pipelines.

Check data in HDFS after transfer to confirm successful integration.