Kafka integration with Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Kafka and Hadoop Integration

Which component is primarily responsible for moving data from Kafka to Hadoop in a typical integration setup?

A Kafka Connect with HDFS Sink Connector

B Apache Spark Streaming directly writing to Kafka

C Hadoop MapReduce reading from Kafka topics

D Kafka Producer sending data to HDFS

💡 Hint

Think about the tool designed to connect Kafka with external systems like Hadoop.

Kafka Connect with HDFS Sink Connector is designed to stream data from Kafka topics directly into Hadoop's HDFS storage.

❓ Predict Output

intermediate

Output of Kafka Connect HDFS Sink Configuration

Given the following Kafka Connect HDFS Sink configuration snippet, what is the expected output directory structure in HDFS?

Hadoop

name=hdfs-sink-connector
connector.class=io.confluent.connect.hdfs.HdfsSinkConnector
tasks.max=1
topics=test-topic
hdfs.url=hdfs://namenode:8020
flush.size=3
format.class=io.confluent.connect.hdfs.avro.AvroFormat
hdfs.topics.dir=/user/kafka/topics

A /user/hdfs/data/test-topic/ with JSON files flushed every 3 records

B /user/kafka/topics/test-topic/partition=0/ with Avro files flushed every 3 records

C /tmp/kafka/test-topic/partition=1/ with Parquet files flushed every 1 record

D /user/kafka/test-topic/partition=0/ with CSV files flushed every 5 records

💡 Hint

Look at the topic name, flush size, and format class in the config.

The connector writes to /user/kafka/topics/{topic}/partition={partition}/ with Avro files flushed after 3 records as per flush.size.

❓ data_output

advanced

Data Output Format from Kafka to Hadoop

When using Kafka Connect HDFS Sink with the default Avro format, what is the schema of the stored data files?

A Avro files containing both data and schema embedded in each file

B Plain text files with JSON strings only

C Binary files without any schema information

D CSV files with header rows describing schema

💡 Hint

Consider how Avro format handles schema and data storage.

Avro files embed the schema along with the data, enabling schema evolution and self-describing files.

🔧 Debug

advanced

Identifying Error in Kafka to Hadoop Data Pipeline

Given a Kafka Connect HDFS Sink connector failing with error 'Failed to write to HDFS: Permission denied', what is the most likely cause?

A HDFS Namenode is down

B Kafka topic does not exist

C Kafka Connect worker does not have write permissions on the target HDFS directory

D Kafka Connect configuration missing flush.size parameter

💡 Hint

Think about file system permissions and access rights.

The error indicates the connector cannot write to HDFS due to insufficient permissions on the target directory.

🚀 Application

expert

Optimizing Kafka to Hadoop Data Ingestion

You want to optimize the Kafka Connect HDFS Sink to reduce small file creation and improve throughput. Which configuration change is most effective?

A Set topics to multiple unrelated topics

B Decrease the tasks.max to 1

C Change format.class to CSVFormat

D Increase the flush.size parameter to a larger number

💡 Hint

Think about how flush.size affects file creation frequency.

Increasing flush.size causes the connector to write larger files less frequently, reducing small file overhead and improving throughput.