Event streaming concept in Kafka - Commands & Configuration

Introduction

Event streaming lets systems send and receive data continuously as events happen. It helps apps react quickly to new information without waiting for batch updates.

When you want to track user actions on a website in real time to personalize content.

When you need to process sensor data from machines immediately to detect faults.

When you want to update inventory levels instantly as sales happen in an online store.

When you want to build a chat app that shows messages instantly to all users.

When you want to collect logs and metrics from many servers continuously for monitoring.

Config File - server.properties

server.properties

broker.id=1
listeners=PLAINTEXT://:9092
log.dirs=/tmp/kafka-logs
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000

This file configures a Kafka broker, which is a server that stores and manages event streams.

broker.id: Unique ID for this Kafka server.

listeners: Network address and port Kafka listens on.

log.dirs: Where Kafka stores event data on disk.

zookeeper.connect: Address of Zookeeper, which helps Kafka manage cluster state.

Commands

Starts the Kafka broker using the configuration file to begin accepting event streams.

Terminal

kafka-server-start.sh server.properties

Expected Output

[2024-06-01 12:00:00,000] INFO Kafka startTimeMs set to 1685611200000 (kafka.server.KafkaServer)

[2024-06-01 12:00:01,000] INFO started (kafka.server.KafkaServer)

Creates a new event stream topic named 'user-actions' where events like user clicks will be sent.

Terminal

kafka-topics.sh --create --topic user-actions --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

Expected Output

Created topic user-actions.

→

--topic - Name of the event stream topic

→

--partitions - Number of partitions for parallelism

→

--replication-factor - Number of copies for fault tolerance

Starts a command line tool to send events (messages) to the 'user-actions' event stream.

Terminal

kafka-console-producer.sh --topic user-actions --bootstrap-server localhost:9092

Expected Output

No output (command runs silently)

→

--topic - Topic to send events to

→

--bootstrap-server - Kafka server address

Starts a command line tool to read all events from the 'user-actions' stream from the start.

Terminal

kafka-console-consumer.sh --topic user-actions --bootstrap-server localhost:9092 --from-beginning

Expected Output

No output (command runs silently)

→

--from-beginning - Read all past events, not just new ones

Key Concept

If you remember nothing else from this pattern, remember: event streaming lets apps send and receive data continuously and instantly as things happen.

Common Mistakes

Not starting the Kafka broker before creating topics or sending events.

Commands fail because the Kafka server is not running to handle requests.

Always start the Kafka broker first with kafka-server-start.sh and confirm it is running.

Creating topics without specifying replication factor or partitions.

Kafka may create topics with default settings that do not meet fault tolerance or performance needs.

Specify --partitions and --replication-factor explicitly when creating topics.

Using the consumer without --from-beginning when you want to see all past events.

Consumer only shows new events arriving after it starts, missing earlier data.

Add --from-beginning flag to see all events from the start of the stream.

Summary

Start the Kafka broker to run the event streaming server.

Create a topic to organize and store related events.

Use the producer command to send events to the topic.

Use the consumer command to read events from the topic, optionally from the beginning.

Practice

1. What is the main purpose of event streaming in Kafka?

easy

A. To batch process data once a day

B. To store data permanently in a database

C. To create static reports from data

D. To send data as it happens for real-time updates

Solution

1. Step 1: Understand event streaming concept

   Event streaming means sending data continuously as events happen, not waiting to batch or store permanently.

2. Step 2: Match purpose with options

   Only To send data as it happens for real-time updates describes sending data as it happens for real-time updates, which is the core of event streaming.

3. Final Answer:

   To send data as it happens for real-time updates -> Option D

4. Quick Check:

   Event streaming = real-time data flow [OK]

Hint: Event streaming means data flows live, not stored or batched [OK]

Common Mistakes:

• Confusing event streaming with batch processing
• Thinking event streaming stores data permanently
• Assuming event streaming creates static reports

2. Which of the following is the correct way to describe a Kafka producer?

easy

A. A database that stores events permanently

B. A component that reads events from topics

C. A component that sends events to topics

D. A tool that visualizes event data

Solution

1. Step 1: Define Kafka producer role

   A Kafka producer is the part that sends or writes events into Kafka topics.

2. Step 2: Compare options with definition

   A component that sends events to topics correctly states the producer sends events to topics; others describe consumers, databases, or tools.

3. Final Answer:

   A component that sends events to topics -> Option C

4. Quick Check:

   Producer = sends events [OK]

Hint: Producer sends events; consumer reads events [OK]

Common Mistakes:

• Mixing up producer and consumer roles
• Thinking producer stores data permanently
• Confusing producer with visualization tools

3. Given this Kafka event flow: Producer sends events to topic 'orders'. Consumer reads from 'orders'. What happens if the consumer is slow?

medium

A. Events are lost immediately

B. Events accumulate in the topic until consumed

C. Producer stops sending events

D. Topic deletes old events instantly

Solution

1. Step 1: Understand Kafka topic behavior

   Kafka stores events in topics until consumers read them; slow consumers cause events to accumulate.

2. Step 2: Analyze options based on behavior

   Events accumulate in the topic until consumed matches this behavior; events wait in topic. Events are not lost immediately, producer keeps sending, and topic deletes old events only after retention period.

3. Final Answer:

   Events accumulate in the topic until consumed -> Option B

4. Quick Check:

   Slow consumer = event backlog in topic [OK]

Hint: Slow consumer causes event backlog, not loss [OK]

Common Mistakes:

• Assuming events are lost if consumer is slow
• Thinking producer stops sending events
• Believing topics delete events instantly

4. You wrote a Kafka consumer but it never receives any events. Which is the most likely mistake?

medium

A. The consumer is subscribed to the wrong topic

B. The producer is sending events correctly

C. The topic has too many partitions

D. The consumer is reading from the correct topic

Solution

1. Step 1: Identify why consumer gets no events

   If a consumer never receives events, a common cause is subscribing to the wrong topic.

2. Step 2: Evaluate options

   The consumer is subscribed to the wrong topic points to wrong topic subscription, which explains no events. Other options do not explain lack of events.

3. Final Answer:

   The consumer is subscribed to the wrong topic -> Option A

4. Quick Check:

   No events = wrong topic subscription [OK]

Hint: Check consumer topic subscription first [OK]

Common Mistakes:

• Ignoring topic subscription errors
• Blaming producer when it sends correctly
• Confusing topic partitions with event delivery

5. You want to build a real-time dashboard that updates instantly when new sales happen. Which Kafka setup best supports this?

hard

A. Producer sends sales events to a topic; consumer reads and updates dashboard immediately

B. Producer batches sales events daily; consumer reads once a day

C. Producer sends events to multiple unrelated topics; consumer reads only one

D. Consumer writes sales events back to producer

Solution

1. Step 1: Identify real-time update needs

   A real-time dashboard requires events sent and consumed instantly to update live.

2. Step 2: Match Kafka setup to real-time flow

   Producer sends sales events to a topic; consumer reads and updates dashboard immediately describes producer sending events to a topic and consumer reading immediately to update dashboard, fitting real-time needs.

3. Step 3: Eliminate other options

   Producer batches sales events daily; consumer reads once a day is batch, not real-time; C is inconsistent topic usage; D is incorrect flow direction.

4. Final Answer:

   Producer sends sales events to a topic; consumer reads and updates dashboard immediately -> Option A

5. Quick Check:

   Real-time dashboard = immediate event flow [OK]

Hint: Real-time needs immediate produce and consume flow [OK]

Common Mistakes:

• Choosing batch processing for real-time needs
• Using multiple unrelated topics incorrectly
• Confusing producer and consumer roles