Amazon MSK in Kafka - Step-by-Step Execution

Process Flow - Amazon MSK

Create MSK Cluster

↓

Configure Topics

↓

Produce Messages

↓

MSK Brokers Receive

↓

Store Messages in Partitions

↓

Consume Messages

↓

Process Data

↓

End

This flow shows how Amazon MSK manages Kafka clusters: create cluster, configure topics, produce messages, store them, and consume for processing.

Execution Sample

Kafka

from kafka import KafkaProducer, KafkaConsumer

producer = KafkaProducer(bootstrap_servers='b-1.msk-cluster:9092')
producer.send('test-topic', b'Hello MSK')
producer.flush()

consumer = KafkaConsumer('test-topic', bootstrap_servers='b-1.msk-cluster:9092', value_deserializer=lambda m: m.decode('utf-8'))
for msg in consumer:
    print(msg.value)

This code sends a message to an MSK topic and then reads messages from it.

Process Table

Step	Action	Evaluation	Result
1	Create KafkaProducer with MSK bootstrap server	Connect to MSK brokers	Producer ready to send messages
2	Send message 'Hello MSK' to 'test-topic'	Message serialized and sent	Message queued in MSK topic partition
3	Create KafkaConsumer for 'test-topic'	Connect to MSK brokers	Consumer ready to receive messages
4	Consumer polls messages	Receives 'Hello MSK' message	Message available for processing
5	Print message value	Decode bytes to string	Output: Hello MSK
6	No more messages	Consumer waits or exits	End of execution

💡 No more messages to consume, consumer stops polling

Status Tracker

Variable	Start	After Step 2	After Step 4	Final
producer	None	KafkaProducer instance connected	KafkaProducer instance connected	KafkaProducer instance connected
consumer	None	None	KafkaConsumer instance connected	KafkaConsumer instance connected
message_sent	None	b'Hello MSK'	b'Hello MSK'	b'Hello MSK'
message_received	None	None	Hello MSK	Hello MSK

Key Moments - 3 Insights

Why does the consumer receive the message after the producer sends it?

Because the producer sends the message to the MSK topic partition, which stores it until the consumer polls and receives it, as shown in steps 2 and 4 of the execution_table.

What happens if the consumer tries to read before the producer sends a message?

The consumer will wait or poll repeatedly but receive no messages until the producer sends one, so no output appears until step 4 in the execution_table.

Why do we need to specify bootstrap_servers in both producer and consumer?

Because both need to connect to the MSK cluster brokers to send or receive messages, as shown in steps 1 and 3 where connections are established.

Visual Quiz - 3 Questions

Look at the execution_table, what is the output printed at step 5?

A b'Hello MSK'

B Hello MSK

C test-topic

D No output

💡 Hint

Check the 'Result' column at step 5 in the execution_table.

At which step does the consumer connect to the MSK brokers?

A Step 3

B Step 2

C Step 1

D Step 4

💡 Hint

Look at the 'Action' and 'Evaluation' columns for consumer creation in the execution_table.

If the producer did not send any message, what would the consumer receive at step 4?

A The last message sent

B An error

C No messages

D A default message

💡 Hint

Refer to the key_moments explanation about consumer behavior before messages are sent.

Concept Snapshot

Amazon MSK manages Kafka clusters in the cloud.
Create clusters, configure topics, produce and consume messages.
Producer sends messages to topic partitions.
Consumer reads messages from topics.
Both connect via bootstrap servers.
MSK handles storage and delivery reliably.

Full Transcript

Amazon MSK is a managed Kafka service that lets you create Kafka clusters easily. You configure topics where messages are sent and received. A producer connects to the MSK brokers and sends messages to a topic. These messages are stored in partitions inside the cluster. A consumer connects to the same brokers and polls messages from the topic partitions. When the consumer receives a message, it can process or print it. This flow ensures reliable message delivery and easy scaling. Both producer and consumer need to know the bootstrap servers to connect to the cluster. If no messages are sent, the consumer waits without output. This example shows sending 'Hello MSK' and receiving it back.