Amazon MSK in Kafka - Mini Project: Build & Apply

Amazon MSK Basic Producer and Consumer

📖 Scenario: You are working with Amazon Managed Streaming for Apache Kafka (MSK) to send and receive messages in a real-time application. You will create a simple Kafka producer and consumer setup to understand how data flows through MSK.

🎯 Goal: Build a basic Kafka producer that sends messages to a topic and a consumer that reads messages from that topic using Amazon MSK.

📋 What You'll Learn

Create a Kafka topic named test-topic

Write a producer script that sends 3 specific messages to test-topic

Write a consumer script that reads messages from test-topic

Print the consumed messages to the console

💡 Why This Matters

🌍 Real World

Amazon MSK is used to build real-time data pipelines and streaming applications that require reliable message delivery and scalability.

💼 Career

Understanding how to produce and consume messages with Amazon MSK is essential for roles in data engineering, backend development, and cloud architecture.

1

Create Kafka topic configuration

Create a variable called topic_name and set it to the string 'test-topic'.

Kafka

# Create a variable called topic_name and set it to 'test-topic'
topic_name = ''
# Your code here

Need a hint?

Use a simple string assignment to set the topic name.

2

Set up Kafka producer messages

Create a list called messages containing these exact strings: 'Hello MSK', 'Kafka is cool', and 'Streaming data'.

Kafka

topic_name = 'test-topic'
# Create a list called messages with the specified strings
messages = []
# Your code here

Need a hint?

Use a Python list with the exact strings in the correct order.

3

Write Kafka producer loop to send messages

Write a for loop using message as the iterator to send each message in messages to the Kafka topic topic_name using producer.send(topic_name, value=message.encode()). Assume producer is already created.

Kafka

topic_name = 'test-topic'
messages = ['Hello MSK', 'Kafka is cool', 'Streaming data']
# Send each message to the Kafka topic using producer.send
# Your code here

Need a hint?

Use a for loop to iterate over messages and send each one encoded as bytes.

4

Consume and print messages from Kafka topic

Write a for loop using msg as the iterator to consume messages from consumer. For each msg, print the decoded message value using print(msg.value.decode()). Assume consumer is already subscribed to topic_name.

Kafka

topic_name = 'test-topic'
messages = ['Hello MSK', 'Kafka is cool', 'Streaming data']
for message in messages:
    producer.send(topic_name, value=message.encode())
# Consume messages from consumer and print each decoded message
# Your code here

Need a hint?

Use a for loop to read messages from consumer and print each decoded message.