Why cloud-native deployment matters in Kafka - See It in Action

Why cloud-native deployment matters

📖 Scenario: You work at a company that uses Apache Kafka to handle real-time data streams. The company wants to move its Kafka deployment to the cloud to improve scalability and reliability.

🎯 Goal: Build a simple Kafka producer and consumer setup that simulates sending and receiving messages in a cloud-native environment.

📋 What You'll Learn

Create a Kafka topic named cloud_native_topic

Create a Kafka producer that sends 3 messages to cloud_native_topic

Create a Kafka consumer that reads messages from cloud_native_topic

Print the received messages to show the flow of data

💡 Why This Matters

🌍 Real World

Cloud-native Kafka deployments allow companies to handle large-scale, real-time data streams with flexibility and reliability.

💼 Career

Understanding Kafka and cloud-native deployment is valuable for roles in data engineering, cloud architecture, and real-time analytics.

1

Create Kafka topic

Create a Kafka topic named cloud_native_topic using the Kafka admin client.

Kafka

# Your code here to create the topic 'cloud_native_topic'

Need a hint?

Use KafkaAdminClient and NewTopic to create the topic.

2

Create Kafka producer

Create a Kafka producer named producer that connects to localhost:9092 and sends these messages to cloud_native_topic: 'Message 1', 'Message 2', 'Message 3'.

Kafka

from kafka.admin import KafkaAdminClient, NewTopic

admin_client = KafkaAdminClient(bootstrap_servers='localhost:9092')
topic = NewTopic(name='cloud_native_topic', num_partitions=1, replication_factor=1)
admin_client.create_topics(new_topics=[topic], validate_only=False)

# Create a Kafka producer and send 3 messages to 'cloud_native_topic'
# Your code here

Need a hint?

Use KafkaProducer and the send method to send messages.

3

Create Kafka consumer

Create a Kafka consumer named consumer that connects to localhost:9092 and subscribes to cloud_native_topic.

Kafka

from kafka.admin import KafkaAdminClient, NewTopic
from kafka import KafkaProducer, KafkaConsumer

admin_client = KafkaAdminClient(bootstrap_servers='localhost:9092')
topic = NewTopic(name='cloud_native_topic', num_partitions=1, replication_factor=1)
admin_client.create_topics(new_topics=[topic], validate_only=False)

producer = KafkaProducer(bootstrap_servers='localhost:9092')
producer.send('cloud_native_topic', b'Message 1')
producer.send('cloud_native_topic', b'Message 2')
producer.send('cloud_native_topic', b'Message 3')
producer.flush()

# Create a Kafka consumer subscribed to 'cloud_native_topic'
# Your code here

Need a hint?

Use KafkaConsumer and subscribe to 'cloud_native_topic'.

4

Print received messages

Use a for loop to read messages from consumer and print each message's value decoded as UTF-8.

Kafka

from kafka.admin import KafkaAdminClient, NewTopic
from kafka import KafkaProducer, KafkaConsumer

admin_client = KafkaAdminClient(bootstrap_servers='localhost:9092')
topic = NewTopic(name='cloud_native_topic', num_partitions=1, replication_factor=1)
admin_client.create_topics(new_topics=[topic], validate_only=False)

producer = KafkaProducer(bootstrap_servers='localhost:9092')
producer.send('cloud_native_topic', b'Message 1')
producer.send('cloud_native_topic', b'Message 2')
producer.send('cloud_native_topic', b'Message 3')
producer.flush()

consumer = KafkaConsumer('cloud_native_topic', bootstrap_servers='localhost:9092', auto_offset_reset='earliest', consumer_timeout_ms=1000)

# Print each message received from the consumer
# Your code here

Need a hint?

Use a for loop over consumer and decode each message's value.