Confluent Cloud overview in Kafka - Mini Project: Build & Apply

Confluent Cloud Overview with Kafka

📖 Scenario: You are working with Confluent Cloud, a managed Kafka service that helps you handle real-time data streams easily without managing servers.Imagine you want to create a simple Kafka topic and produce some messages to it, then consume those messages to see how data flows in Confluent Cloud.

🎯 Goal: Build a simple Kafka setup in Confluent Cloud by creating a topic, producing messages, and consuming them to understand how Confluent Cloud works.

📋 What You'll Learn

Create a Kafka topic named test-topic

Produce three messages to test-topic with exact values: 'Hello', 'Confluent', 'Cloud'

Consume messages from test-topic and store them in a list called received_messages

Print the received_messages list to show the consumed messages

💡 Why This Matters

🌍 Real World

Confluent Cloud is used by companies to handle real-time data streams like user activity, logs, or sensor data without managing Kafka servers themselves.

💼 Career

Understanding how to create topics, produce, and consume messages in Confluent Cloud is essential for roles in data engineering, backend development, and real-time analytics.

1

Create the Kafka topic

Create a list called topics with a single string element 'test-topic' to represent the Kafka topic you will use.

Kafka

# Create the topics list with 'test-topic'
topics = []
# Your code here

Hint

Think of topics as a list holding the names of Kafka topics you want to work with.

2

Prepare messages to produce

Create a list called messages_to_send with the exact strings 'Hello', 'Confluent', and 'Cloud' to represent messages you will send to the topic.

Kafka

topics = ['test-topic']
# Create the messages_to_send list with 'Hello', 'Confluent', 'Cloud'
messages_to_send = []
# Your code here

Hint

Think of messages_to_send as the data you want to send into your Kafka topic.

3

Simulate consuming messages

Create a list called received_messages and assign it the value of messages_to_send to simulate consuming the messages from the topic.

Kafka

topics = ['test-topic']
messages_to_send = ['Hello', 'Confluent', 'Cloud']
# Assign messages_to_send to received_messages to simulate consumption
received_messages = []
# Your code here

Hint

Since this is a simple overview, just copy the sent messages to received messages to simulate consumption.

4

Print the consumed messages

Write a print statement to display the received_messages list.

Kafka

topics = ['test-topic']
messages_to_send = ['Hello', 'Confluent', 'Cloud']
received_messages = messages_to_send
# Print the received_messages list
# Your code here

Hint

Use print(received_messages) to show the messages you consumed.

Practice

1. What is the main purpose of Confluent Cloud in Kafka usage?

easy

A. It provides a managed Kafka service for easy data streaming.

B. It is a tool to write SQL queries on Kafka topics.

C. It is a database for storing Kafka messages permanently.

D. It is a programming language for Kafka applications.

Solution

1. Step 1: Understand Confluent Cloud's role

   Confluent Cloud is described as a managed Kafka service that simplifies data streaming.

2. Step 2: Compare options with this role

   Only It provides a managed Kafka service for easy data streaming. matches this description; others describe unrelated tools or concepts.

3. Final Answer:

   It provides a managed Kafka service for easy data streaming. -> Option A

4. Quick Check:

   Managed Kafka service = It provides a managed Kafka service for easy data streaming. [OK]

Hint: Remember: Confluent Cloud manages Kafka for you [OK]

Common Mistakes:

• Confusing Confluent Cloud with a database
• Thinking it is a programming language
• Assuming it runs SQL queries directly

2. Which of the following is the correct way to describe a Kafka topic in Confluent Cloud?

easy

A. A topic is a user account in Confluent Cloud.

B. A topic is a programming function to send messages.

C. A topic is a type of database index.

D. A topic is a place where messages are stored and organized.

Solution

1. Step 1: Define Kafka topic concept

   A Kafka topic is a category or feed name to which messages are published and stored.

2. Step 2: Match definition to options

   A topic is a place where messages are stored and organized. correctly states that a topic stores and organizes messages; others are incorrect.

3. Final Answer:

   A topic is a place where messages are stored and organized. -> Option D

4. Quick Check:

   Topic = message storage location [OK]

Hint: Topics hold messages, not functions or users [OK]

Common Mistakes:

• Confusing topics with user accounts
• Thinking topics are functions or indexes
• Mixing topics with database concepts

3. Given the following code snippet for producing a message in Confluent Cloud Kafka client:

producer.produce(topic='orders', value='order123')

What happens when this code runs successfully?

medium

A. The 'orders' topic is deleted.

B. The message 'order123' is sent to the 'orders' topic.

C. The producer client shuts down.

D. An error occurs because 'value' must be a number.

Solution

1. Step 1: Understand produce method

   The produce method sends a message to a specified topic with the given value.

2. Step 2: Analyze the code behavior

   Here, 'order123' is sent as a message to the 'orders' topic; no deletion or shutdown occurs.

3. Final Answer:

   The message 'order123' is sent to the 'orders' topic. -> Option B

4. Quick Check:

   Produce sends message to topic [OK]

Hint: Produce sends data to topic, no deletion or shutdown [OK]

Common Mistakes:

• Thinking produce deletes topics
• Assuming value must be numeric
• Confusing produce with client shutdown

4. You wrote this code to consume messages from a Confluent Cloud topic:

consumer.subscribe(['payments'])
message = consumer.poll(1.0)
print(message.value())

But you get an error: 'NoneType' object has no attribute 'value'. What is the likely cause?

medium

A. The print statement syntax is incorrect.

B. The topic 'payments' does not exist.

C. No messages were available to consume during poll timeout.

D. The consumer client was not connected to Confluent Cloud.

Solution

1. Step 1: Understand poll method behavior

   poll returns None if no messages are received within the timeout period.

2. Step 2: Analyze error cause

   Calling value() on None causes the error; this means no messages arrived during poll.

3. Final Answer:

   No messages were available to consume during poll timeout. -> Option C

4. Quick Check:

   poll returns None if no messages [OK]

Hint: poll returns None if no messages, check before accessing [OK]

Common Mistakes:

• Assuming topic absence causes this error
• Blaming connection issues without checking poll
• Thinking print syntax is wrong

5. You want to create a Confluent Cloud Kafka topic that automatically deletes messages after 7 days. Which configuration should you apply when creating the topic?

hard

A. Set the topic's retention.ms to 604800000 (7 days in milliseconds).

B. Set the topic's cleanup.policy to 'compact'.

C. Set the topic's max.message.bytes to 7.

D. Set the topic's partitions to 7.

Solution

1. Step 1: Understand retention.ms setting

   retention.ms controls how long messages are kept before deletion, in milliseconds.

2. Step 2: Calculate 7 days in milliseconds

   7 days x 24 hours x 60 minutes x 60 seconds x 1000 = 604800000 ms.

3. Step 3: Match configuration to requirement

   Setting retention.ms to 604800000 ensures messages delete after 7 days; other options do not control retention time.

4. Final Answer:

   Set the topic's retention.ms to 604800000 (7 days in milliseconds). -> Option A

5. Quick Check:

   retention.ms = 7 days in ms [OK]

Hint: Use retention.ms for message lifetime in ms [OK]

Common Mistakes:

• Confusing cleanup.policy with retention time
• Setting max.message.bytes instead of retention
• Changing partitions unrelated to retention