Amazon MSK in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this Kafka consumer configuration?

Consider the following Kafka consumer configuration snippet for Amazon MSK. What will be the output when the consumer tries to read messages from a topic that does not exist?

Kafka

Properties props = new Properties();
props.put("bootstrap.servers", "b-1.mskcluster.example.com:9092,b-2.mskcluster.example.com:9092");
props.put("group.id", "test-group");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Collections.singletonList("nonexistent-topic"));
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5));
System.out.println("Number of records: " + records.count());

A Number of records: 0

B Exception: org.apache.kafka.common.errors.UnknownTopicOrPartitionException

C Number of records: null

D Consumer hangs indefinitely waiting for messages

💡 Hint

Think about how Kafka handles subscriptions to topics that do not exist yet.

When a Kafka consumer subscribes to a topic that does not exist, it does not throw an error immediately. Instead, it waits and polls, returning zero records until the topic is created.

🧠 Conceptual

intermediate

Which Amazon MSK feature ensures data durability across multiple availability zones?

Amazon MSK is designed to be highly available and durable. Which feature primarily ensures that your Kafka data is safe even if one availability zone fails?

A Replication factor configured on Kafka topics

B Using a single broker in one availability zone

C Enabling auto-scaling on the MSK cluster

D Using client-side caching in Kafka producers

💡 Hint

Think about how Kafka replicates data to avoid data loss.

Kafka topics have a replication factor that determines how many copies of each partition exist across brokers in different availability zones, ensuring durability.

🔧 Debug

advanced

Why does this Kafka producer fail to send messages to Amazon MSK?

Examine the following Kafka producer code snippet. It fails to send messages to the MSK cluster. What is the most likely cause?

Kafka

Properties props = new Properties();
props.put("bootstrap.servers", "b-1.mskcluster.example.com:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
ProducerRecord<String, String> record = new ProducerRecord<>("my-topic", "key1", "value1");
producer.send(record);
producer.close();

A Incorrect topic name format; topics must start with 'msk-' prefix

B KafkaProducer requires a consumer group id to send messages

C Producer is missing a call to flush() before close()

D Missing security configuration for SASL/SCRAM authentication

💡 Hint

Amazon MSK clusters often require authentication settings in the client configuration.

If the MSK cluster is configured with SASL/SCRAM authentication, the producer must include the proper security settings to connect and send messages.

📝 Syntax

advanced

Identify the syntax error in this Kafka topic creation command for Amazon MSK.

Which option contains the correct syntax to create a Kafka topic named 'orders' with 3 partitions and a replication factor of 2 using kafka-topics.sh CLI?

A kafka-topics.sh --create --topic orders --partition 3 --replication 2 --bootstrap-server b-1.mskcluster.example.com:9092

B kafka-topics.sh --create --topic orders --partitions 3 --replication-factor 2 --bootstrap-server b-1.mskcluster.example.com:9092

C kafka-topics.sh --create --topic orders --partitions=3 --replication-factor=2 --bootstrap-server b-1.mskcluster.example.com:9092

D kafka-topics.sh --create orders --partitions 3 --replication-factor 2 --bootstrap-server b-1.mskcluster.example.com:9092

💡 Hint

Check the exact option names and their spelling for partitions and replication factor.

The correct flags are --partitions and --replication-factor without equals signs. The topic name must follow the --topic flag.

🚀 Application

expert

How to configure an Amazon MSK cluster for encryption in transit and at rest?

You want to secure your Amazon MSK cluster so that all data is encrypted both when moving between clients and brokers, and when stored on disk. Which configuration steps are required?

A Use plaintext communication and enable EBS encryption only

B Enable SASL authentication only; encryption is automatic

C Enable TLS encryption for client-broker communication and enable AWS KMS encryption for EBS volumes

D Configure client-side encryption libraries; no MSK cluster changes needed

💡 Hint

Think about both network encryption and storage encryption options in MSK.

Encryption in transit is achieved by enabling TLS on the MSK cluster and clients. Encryption at rest is done by enabling AWS KMS encryption on the EBS volumes used by the brokers.

Practice

1. What is the main benefit of using Amazon MSK for Kafka users?

easy

A. It manages Kafka servers for you, so you don't have to.

B. It replaces Kafka with a different messaging system.

C. It requires you to manually configure all Kafka brokers.

D. It only works with AWS Lambda functions.

Solution

1. Step 1: Understand Amazon MSK's purpose

   Amazon MSK is a managed service that runs Kafka clusters for you.

2. Step 2: Compare options with MSK features

   Only It manages Kafka servers for you, so you don't have to. correctly states that MSK manages Kafka servers, removing manual server management.

3. Final Answer:

   It manages Kafka servers for you, so you don't have to. -> Option A

4. Quick Check:

   Managed Kafka service = It manages servers [OK]

Hint: Remember MSK means managed Kafka servers [OK]

Common Mistakes:

• Thinking MSK replaces Kafka
• Assuming manual server setup is needed
• Believing MSK only works with Lambda

2. Which of the following is the correct way to specify the bootstrap servers when connecting to an Amazon MSK cluster in Kafka client configuration?

easy

A. "localhost:9092"

B. "msk://cluster-endpoint"

C. "127.0.0.1:2181"

D. "b-1.mskcluster.abcd1234.kafka.us-east-1.amazonaws.com:9092,b-2.mskcluster.abcd1234.kafka.us-east-1.amazonaws.com:9092"

Solution

1. Step 1: Identify MSK cluster endpoints format

   Amazon MSK provides bootstrap servers as DNS names with ports, like in "b-1.mskcluster.abcd1234.kafka.us-east-1.amazonaws.com:9092,b-2.mskcluster.abcd1234.kafka.us-east-1.amazonaws.com:9092".

2. Step 2: Eliminate incorrect formats

   "localhost:9092" is localhost, not MSK. "127.0.0.1:2181" is Zookeeper port, not Kafka. "msk://cluster-endpoint" is not a valid Kafka bootstrap server format.

3. Final Answer:

   "b-1.mskcluster.abcd1234.kafka.us-east-1.amazonaws.com:9092,b-2.mskcluster.abcd1234.kafka.us-east-1.amazonaws.com:9092" -> Option D

4. Quick Check:

   MSK bootstrap servers = DNS names with ports [OK]

Hint: MSK bootstrap servers are DNS names with port 9092 [OK]

Common Mistakes:

• Using localhost instead of MSK endpoints
• Confusing Zookeeper port with Kafka port
• Using invalid URI schemes like msk://

3. Given the following Kafka producer code snippet connecting to Amazon MSK, what will be the output if the message is sent successfully?

producer.send(new ProducerRecord<>("my-topic", "key1", "Hello MSK"));
System.out.println("Message sent successfully");

medium

A. Error: Connection refused

B. Message sent successfully

C. No output

D. Message failed to send

Solution

1. Step 1: Understand producer.send behavior

   The send method is asynchronous but the print statement runs immediately after.

2. Step 2: Check output after send call

   Since no error handling is shown and send is called, the print statement will output "Message sent successfully".

3. Final Answer:

   Message sent successfully -> Option B

4. Quick Check:

   Producer send + print = Message sent successfully [OK]

Hint: Print after send shows success unless exception thrown [OK]

Common Mistakes:

• Assuming send blocks until confirmed
• Expecting error without exception handling
• Thinking no output means success

4. You have this Kafka consumer code connecting to Amazon MSK:

Properties props = new Properties();
props.put("bootstrap.servers", "b-1.mskcluster.kafka:9092");
props.put("group.id", "test-group");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer consumer = new KafkaConsumer<>(props);
consumer.subscribe(Collections.singletonList("my-topic"));
consumer.poll(Duration.ofMillis(100));

The consumer throws a runtime error about deserializer class not found. What is the likely cause?

medium

A. The topic name is invalid.

B. The bootstrap server address is wrong.

C. The deserializer class names are incorrect or misspelled.

D. The group.id property is missing.

Solution

1. Step 1: Analyze the error message

   The error says deserializer class not found, so the problem is with deserializer class names.

2. Step 2: Check deserializer class names in code

   They must be fully qualified and spelled exactly. Any typo causes class not found error.

3. Final Answer:

   The deserializer class names are incorrect or misspelled. -> Option C

4. Quick Check:

   Deserializer class error = wrong class name [OK]

Hint: Check spelling of deserializer class names first [OK]

Common Mistakes:

• Ignoring class name typos
• Assuming bootstrap server causes deserializer error
• Missing group.id but error is about deserializer

5. You want to create a Kafka consumer in Java that connects to an Amazon MSK cluster and only processes messages with keys starting with "user-". Which approach correctly applies this filter after receiving messages?

hard

A. Use a Kafka Streams application with a filter on the key prefix.

B. Set the consumer property "key.filter.prefix" to "user-".

C. Subscribe only to topics named "user-*".

D. Configure the producer to send only keys starting with "user-".

Solution

1. Step 1: Understand Kafka consumer filtering options

   Kafka consumers do not support filtering by key prefix via configuration properties.

2. Step 2: Identify correct filtering method

   Kafka Streams allows processing and filtering messages by key prefix programmatically.

3. Step 3: Evaluate other options

   There is no Kafka consumer property like "key.filter.prefix". Subscribing to topics named "user-*" filters by topic name, not message keys. Producer configuration controls what is sent, not what the consumer processes.

4. Final Answer:

   Use a Kafka Streams application with a filter on the key prefix. -> Option A

5. Quick Check:

   Filter keys after consuming = Kafka Streams filter [OK]

Hint: Filter keys in Kafka Streams, not consumer config [OK]

Common Mistakes:

• Expecting consumer config to filter keys
• Filtering by topic name instead of message key
• Trying to filter on producer side only