Bird
Raised Fist0
Kafkadevops~5 mins

Amazon MSK in Kafka - Commands & Configuration

Choose your learning style10 modes available
LearnWhyDeepVisualTryChallengeProjectRecallTime
Introduction
Amazon MSK is a service that makes it easy to run Apache Kafka without managing servers. It solves the problem of setting up and maintaining Kafka clusters by handling the infrastructure for you.
When you want to stream data between applications without managing Kafka servers yourself
When you need a reliable system to process real-time data like logs or user activity
When you want to scale Kafka easily as your data grows
When you want to integrate Kafka with other AWS services securely
When you want to reduce operational overhead of running Kafka clusters
Commands
This command creates a new Amazon MSK cluster named 'example-cluster' with 3 broker nodes of type kafka.m5.large in the specified subnet and security group.
Terminal
aws kafka create-cluster --cluster-name example-cluster --kafka-version 3.4.0 --number-of-broker-nodes 3 --broker-node-group-info InstanceType=kafka.m5.large,ClientSubnets=subnet-0bb1c79de3EXAMPLE,SecurityGroups=sg-0a123b456c789d012
Expected OutputExpected
{ "ClusterArn": "arn:aws:kafka:us-east-1:123456789012:cluster/example-cluster/abcd1234-abcd-1234-abcd-1234abcd5678", "State": "CREATING" }
→
--cluster-name - Sets the name of the Kafka cluster
→
--kafka-version - Specifies the Kafka version to use
→
--number-of-broker-nodes - Defines how many broker nodes the cluster will have
This command checks the status and details of the Kafka cluster to see if it is ready to use.
Terminal
aws kafka describe-cluster --cluster-arn arn:aws:kafka:us-east-1:123456789012:cluster/example-cluster/abcd1234-abcd-1234-abcd-1234abcd5678
Expected OutputExpected
{ "ClusterInfo": { "ClusterName": "example-cluster", "State": "ACTIVE", "NumberOfBrokerNodes": 3, "KafkaVersion": "3.4.0" } }
→
--cluster-arn - Specifies the unique identifier of the Kafka cluster
This command lists all the Kafka clusters in your AWS account to confirm your cluster is created.
Terminal
aws kafka list-clusters
Expected OutputExpected
{ "ClusterInfoList": [ { "ClusterName": "example-cluster", "ClusterArn": "arn:aws:kafka:us-east-1:123456789012:cluster/example-cluster/abcd1234-abcd-1234-abcd-1234abcd5678", "State": "ACTIVE" } ] }
Key Concept

If you remember nothing else from this pattern, remember: Amazon MSK lets you run Kafka clusters easily by managing the servers and scaling for you.

Common Mistakes
Using incorrect subnet IDs or security group IDs when creating the cluster
The cluster creation will fail or the brokers won't be accessible if network settings are wrong
Verify subnet and security group IDs exist and belong to the correct VPC before running the create command
Not waiting for the cluster state to become ACTIVE before using it
Trying to produce or consume messages before the cluster is ready will cause connection errors
Use the describe-cluster command to check the cluster state and wait until it shows ACTIVE
Summary
Create an Amazon MSK cluster with aws kafka create-cluster specifying name, version, nodes, and network settings.
Check the cluster status with aws kafka describe-cluster to ensure it is ACTIVE before use.
List all clusters with aws kafka list-clusters to confirm your cluster exists and is running.

Practice

(1/5)
1. What is the main benefit of using Amazon MSK for Kafka users?
easy
A. It manages Kafka servers for you, so you don't have to.
B. It replaces Kafka with a different messaging system.
C. It requires you to manually configure all Kafka brokers.
D. It only works with AWS Lambda functions.

Solution

  1. Step 1: Understand Amazon MSK's purpose

    Amazon MSK is a managed service that runs Kafka clusters for you.

  2. Step 2: Compare options with MSK features

    Only It manages Kafka servers for you, so you don't have to. correctly states that MSK manages Kafka servers, removing manual server management.

  3. Final Answer:

    It manages Kafka servers for you, so you don't have to. -> Option A

  4. Quick Check:

    Managed Kafka service = It manages servers [OK]
Hint: Remember MSK means managed Kafka servers [OK]
Common Mistakes:
  • Thinking MSK replaces Kafka
  • Assuming manual server setup is needed
  • Believing MSK only works with Lambda
2. Which of the following is the correct way to specify the bootstrap servers when connecting to an Amazon MSK cluster in Kafka client configuration?
easy
A. "localhost:9092"
B. "msk://cluster-endpoint"
C. "127.0.0.1:2181"
D. "b-1.mskcluster.abcd1234.kafka.us-east-1.amazonaws.com:9092,b-2.mskcluster.abcd1234.kafka.us-east-1.amazonaws.com:9092"

Solution

  1. Step 1: Identify MSK cluster endpoints format

    Amazon MSK provides bootstrap servers as DNS names with ports, like in "b-1.mskcluster.abcd1234.kafka.us-east-1.amazonaws.com:9092,b-2.mskcluster.abcd1234.kafka.us-east-1.amazonaws.com:9092".

  2. Step 2: Eliminate incorrect formats

    "localhost:9092" is localhost, not MSK. "127.0.0.1:2181" is Zookeeper port, not Kafka. "msk://cluster-endpoint" is not a valid Kafka bootstrap server format.

  3. Final Answer:

    "b-1.mskcluster.abcd1234.kafka.us-east-1.amazonaws.com:9092,b-2.mskcluster.abcd1234.kafka.us-east-1.amazonaws.com:9092" -> Option D

  4. Quick Check:

    MSK bootstrap servers = DNS names with ports [OK]
Hint: MSK bootstrap servers are DNS names with port 9092 [OK]
Common Mistakes:
  • Using localhost instead of MSK endpoints
  • Confusing Zookeeper port with Kafka port
  • Using invalid URI schemes like msk://
3. Given the following Kafka producer code snippet connecting to Amazon MSK, what will be the output if the message is sent successfully? 
producer.send(new ProducerRecord<>("my-topic", "key1", "Hello MSK"));
System.out.println("Message sent successfully");
medium
A. Error: Connection refused
B. Message sent successfully
C. No output
D. Message failed to send

Solution

  1. Step 1: Understand producer.send behavior

    The send method is asynchronous but the print statement runs immediately after.

  2. Step 2: Check output after send call

    Since no error handling is shown and send is called, the print statement will output "Message sent successfully".

  3. Final Answer:

    Message sent successfully -> Option B

  4. Quick Check:

    Producer send + print = Message sent successfully [OK]
Hint: Print after send shows success unless exception thrown [OK]
Common Mistakes:
  • Assuming send blocks until confirmed
  • Expecting error without exception handling
  • Thinking no output means success
4. You have this Kafka consumer code connecting to Amazon MSK: 
Properties props = new Properties();
props.put("bootstrap.servers", "b-1.mskcluster.kafka:9092");
props.put("group.id", "test-group");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer consumer = new KafkaConsumer<>(props);
consumer.subscribe(Collections.singletonList("my-topic"));
consumer.poll(Duration.ofMillis(100));
The consumer throws a runtime error about deserializer class not found. What is the likely cause?
medium
A. The topic name is invalid.
B. The bootstrap server address is wrong.
C. The deserializer class names are incorrect or misspelled.
D. The group.id property is missing.

Solution

  1. Step 1: Analyze the error message

    The error says deserializer class not found, so the problem is with deserializer class names.

  2. Step 2: Check deserializer class names in code

    They must be fully qualified and spelled exactly. Any typo causes class not found error.

  3. Final Answer:

    The deserializer class names are incorrect or misspelled. -> Option C

  4. Quick Check:

    Deserializer class error = wrong class name [OK]
Hint: Check spelling of deserializer class names first [OK]
Common Mistakes:
  • Ignoring class name typos
  • Assuming bootstrap server causes deserializer error
  • Missing group.id but error is about deserializer
5. You want to create a Kafka consumer in Java that connects to an Amazon MSK cluster and only processes messages with keys starting with "user-". Which approach correctly applies this filter after receiving messages?
hard
A. Use a Kafka Streams application with a filter on the key prefix.
B. Set the consumer property "key.filter.prefix" to "user-".
C. Subscribe only to topics named "user-*".
D. Configure the producer to send only keys starting with "user-".

Solution

  1. Step 1: Understand Kafka consumer filtering options

    Kafka consumers do not support filtering by key prefix via configuration properties.

  2. Step 2: Identify correct filtering method

    Kafka Streams allows processing and filtering messages by key prefix programmatically.

  3. Step 3: Evaluate other options

    There is no Kafka consumer property like "key.filter.prefix". Subscribing to topics named "user-*" filters by topic name, not message keys. Producer configuration controls what is sent, not what the consumer processes.

  4. Final Answer:

    Use a Kafka Streams application with a filter on the key prefix. -> Option A

  5. Quick Check:

    Filter keys after consuming = Kafka Streams filter [OK]
Hint: Filter keys in Kafka Streams, not consumer config [OK]
Common Mistakes:
  • Expecting consumer config to filter keys
  • Filtering by topic name instead of message key
  • Trying to filter on producer side only
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard