Topic deletion and cleanup in Kafka - Step-by-Step Execution

Process Flow - Topic deletion and cleanup

Start

↓

Request Topic Deletion

↓

Mark Topic for Deletion

↓

Broker Deletes Topic Data

↓

Cleanup Logs and Metadata

↓

Topic Removed from Cluster

↓

End

The flow shows how a topic deletion request marks the topic, then brokers delete data and clean metadata, finally removing the topic from the cluster.

Execution Sample

Kafka

kafka-topics.sh --delete --topic my-topic --bootstrap-server localhost:9092

This command requests deletion of 'my-topic' from the Kafka cluster.

Process Table

Step	Action	Internal State Change	Result
1	Send delete request for 'my-topic'	Topic marked for deletion in controller	Deletion initiated
2	Controller notifies brokers	Brokers receive deletion command	Prepare to delete topic data
3	Brokers delete topic partitions data	Topic data files removed from disk	Storage freed
4	Brokers clean up topic metadata	Topic metadata removed from logs	Metadata cleaned
5	Controller updates cluster state	Topic removed from cluster metadata	Topic no longer listed
6	Clients receive updated metadata	No references to 'my-topic'	Topic deletion complete

💡 Topic fully deleted and cleaned up from cluster and brokers

Status Tracker

Variable	Start	After Step 1	After Step 3	After Step 5	Final
Topic State	Exists	Marked for deletion	Data deleted	Metadata removed	Deleted
Broker Storage	Contains topic data	Contains topic data	Topic data removed	Cleaned metadata	No topic data
Cluster Metadata	Includes topic	Includes topic	Includes topic	Topic removed	No topic

Key Moments - 3 Insights

Why does the topic still appear briefly after deletion is requested?

Because the topic is first marked for deletion (Step 1) and brokers need time to delete data and metadata (Steps 3 and 4). It is removed from cluster metadata only after cleanup (Step 5).

What happens if a client tries to produce to the topic during deletion?

Clients receive updated metadata after deletion (Step 6). Before that, produce requests may fail or be rejected because the topic is marked for deletion and data is being removed.

Is topic deletion immediate?

No, deletion involves multiple steps: marking, data deletion, metadata cleanup, and cluster update. This ensures safe removal without data loss.

Visual Quiz - 3 Questions

Look at the execution table, at which step is the topic data actually removed from disk?

A Step 3

B Step 2

C Step 4

D Step 5

💡 Hint

Check the 'Internal State Change' column for data removal details.

According to the variable tracker, what is the state of 'Cluster Metadata' after Step 5?

A Includes topic

B Marked for deletion

C Topic removed

D No topic data

💡 Hint

Look at the 'Cluster Metadata' row under 'After Step 5' column.

If the deletion command is sent but brokers fail to delete data, what step would not complete successfully?

A Step 1

B Step 3

C Step 5

D Step 6

💡 Hint

Step 3 is when brokers delete topic partitions data.

Concept Snapshot

Topic deletion in Kafka:
- Use kafka-topics.sh --delete to request deletion
- Topic is marked for deletion by controller
- Brokers delete topic data and clean metadata
- Controller updates cluster metadata
- Clients see updated metadata without the topic
- Deletion is asynchronous and involves multiple steps

Full Transcript

This visual execution trace shows how Kafka deletes a topic. First, a delete request marks the topic for deletion. Then the controller tells brokers to delete the topic data from disk. Brokers remove the data files and clean up metadata. After that, the controller updates the cluster metadata to remove the topic. Finally, clients receive updated metadata and the topic is fully deleted. The process is not instant but happens step-by-step to ensure safe cleanup. Variables like topic state, broker storage, and cluster metadata change gradually from existing to deleted. Common confusions include why the topic still appears briefly after deletion starts and what happens if clients try to use the topic during deletion. The quizzes test understanding of when data is deleted, cluster metadata state, and failure points in deletion.

Practice

1. What is the correct Kafka configuration to enable topic deletion?

easy

A. topic.delete.enabled=false

B. enable.topic.removal=false

C. delete.topic.enable=true

D. allow.topic.delete=no

Solution

1. Step 1: Understand Kafka topic deletion setting

   The Kafka broker must have a configuration that explicitly allows topic deletion.

2. Step 2: Identify the correct configuration key

   The correct key is delete.topic.enable and it must be set to true to allow deletion.

3. Final Answer:

   delete.topic.enable=true -> Option C

4. Quick Check:

   Topic deletion enabled = delete.topic.enable=true [OK]

Hint: Look for 'delete.topic.enable' set to true in config [OK]

Common Mistakes:

• Confusing the config key name
• Setting the value to false
• Assuming deletion works without config change

2. Which command correctly deletes a Kafka topic named my-topic?

easy

A. kafka-delete-topic.sh --topic my-topic

B. kafka-topics.sh --delete --topic my-topic

C. kafka-topics.sh --remove --topic my-topic

D. kafka-topics.sh --drop --topic my-topic

Solution

1. Step 1: Identify the correct Kafka topic deletion command

   The official Kafka command to delete a topic uses kafka-topics.sh with the --delete flag.

2. Step 2: Verify the command syntax

   The correct syntax is kafka-topics.sh --delete --topic my-topic.

3. Final Answer:

   kafka-topics.sh --delete --topic my-topic -> Option B

4. Quick Check:

   Delete topic command = kafka-topics.sh --delete --topic [OK]

Hint: Use --delete flag with kafka-topics.sh to remove topics [OK]

Common Mistakes:

• Using wrong flags like --remove or --drop
• Using a non-existent script
• Omitting the --topic option

3. After running kafka-topics.sh --delete --topic test-topic, which command confirms the topic is deleted?

medium

A. kafka-topics.sh --alter --topic test-topic

B. kafka-topics.sh --describe --topic test-topic

C. kafka-topics.sh --create --topic test-topic

D. kafka-topics.sh --list

Solution

1. Step 1: Understand how to check existing topics

   Listing topics shows all current topics in Kafka, so deleted topics won't appear.

2. Step 2: Choose the command that lists topics

   kafka-topics.sh --list lists all topics, confirming if test-topic is gone.

3. Final Answer:

   kafka-topics.sh --list -> Option D

4. Quick Check:

   List topics to confirm deletion = kafka-topics.sh --list [OK]

Hint: List topics to verify deletion, deleted ones won't show [OK]

Common Mistakes:

• Using describe which errors if topic missing
• Trying to create or alter instead of checking
• Assuming deletion is immediate without checking

4. You tried to delete a topic but it still appears in the list. What is the most likely cause?

medium

A. The broker configuration delete.topic.enable is set to false

B. You used the wrong topic name in the delete command

C. Kafka does not support topic deletion

D. The topic is currently being produced to

Solution

1. Step 1: Check broker configuration for topic deletion

   If delete.topic.enable is false, Kafka ignores delete commands and topics remain.

2. Step 2: Understand other options

   Wrong topic name causes error, Kafka supports deletion if enabled, and producing to topic doesn't block deletion.

3. Final Answer:

   The broker configuration delete.topic.enable is set to false -> Option A

4. Quick Check:

   Deletion blocked if delete.topic.enable=false [OK]

Hint: Check delete.topic.enable=true to allow deletion [OK]

Common Mistakes:

• Assuming Kafka never supports deletion
• Ignoring broker config settings
• Thinking active producers block deletion

5. You want to automate cleanup of unused topics older than 30 days. Which approach best uses Kafka's topic deletion and cleanup features?

hard

A. Enable delete.topic.enable=true and run a script that lists topics, checks their creation date, and deletes old ones

B. Set log.retention.ms to 30 days and rely on Kafka to delete topics automatically

C. Manually delete topics from Kafka config files after 30 days

D. Disable topic deletion and archive topics instead

Solution

1. Step 1: Understand Kafka's automatic cleanup capabilities

   Kafka deletes old log segments inside topics but does not delete entire topics automatically based on age.

2. Step 2: Choose an automation method for topic deletion

   Enabling delete.topic.enable=true and scripting topic listing with date checks allows controlled deletion of old topics.

3. Final Answer:

   Enable delete.topic.enable=true and run a script that lists topics, checks their creation date, and deletes old ones -> Option A

4. Quick Check:

   Automate deletion with config + script, not just retention settings [OK]

Hint: Use delete.topic.enable plus scripts for automated topic cleanup [OK]

Common Mistakes:

• Expecting Kafka to delete topics automatically by age
• Editing config files manually for topic deletion
• Disabling deletion and expecting cleanup