Exactly-once stream processing in Kafka - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to enable exactly-once processing in Kafka Streams.

Kafka

Properties props = new Properties();
props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, [1]);

Drag options to blanks, or click blank then click option'

A"at_least_once"

B"at_most_once"

C"none"

D"exactly_once"

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'at_least_once' which allows duplicates.

Using 'none' which disables guarantees.

Setting PROCESSING_GUARANTEE_CONFIG to "exactly_once" enables exactly-once semantics in Kafka Streams.

2fill in blank

medium

Complete the code to commit Kafka Streams transactions for exactly-once processing.

Kafka

KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.[1]();

Drag options to blanks, or click blank then click option'

Aflush

Bstart

Cclose

Dcommit

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'commit()' which is not a KafkaStreams method.

Using 'flush()' which only flushes records.

The start() method begins the Kafka Streams application and manages transactions for exactly-once processing.

3fill in blank

hard

Fix the error in the code to enable exactly-once processing with idempotent producer.

Kafka

props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, [1]);

Drag options to blanks, or click blank then click option'

A"true"

Bfalse

Ctrue

D"false"

Attempts:

3 left

💡 Hint

Common Mistakes

Using boolean true instead of string "true".

Using "false" which disables idempotence.

The value must be a string "true" to correctly configure the producer for idempotence in Kafka Streams properties.

4fill in blank

hard

Fill both blanks to create a transactional producer with exactly-once semantics.

Kafka

props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, [1]);
props.put(ProducerConfig.ACKS_CONFIG, [2]);

Drag options to blanks, or click blank then click option'

A"txn-id-1"

B"all"

C"1"

D"0"

Attempts:

3 left

💡 Hint

Common Mistakes

Using numeric values for transactional ID.

Setting acks to '1' or '0' which weakens guarantees.

Setting a unique TRANSACTIONAL_ID_CONFIG enables transactions, and ACKS_CONFIG set to "all" ensures full commit acknowledgment for exactly-once.

5fill in blank

hard

Fill all three blanks to configure Kafka Streams for exactly-once processing with proper commit interval and retries.

Kafka

props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, [1]);
props.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, [2]);
props.put(ProducerConfig.RETRIES_CONFIG, [3]);

Drag options to blanks, or click blank then click option'

A"exactly_once"

B1000

C5

D"at_least_once"

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'at_least_once' instead of 'exactly_once'.

Setting commit interval too high or retries too low.

Exactly-once processing requires PROCESSING_GUARANTEE_CONFIG set to "exactly_once", a commit interval of 1000 ms for timely commits, and retries set to 5 for robustness.

Practice

1. What is the main benefit of exactly-once stream processing in Kafka?

easy

A. It ensures messages are processed only once, avoiding duplicates.

B. It speeds up message delivery by skipping acknowledgments.

C. It allows messages to be processed multiple times for safety.

D. It disables message ordering to improve throughput.

Solution

1. Step 1: Understand exactly-once processing goal

   Exactly-once processing means each message affects the system only once, preventing duplicates.

2. Step 2: Compare options with this goal

   Only It ensures messages are processed only once, avoiding duplicates. describes avoiding duplicates by processing messages once.

3. Final Answer:

   It ensures messages are processed only once, avoiding duplicates. -> Option A

4. Quick Check:

   Exactly-once = no duplicates [OK]

Hint: Exactly-once means no duplicate effects on data [OK]

Common Mistakes:

• Confusing exactly-once with at-least-once processing
• Thinking it speeds up delivery by skipping steps
• Assuming it allows multiple processing for safety

2. Which Kafka producer configuration enables exactly-once semantics?

easy

A. compression.type = none

B. enable.idempotence = true

C. acks = 0

D. max.in.flight.requests.per.connection = 10

Solution

1. Step 1: Identify config for exactly-once

   Enabling idempotence ensures the producer can retry without duplicates.

2. Step 2: Match config options

   Only 'enable.idempotence = true' enables this feature.

3. Final Answer:

   enable.idempotence = true -> Option B

4. Quick Check:

   Idempotence enabled = exactly-once [OK]

Hint: Idempotence true enables exactly-once in Kafka producer [OK]

Common Mistakes:

• Choosing acks=0 which disables acknowledgments
• Confusing compression with processing guarantees
• Setting max.in.flight.requests without idempotence

3. Given this Kafka Streams code snippet, what is the output if a failure occurs before commit?

streams.start();
try {
  streams.process(record);
  streams.commitTransaction();
} catch (Exception e) {
  streams.abortTransaction();
}

medium

A. Processed record is committed multiple times.

B. Processed record may be duplicated after restart.

C. Processed record is committed exactly once.

D. Processed record is lost and not committed.

Solution

1. Step 1: Analyze transaction handling

   If an exception occurs before commit, abortTransaction() is called, rolling back changes.

2. Step 2: Determine output effect

   Rollback means the processed record is not committed and effectively lost in this run.

3. Final Answer:

   Processed record is lost and not committed. -> Option D

4. Quick Check:

   Abort transaction = no commit = lost record [OK]

Hint: Abort transaction means no commit, so data is lost [OK]

Common Mistakes:

• Assuming record is committed despite failure
• Thinking abortTransaction duplicates records
• Confusing commitTransaction with abortTransaction

4. Identify the error in this Kafka Streams code snippet for exactly-once processing:

props.put("enable.idempotence", "true");
props.put("transactional.id", "txn-1");
KafkaProducer producer = new KafkaProducer(props);
producer.beginTransaction();
producer.send(record);
// Missing commit or abort here
producer.close();

medium

A. Missing commitTransaction or abortTransaction before close.

B. enable.idempotence should be false for transactions.

C. transactional.id must be unique per message.

D. KafkaProducer cannot be closed after sending messages.

Solution

1. Step 1: Review transaction lifecycle

   After beginTransaction and send, a commit or abort must finalize the transaction.

2. Step 2: Identify missing step

   The code closes producer without committing or aborting, causing incomplete transaction.

3. Final Answer:

   Missing commitTransaction or abortTransaction before close. -> Option A

4. Quick Check:

   Transactions need commit or abort before close [OK]

Hint: Always commit or abort before closing producer [OK]

Common Mistakes:

• Thinking idempotence false is needed for transactions
• Believing transactional.id changes per message
• Assuming producer close auto-commits transactions

5. You want to build a Kafka Streams application that updates user balances exactly once, even if failures occur. Which approach ensures this behavior?

hard

A. Process messages without committing offsets to avoid duplicates.

B. Disable transactions and rely on at-least-once processing with retries.

C. Use idempotent producer with transactions and commit offsets atomically.

D. Use multiple producers without transactions to increase throughput.

Solution

1. Step 1: Understand exactly-once requirements

   Exactly-once needs atomic commit of data and offsets to avoid duplicates or loss.

2. Step 2: Match approach to requirements

   Using idempotent producer with transactions and atomic offset commits ensures exactly-once updates.

3. Final Answer:

   Use idempotent producer with transactions and commit offsets atomically. -> Option C

4. Quick Check:

   Idempotent + transactions + atomic commit = exactly-once [OK]

Hint: Atomic commit with idempotent producer ensures exactly-once [OK]

Common Mistakes:

• Relying on retries without transactions causes duplicates
• Using multiple producers without coordination breaks exactly-once
• Not committing offsets leads to message reprocessing