Resource planning and capacity in Kafka - Time & Space Complexity

Time Complexity: Resource planning and capacity

O(n)

Understanding Time Complexity

When working with Kafka, resource planning and capacity affect how fast and smoothly data flows through the system.

We want to understand how the time to process messages grows as the number of messages or partitions increases.

Scenario Under Consideration

Analyze the time complexity of the following Kafka consumer processing loop.

// Kafka consumer polling loop
while (true) {
  ConsumerRecords records = consumer.poll(Duration.ofMillis(100));
  for (ConsumerRecord record : records) {
    process(record);
  }
}

This code continuously polls Kafka for new messages and processes each message one by one.

Identify Repeating Operations

Look at what repeats as input grows.

• Primary operation: Loop over all messages received in each poll.
• How many times: Once per message in the batch, repeated every poll cycle.

How Execution Grows With Input

As the number of messages per poll increases, the processing time grows proportionally.

Input Size (messages per poll)	Approx. Operations
10	10 process calls
100	100 process calls
1000	1000 process calls

Pattern observation: Processing time grows linearly with the number of messages.

Final Time Complexity

Time Complexity: O(n)

This means the time to process messages grows directly in proportion to how many messages arrive.

Common Mistake

[X] Wrong: "Processing time stays the same no matter how many messages arrive."

[OK] Correct: Each message needs its own processing time, so more messages mean more total work.

Interview Connect

Understanding how processing time scales with message volume helps you design systems that handle load smoothly and predict performance.

Self-Check

"What if we batch process messages instead of one by one? How would the time complexity change?"