In-sync replicas (ISR) in Kafka - Deep Dive

Overview - In-sync replicas (ISR)

What is it?

In Kafka, In-sync replicas (ISR) are the set of replicas of a partition that are fully caught up with the leader. These replicas have all the latest messages that the leader has acknowledged. ISR ensures data durability and availability by keeping multiple copies of data synchronized.

Why it matters

ISR exists to prevent data loss and maintain high availability in Kafka clusters. Without ISR, if a leader fails, followers might not have the latest data, causing message loss or inconsistent reads. This would make Kafka unreliable for critical data streaming applications.

Where it fits

Before learning about ISR, you should understand Kafka basics like partitions, leaders, and replicas. After ISR, you can explore Kafka's replication protocols, leader election, and fault tolerance mechanisms.

Mental Model

Core Idea

ISR is the group of replicas that have caught up with the leader and can safely take over without data loss.

Think of it like...

ISR is like a team of backup singers who always stay in perfect harmony with the lead singer, ready to step in without missing a beat if the lead stops singing.

Partition Leader
  │
  ├─ Replica 1 (In-sync)
  ├─ Replica 2 (In-sync)
  └─ Replica 3 (Out-of-sync)

ISR = {Replica 1, Replica 2}
Out-of-sync replicas are excluded from ISR until they catch up.

Build-Up - 7 Steps

1

Foundation

Kafka Replication Basics

Concept: Introduce Kafka partitions, leaders, and replicas.

Kafka stores data in partitions. Each partition has one leader and multiple replicas. The leader handles all reads and writes. Replicas copy data from the leader to provide fault tolerance.

Result

Learner understands the roles of leader and replicas in Kafka partitions.

Understanding leader and replicas is essential to grasp why synchronization matters for data safety.

2

Foundation

What Does Synchronization Mean?

Concept: Explain what it means for replicas to be synchronized with the leader.

A replica is synchronized if it has copied all messages the leader has committed. If a replica lags behind, it is out-of-sync and cannot guarantee it has the latest data.

Result

Learner knows the difference between in-sync and out-of-sync replicas.

Knowing synchronization status helps understand how Kafka decides which replicas can safely serve data.

3

Intermediate

Defining the ISR Set

🤔Before reading on: do you think all replicas are always in the ISR? Commit to yes or no.

Concept: Introduce the ISR as the set of replicas fully caught up with the leader.

Kafka maintains an ISR list per partition. Only replicas that have fully caught up with the leader are in this list. Replicas that fall behind are removed until they catch up again.

Result

Learner understands that ISR is a dynamic set reflecting replica health.

Recognizing ISR as a dynamic group clarifies how Kafka manages data consistency and availability.

4

Intermediate

ISR and Leader Election

🤔Before reading on: do you think Kafka can elect any replica as leader or only those in ISR? Commit to your answer.

Concept: Explain that only replicas in ISR can become leaders during failover.

When a leader fails, Kafka chooses a new leader from the ISR. This ensures the new leader has all committed data, preventing data loss.

Result

Learner sees how ISR protects data during leader changes.

Knowing leader election depends on ISR helps understand Kafka's fault tolerance guarantees.

5

Intermediate

ISR Shrinking and Expansion

Concept: Show how replicas join or leave ISR based on their sync status.

If a replica falls behind due to network or load, Kafka removes it from ISR. When it catches up, Kafka adds it back. This keeps ISR accurate and reliable.

Result

Learner understands ISR updates reflect real-time replica health.

Understanding ISR dynamics helps predict Kafka behavior under load or failures.

6

Advanced

ISR Impact on Acknowledgments

🤔Before reading on: do you think Kafka waits for all replicas or only ISR replicas to acknowledge writes? Commit to your answer.

Concept: Explain how ISR affects message acknowledgment and durability.

Kafka's producer acks can be configured. For 'acks=all', Kafka waits for all ISR replicas to confirm writes before acknowledging. This ensures data is safely replicated.

Result

Learner knows how ISR size affects write durability and latency.

Understanding ISR's role in acknowledgments reveals the tradeoff between safety and performance.

7

Expert

ISR and Partition Reassignment Surprises

🤔Before reading on: do you think ISR is preserved during partition reassignment? Commit to yes or no.

Concept: Discuss how ISR behaves during partition reassignment and broker restarts.

During partition reassignment, ISR can shrink because new replicas start empty and must catch up. Also, if a broker restarts slowly, its replicas may be removed from ISR, affecting availability.

Result

Learner understands ISR can fluctuate unexpectedly in production.

Knowing ISR behavior during reassignment helps prevent downtime and data loss surprises.

Under the Hood

Kafka tracks the offset of the last message each replica has replicated. The leader maintains a high watermark representing the highest offset committed by all ISR replicas. Replicas send periodic heartbeats and fetch requests to the leader. If a replica stops responding or falls behind beyond a configured threshold, the leader removes it from ISR. This mechanism ensures only replicas with up-to-date data are trusted.

Why designed this way?

Kafka was designed for high throughput and fault tolerance. Using ISR allows Kafka to balance data safety and availability by only trusting replicas that are fully caught up. Alternatives like waiting for all replicas would reduce availability, while ignoring replica lag would risk data loss.

┌─────────────┐      ┌─────────────┐      ┌─────────────┐
│  Leader     │─────▶│ Replica 1   │
│ (Partition) │      │ (In-sync)   │
└─────────────┘      └─────────────┘
       │                  ▲
       │                  │
       │                  │
       │          ┌─────────────┐
       │          │ Replica 2   │
       │          │ (In-sync)   │
       │          └─────────────┘
       │                  ▲
       │                  │
       │          ┌─────────────┐
       │          │ Replica 3   │
       └─────────▶│ (Out-of-sync)│
                  └─────────────┘

ISR = {Replica 1, Replica 2}
Leader tracks offsets and heartbeats to maintain ISR.

Myth Busters - 4 Common Misconceptions

Quick: Do you think all replicas in Kafka are always in-sync? Commit yes or no.

Common Belief: All replicas are always in-sync with the leader.

Reality: Only replicas that have fully caught up are in the ISR; others are excluded until they catch up.

Why it matters: Assuming all replicas are in-sync can lead to overestimating data safety and cause unexpected data loss during failover.

Quick: Can a replica outside ISR become leader during failover? Commit yes or no.

Common Belief: Any replica can become leader regardless of sync status.

Reality: Only replicas in ISR can be elected leader to ensure no data loss.

Why it matters: Believing otherwise risks choosing a replica missing data, causing inconsistent reads and data loss.

Quick: Does ISR size always stay constant? Commit yes or no.

Common Belief: ISR size is fixed and does not change dynamically.

Reality: ISR size changes as replicas fall behind or catch up, reflecting real-time health.

Why it matters: Ignoring ISR dynamics can cause misinterpretation of cluster health and lead to wrong operational decisions.

Quick: Does ISR guarantee zero data loss in all failure cases? Commit yes or no.

Common Belief: ISR guarantees zero data loss no matter what.

Reality: ISR reduces data loss risk but cannot guarantee zero loss if misconfigured or under extreme failures.

Why it matters: Overtrusting ISR can cause complacency in backup and monitoring strategies, risking data integrity.

Expert Zone

1

ISR membership depends on replica fetch latency and heartbeat intervals, so network issues can cause temporary ISR shrinkage.

2

The leader's high watermark advances only when all ISR replicas acknowledge, affecting consumer visibility of messages.

3

During partition reassignment, ISR resets and replicas must catch up, which can temporarily reduce fault tolerance.

When NOT to use

ISR is not applicable when using Kafka in non-replicated mode or with unclean leader election enabled, which sacrifices data safety for availability. In such cases, alternatives like external replication or backup systems should be used.

Production Patterns

In production, operators monitor ISR size to detect replica lag or failures. They tune replica.lag.time.max.ms and min.insync.replicas to balance durability and availability. During maintenance, careful partition reassignment is done to avoid ISR shrinkage causing downtime.

Connections

Consensus Algorithms (e.g., Raft, Paxos)

ISR is similar to quorum sets in consensus algorithms that ensure agreement among nodes before committing data.

Understanding ISR helps grasp how distributed systems achieve consistency and fault tolerance through majority agreement.

Database Replication

ISR parallels synchronous replication in databases where replicas must confirm writes before commit.

Knowing ISR clarifies tradeoffs between synchronous and asynchronous replication in data durability.

Team Backup Systems

ISR resembles backup team members who must be fully prepared before taking over a task.

This cross-domain view highlights the importance of readiness and synchronization in reliable failover.

Common Pitfalls

#1 Assuming all replicas are always in ISR and ignoring replica lag.

Wrong approach:

Setting min.insync.replicas=3 with only 2 replicas in ISR, expecting full durability.

Correct approach:

Monitor ISR size and adjust min.insync.replicas to match actual ISR count to avoid write failures.

Root cause:

Misunderstanding that ISR is dynamic and can shrink under load or network issues.

#2 Enabling unclean leader election and ignoring ISR implications.

Wrong approach:

Setting unclean.leader.election.enable=true to improve availability without considering data loss.

Correct approach:

Keep unclean.leader.election.enable=false to ensure only ISR replicas become leaders, preserving data safety.

Root cause:

Lack of awareness that unclean leader election can cause data loss by electing out-of-sync replicas.

#3 Ignoring ISR changes during partition reassignment.

Wrong approach:

Reassigning partitions without monitoring ISR, causing ISR to shrink and availability to drop.

Correct approach:

Perform reassignment gradually and monitor ISR to ensure replicas catch up before leader switches.

Root cause:

Not understanding that new replicas start empty and must rejoin ISR to maintain fault tolerance.

Key Takeaways

In-sync replicas (ISR) are the set of Kafka replicas fully caught up with the leader, ensuring data safety.

Only replicas in ISR can become leaders, preventing data loss during failover.

ISR membership changes dynamically based on replica health and synchronization status.

ISR size affects write durability and latency through producer acknowledgment settings.

Understanding ISR behavior during partition reassignment and failures is critical for reliable Kafka operations.

Practice

1. What does In-sync replicas (ISR) mean in Kafka?

easy

A. Replicas that are offline

B. Replicas that never receive data

C. Replicas that have the latest data from the leader

D. Replicas that only store old data

Solution

1. Step 1: Understand the role of ISR

   ISR are replicas that keep up-to-date with the leader's data.

2. Step 2: Compare options

   Only Replicas that have the latest data from the leader correctly describes ISR as replicas with the latest data.

3. Final Answer:

   Replicas that have the latest data from the leader -> Option C

4. Quick Check:

   ISR = Latest data replicas [OK]

Hint: ISR means replicas fully caught up with leader data [OK]

Common Mistakes:

• Thinking ISR includes offline replicas
• Confusing ISR with replicas that lag behind
• Assuming ISR means replicas with old data

2. Which Kafka configuration controls the minimum number of ISR replicas required to acknowledge a write?

easy

A. replica.fetch.max.bytes

B. min.insync.replicas

C. max.insync.replicas

D. offsets.retention.minutes

Solution

1. Step 1: Identify configuration related to ISR acknowledgments

   The setting min.insync.replicas defines the minimum ISR count for write success.

2. Step 2: Eliminate unrelated options

   Other options control fetch size or retention, not ISR acknowledgments.

3. Final Answer:

   min.insync.replicas -> Option B

4. Quick Check:

   Minimum ISR replicas = min.insync.replicas [OK]

Hint: min.insync.replicas sets minimum ISR for writes [OK]

Common Mistakes:

• Confusing min.insync.replicas with max.insync.replicas
• Choosing unrelated configs like offsets.retention.minutes
• Assuming replica.fetch.max.bytes controls ISR count

3. Given a Kafka topic with 3 replicas and ISR = [replica1, replica2], what happens if replica2 falls behind and leaves ISR?

medium

A. Only replica1 remains in ISR; leader continues with one ISR

B. Leader stops producing until replica2 catches up

C. All replicas are removed from ISR

D. Replica2 becomes the new leader immediately

Solution

1. Step 1: Understand ISR behavior when a replica falls behind

   If a replica lags, it is removed from ISR, leaving others in ISR.

2. Step 2: Analyze leader and ISR state

   The leader continues with remaining ISR replicas; it does not stop or change leader immediately.

3. Final Answer:

   Only replica1 remains in ISR; leader continues with one ISR -> Option A

4. Quick Check:

   Lagging replica leaves ISR, leader continues [OK]

Hint: Lagging replica leaves ISR; leader stays with remaining ISR [OK]

Common Mistakes:

• Thinking leader stops producing when ISR shrinks
• Assuming all replicas leave ISR together
• Believing lagging replica becomes leader immediately

4. You see this error in Kafka logs: "Not enough replicas are in ISR to satisfy min.insync.replicas". What is the likely cause?

medium

A. The consumer group is not committed

B. The leader replica is offline

C. The topic has zero partitions

D. Too few replicas are currently in ISR compared to min.insync.replicas

Solution

1. Step 1: Interpret the error message

   The error means the number of replicas in ISR is less than the configured minimum.

2. Step 2: Match cause with options

   Too few replicas are currently in ISR compared to min.insync.replicas directly matches the error cause; other options are unrelated to ISR count.

3. Final Answer:

   Too few replicas are currently in ISR compared to min.insync.replicas -> Option D

4. Quick Check:

   ISR count below min.insync.replicas causes error [OK]

Hint: Error means ISR count < min.insync.replicas [OK]

Common Mistakes:

• Blaming leader offline instead of ISR count
• Confusing consumer group commit with ISR errors
• Thinking zero partitions cause this error

5. You want to ensure high availability in Kafka by allowing leader failover only if at least 2 replicas are in ISR. Which configuration and setup achieve this?

hard

A. Set min.insync.replicas=2 and have at least 3 replicas per partition

B. Set min.insync.replicas=1 and have 2 replicas per partition

C. Set min.insync.replicas=3 with 2 replicas per partition

D. Set min.insync.replicas=0 and 1 replica per partition

Solution

1. Step 1: Understand min.insync.replicas meaning

   This setting defines the minimum ISR count required for leader failover and write success.

2. Step 2: Match configuration to requirement

   To require at least 2 ISR replicas, set min.insync.replicas=2 and have at least 3 replicas to allow failover.

3. Final Answer:

   Set min.insync.replicas=2 and have at least 3 replicas per partition -> Option A

4. Quick Check:

   High availability needs min.insync.replicas=2 with 3 replicas [OK]

Hint: min.insync.replicas ≤ replica count for failover [OK]

Common Mistakes:

• Setting min.insync.replicas higher than replica count
• Using too few replicas for desired availability
• Setting min.insync.replicas to zero disables safety