Consumer groups concept in Redis - Step-by-Step Execution

Concept Flow - Consumer groups concept

Messages added to stream

↓

Create consumer group

↓

Consumers read messages

↓

Messages assigned to consumers

↓

Consumers acknowledge messages

↓

Pending messages tracked

↓

Reassign unacknowledged messages if needed

Messages enter a stream, a consumer group is created, consumers read and acknowledge messages, and pending messages are tracked for reliability.

Execution Sample

Redis

XGROUP CREATE mystream mygroup 0
XREADGROUP GROUP mygroup Alice COUNT 2 STREAMS mystream >
XACK mystream mygroup 1526985058136-0

Create a consumer group, read messages as a consumer, then acknowledge processed messages.

Execution Table

Step	Action	Input/Command	Result/State Change	Output
1	Add messages to stream	XADD mystream * name Alice	Message added with ID 1526985058136-0	1526985058136-0
2	Create consumer group	XGROUP CREATE mystream mygroup 0	Group 'mygroup' created starting from the first message	OK
3	Consumer reads messages	XREADGROUP GROUP mygroup Alice COUNT 2 STREAMS mystream >	Consumer 'Alice' assigned new messages	Messages with IDs 1526985058136-0
4	Consumer acknowledges message	XACK mystream mygroup 1526985058136-0	Message marked as processed, removed from pending	1
5	Check pending messages	XPENDING mystream mygroup	No pending messages for 'Alice'	Empty list
6	Exit	No more messages	No new messages to assign	End of processing

💡 No new messages available, consumer group processing ends

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
Stream 'mystream'	empty	1 message (1526985058136-0)	1 message	1 message assigned to Alice	1 message acknowledged (removed from pending)	1 message in stream, none pending
Consumer group 'mygroup'	none	none	created	active with Alice assigned message	message acknowledged	active, no pending messages
Pending messages	none	none	none	1 message pending for Alice	0 pending	0 pending

Key Moments - 3 Insights

Why does the consumer group start reading from '0' when created?

Starting from '0' means the group reads messages from the beginning of the stream, as shown in Step 2 and Step 3 of the execution_table.

What happens if a consumer does not acknowledge a message?

The message stays in the pending list (Step 4 and 5), so it can be reassigned or retried later, ensuring no message is lost.

Why do we use '>' in the XREADGROUP command?

'>' tells Redis to deliver only new messages that were never delivered to any consumer, as seen in Step 3.

Visual Quiz - 3 Questions

Look at the execution_table, what is the state of pending messages after Step 3?

A One message pending for consumer Alice

B No pending messages

C All messages acknowledged

D Pending messages for multiple consumers

💡 Hint

Check the 'Result/State Change' column in Step 3 and Step 4

At which step does the consumer group get created?

A Step 1

B Step 3

C Step 2

D Step 4

💡 Hint

Look for the 'Create consumer group' action in the execution_table

If the consumer never acknowledges messages, what would happen to the pending messages?

A They get deleted immediately

B They remain pending and can be reassigned

C They are delivered to other consumers instantly

D They disappear from the stream

💡 Hint

Refer to the key_moments explanation about unacknowledged messages

Concept Snapshot

Consumer groups let multiple consumers share a stream.
Create a group with XGROUP CREATE.
Consumers read with XREADGROUP using '>' for new messages.
Consumers acknowledge messages with XACK.
Pending messages track unacknowledged messages.
This ensures reliable, distributed message processing.

Full Transcript

In Redis streams, messages are added to a stream. A consumer group is created to allow multiple consumers to read from the stream without duplicating work. When a consumer reads messages using XREADGROUP with the '>' symbol, it receives new messages not yet delivered to any consumer. After processing, the consumer acknowledges messages with XACK, removing them from the pending list. Pending messages are tracked to handle cases where consumers fail to acknowledge, allowing reassignment and ensuring no message is lost. This flow supports reliable and scalable message consumption.

Practice

1. What is the main purpose of Redis consumer groups in streams?

easy

A. Allow multiple clients to read the same stream without duplicating work

B. Store data permanently in Redis

C. Encrypt messages in the stream

D. Automatically delete old messages from the stream

Solution

1. Step 1: Understand consumer groups role

   Consumer groups let many clients read from the same stream safely and efficiently.

2. Step 2: Identify main benefit

   They avoid duplicate processing by tracking which messages each consumer has read.

3. Final Answer:

   Allow multiple clients to read the same stream without duplicating work -> Option A

4. Quick Check:

   Consumer groups = safe multi-client reading [OK]

Hint: Consumer groups = shared reading without duplicates [OK]

Common Mistakes:

• Thinking consumer groups store data permanently
• Confusing consumer groups with encryption
• Assuming consumer groups delete old messages automatically

2. Which Redis command is used to create a consumer group for a stream named mystream?

easy

A. XGROUP NEW mystream mygroup $

B. XGROUP CREATE mystream mygroup $

C. XGROUP ADD mystream mygroup $

D. XREADGROUP CREATE mystream mygroup $

Solution

1. Step 1: Recall command to create consumer group

   The correct command is XGROUP CREATE followed by stream name, group name, and ID.

2. Step 2: Match command syntax

   XGROUP CREATE mystream mygroup $ correctly creates group mygroup on mystream.

3. Final Answer:

   XGROUP CREATE mystream mygroup $ -> Option B

4. Quick Check:

   Create group = XGROUP CREATE [OK]

Hint: Use XGROUP CREATE to make groups [OK]

Common Mistakes:

• Using XREADGROUP instead of XGROUP CREATE
• Using non-existent commands like XGROUP ADD or XGROUP NEW
• Mixing command order or missing parameters

3. Given a consumer group mygroup on stream mystream, what does this command return?

XREADGROUP GROUP mygroup consumer1 COUNT 2 STREAMS mystream > 

medium

A. Returns up to 2 new messages never delivered to any consumer

B. Returns all messages in the stream from the beginning

C. Returns the last 2 messages already acknowledged

D. Returns an error because of wrong syntax

Solution

1. Step 1: Understand XREADGROUP with '>' ID

   Using '>' means read only new messages never delivered to any consumer in the group.

2. Step 2: COUNT 2 limits messages returned

   The command returns up to 2 such new messages for consumer1.

3. Final Answer:

   Returns up to 2 new messages never delivered to any consumer -> Option A

4. Quick Check:

   XREADGROUP with '>' = new messages only [OK]

Hint: '>' ID in XREADGROUP means new messages only [OK]

Common Mistakes:

• Thinking it returns all messages from start
• Assuming it returns acknowledged messages
• Believing the command syntax is incorrect

4. You run this command but get an error:

XREADGROUP GROUP mygroup consumer1 STREAMS mystream >

What is the likely cause?

medium

A. Missing COUNT argument causes syntax error

B. The '>' symbol is invalid here

C. The command is missing the ID after STREAMS

D. Consumer group 'mygroup' does not exist

Solution

1. Step 1: Verify command syntax

   The syntax is correct: XREADGROUP GROUP mygroup consumer1 STREAMS mystream > uses '>' ID for new messages. COUNT is optional.

2. Step 2: Identify prerequisite error

   XREADGROUP requires an existing consumer group created with XGROUP CREATE. If missing, it returns a NOGROUP error.

3. Final Answer:

   Consumer group 'mygroup' does not exist -> Option D

4. Quick Check:

   XREADGROUP needs existing group [OK]

Hint: XREADGROUP requires existing consumer group [OK]

Common Mistakes:

• Thinking COUNT is mandatory (it's optional)
• Believing '>' is invalid
• Mistakenly thinking ID is missing after STREAMS

5. You have a stream orders and consumer group ordergroup. You want to ensure that if a consumer crashes before acknowledging messages, another consumer can claim and process those pending messages. Which Redis command helps you do this?

hard

A. XREADGROUP GROUP ordergroup consumer2 STREAMS orders >

B. XACK orders ordergroup 1526985058136

C. XCLAIM orders ordergroup consumer2 60000 1526985058136

D. XGROUP CREATE orders ordergroup $

Solution

1. Step 1: Understand the problem of crashed consumers

   If a consumer crashes, its pending messages remain unacknowledged and need to be claimed by others.

2. Step 2: Identify command to claim pending messages

   XCLAIM lets another consumer take ownership of pending messages after a timeout.

3. Step 3: Match command usage

   XCLAIM orders ordergroup consumer2 60000 1526985058136 claims message ID 1526985058136 for consumer2.

4. Final Answer:

   XCLAIM orders ordergroup consumer2 60000 1526985058136 -> Option C

5. Quick Check:

   Claim pending messages = XCLAIM [OK]

Hint: Use XCLAIM to recover unacknowledged messages [OK]

Common Mistakes:

• Using XACK which only acknowledges, not claims
• Using XREADGROUP with '>' which reads new messages only
• Trying to create group again instead of claiming