Consumer groups concept in Redis - Mini Project: Build & Apply

Working with Redis Consumer Groups

📖 Scenario: You are building a simple message processing system using Redis streams. Multiple consumers will read messages from the same stream using a consumer group to share the workload.

🎯 Goal: Create a Redis stream and a consumer group, then read messages from the stream using the consumer group concept.

📋 What You'll Learn

Create a Redis stream called mystream with two messages having fields name and age.

Create a consumer group called mygroup for the stream mystream starting from the beginning of the stream.

Read messages from mystream using the consumer group mygroup and consumer name consumer1.

Acknowledge the messages read by consumer1 to mark them as processed.

💡 Why This Matters

🌍 Real World

Redis streams with consumer groups are used in real-time data processing systems where multiple workers share the workload of processing messages efficiently.

💼 Career

Understanding consumer groups in Redis is valuable for backend developers and system architects building scalable, fault-tolerant message processing pipelines.

1

Create the Redis stream with initial messages

Use the Redis command XADD mystream * to add two messages to the stream mystream. Each message should have fields name and age with these exact values: first message name=Alice, age=30; second message name=Bob, age=25.

Redis

# Add two messages to the Redis stream mystream
# Your code here

Hint

Use XADD mystream * to add messages with fields name and age.

2

Create a consumer group for the stream

Use the Redis command XGROUP CREATE mystream mygroup 0 to create a consumer group called mygroup for the stream mystream starting from the beginning (ID 0).

Redis

XADD mystream * name Alice age 30
XADD mystream * name Bob age 25
# Create the consumer group mygroup starting from 0
# Your code here

Hint

Use XGROUP CREATE mystream mygroup 0 to create the group starting from the first message.

3

Read messages from the stream using the consumer group

Use the Redis command XREADGROUP GROUP mygroup consumer1 COUNT 2 STREAMS mystream > to read up to 2 new messages from the stream mystream as consumer consumer1 in the group mygroup.

Redis

XADD mystream * name Alice age 30
XADD mystream * name Bob age 25
XGROUP CREATE mystream mygroup 0 MKSTREAM
# Read messages as consumer1 from mygroup
# Your code here

Hint

Use XREADGROUP GROUP mygroup consumer1 COUNT 2 STREAMS mystream > to read new messages.

4

Acknowledge the messages after processing

Use the Redis command XACK mystream mygroup followed by the message IDs you read to acknowledge them. For example, if the message IDs are 1526985058136-0 and 1526985058137-0, the command is XACK mystream mygroup 1526985058136-0 1526985058137-0. Add this command with the exact IDs you got from reading the messages.

Redis

XADD mystream * name Alice age 30
XADD mystream * name Bob age 25
XGROUP CREATE mystream mygroup 0 MKSTREAM
XREADGROUP GROUP mygroup consumer1 COUNT 2 STREAMS mystream >
# Acknowledge the messages by their IDs
# Your code here

Hint

Use XACK mystream mygroup <message-id> to acknowledge messages after processing.

Practice

1. What is the main purpose of Redis consumer groups in streams?

easy

A. Allow multiple clients to read the same stream without duplicating work

B. Store data permanently in Redis

C. Encrypt messages in the stream

D. Automatically delete old messages from the stream

Solution

1. Step 1: Understand consumer groups role

   Consumer groups let many clients read from the same stream safely and efficiently.

2. Step 2: Identify main benefit

   They avoid duplicate processing by tracking which messages each consumer has read.

3. Final Answer:

   Allow multiple clients to read the same stream without duplicating work -> Option A

4. Quick Check:

   Consumer groups = safe multi-client reading [OK]

Hint: Consumer groups = shared reading without duplicates [OK]

Common Mistakes:

• Thinking consumer groups store data permanently
• Confusing consumer groups with encryption
• Assuming consumer groups delete old messages automatically

2. Which Redis command is used to create a consumer group for a stream named mystream?

easy

A. XGROUP NEW mystream mygroup $

B. XGROUP CREATE mystream mygroup $

C. XGROUP ADD mystream mygroup $

D. XREADGROUP CREATE mystream mygroup $

Solution

1. Step 1: Recall command to create consumer group

   The correct command is XGROUP CREATE followed by stream name, group name, and ID.

2. Step 2: Match command syntax

   XGROUP CREATE mystream mygroup $ correctly creates group mygroup on mystream.

3. Final Answer:

   XGROUP CREATE mystream mygroup $ -> Option B

4. Quick Check:

   Create group = XGROUP CREATE [OK]

Hint: Use XGROUP CREATE to make groups [OK]

Common Mistakes:

• Using XREADGROUP instead of XGROUP CREATE
• Using non-existent commands like XGROUP ADD or XGROUP NEW
• Mixing command order or missing parameters

3. Given a consumer group mygroup on stream mystream, what does this command return?

XREADGROUP GROUP mygroup consumer1 COUNT 2 STREAMS mystream > 

medium

A. Returns up to 2 new messages never delivered to any consumer

B. Returns all messages in the stream from the beginning

C. Returns the last 2 messages already acknowledged

D. Returns an error because of wrong syntax

Solution

1. Step 1: Understand XREADGROUP with '>' ID

   Using '>' means read only new messages never delivered to any consumer in the group.

2. Step 2: COUNT 2 limits messages returned

   The command returns up to 2 such new messages for consumer1.

3. Final Answer:

   Returns up to 2 new messages never delivered to any consumer -> Option A

4. Quick Check:

   XREADGROUP with '>' = new messages only [OK]

Hint: '>' ID in XREADGROUP means new messages only [OK]

Common Mistakes:

• Thinking it returns all messages from start
• Assuming it returns acknowledged messages
• Believing the command syntax is incorrect

4. You run this command but get an error:

XREADGROUP GROUP mygroup consumer1 STREAMS mystream >

What is the likely cause?

medium

A. Missing COUNT argument causes syntax error

B. The '>' symbol is invalid here

C. The command is missing the ID after STREAMS

D. Consumer group 'mygroup' does not exist

Solution

1. Step 1: Verify command syntax

   The syntax is correct: XREADGROUP GROUP mygroup consumer1 STREAMS mystream > uses '>' ID for new messages. COUNT is optional.

2. Step 2: Identify prerequisite error

   XREADGROUP requires an existing consumer group created with XGROUP CREATE. If missing, it returns a NOGROUP error.

3. Final Answer:

   Consumer group 'mygroup' does not exist -> Option D

4. Quick Check:

   XREADGROUP needs existing group [OK]

Hint: XREADGROUP requires existing consumer group [OK]

Common Mistakes:

• Thinking COUNT is mandatory (it's optional)
• Believing '>' is invalid
• Mistakenly thinking ID is missing after STREAMS

5. You have a stream orders and consumer group ordergroup. You want to ensure that if a consumer crashes before acknowledging messages, another consumer can claim and process those pending messages. Which Redis command helps you do this?

hard

A. XREADGROUP GROUP ordergroup consumer2 STREAMS orders >

B. XACK orders ordergroup 1526985058136

C. XCLAIM orders ordergroup consumer2 60000 1526985058136

D. XGROUP CREATE orders ordergroup $

Solution

1. Step 1: Understand the problem of crashed consumers

   If a consumer crashes, its pending messages remain unacknowledged and need to be claimed by others.

2. Step 2: Identify command to claim pending messages

   XCLAIM lets another consumer take ownership of pending messages after a timeout.

3. Step 3: Match command usage

   XCLAIM orders ordergroup consumer2 60000 1526985058136 claims message ID 1526985058136 for consumer2.

4. Final Answer:

   XCLAIM orders ordergroup consumer2 60000 1526985058136 -> Option C

5. Quick Check:

   Claim pending messages = XCLAIM [OK]

Hint: Use XCLAIM to recover unacknowledged messages [OK]

Common Mistakes:

• Using XACK which only acknowledges, not claims
• Using XREADGROUP with '>' which reads new messages only
• Trying to create group again instead of claiming