Write sharding in DynamoDB - Step-by-Step Execution

Concept Flow - Write sharding

Start Write Request

↓

Determine Shard Key

↓

Select Shard Table/Partition

↓

Write Data to Selected Shard

↓

Confirm Write Success

↓

End Write Request

Write sharding splits data writes across multiple partitions or tables to spread load and improve performance.

Execution Sample

DynamoDB

function writeData(item) {
  const shardKey = hash(item.userId) % 3;
  const tableName = 'UserDataShard' + shardKey;
  dynamoDB.put({TableName: tableName, Item: item});
}

This code chooses a shard based on userId hash and writes the item to that shard's table.

Execution Table

Step	Input Item userId	Shard Key Calculation	Selected Shard Table	Write Action	Result
1	userId=101	hash(101)%3=2	UserDataShard2	Write item to UserDataShard2	Success
2	userId=205	hash(205)%3=1	UserDataShard1	Write item to UserDataShard1	Success
3	userId=309	hash(309)%3=0	UserDataShard0	Write item to UserDataShard0	Success
4	userId=412	hash(412)%3=1	UserDataShard1	Write item to UserDataShard1	Success
5	userId=523	hash(523)%3=2	UserDataShard2	Write item to UserDataShard2	Success
6	-	-	-	-	All writes completed successfully

💡 All items written to their respective shards; no more writes pending.

Variable Tracker

Variable	Start	After 1	After 2	After 3	After 4	After 5	Final
shardKey	-	2	1	0	1	2	N/A
tableName	-	UserDataShard2	UserDataShard1	UserDataShard0	UserDataShard1	UserDataShard2	N/A

Key Moments - 2 Insights

Why do we use hash(userId) % 3 to select the shard?

Using hash(userId) % 3 evenly distributes writes across 3 shards, preventing overload on a single shard as shown in execution_table rows 1-5.

What happens if we write all data to one shard?

All writes would go to one shard causing slow performance and possible throttling, unlike the balanced writes shown in the execution_table.

Visual Quiz - 3 Questions

Look at the execution_table, what shard table is selected for userId=309?

A UserDataShard1

B UserDataShard2

C UserDataShard0

D UserDataShard3

💡 Hint

Check row 3 in the execution_table under 'Selected Shard Table'

At which step does the shardKey equal 1 for the first time?

A Step 2

B Step 1

C Step 3

D Step 4

💡 Hint

Look at the 'Shard Key Calculation' column in execution_table rows 1-4

If we increase shards to 4, how would the shardKey calculation change?

A hash(userId) * 4

B hash(userId) % 4

C hash(userId) % 3

D hash(userId) + 4

💡 Hint

Shard key uses modulo operation with number of shards as divisor, see code in execution_sample

Concept Snapshot

Write Sharding splits data writes across multiple shards.
Use a shard key (like hash(userId) % number_of_shards) to pick shard.
Write data to selected shard to balance load.
Improves write throughput and reduces bottlenecks.
Each shard is a separate partition or table.

Full Transcript

Write sharding in DynamoDB means splitting write requests across multiple shards or tables. Each write uses a shard key, often a hash of a unique identifier like userId, modulo the number of shards. This calculation decides which shard to write to. The write then goes to that shard's table or partition. This spreads the load evenly, avoiding bottlenecks. The execution table shows how different userIds map to different shards and writes succeed. Variables shardKey and tableName change per write to reflect the shard chosen. Beginners often wonder why modulo is used; it ensures even distribution. Also, writing all data to one shard causes slowdowns, which sharding prevents. The quiz checks understanding of shard selection and shard key calculation. The snapshot summarizes the key points for quick review.

Practice

1. What is the main purpose of write sharding in DynamoDB?

easy

A. To backup data automatically to S3

B. To spread write load across multiple partitions by adding shard IDs

C. To encrypt data at rest for security

D. To reduce read latency by caching data locally

Solution

1. Step 1: Understand write sharding concept

   Write sharding adds shard IDs to partition keys to distribute writes evenly.

2. Step 2: Identify the main goal

   This distribution avoids hot partitions and improves write performance.

3. Final Answer:

   To spread write load across multiple partitions by adding shard IDs -> Option B

4. Quick Check:

   Write sharding = spreading writes [OK]

Hint: Write sharding spreads writes by adding shard IDs [OK]

Common Mistakes:

• Confusing write sharding with read caching
• Thinking it encrypts data
• Assuming it handles backups

2. Which of the following is the correct way to implement write sharding in a DynamoDB partition key?

easy

A. Use a fixed partition key without any shard ID

B. Use the sort key as the shard ID

C. Append a random shard ID to the partition key before writing

D. Write all data to a single partition key

Solution

1. Step 1: Recall write sharding method

   Write sharding requires adding a shard ID to the partition key to distribute writes.

2. Step 2: Identify correct implementation

   Appending a random shard ID to the partition key spreads writes across partitions.

3. Final Answer:

   Append a random shard ID to the partition key before writing -> Option C

4. Quick Check:

   Shard ID + partition key = write sharding [OK]

Hint: Add random shard ID to partition key for sharding [OK]

Common Mistakes:

• Not adding shard ID at all
• Using sort key as shard ID incorrectly
• Writing all data to one partition

3. Given the following code snippet for write sharding, what is the expected effect?

shard_id = random.randint(0, 4)
partition_key = f"user#{user_id}#{shard_id}"
dynamodb_table.put_item(Item={"PK": partition_key, "Data": data})

medium

A. Writes are distributed across 5 partitions reducing hot spots

B. All writes go to the same partition causing throttling

C. Writes fail due to invalid partition key format

D. Data is only readable by shard_id 0

Solution

1. Step 1: Analyze shard ID usage

   The shard_id is a random number from 0 to 4, appended to the partition key.

2. Step 2: Understand partition key effect

   This creates 5 different partition keys for the same user, spreading writes across 5 partitions.

3. Final Answer:

   Writes are distributed across 5 partitions reducing hot spots -> Option A

4. Quick Check:

   Random shard ID 0-4 = 5 partitions [OK]

Hint: Random shard ID 0-4 means 5 partitions [OK]

Common Mistakes:

• Assuming all writes go to one partition
• Thinking partition key format is invalid
• Believing data is restricted by shard ID

4. You implemented write sharding by appending shard IDs to partition keys, but you notice some writes still cause throttling. What is the likely cause?

medium

A. Shard ID range is too small causing uneven distribution

B. You forgot to add shard ID to the sort key

C. You used a random shard ID but did not update the partition key

D. DynamoDB does not support write sharding

Solution

1. Step 1: Check shard ID range

   If the shard ID range is too small, writes cluster on few partitions causing throttling.

2. Step 2: Verify correct key usage

   Shard ID must be part of the partition key, not sort key, and must cover enough range.

3. Final Answer:

   Shard ID range is too small causing uneven distribution -> Option A

4. Quick Check:

   Small shard range = hot partitions [OK]

Hint: Use enough shard IDs to spread writes evenly [OK]

Common Mistakes:

• Adding shard ID to sort key instead of partition key
• Not updating partition key with shard ID
• Believing DynamoDB disallows sharding

5. You want to design a write sharding strategy for a DynamoDB table with very high write traffic on user data. Which approach best balances write load and allows efficient reads?

hard

A. Write all data to a single partition key and batch writes

B. Use user ID as partition key without sharding; rely on DynamoDB auto-scaling

C. Add shard ID to sort key and keep partition key as user ID

D. Add a random shard ID (0-9) to the partition key and store user data in each shard; aggregate reads by querying all shards

Solution

1. Step 1: Evaluate write sharding options

   Adding a random shard ID (0-9) to partition key spreads writes across 10 partitions.

2. Step 2: Consider read strategy

   Reads require querying all shards and aggregating results, which is manageable for balanced load.

3. Step 3: Compare other options

   Using no sharding risks throttling; shard ID in sort key does not distribute writes; single partition causes hot spots.

4. Final Answer:

   Add a random shard ID (0-9) to the partition key and store user data in each shard; aggregate reads by querying all shards -> Option D

5. Quick Check:

   Shard ID in partition key + aggregate reads = balanced writes [OK]

Hint: Shard in partition key, aggregate reads from all shards [OK]

Common Mistakes:

• Not sharding partition key causing hot partitions
• Adding shard ID to sort key only
• Ignoring read complexity after sharding