LSM trees in write-heavy systems in Data Structures Theory - Step-by-Step Execution

Concept Flow - LSM trees in write-heavy systems

Write request arrives

↓

Write to in-memory structure (MemTable)

↓

MemTable full?

No→Wait for next write

Yes↓

Flush MemTable to disk as SSTable

↓

Merge/Compaction of SSTables

↓

Optimized disk storage for reads

↓

Next write request

Writes first go to fast memory, then flush to disk when full, followed by merging files to keep reads efficient.

Execution Sample

Data Structures Theory

write(data)
  memtable.add(data)
  if memtable.is_full():
    flush_to_disk(memtable)
    compact_sstables()

This shows how data is written to memory, flushed to disk when full, and then compacted.

Analysis Table

Step	Action	MemTable State	Disk SSTables	Notes
1	Write data1	data1	empty	MemTable stores data1
2	Write data2	data1, data2	empty	MemTable stores data2
3	MemTable full check	data1, data2	empty	MemTable not full yet
4	Write data3	data1, data2, data3	empty	MemTable stores data3
5	MemTable full check	data1, data2, data3	empty	MemTable full now
6	Flush MemTable	empty	SSTable1(data1, data2, data3)	Data moved to disk
7	Write data4	data4	SSTable1(data1, data2, data3)	New MemTable starts
8	Write data5	data4, data5	SSTable1(data1, data2, data3)	MemTable stores data5
9	MemTable full check	data4, data5	SSTable1(data1, data2, data3)	MemTable not full
10	Write data6	data4, data5, data6	SSTable1(data1, data2, data3)	MemTable stores data6
11	MemTable full check	data4, data5, data6	SSTable1(data1, data2, data3)	MemTable full again
12	Flush MemTable	empty	SSTable1(data1, data2, data3), SSTable2(data4, data5, data6)	Second flush to disk
13	Compaction	empty	SSTable_merged(data1..data6)	Merge SSTables for efficiency
14	Wait for next write	empty	SSTable_merged(data1..data6)	Ready for new writes

💡 Process repeats as new writes come in, keeping memory fast and disk organized.

State Tracker

Variable	Start	After Step 1	After Step 5	After Step 6	After Step 12	After Step 13	Final
MemTable	empty	data1	data1, data2, data3	empty	empty	empty	empty
Disk SSTables	empty	empty	empty	SSTable1(data1, data2, data3)	SSTable1(data1, data2, data3), SSTable2(data4, data5, data6)	SSTable_merged(data1..data6)	SSTable_merged(data1..data6)

Key Insights - 3 Insights

Why do writes go first to memory (MemTable) instead of directly to disk?

Writing to memory is much faster than disk. The execution_table shows steps 1-5 where data accumulates in MemTable before flushing, improving write speed.

What happens when the MemTable becomes full?

As seen in steps 5 and 11, once MemTable is full, it is flushed to disk as an SSTable. This prevents memory overflow and organizes data on disk.

Why is compaction needed after multiple SSTables are created?

Compaction merges multiple SSTables into one (step 13) to reduce disk reads and improve read performance by keeping data organized.

Visual Quiz - 3 Questions

Look at the execution_table at step 6. What is the state of the MemTable?

A Contains data4

B Empty

C Contains data1, data2, data3

D Contains data4, data5, data6

💡 Hint

Check the 'MemTable State' column at step 6 in the execution_table.

At which step does the first flush of MemTable to disk happen?

A Step 6

B Step 5

C Step 12

D Step 13

💡 Hint

Look for the action 'Flush MemTable' in the execution_table.

If the MemTable size limit was larger, how would the execution_table change?

A Compaction would not be needed

B Flush steps would happen earlier with less data

C Flush steps would happen later with more data in MemTable

D MemTable would never flush

💡 Hint

Consider how MemTable fullness triggers flush in steps 5 and 11.

Concept Snapshot

LSM trees handle many writes by first storing data in fast memory (MemTable).
When full, MemTable is flushed to disk as sorted files (SSTables).
Multiple SSTables are merged (compacted) to keep reads efficient.
This design optimizes write speed and manages disk storage well.

Full Transcript

LSM trees are designed for systems with many writes. When data is written, it first goes into an in-memory structure called MemTable. This is fast and efficient. When the MemTable fills up, it is saved to disk as a sorted file called an SSTable. Over time, many SSTables accumulate, so the system merges them in a process called compaction. This keeps the disk organized and makes reading data faster. The process then repeats for new writes, balancing fast memory writes and efficient disk storage.