Why databases persist data in FastAPI - Why It Works This Way

Overview - Why databases persist data

What is it?

Databases are systems that store data so it can be saved and used later. Persisting data means keeping it safe even when the computer or app is turned off. This allows apps to remember information like user accounts, messages, or settings over time. Without persistence, all data would disappear when the app closes.

Why it matters

Persisting data solves the problem of losing important information every time an app stops running. Imagine writing a message and losing it when your phone restarts. Databases keep data safe and organized so apps can work reliably and users don’t lose their work or preferences. Without this, apps would be forgetful and frustrating.

Where it fits

Before learning why databases persist data, you should understand basic programming and how apps handle data temporarily. After this, you can learn about different types of databases, how to connect them to apps like FastAPI, and how to query and update data safely.

Mental Model

Core Idea

Databases persist data by saving it permanently on storage so it remains available even after the app or computer stops running.

Think of it like...

Persisting data in a database is like writing notes in a notebook instead of on a whiteboard. The notebook keeps your notes safe over time, while the whiteboard erases everything when you clean it.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│   App Memory  │──────▶│   Database    │──────▶│  Storage Disk │
│ (Temporary)   │       │ (Persistent)  │       │ (Permanent)   │
└───────────────┘       └───────────────┘       └───────────────┘

Build-Up - 6 Steps

1

Foundation

What is data persistence

Concept: Data persistence means saving data so it lasts beyond the current program run.

When you use an app, it stores some data temporarily in memory. But when the app closes, that data disappears. Persistence means saving data to a place that keeps it safe even when the app is off, like a file or database.

Result

You understand that persistence is about saving data permanently, not just temporarily.

Understanding persistence is key to knowing why databases exist and how they help apps remember information.

2

Foundation

Difference between memory and storage

Concept: Memory is fast but temporary; storage is slower but permanent.

Computers have memory (RAM) that apps use to work quickly. But RAM forgets everything when power is lost. Storage devices like hard drives or SSDs keep data safe even when the computer is off.

Result

You can explain why data in memory disappears but data in storage stays.

Knowing this difference helps you see why databases use storage to persist data.

3

Intermediate

How databases save data permanently

🤔Before reading on: do you think databases save data directly in memory or on disk? Commit to your answer.

Concept: Databases write data to storage disks to keep it safe and permanent.

When you save data in a database, it writes that data to files on a disk. This way, even if the app or computer restarts, the data is still there. Databases also organize data so it can be found and updated efficiently.

Result

You understand that databases use storage devices to persist data beyond app runs.

Knowing that databases rely on disk storage explains how they keep data safe long-term.

4

Intermediate

Why apps use databases for persistence

🤔Before reading on: do you think apps can just save data in files instead of databases? Commit to your answer.

Concept: Databases provide organized, reliable, and fast ways to save and retrieve data compared to simple files.

While apps can save data in files, databases offer features like searching, updating, and protecting data from errors. They handle many users and large data efficiently, which simple files cannot do well.

Result

You see why databases are preferred for complex or shared data persistence.

Understanding the advantages of databases over files helps you appreciate their role in app development.

5

Advanced

How databases ensure data safety

🤔Before reading on: do you think databases always save data instantly or sometimes delay it? Commit to your answer.

Concept: Databases use techniques like transactions and logging to keep data safe and consistent even if errors or crashes happen.

Databases group changes into transactions that either fully happen or not at all. They also keep logs to recover data if the system crashes. This prevents data loss or corruption.

Result

You understand how databases protect data integrity during saves.

Knowing these safety mechanisms explains why databases are trusted for critical data.

6

Expert

Trade-offs in database persistence design

🤔Before reading on: do you think faster data saving always means safer data? Commit to your answer.

Concept: Database persistence balances speed, safety, and complexity with design choices like caching and write strategies.

To save data quickly, databases may delay writing to disk or use memory caches. But this risks losing recent changes if a crash occurs. Designers choose trade-offs based on app needs, balancing performance and data safety.

Result

You see why database persistence is a complex design problem with no perfect solution.

Understanding these trade-offs helps you make informed choices when building or using databases.

Under the Hood

Databases persist data by writing structured data to storage media like hard drives or SSDs. They use files and indexes to organize data for fast access. When data changes, databases write logs and use transactions to ensure all changes complete fully or not at all. This prevents partial updates and corruption. The database engine manages caching data in memory for speed but flushes changes to disk to keep data permanent.

Why designed this way?

Databases were designed to solve the problem of safely storing and retrieving large amounts of data reliably. Early systems used simple files but faced issues with data loss, corruption, and slow access. The transactional model and logging were introduced to guarantee data integrity. Balancing speed and safety led to caching and delayed writes. These design choices evolved to meet the needs of multi-user, high-availability systems.

┌───────────────┐
│   Application │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Database Engine│
│ ┌───────────┐ │
│ │ Transaction│ │
│ │  Manager   │ │
│ └────┬──────┘ │
│      │        │
│ ┌────▼──────┐ │
│ │  Cache    │ │
│ └────┬──────┘ │
└──────┼────────┘
       │ Writes
       ▼
┌───────────────┐
│ Storage Disk  │
│ (Files & Logs)│
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do databases always save data immediately to disk? Commit to yes or no.

Common Belief: Databases write data instantly to disk every time you save.

Reality: Databases often cache data in memory and write to disk later to improve speed.

Why it matters: Assuming instant writes can lead to misunderstanding data loss risks during crashes.

Quick: Can you use any file as a database? Commit to yes or no.

Common Belief: Any file can act as a database if you save data there.

Reality: Databases use special structures and indexes to organize data efficiently, unlike simple files.

Why it matters: Treating files as databases can cause slow, unreliable data access in real apps.

Quick: Does persisting data mean it never changes? Commit to yes or no.

Common Belief: Persisted data is fixed and cannot be updated.

Reality: Databases allow data to be updated, deleted, or added while keeping it persistent.

Why it matters: Thinking persistence means unchangeable data limits understanding of dynamic apps.

Quick: Is data persistence only about saving data? Commit to yes or no.

Common Belief: Persistence just means saving data somewhere.

Reality: Persistence also involves organizing, protecting, and efficiently accessing data over time.

Why it matters: Ignoring these aspects leads to poor app performance and data errors.

Expert Zone

1

Some databases use write-ahead logging to ensure data can be recovered after crashes without losing committed changes.

2

The choice between synchronous and asynchronous disk writes affects both performance and risk of data loss.

3

Database caching strategies can cause data to appear saved but not yet be on disk, requiring careful transaction management.

When NOT to use

Databases are not ideal for very simple or temporary data storage where file systems or in-memory caches suffice. For high-speed, ephemeral data, in-memory stores like Redis are better. For unstructured large files, object storage is preferred.

Production Patterns

In real systems, databases are combined with caching layers, backups, and replication to ensure data persistence, availability, and performance. Developers use ORMs or query builders in FastAPI to interact with databases safely and efficiently.

Connections

File Systems

Databases build on file systems by adding structure and management for data persistence.

Understanding file systems helps grasp how databases store data physically and why they need extra layers.

Memory Management

Databases use memory management techniques like caching to balance speed and persistence.

Knowing memory management clarifies why databases delay writes and how they optimize performance.

Human Memory and Note Taking

Persistence in databases is like how humans write notes to remember information beyond short-term memory.

This connection shows how persistence solves the universal problem of retaining information over time.

Common Pitfalls

#1 Assuming data is saved permanently immediately after a save command.

Wrong approach:

await db.save(data)
// Assume data is safely on disk now

Correct approach:

await db.transaction(async () => {
await db.save(data)
})
// Ensures data is committed safely

Root cause:

Misunderstanding that database writes may be cached and not instantly persisted.

#2 Using plain files to store complex app data without structure.

Wrong approach:

fs.writeFile('data.txt', JSON.stringify(data))
// No indexing or query support

Correct approach:

Use a database like SQLite or PostgreSQL to store and query data efficiently.

Root cause:

Not recognizing the need for organized data storage and retrieval.

#3 Ignoring transaction management leading to partial data updates.

Wrong approach:

db.updatePart1()
db.updatePart2()
// No transaction wrapping

Correct approach:

db.transaction(async () => {
await db.updatePart1()
await db.updatePart2()
})

Root cause:

Not understanding atomicity and consistency in data persistence.

Key Takeaways

Databases persist data by saving it permanently on storage devices, allowing apps to remember information beyond their runtime.

Persistence solves the problem of data loss when apps or computers shut down, making apps reliable and user-friendly.

Databases organize, protect, and efficiently manage data, which simple files cannot do well for complex or shared data.

Database persistence involves trade-offs between speed and safety, using techniques like caching, transactions, and logging.

Understanding how databases persist data helps developers build better apps and avoid common mistakes with data loss or corruption.