Database migration in production in Django - Deep Dive

Overview - Database migration in production

What is it?

Database migration in production means safely changing the structure of a live database while the application is running. It involves updating tables, columns, or indexes without losing data or causing downtime. This process uses tools to apply changes step-by-step and keep the database and code in sync. It ensures the app continues working smoothly as new features or fixes require database updates.

Why it matters

Without proper database migration in production, changes can break the app, cause data loss, or force downtime that frustrates users. It solves the problem of evolving a database safely as the app grows. Imagine a busy store changing its shelves without closing; migrations let developers rearrange data storage without stopping service. Without this, updates would be risky and slow, hurting user trust and business.

Where it fits

Before learning this, you should understand basic Django models and how databases work. After mastering migrations, you can explore advanced topics like database optimization, backup strategies, and continuous deployment. This topic sits between learning Django ORM basics and deploying reliable, scalable applications.

Mental Model

Core Idea

Database migration in production is a careful, step-by-step update of the live database structure that keeps data safe and the app running without interruption.

Think of it like...

It's like renovating a kitchen while still cooking meals: you plan small changes, do them one at a time, and keep everything usable so no dinner is ruined.

┌─────────────────────────────┐
│   Current Database Schema   │
├──────────────┬──────────────┤
│   Tables     │   Data       │
├──────────────┼──────────────┤
│   Users      │   1000 rows  │
│   Orders     │   5000 rows  │
└──────────────┴──────────────┘
          │
          ▼
┌─────────────────────────────┐
│   Migration Script Runs      │
│  - Add column                │
│  - Rename table              │
│  - Modify index              │
└──────────────┬──────────────┘
               ▼
┌─────────────────────────────┐
│ Updated Database Schema      │
│  - New columns/tables       │
│  - Data preserved           │
│  - App continues working    │
└─────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding Django Models and ORM

Concept: Learn how Django models define database tables and how the ORM maps Python code to database operations.

Django models are Python classes that describe the structure of your database tables. Each attribute in a model corresponds to a database column. The ORM (Object-Relational Mapper) lets you work with database data using Python code instead of SQL. For example, creating a User model with fields like name and email creates a users table with those columns.

Result

You can create, read, update, and delete database records using Python objects without writing SQL.

Understanding models and ORM is essential because migrations are generated from changes in these models.

2

Foundation

What Are Migrations in Django?

Concept: Migrations are files that describe changes to the database schema based on model changes.

When you change a Django model, you create a migration file using 'python manage.py makemigrations'. This file contains instructions to update the database schema, like adding a column or creating a table. Applying migrations with 'python manage.py migrate' runs these instructions on the database.

Result

Your database schema updates to match your Django models automatically.

Migrations automate database changes, preventing manual errors and keeping code and database in sync.

3

Intermediate

Preparing Migrations for Production

🤔Before reading on: do you think you should run migrations directly on production without testing? Commit to yes or no.

Concept: Learn how to prepare and test migrations safely before applying them to a live database.

Before running migrations in production, generate them locally and test on a staging database that mirrors production. Review migration files to understand what changes will happen. Use version control to track migration files. Avoid destructive operations like dropping columns without backups. Plan for downtime or use zero-downtime techniques if needed.

Result

You reduce risks of errors or downtime when applying migrations to production.

Knowing how to prepare migrations prevents costly mistakes and ensures smooth updates.

4

Intermediate

Applying Migrations Without Downtime

🤔Before reading on: do you think all migrations can be applied instantly without affecting users? Commit to yes or no.

Concept: Techniques to apply migrations while keeping the app available to users.

Some migrations, like adding nullable columns or indexes, can run without downtime. Others, like renaming columns or removing fields, may require careful steps: add new columns, update code to use them, migrate data, then remove old columns in a later deployment. Use database locks and monitor performance. Tools like Django's 'RunPython' allow custom data migrations.

Result

You can update the database schema with minimal or no downtime.

Understanding migration impact on live systems helps maintain user experience during updates.

5

Intermediate

Handling Migration Conflicts and Dependencies

🤔Before reading on: do you think migrations from different branches always merge cleanly? Commit to yes or no.

Concept: How to manage migration files when multiple developers work on the same project.

When multiple branches add migrations, conflicts can occur if they modify the same models. Django tracks migration dependencies to apply them in order. To resolve conflicts, merge migration files carefully or create new migrations that combine changes. Use descriptive names and keep migrations small. Communicate with your team to avoid overlapping changes.

Result

Your migration history stays consistent and avoids errors during deployment.

Managing migration dependencies is crucial for teamwork and stable production updates.

6

Advanced

Customizing Migrations with RunPython

🤔Before reading on: do you think migrations only change schema, not data? Commit to yes or no.

Concept: Using Django's RunPython to perform data transformations during migrations.

Sometimes schema changes require updating existing data. Django's RunPython lets you write Python code inside migrations to modify data safely. For example, filling a new column based on old data or cleaning inconsistent records. This code runs as part of the migration process, ensuring schema and data stay consistent.

Result

You can perform complex data updates automatically during migrations.

Knowing how to combine schema and data changes prevents manual fixes and keeps production data reliable.

7

Expert

Zero-Downtime Migrations and Database Locks

🤔Before reading on: do you think all database schema changes are fast and non-blocking? Commit to yes or no.

Concept: Understanding how database locks affect migrations and strategies to avoid downtime.

Some schema changes lock tables, blocking reads and writes, causing downtime. For example, adding a column with a default value can lock large tables. Experts use techniques like adding nullable columns first, backfilling data in batches, then making columns non-nullable later. Tools like pt-online-schema-change or Django's separate migrations help. Monitoring locks and query performance is essential during migration.

Result

You minimize or eliminate downtime caused by database schema changes.

Understanding database internals and lock behavior is key to reliable production migrations.

Under the Hood

Django migrations work by generating Python files that describe database schema changes as operations. When you run 'migrate', Django reads these files and executes SQL commands in order, tracking applied migrations in a special table. This ensures each migration runs once and in the correct sequence. The migration system also supports dependencies and custom code for data changes.

Why designed this way?

Django migrations were designed to automate and standardize database schema changes, reducing manual errors and improving developer productivity. The use of Python files allows flexibility, including custom data operations. Tracking applied migrations prevents repeated changes and supports collaborative development. Alternatives like raw SQL scripts were error-prone and hard to maintain.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│  Model Change │──────▶│ Makemigrations│──────▶│ Migration File│
└───────────────┘       └───────────────┘       └───────────────┘
                                                      │
                                                      ▼
                                              ┌───────────────┐
                                              │   Migrate     │
                                              └───────────────┘
                                                      │
                                                      ▼
                                              ┌───────────────┐
                                              │ Database SQL  │
                                              └───────────────┘
                                                      │
                                                      ▼
                                              ┌───────────────┐
                                              │  Updated DB   │
                                              └───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do you think running migrations always happens instantly without affecting users? Commit to yes or no.

Common Belief: Running migrations is always fast and safe to do anytime on production.

Reality: Some migrations lock tables or take a long time, causing downtime or slow queries.

Why it matters: Ignoring this can cause your app to freeze or crash during migration, hurting users and revenue.

Quick: Do you think migrations only change database structure, not data? Commit to yes or no.

Common Belief: Migrations only modify tables and columns, not the actual data inside.

Reality: Migrations can include data changes using custom Python code to keep data consistent with schema.

Why it matters: Not knowing this leads to manual data fixes and inconsistent production data.

Quick: Do you think you can safely delete old migration files after applying them? Commit to yes or no.

Common Belief: Once migrations run, old migration files are no longer needed and can be deleted.

Reality: Migration files must be kept for future deployments and to keep migration history intact.

Why it matters: Deleting migration files breaks deployment on new environments and causes errors.

Quick: Do you think all migration conflicts are easy to fix by merging files? Commit to yes or no.

Common Belief: Migrations from different branches always merge cleanly without extra work.

Reality: Conflicts can be complex and require careful resolution or creating new combined migrations.

Why it matters: Poor conflict handling causes broken migrations and deployment failures.

Expert Zone

1

Some database engines handle schema changes differently; knowing your database's locking and indexing behavior is crucial for safe migrations.

2

Splitting complex migrations into smaller steps reduces risk and makes rollback easier if something goes wrong.

3

Using feature flags alongside migrations allows gradual rollout of schema changes and code updates, minimizing user impact.

When NOT to use

Avoid running large or destructive migrations directly on production during peak hours; instead, use blue-green deployments, shadow databases, or offline maintenance windows. For very complex schema changes, consider database refactoring tools or manual SQL scripts with expert supervision.

Production Patterns

In production, teams often use a multi-step migration process: first add nullable columns, deploy code to use new columns, backfill data asynchronously, then remove old columns in a later migration. Continuous integration pipelines run migrations on staging before production. Monitoring tools track migration impact on database performance.

Connections

Continuous Integration/Continuous Deployment (CI/CD)

Database migrations are integrated into CI/CD pipelines to automate safe deployment of schema changes.

Understanding migrations helps you design deployment pipelines that keep app and database in sync without manual steps.

Version Control Systems (Git)

Migration files are tracked in version control to coordinate schema changes among developers.

Knowing how migrations relate to Git branches and merges prevents conflicts and deployment errors.

Project Management and Change Control

Database migrations require planning and coordination similar to managing changes in complex projects.

Viewing migrations as controlled changes helps avoid surprises and ensures smooth collaboration.

Common Pitfalls

#1 Running migrations directly on production without testing.

Wrong approach:

python manage.py migrate --noinput

Correct approach:

Test migrations on staging first, review migration files, then run migrate on production during low traffic.

Root cause:

Underestimating risks and skipping testing leads to unexpected downtime or data loss.

#2 Dropping columns or tables in one migration step without data backup.

Wrong approach:

class Migration(migrations.Migration):
operations = [
migrations.RemoveField(model_name='user', name='old_field'),
]

Correct approach:

First mark field as deprecated in code, migrate data out, then remove field in a later migration with backup.

Root cause:

Not understanding data dependencies causes irreversible data loss.

#3 Ignoring migration conflicts from multiple branches.

Wrong approach:

Merging branches with conflicting migration files without resolving dependencies.

Correct approach:

Manually merge migrations or create new combined migration files to resolve conflicts before deployment.

Root cause:

Lack of communication and understanding of migration dependencies causes broken migration history.

Key Takeaways

Database migrations in production let you safely update your live database structure without stopping your app.

Always prepare, test, and review migrations before applying them to avoid downtime and data loss.

Some migrations can lock the database and cause delays; plan steps carefully to minimize impact.

Migrations can change both schema and data, so use custom code when needed to keep data consistent.

Managing migration files and conflicts is essential for teamwork and smooth production deployments.