Oplog and replication mechanism in MongoDB - Mini Project: Build & Apply

Understanding MongoDB Oplog and Replication Mechanism

📖 Scenario: You are managing a MongoDB replica set for a small online store. To ensure data safety and availability, you want to understand how MongoDB replicates data changes across servers using the oplog (operation log).This project will guide you through creating a simple oplog-like collection, configuring a replication threshold, querying the oplog for recent operations, and completing the replication setup.

🎯 Goal: Build a basic MongoDB oplog simulation and query it to understand how replication tracks and applies data changes.

📋 What You'll Learn

Create a collection named oplog with specific operation entries

Add a variable replicationThreshold to filter recent operations

Write a query to find oplog entries newer than the threshold

Complete the setup by creating an index on the ts field for efficient querying

💡 Why This Matters

🌍 Real World

MongoDB replica sets use the oplog to replicate data changes from the primary to secondary servers, ensuring data consistency and high availability.

💼 Career

Understanding the oplog and replication mechanism is essential for database administrators and backend developers managing MongoDB clusters to ensure data durability and fault tolerance.

1

Create the oplog collection with sample operations

Create a collection called oplog and insert these exact documents: { ts: 1001, op: 'i', ns: 'store.products', o: { _id: 1, name: 'Pen' } }, { ts: 1002, op: 'u', ns: 'store.products', o: { _id: 1, price: 1.5 } }, and { ts: 1003, op: 'd', ns: 'store.products', o: { _id: 2 } }.

MongoDB

// Insert the specified documents into the oplog collection
// Your code here

Hint

Use db.oplog.insertMany() with an array of documents.

2

Set the replication threshold timestamp

Create a variable called replicationThreshold and set it to 1001 to represent the minimum timestamp for replication.

MongoDB

db.oplog.insertMany([
  { ts: 1001, op: 'i', ns: 'store.products', o: { _id: 1, name: 'Pen' } },
  { ts: 1002, op: 'u', ns: 'store.products', o: { _id: 1, price: 1.5 } },
  { ts: 1003, op: 'd', ns: 'store.products', o: { _id: 2 } }
])

// Define the replication threshold timestamp
// Your code here

Hint

Use const replicationThreshold = 1001 to set the threshold.

3

Query oplog entries newer than the threshold

Write a query called recentOps that finds all documents in oplog where the ts field is greater than or equal to replicationThreshold.

MongoDB

db.oplog.insertMany([
  { ts: 1001, op: 'i', ns: 'store.products', o: { _id: 1, name: 'Pen' } },
  { ts: 1002, op: 'u', ns: 'store.products', o: { _id: 1, price: 1.5 } },
  { ts: 1003, op: 'd', ns: 'store.products', o: { _id: 2 } }
])

const replicationThreshold = 1001;

// Query oplog for entries with ts >= replicationThreshold
// Your code here

Hint

Use db.oplog.find({ ts: { $gte: replicationThreshold } }) to get recent operations.

4

Create an index on the ts field for efficient replication queries

Create an index on the ts field of the oplog collection using db.oplog.createIndex to speed up queries filtering by timestamp.

MongoDB

db.oplog.insertMany([
  { ts: 1001, op: 'i', ns: 'store.products', o: { _id: 1, name: 'Pen' } },
  { ts: 1002, op: 'u', ns: 'store.products', o: { _id: 1, price: 1.5 } },
  { ts: 1003, op: 'd', ns: 'store.products', o: { _id: 2 } }
])

const replicationThreshold = 1001;

const recentOps = db.oplog.find({ ts: { $gte: replicationThreshold } });

// Create an index on ts field
// Your code here

Hint

Use db.oplog.createIndex({ ts: 1 }) to create an ascending index on ts.

Practice

1. What is the primary purpose of the oplog in MongoDB replica sets?

easy

A. To manage database indexes

B. To store user authentication details

C. To record all data changes for replication

D. To backup data to external storage

Solution

1. Step 1: Understand the role of oplog in MongoDB

   The oplog is a special log that records every data change in a replica set.

2. Step 2: Connect oplog to replication

   Replication uses the oplog to copy changes from the primary to secondary servers.

3. Final Answer:

   To record all data changes for replication -> Option C

4. Quick Check:

   Oplog records changes [OK]

Hint: Oplog logs changes for syncing replicas [OK]

Common Mistakes:

• Confusing oplog with user data storage
• Thinking oplog manages backups
• Assuming oplog stores indexes

2. Which of the following is the correct command to view the oplog entries in a MongoDB replica set?

easy

A. db.oplog.rs.find()

B. db.oplog.find()

C. db.replica.oplog()

D. db.oplogrs.find()

Solution

1. Step 1: Recall the oplog collection name

   The oplog is stored in the local database under the collection named oplog.rs.

2. Step 2: Identify the correct query syntax

   To view entries, use db.oplog.rs.find() in the mongo shell.

3. Final Answer:

   db.oplog.rs.find() -> Option A

4. Quick Check:

   Correct oplog collection query [OK]

Hint: Oplog collection is oplog.rs in local DB [OK]

Common Mistakes:

• Using wrong collection name like oplog or oplogrs
• Trying to call oplog as a function
• Missing the dot between oplog and rs

3. Given a MongoDB replica set, what will happen if the oplog on the primary server is deleted accidentally?

medium

A. Secondaries will stop replicating and may become stale

B. Secondaries will continue replicating normally

C. Primary will automatically recreate the oplog with old data

D. The replica set will elect a new primary immediately

Solution

1. Step 1: Understand oplog role in replication

   The oplog stores all changes that secondaries use to replicate data.

2. Step 2: Consequence of oplog deletion on primary

   If oplog is deleted, secondaries lose the source of changes and stop replicating, causing stale data.

3. Final Answer:

   Secondaries will stop replicating and may become stale -> Option A

4. Quick Check:

   Oplog missing = replication stops [OK]

Hint: No oplog means no replication updates [OK]

Common Mistakes:

• Assuming oplog auto-recreates with old data
• Thinking secondaries replicate without oplog
• Confusing oplog deletion with primary election

4. You run db.oplog.rs.find().limit(1) on a secondary node and get an error. What is the most likely cause?

medium

A. The oplog collection does not exist on secondaries

B. You need to switch to the local database first

C. The oplog is empty on secondaries

D. You must run the command on the primary node

Solution

1. Step 1: Check database context for oplog

   The oplog is stored in the local database, so you must switch to it before querying.

2. Step 2: Understand error cause

   If you run the command without use local, the shell looks in the wrong database and errors.

3. Final Answer:

   You need to switch to the local database first -> Option B

4. Quick Check:

   Oplog in local DB = switch DB first [OK]

Hint: Always use 'local' DB to query oplog [OK]

Common Mistakes:

• Trying to query oplog in wrong database
• Assuming oplog only exists on primary
• Thinking oplog is empty on secondaries

5. In a MongoDB replica set, how does the oplog help maintain data consistency during failover when the primary goes down?

hard

A. Oplog automatically merges conflicting writes from multiple primaries

B. Oplog backs up data externally to restore after failover

C. Oplog prevents any writes during failover to avoid conflicts

D. Secondaries use oplog entries to catch up and elect a new primary with latest data

Solution

1. Step 1: Understand failover in replica sets

   When primary fails, secondaries elect a new primary to continue operations.

2. Step 2: Role of oplog in failover

   Secondaries use oplog entries to apply all changes and ensure they have the latest data before election.

3. Step 3: Why other options are incorrect

   Oplog does not merge conflicting writes or prevent writes during failover; it also does not backup externally.

4. Final Answer:

   Secondaries use oplog entries to catch up and elect a new primary with latest data -> Option D

5. Quick Check:

   Oplog syncs data for failover [OK]

Hint: Oplog helps secondaries catch up before election [OK]

Common Mistakes:

• Thinking oplog merges conflicting writes
• Assuming oplog blocks writes during failover
• Confusing oplog with backup system