MongoDB Mastery lessons / Module 25: Oplog & Replication

The oplog, initial sync, lag and rollback

Level 4: Database Engineer28 min

The capped collection at the heart of replication: what entries look like, how big it should be, what causes replication lag and how rollback works.

This lesson is interactive. Run every example against an in-browser MongoDB, see explain() plans drawn as trees, and take the quiz.

Open the interactive lesson

You will learn to

Read oplog entries and know why they are idempotent
Size the oplog by window, not by bytes
Diagnose replication lag
Explain initial sync and rollback

conceptThe concept

Open the interactive lesson to use this diagram.

The oplog (local.oplog.rs) is a special capped collection. Every write on the primary appends entries describing the result of the change: an $inc becomes the new value, so applying an entry twice gives the same result (idempotent). Secondaries fetch entries in batches, apply them in parallel by document, and record their progress (optime).

Default size: 5% of free disk, between 990 MB and 50 GB. Resize online with replSetResizeOplog. Since 4.4, oplogMinRetentionHours keeps entries for a minimum time.
The oplog window is the time between the first and last entry. A member that is offline longer than the window cannot catch up and needs a full initial sync.
Initial sync clones every database and then applies the oplog accumulated during the copy. File copy based initial sync (Enterprise 5.2+) copies data files instead.
Rollback: if a primary accepted writes that never replicated to the majority, then failed over, those writes are undone when it rejoins and saved to rollback/ files.

whyWhy it matters

The oplog window is your safety margin for maintenance, network partitions, change stream consumers and backups based on oplog replay. Many teams discover it is only 2 hours during a heavy batch, exactly when a secondary is down for patching.

Open the interactive lesson to use this diagram.

Use caseReal-world scenario

A secondary is taken down for an OS upgrade that runs long. The oplog window during normal traffic is 30 hours, but a bulk re-indexing job generates 10x the usual writes, shrinking the window to 3 hours. The secondary falls off the oplog and needs a 6-hour initial sync of 2 TB. Monitoring the window during batch jobs and sizing the oplog for the worst day avoids this.

Try itExamples

Run and edit these in the interactive lesson.

See your own oplog entries

db.orders.updateOne({ orderNo: "ORD-100050" }, { $inc: { total: 10 } })
db.customers.updateOne({ _id: 12 }, { $unset: { phone: "" }, $set: { tier: "gold" } })
db.getSiblingDB("local").oplog.rs.find({ ns: /^shop\./ }).sort({ $natural: -1 }).limit(2)

$inc: 10 is recorded as the resulting total. Re-applying the entry is harmless. (This oplog uses a simplified diff format.)

Oplog window and lag

rs.printReplicationInfo()
rs.printSecondaryReplicationInfo()

Operations on a real deployment

// resize oplog to 100 GB online (run on each member)
db.adminCommand({ replSetResizeOplog: 1, size: 102400 })
// keep at least 48 hours regardless of size
db.adminCommand({ replSetResizeOplog: 1, minRetentionHours: 48 })
// flow control status
db.serverStatus().flowControl
// replication metrics on a secondary
db.serverStatus().metrics.repl.apply

internalsUnder the hood

Replication lag causes: under-provisioned secondaries (slower disks), heavy single-document hot spots (cannot be applied in parallel), large multi-updates generating many entries, index builds on secondaries, network bandwidth between data centres, and read load on secondaries competing with oplog application. Flow control (4.2+) throttles writes on the primary when the majority commit point lags more than flowControlTargetLagSeconds (10 s by default), trading write latency for bounded lag.

Oplog window calculator

Open the interactive lesson to use this diagram.

Avoid / preferDo this, not that

Avoid

replication:
  oplogSizeMB: 1024    # busy 2 TB cluster

Prefer

db.adminCommand({ replSetResizeOplog: 1, size: 51200, minRetentionHours: 48 })

The oplog window must outlast your longest maintenance, backup or resync. A tiny oplog on a busy cluster means any secondary down for an hour needs a full initial sync.

Avoid

db.getSiblingDB("local").oplog.rs.find({ ns: "shop.orders" })

Prefer

db.getSiblingDB("local").oplog.rs.find({
  ts: { $gte: Timestamp({ t: 1791446000, i: 0 }) }, ns: "shop.orders"
}).limit(100)

The oplog has no secondary indexes, but it is stored in ts order, so a ts lower bound becomes a cheap bounded scan. An ns-only filter reads the whole oplog.

PitfallsCommon mistakes

Sizing the oplog by bytes without measuring the window under peak write load.
Running huge updateMany operations that produce millions of oplog entries in minutes.
Serving heavy analytics from the same secondaries that must keep up with replication.
Not alerting on the oplog window and on lag (seconds behind primary).

LimitsWhen not to use it

Applications should not read the oplog directly for change data capture: its format is internal and can change between versions. Use change streams, which are resumable, permission-checked and stable.

SecuritySecurity notes

The oplog contains every write, including sensitive fields. Only backup and CDC tools should be able to read the local database; no application role needs it.

ChallengeTry it yourself

Insert a document into a scratch collection, update it with $inc, delete it, then list the matching oplog entries (op and o) in order.

Show one solution

One solution

db.oplog_lab.drop()
db.oplog_lab.insertOne({ _id: 1, qty: 1 })
db.oplog_lab.updateOne({ _id: 1 }, { $inc: { qty: 2 } })
db.oplog_lab.deleteOne({ _id: 1 })
db.getSiblingDB("local").oplog.rs.find({ ns: "shop.oplog_lab" }, { _id: 0, op: 1, o: 1 }).sort({ $natural: 1 })

InterviewQuestions you should be able to answer

Why must oplog entries be idempotent?

Secondaries may apply an entry more than once (after a crash during a batch, during initial sync while the copy overlaps the oplog). Recording results (set field to 42) rather than operations (increment by 1) makes repeated application safe.

A secondary is lagging. How do you troubleshoot?

Check rs.printSecondaryReplicationInfo() and lag trend; compare hardware and disk latency of primary vs secondary; look at the write workload (bulk updates, hot documents); check whether secondary reads or index builds compete; check network between sites; inspect metrics.repl.apply and buffer; verify the oplog window is not about to be exceeded. Short term: reduce secondary read load, throttle batches; long term: match hardware, fix workload patterns, enable or tune flow control.

Explain rollback in a replica set.

A primary accepts writes acknowledged with w:1, crashes or is partitioned before replicating them, and another member becomes primary and takes new writes. When the old primary returns, its oplog has diverged; it finds the common point, undoes its unreplicated writes (recover-to-timestamp in modern versions), saves them as BSON in the rollback directory, and resyncs. Majority write concern avoids losing acknowledged writes this way.

QuizCheck your understanding

4 questions with instant feedback in the interactive lesson; a wrong answer goes into your review deck.

← Replica sets, elections and failover Backup, restore and point-in-time recovery →