Replica sets, elections and failover
How members replicate, how a new primary is elected, what priorities, votes, hidden and delayed members do, and how failover looks from the application.
- Describe member types: primary, secondary, arbiter, hidden, delayed
- Explain the election protocol, terms, priorities and timeouts
- Design a fault-tolerant voting topology
- Predict application behaviour during failover
conceptThe concept
Open the interactive lesson to use this diagram.
- A replica set has one primary that accepts writes and records them in the oplog, and secondaries that copy and apply the oplog.
- Up to 50 members, at most 7 voting. Elections need a majority of voting members: 3 members tolerate 1 failure, 5 tolerate 2.
- priority (0 to 1000) influences who becomes primary; priority 0 members never become primary. hidden members are invisible to clients (good for backups and analytics). delayed members (
secondaryDelaySecs) keep an older copy, a protection against human error. arbiters vote but hold no data. - Members send heartbeats every 2 s. If a secondary has not heard from the primary for
electionTimeoutMillis(10 s), it calls an election: increments the term, votes for itself, and asks others. A voter grants its vote only if the candidate's oplog is at least as recent as its own.
Replica set simulator
Write with w:1 or w:"majority", kill members, partition the primary, add lag, and watch elections, commit points and rollbacks happen.
Open the interactive lesson to use this diagram.
whyWhy it matters
Replica sets are how MongoDB provides high availability and safe backups. Most production outages that lose data involve a misunderstood topology: two data centres with an even split, arbiters in PSA sets, or w:1 writes during failover.
Use caseReal-world scenario
Three data-bearing members across three availability zones, priorities 2/1/1 so the preferred zone holds the primary, plus a hidden delayed member (1 hour) in a fourth location with priority 0 and votes 0 for recovering from accidental deletes. The application uses w: "majority" and retryable writes, so a zone outage causes a ~10 s write pause and no lost acknowledged data.
Try itExamples
Run and edit these in the interactive lesson.
rs.status().members.map(m => ({ name: m.name, state: m.stateStr, health: m.health, lagSecs: m.optimeDate ? Math.round((rs.status().date - m.optimeDate) / 1000) : null }))
rs.conf().members.map(m => ({ host: m.host, priority: m.priority, votes: m.votes, hidden: m.hidden, delaySecs: m.secondaryDelaySecs }))
rs.conf().settingscfg = rs.conf()
cfg.members[0].priority = 2 // prefer this member as primary
cfg.members[3].priority = 0
cfg.members[3].hidden = true
cfg.members[3].secondaryDelaySecs = 3600 // delayed member, also set votes: 0
cfg.members[3].votes = 0
rs.reconfig(cfg)
rs.stepDown(60) // planned failover: primary steps down for 60 s
rs.freeze(120) // keep a secondary from seeking electionmongodb://app:***@mongo-0:27017,mongo-1:27017,mongo-2:27017/shop?replicaSet=rs0&retryWrites=true&w=majority&readPreference=primaryinternalsUnder the hood
The election protocol (pv1) is Raft-like: terms prevent split brain, a candidate needs a majority of votes, and a primary that cannot see a majority steps down by itself. After an election, the new primary may catch up from other secondaries (catchUpTimeoutMillis) before accepting writes. Typical failover: 10 s to detect plus around 1-2 s to elect; drivers detect the new primary through server monitoring and retry writes once (retryable writes) and reads (retryable reads).
Avoid / preferDo this, not that
rs.addArb("arb1:27017") // primary + secondary + arbiterrs.add({ host: "db3:27017", priority: 1 }) // three data-bearing membersWith PSA, losing the one secondary leaves no majority of data-bearing members, so w:"majority" writes stall and the majority commit point stops moving.
// DC-A: db1, db2 DC-B: db3, db4
// DC-A: db1, db2 DC-B: db3, db4 DC-C: db5
An even split across two sites cannot survive losing either site: neither side has a majority. A fifth voter in a third site breaks the tie.
PitfallsCommon mistakes
- Even number of voting members across two data centres: losing the bigger side (or the link) leaves no majority anywhere.
- Primary-Secondary-Arbiter (PSA) in production: when one data member is down, majority writes block and the cache fills with history (the majority commit point cannot advance).
- Using hidden or delayed members with votes that matter for elections.
- Changing several voting members in a single reconfig instead of one at a time.
- Testing failover only on paper; run
rs.stepDown()drills in staging and production maintenance windows.
LimitsWhen not to use it
Do not raise electionTimeoutMillis to hide a flaky network: it makes genuine failovers slower. Fix the network instead. Hidden and priority 0 members are for backups, analytics and delayed copies, not for serving fresh reads.
SecuritySecurity notes
Members must authenticate to each other: use x.509 certificates (security.clusterAuthMode: x509) or at least a keyfile, and encrypt replication traffic with TLS. Without internal authentication, anything that can reach the port can pose as a member.
ChallengeTry it yourself
From rs.status(), list each member with its state and how many seconds it is behind the primary.
Show one solution
const st = rs.status()
const p = st.members.find(m => m.stateStr === "PRIMARY")
st.members.map(m => ({ name: m.name, state: m.stateStr, lagSecs: (p.optimeDate - m.optimeDate) / 1000 }))InterviewQuestions you should be able to answer
How does a MongoDB replica set elect a new primary?
When secondaries miss heartbeats from the primary for electionTimeoutMillis (10 s default), an eligible member (priority > 0, up to date enough) increments the term and requests votes. Each voter votes once per term and only for a candidate whose oplog is at least as recent as its own. A majority of votes wins. Higher-priority members can later trigger a takeover once caught up. A primary that loses contact with a majority steps down to avoid split brain.
Why are arbiters discouraged?
They add a vote without data. In a PSA set with one data member down you still have a primary (2 of 3 votes), but w: "majority" writes cannot be acknowledged, majority reads go stale, and the primary keeps history for the majority commit point, causing cache pressure. A third data-bearing member is safer. If you must use an arbiter, consider setting the secondary to votes: 0 during long outages, carefully.
What is a delayed member used for?
It applies the oplog with a delay (e.g. 1 hour), so after a human error such as a dropped collection you can recover data from it before the mistake replicates. It must be priority 0 and should be hidden, with votes 0 in most designs. It is not a replacement for backups.
What does the application see during a failover?
Writes fail or block for the election window (roughly 10-12 s with defaults). Drivers with retryable writes retry once after the new primary is discovered, so most single-statement writes succeed transparently. In-flight transactions abort with TransientTransactionError and are retried by the callback API. Reads with primaryPreferred/secondaryPreferred can continue on secondaries.
QuizCheck your understanding
5 questions with instant feedback in the interactive lesson; a wrong answer goes into your review deck.