HA, DR and multi-region topologies
Design for data-centre and region failures: member placement, majority math, disaster recovery runbooks, forced reconfiguration and RPO/RTO trade-offs.
- Place members across zones and regions so a majority survives
- Distinguish HA (automatic) from DR (planned, possibly manual)
- Write a region-loss runbook including forced reconfiguration
- Combine replication, backups and cluster-to-cluster sync for DR
conceptThe concept
The majority rule drives everything. A replica set can elect a primary only where a majority of voting members can talk to each other.
| Topology | Survives | Notes |
|---|---|---|
| 3 members, 3 zones (1-1-1) | any one zone | Standard single-region HA |
| 5 members, 3 regions (2-2-1) | any one region | Third region can be a small tie-breaker member |
| 2 regions (2-1 or 3-2) | only loss of the minority side automatically | Losing the majority side needs manual forced reconfig (possible data loss) |
| Separate DR cluster (mongosync / backups) | full region or cluster loss, logical corruption depending on lag | Manual failover, RPO = sync lag |
HA is automatic and handles member or zone failures with RPO 0 (majority writes) and RTO ~ seconds. DR handles region loss, corruption and human error with defined RPO/RTO and a runbook that people have practised.
whyWhy it matters
Architectures that look redundant can still fail badly: two data centres with 2+2 members and an arbiter in one of them; a region outage leaving the minority side read-only for hours because nobody knew how to force a reconfig. Interviewers for senior roles love this topic because it reveals whether you understand quorum.
Use caseReal-world scenario
A payments platform runs 5 members: 2 in Mumbai, 2 in Hyderabad, 1 in Chennai (priority 0, small instance). Writes use w: "majority", which needs 3 acknowledgements, so at least one is cross-region; latency budget accounts for it. Losing Mumbai entirely leaves 3 of 5 votes: Hyderabad elects a primary automatically. Nightly PBM backups and continuous PITR go to object storage in a fourth region.
Try itExamples
Run and edit these in the interactive lesson.
# 0. Declare incident; confirm the majority region is really down (not a monitoring blip).
# 1. Fence the failed side: block network / stop instances so it cannot come back as primary.
# 2. On a surviving member:
cfg = rs.conf()
cfg.members = cfg.members.filter(m => ["hyd-1:27017", "hyd-2:27017", "chn-1:27017"].includes(m.host))
rs.reconfig(cfg, { force: true })
# 3. Verify a primary is elected; check rs.status() optimes to understand data loss window.
# 4. Point applications (DNS / service discovery / connection string) at the new set.
# 5. When the failed region returns: wipe and resync its members, then re-add with rs.add().
# 6. Post-incident: reconcile lost writes from rollback files / upstream sources.cfg = rs.conf()
cfg.members[0].tags = { region: "mumbai", zone: "a" }
cfg.members[1].tags = { region: "mumbai", zone: "b" }
cfg.members[2].tags = { region: "hyderabad", zone: "a" }
cfg.settings.getLastErrorModes = { multiRegion: { region: 2 } } // custom write concern: 2 distinct regions
rs.reconfig(cfg)
db.payments.insertOne(doc, { writeConcern: { w: "multiRegion" } })
// local reads
db.orders.find(q).readPref("nearest", [{ region: "hyderabad" }])internalsUnder the hood
Forced reconfiguration (rs.reconfig(cfg, { force: true }) on a surviving member) lets a minority become a replica set again. It can lose writes that only the unreachable majority had, and if the old majority comes back without being reconfigured you risk split brain, so the runbook must fence the old side first. Chained replication, network latency and write concern choices determine actual RPO for cross-region members.
Open the interactive lesson to use this diagram.
Avoid / preferDo this, not that
// rs0-0: ap-south-1a // rs0-1: ap-south-1a // rs0-2: ap-south-1a
// rs0-0: ap-south-1a // rs0-1: ap-south-1b // rs0-2: ap-south-1c
Three members in one zone survive a disk or node failure but not the zone going away. Spreading across zones costs a few milliseconds of write latency and removes a common single point of failure.
// primary region: 1 voter // DR region: 2 voters
// primary region: 2 voters (priority 2) // DR region: 2 voters (priority 1) // third site: 1 voter (priority 0 or arbiter-free tie-breaker)
Where the majority of votes lives decides where a primary can be elected. With the majority in the DR region, a network cut between regions moves the primary away from the application, and every w:"majority" write waits on cross-region round trips.
PitfallsCommon mistakes
- Two-region designs that assume automatic failover in both directions.
- Arbiters as the tie-breaker in a separate region while using majority write concern (PSA problems).
- DR plans that were never executed; the first run during a real outage takes hours.
- Forgetting that application servers, DNS, secrets and monitoring must also fail over.
- Counting replication as DR against logical corruption; it replicates the corruption.
LimitsWhen not to use it
Do not stretch one replica set across continents if the application cannot tolerate cross-region latency on every majority write. Separate regional clusters with asynchronous replication (for example Cluster-to-Cluster Sync) give lower latency in exchange for a non-zero recovery point. Pick the trade-off from your stated RPO and RTO, not from what looks most resilient on a diagram.
SecuritySecurity notes
DR runbooks include rs.reconfig(cfg, { force: true }). Limit replSetReconfig to a small admin group and fence the failed side (stop it or cut its network) before forcing, or two primaries can accept writes. Make sure the DR site has what it needs to start: TLS certificates, keyfile, KMS keys and credentials replicated there, not only in the primary region.
ChallengeTry it yourself
Take a copy of rs.conf() (do not apply it), tag the members with regions mumbai, mumbai, hyderabad, then count votes per region and the votes needed for a majority.
Show one solution
const cfg = rs.conf()
const regions = ["mumbai", "mumbai", "hyderabad"]
cfg.members.forEach((m, i) => { m.tags = { region: regions[i] } })
const votes = {}
cfg.members.forEach(m => { votes[m.tags.region] = (votes[m.tags.region] || 0) + m.votes })
const total = cfg.members.reduce((a, m) => a + m.votes, 0)
printjson({ votes, majority: Math.floor(total / 2) + 1 })InterviewQuestions you should be able to answer
Design a MongoDB deployment that survives the loss of a full region.
At least 3 regions with a 2-2-1 (5 members) or 1-1-1 (3 members) voting distribution so any single region loss leaves a majority. Majority write concern, retryable writes, drivers configured with all members. Cross-region latency adds to majority write latency, so place the two larger groups close together. Add tagged read preferences for local reads, PITR backups in another region, and a practised runbook for application and DNS failover.
What happens with 3 members in DC1 and 2 in DC2 if DC1 fails?
DC2 holds 2 of 5 votes, not a majority, so no primary is elected; the set is read-only. Restoring writes needs a forced reconfiguration to the DC2 members, accepting that writes acknowledged only by DC1 members may be lost, and fencing DC1 so it cannot return as a second primary.
QuizCheck your understanding
4 questions with instant feedback in the interactive lesson; a wrong answer goes into your review deck.