Cluster migrations and cutover

conceptThe concept

Phases: initial copy → continuous catch-up (oplog or change streams) → validation → cutover (stop writes, wait for zero lag, switch connection strings) → monitoring with a rollback window.

whyWhy it matters

Migrations happen more often than people expect: hardware refresh, cloud moves, version upgrades that skip several majors, consolidations, or moving off a managed service to self-hosted infrastructure. They fail on details: missing indexes, a collection nobody knew about, users and roles not copied, or no way back.

Use caseReal-world scenario

Moving 4 TB from an on-premises replica set to a self-managed Kubernetes cluster: mongosync performs the initial sync over three days and keeps up with changes; validation compares counts and per-collection hashes; on the cutover night writes are paused for 4 minutes, lag reaches zero, the commit completes, and applications switch connection strings. Reverse sync is kept running for a week as the rollback path.

Try itExamples

Run and edit these in the interactive lesson.

mongosync --cluster0 "mongodb://src-0,src-1,src-2/?replicaSet=rs0" \
          --cluster1 "mongodb://dst-0,dst-1,dst-2/?replicaSet=rs0" &
curl -X POST localhost:27182/api/v1/start -d '{ "source": "cluster0", "destination": "cluster1" }'
curl localhost:27182/api/v1/progress            # state, lagTimeSeconds, canCommit
# cutover: stop application writes, wait for lagTimeSeconds ~ 0
curl -X POST localhost:27182/api/v1/commit -d '{}'

db.getCollectionNames().filter(c => !c.startsWith("system.")).map(c => ({ coll: c, count: db.getCollection(c).countDocuments(), indexes: db.getCollection(c).getIndexes().length }))

db.orders.aggregate([
  { $project: { k: { $concat: [{ $toString: "$_id" }, ":", { $toString: "$total" }, ":", "$status"] } } },
  { $sort: { k: 1 } },
  { $group: { _id: null, n: { $sum: 1 }, sample: { $first: "$k" }, last: { $last: "$k" } } }
])

internalsUnder the hood

Catch-up depends on the source oplog window: if the initial copy takes longer than the window, the sync cannot resume and must start over. Index builds on the destination can dominate copy time, so tools build them at the right point. Validation methods: countDocuments per collection, dbHash on quiesced data, sampled document comparison, and application-level checks.

Avoid / preferDo this, not that

mongodump --uri "$SRC" --archive | mongorestore --uri "$DST" --archive
# while the application keeps writing to the source

mongosync --cluster0 "$SRC" --cluster1 "$DST"
curl -X POST localhost:27182/api/v1/start --data '{ "source": "cluster0", "destination": "cluster1" }'
# continuous sync, then a short planned cutover

// counts look similar, switch the application now

// pause writes or commit the sync, then compare per collection:
// counts, index lists, and sampled or hashed documents on both sides

PitfallsCommon mistakes

• Forgetting users, roles, indexes, validators, TTL settings and views in the destination.
• Initial copy longer than the oplog window.
• No rehearsal with production-sized data.
• No rollback plan after cutover (writes have already happened on the new cluster).
• Changing versions, topology and schema all in one migration.

LimitsWhen not to use it

Live sync tools have prerequisites (versions, feature compatibility, restrictions on DDL during sync). For a small database with an agreed downtime window, dump and restore during the window is simpler and easier to reason about. Do not change topology, version and schema in one migration; split them.

SecuritySecurity notes

Migration tools need broad read and write privileges on both clusters. Create temporary dedicated users, revoke them after cutover, use TLS on both legs, and do not leave dump archives on shared disks or buckets without encryption and a deletion date.

ChallengeTry it yourself

const edge = (coll, dir) => {
  const d = coll.find({}, { _id: 1 }).sort({ _id: dir }).limit(1).toArray()[0]
  return d ? d._id : null
}
const fingerprint = db.getCollectionNames().sort().map(c => {
  const coll = db.getCollection(c)
  return { coll: c, count: coll.countDocuments(), firstId: edge(coll, 1), lastId: edge(coll, -1) }
})
fingerprint

InterviewQuestions you should be able to answer

QuizCheck your understanding

4 questions with instant feedback in the interactive lesson; a wrong answer goes into your review deck.