Maintenance and data checks: validate, compact, renameCollection and $redact

conceptThe concept

validate checks a collection's data and indexes: every document is readable, every index has exactly the entries it should, and (with { full: true }) the underlying WiredTiger structures are sound. It reports valid, nInvalidDocuments, nNonCompliantDocuments (documents that fail the schema validator), keysPerIndex, and lists such as missingIndexEntries and extraIndexEntries. Plain validate takes an exclusive lock on the collection while it runs; { background: true } (5.0) checks a checkpoint without blocking, and { metadata: true } only checks the collection's metadata.

Free space. Deleting documents does not shrink the data files. WiredTiger keeps the space for reuse and reports it as freeStorageSize in collection stats. compact rewrites the collection and its indexes to give that space back to the operating system. Since 4.4 it does not block reads and writes on the collection. In 8.0, { dryRun: true } estimates how much it would free, and autoCompact runs compaction in the background on files with enough reusable space.

renameCollection renames within a database as a metadata change. With dropTarget: true it replaces an existing collection in one step, which makes it the standard way to swap in a rebuilt collection. Renaming across databases copies all data and is slow.

$redact walks each document level by level. An expression decides for each level: $$DESCEND keeps it and checks the levels below, $$PRUNE removes it and everything inside, $$KEEP keeps it without further checks.

whyWhy it matters

Validation is how you prove a node is healthy after a crash, a storage incident or a suspicious restore. Compact is how you get disk back after deleting most of a large collection, which matters when a volume is close to full. Collection swaps let you rebuild data (a backfill, a schema change, a re-import) without the application seeing a half-built collection. And redaction is how analysts and support staff get useful data without seeing fields they must not see.

Use caseReal-world scenario

A retention job deletes three years of events: 1.2 TB of a 1.5 TB collection. Disk usage does not move, and the volume alert keeps firing. stats().freeStorageSize shows 1.1 TB reusable. The team runs compact with dryRun on each secondary to confirm the estimate, then compacts the secondaries one at a time, steps the primary down and compacts it last. Before and after, validate on a restored copy confirms the data and indexes are consistent.

Try itExamples

Run and edit these in the interactive lesson.

const v = db.orders.validate({ full: true })
;({ valid: v.valid, nrecords: v.nrecords, nInvalidDocuments: v.nInvalidDocuments, keysPerIndex: v.keysPerIndex, missing: v.missingIndexEntries.length, extra: v.extraIndexEntries.length })

db.lab_val.drop()
db.lab_val.insertMany([{ sku: "A1", price: 10 }, { sku: "A2" }]).acknowledged
db.runCommand({ collMod: "lab_val", validator: { $jsonSchema: { required: ["sku", "price"] } }, validationLevel: "moderate" })
db.lab_val.validate().nNonCompliantDocuments

db.lab_cmp.drop()
db.lab_cmp.insertMany(db.orders.find({}, { _id: 0 }).limit(2000).toArray()).acknowledged
db.lab_cmp.deleteMany({ status: { $ne: "delivered" } }).deletedCount
const s1 = db.lab_cmp.stats()
const est = db.runCommand({ compact: "lab_cmp", dryRun: true })
const done = db.runCommand({ compact: "lab_cmp" })
;({ storageBefore: s1.storageSize, freeBefore: s1.freeStorageSize, estimated: est.estimatedBytesFreed, freed: done.bytesFreed, storageAfter: db.lab_cmp.stats().storageSize })

db.lab_prices.drop()
db.lab_prices_new.drop()
db.lab_prices.insertMany([{ _id: "P001", price: 200 }, { _id: "P002", price: 50 }]).acknowledged
db.lab_prices_new.insertMany([{ _id: "P001", price: 199 }, { _id: "P002", price: 55 }, { _id: "P003", price: 12 }]).acknowledged
db.lab_prices_new.createIndex({ price: 1 })
db.adminCommand({ renameCollection: "shop.lab_prices_new", to: "shop.lab_prices", dropTarget: true })
;({ docs: db.lab_prices.countDocuments(), indexes: db.lab_prices.getIndexes().map(i => i.name) })

db.lab_a.drop(); db.lab_b.drop()
db.lab_a.insertOne({ x: 1 }); db.lab_b.insertOne({ x: 2 })
db.lab_a.renameCollection("lab_b")

db.lab_docs.drop()
db.lab_docs.insertOne({
  _id: 1, title: "Q3 plan", level: 1,
  budget: { level: 3, amount: 2500000 },
  sections: [{ level: 1, text: "Hiring" }, { level: 3, text: "Acquisition target" }]
})
db.lab_docs.aggregate([
  { $redact: { $cond: { if: { $lte: [{ $ifNull: ["$level", 1] }, 2] }, then: "$$DESCEND", else: "$$PRUNE" } } }
])

// 8.0: background compaction on files with at least 100 MB reusable
db.adminCommand({ autoCompact: true, freeSpaceTargetMB: 100 })
db.adminCommand({ autoCompact: false })

// manual compact on a replica set: secondaries first, primary last
// on each secondary:
db.getSiblingDB("shop").runCommand({ compact: "events", dryRun: true })
db.getSiblingDB("shop").runCommand({ compact: "events" })
// then on the primary
rs.stepDown()
db.getSiblingDB("shop").runCommand({ compact: "events" })

// without blocking: checks the last checkpoint (5.0+)
db.events.validate({ background: true })

// full structural check: on a hidden or out-of-rotation secondary, or a restored copy
db.events.validate({ full: true })

// metadata only, very fast
db.runCommand({ validate: "events", metadata: true })

// sharded clusters: compare config metadata with the shards (7.0+)
db.getSiblingDB("admin").runCommand({ checkMetadataConsistency: 1 })

internalsUnder the hood

Every collection and index is a separate WiredTiger file. Freed blocks inside a file are reused by later inserts in that same file; compact rewrites live data towards the start of the file and truncates the end. It needs free disk to work and generates I/O, so run it member by member. A member resync also rebuilds every file compactly, which can be faster than compacting many collections.

Validate reads every document and every index entry. On large collections it is slow and, without background mode, blocks the collection, so run it on a secondary taken out of rotation or on a restored copy.

Avoid / preferDo this, not that

// "free up disk" on the primary at noon
db.events.deleteMany({ ts: { $lt: cutoff } })
db.runCommand({ compact: "events" })

db.runCommand({ compact: "events", dryRun: true })   // on each secondary first
// compact secondaries one at a time, then rs.stepDown() and compact the old primary

db.products.deleteMany({})
db.products.insertMany(fresh)

db.products_new.insertMany(fresh)
db.products_new.createIndex({ sku: 1 }, { unique: true })
db.adminCommand({ renameCollection: "shop.products_new", to: "shop.products", dropTarget: true })

PitfallsCommon mistakes

• Running validate({ full: true }) on a busy primary and blocking the collection.
• Expecting deleteMany or a TTL index to shrink the data files.
• Compacting every member at the same time, or the primary first.
• Building a replacement collection in place, so readers see it half-loaded, instead of renaming at the end.
• Using $redact for access control alone: redaction must sit behind a view or a role, or users can query the base collection.

LimitsWhen not to use it

Do not compact routinely: reusable space is normal and is filled by new data. Do not run validate as a health check on every collection every night on the primary. And do not rename across databases on large collections in production; it copies all data. Export and import, or rebuild in the target database, under your own control.

SecuritySecurity notes

validate, compact and renameCollection need administrative privileges (dbAdmin on the database, clusterManager or equivalent for some options); keep them out of application roles. For redaction, put the $redact pipeline in a view and grant readers access to the view only. A user with find on the base collection bypasses redaction.

ChallengeTry it yourself

db.lab_tickets.drop()
db.lab_tickets.insertOne({ _id: 1, subject: "Refund", level: 1, customer: { level: 2, email: "a@example.com" }, notes: [{ level: 1, text: "called" }, { level: 3, text: "fraud check" }] })
db.lab_tickets.aggregate([
  { $redact: { $cond: { if: { $lte: [{ $ifNull: ["$level", 1] }, 2] }, then: "$$DESCEND", else: "$$PRUNE" } } }
]).toArray()

InterviewQuestions you should be able to answer

QuizCheck your understanding

5 questions with instant feedback in the interactive lesson; a wrong answer goes into your review deck.