Monitoring and diagnostics

conceptThe concept

FTDC (full-time diagnostic data capture) records serverStatus, replSetGetStatus, system metrics and more every second into diagnostic.data/ in a compact format. It is the first thing MongoDB and Percona support ask for after an incident.

whyWhy it matters

Good monitoring turns "the database is slow" into "cache dirty hit 20% at 14:02 when the batch started, checkout p99 rose to 900 ms". Without history you are guessing.

Use caseReal-world scenario

Incident triage order: (1) is the replica set healthy and is there a primary; (2) resource saturation: CPU, disk latency, cache, tickets and queues; (3) what changed: deploy, batch, traffic; (4) top operations: currentOp and slow log by shape; (5) fix or mitigate: kill the runaway, throttle the batch, add the index, scale.

Try itExamples

Run and edit these in the interactive lesson.

const ss = db.serverStatus()
printjson({ uptime: ss.uptime, connections: ss.connections, opcounters: ss.opcounters, queue: ss.globalLock && ss.globalLock.currentQueue })

db.currentOp({ active: true })
db.currentOp({ active: true, secs_running: { $gt: 3 }, op: { $in: ["query", "update", "command"] } })

mongostat --uri "$MONGO_URI" 1          # per-second ops, dirty %, used %, qrw/arw queues, conn
mongotop --uri "$MONGO_URI" 5           # read/write time per collection
# slow queries from the JSON log, grouped by query shape
jq -c 'select(.msg=="Slow query") | {ns: .attr.ns, hash: .attr.queryHash, ms: .attr.durationMillis, plan: .attr.planSummary}' /var/log/mongodb/mongod.log

db.orders.aggregate([{ $collStats: { storageStats: {}, count: {} } }, { $project: { ns: 1, count: 1, "storageStats.size": 1, "storageStats.totalIndexSize": 1 } }])

internalsUnder the hood

Logs are structured JSON since 4.4 ({"t":..., "s":"I", "c":"COMMAND", "id":51803, "msg":"Slow query", "attr":{...}}), making them easy to parse with jq or ship to a log platform. Each slow query line includes planSummary, keysExamined, docsExamined, durationMillis, queryHash and storage wait times. Per-collection latency is available with $collStats: { latencyStats: {} }.

Avoid / preferDo this, not that

// alert: CPU > 90%

// alerts: replication lag > 30 s, oplog window < 24 h,
//         dirty cache > 5%, sustained operation queues,
//         connections > 80% of the limit

db.currentOp()

db.currentOp({ active: true, secs_running: { $gte: 5 } })

PitfallsCommon mistakes

• Alerting only on CPU; cache dirty %, tickets, queues and replication lag are earlier signals.
• No baseline: you cannot tell what is abnormal without normal.
• Deleting diagnostic.data to save space; it is capped (about 200 MB) and invaluable.
• Killing operations without recording what they were (capture currentOp output first).

LimitsWhen not to use it

mongostat and mongotop are live triage tools, not a monitoring system: they keep no history. Use PMM, Prometheus exporters or Ops Manager to keep metrics over time and alert on them.

SecuritySecurity notes

Give monitoring exporters the clusterMonitor role, never root; exporter credentials are often stored in plain configuration files. currentOp and logs show query filters with literal values, so treat them as sensitive.

ChallengeTry it yourself

db.currentOp({ active: true, secs_running: { $gte: 5 } }).inprog
  .map(op => ({ opid: op.opid, ns: op.ns, plan: op.planSummary, secs: op.secs_running }))

InterviewQuestions you should be able to answer

QuizCheck your understanding

4 questions with instant feedback in the interactive lesson; a wrong answer goes into your review deck.