Monitoring with Percona Monitoring and Management (PMM)

conceptThe concept

PMM is an open-source monitoring platform. PMM Server (usually a container) stores metrics and runs Grafana dashboards, alerting and Query Analytics (QAN). PMM Client (pmm-agent plus exporters) runs on each database host and ships metrics and query samples.

For MongoDB, pmm-admin add mongodb registers a service; metrics come from the MongoDB exporter, and QAN reads the profiler (--query-source=profiler) or slow-query logs. Typical dashboards: instances overview and summary, replica set summary (lag, elections, oplog window), WiredTiger (cache, tickets, eviction), sharded cluster summary, and node-level CPU, disk and memory.

whyWhy it matters

Self-hosted MongoDB needs the same visibility that a managed service offers: per-query-shape latency, replication health and storage engine internals in one place. PMM provides that without licence costs and integrates with the Percona Operator.

Use caseReal-world scenario

Latency rises every day at 09:00. QAN ranks query shapes by total time: one find on orders by status and createdAt dominates, with a high documents-examined-to-returned ratio. Its example query plus explain confirms a blocking sort; an ESR index fixes it, and the WiredTiger dashboard shows application-thread eviction dropping back to zero.

Try itExamples

Run and edit these in the interactive lesson.

use admin
try { db.dropUser("pmm") } catch (e) {}
try { db.dropRole("explainRole") } catch (e) {}
db.createRole({
  role: "explainRole",
  privileges: [{ resource: { db: "", collection: "" }, actions: ["dbHash", "find", "listIndexes", "listCollections", "collStats", "dbStats", "indexStats"] },
               { resource: { db: "", collection: "system.version" }, actions: ["find"] }],
  roles: []
})
db.createUser({ user: "pmm", pwd: "change-me", roles: [{ role: "explainRole", db: "admin" }, { role: "clusterMonitor", db: "admin" }, { role: "read", db: "local" }] })
db.getUser("pmm").roles.map(r => r.role + "@" + r.db)

# on the database host, after installing pmm-client
pmm-admin config --server-insecure-tls --server-url=https://admin:****@pmm.internal:443
pmm-admin add mongodb --username=pmm --password=**** \
  --host=127.0.0.1 --port=27017 --cluster=prod-rs0 \
  --replication-set=rs0 --environment=prod \
  --query-source=profiler --service-name=prod-rs0-node1
pmm-admin list

db.setProfilingLevel(1, { slowms: 50, sampleRate: 0.5 })
db.getProfilingStatus()

internalsUnder the hood

The profiler query source adds load proportional to profiled operations, so keep slowms or the sample rate sensible on busy clusters. Dashboards are only as good as the scrape interval; for second-level incidents, combine PMM with FTDC data. The monitoring user needs clusterMonitor, read access to local for oplog metrics, and a small custom role for collection and index statistics.

Avoid / preferDo this, not that

pmm-admin add mongodb --username=root --password=...

pmm-admin add mongodb --username=pmm --password="$PMM_PW" --query-source=profiler

db.setProfilingLevel(2)   // profile everything for Query Analytics

db.setProfilingLevel(1, { slowms: 100 })

PitfallsCommon mistakes

• Profiling every operation (level 2) on a busy production cluster to feed QAN.
• Monitoring only the primary; lag and eviction problems often start on secondaries.
• Giving the monitoring user root or readWriteAnyDatabase instead of a minimal role.

LimitsWhen not to use it

PMM is a monitoring and query analytics stack; it does not replace alert routing and on-call tooling, or capacity planning on its own. Keep its retention and scrape intervals reasonable: second-level scrapes across many nodes make PMM itself the bottleneck.

SecuritySecurity notes

Give the PMM user only monitoring roles, serve the PMM server over HTTPS behind SSO, and change the default admin password at install. Query Analytics shows real query text and examples, which can contain customer data: restrict who can open QAN.

ChallengeTry it yourself

db.setProfilingLevel(1, { slowms: 100 })
const admin = db.getSiblingDB("admin")
try { admin.dropUser("pmm_lab") } catch (e) {}
admin.createUser({ user: "pmm_lab", pwd: "change-me", roles: [{ role: "clusterMonitor", db: "admin" }, { role: "read", db: "local" }] })
printjson({ profile: db.getProfilingStatus(), roles: admin.getUser("pmm_lab").roles.map(r => r.role) })

InterviewQuestions you should be able to answer

QuizCheck your understanding

4 questions with instant feedback in the interactive lesson; a wrong answer goes into your review deck.