MongoDB on Kubernetes with operators

conceptThe concept

Open the interactive lesson to use this diagram.

An operator watches a custom resource (for example a PerconaServerMongoDB or MongoDBCommunity object) and reconciles StatefulSets, Services, Secrets and config to match it: creating replica sets, adding members, rotating certificates, performing rolling upgrades and scheduling backups.

• Storage: one PVC per member from a StorageClass with volumeBindingMode: WaitForFirstConsumer (so the volume is created in the pod's zone) and allowVolumeExpansion: true. Use local NVMe only if you accept member rebuilds on node loss.
• Scheduling: pod anti-affinity or topology spread constraints across nodes and zones; PodDisruptionBudget allowing at most one member down.
• Resources: guaranteed QoS (requests = limits for memory); set cacheSizeGB explicitly to about half of the memory limit.
• Networking: clients inside the cluster use the headless service; external clients need per-member endpoints (split horizon / LoadBalancer per pod) because drivers connect to each member by name.
• Backups: operator-integrated PBM or Ops Manager to object storage, plus PITR.

whyWhy it matters

Many organisations standardise on Kubernetes for everything, including databases. Operators make that reasonable, but stateful workloads punish naive settings: a node drain that evicts two members at once, a volume in the wrong zone, or a memory limit that OOM-kills mongod because the cache was sized from host RAM.

Use caseReal-world scenario

A platform team runs 30 Percona Server for MongoDB replica sets via the Percona Operator. Each has 3 members spread across zones, PDB maxUnavailable 1, 16 Gi memory limits with cacheSizeGB: 7, gp3 volumes with expansion enabled, nightly physical backups plus PITR to S3, and upgrades done by changing the image in the custom resource during maintenance windows.

Try itExamples

Run and edit these in the interactive lesson.

apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB
metadata:
  name: orders-db
spec:
  crVersion: 1.20.0
  image: percona/percona-server-mongodb:8.0.4-1
  replsets:
    - name: rs0
      size: 3
      affinity:
        antiAffinityTopologyKey: topology.kubernetes.io/zone
      podDisruptionBudget:
        maxUnavailable: 1
      resources:
        limits:   { cpu: "4", memory: 16Gi }
        requests: { cpu: "4", memory: 16Gi }
      configuration: |
        storage:
          wiredTiger:
            engineConfig:
              cacheSizeGB: 7
      volumeSpec:
        persistentVolumeClaim:
          storageClassName: gp3-wffc
          resources:
            requests: { storage: 500Gi }
  backup:
    enabled: true
    pitr: { enabled: true }
    storages:
      s3-backups:
        type: s3
        s3: { bucket: orders-db-backups, region: ap-south-1, credentialsSecret: backup-s3 }
    tasks:
      - name: nightly
        schedule: "0 2 * * *"
        storageName: s3-backups
        type: physical

kubectl get psmdb -n db                      # cluster status from the operator
kubectl get pods -n db -o wide               # which node and zone each member is on
kubectl exec -n db orders-db-rs0-0 -c mongod -- mongosh --quiet --eval 'rs.status().members.map(m => [m.name, m.stateStr])'
kubectl patch psmdb orders-db -n db --type merge -p '{"spec":{"image":"percona/percona-server-mongodb:8.0.5-2"}}'
kubectl get psmdb-backup -n db

internalsUnder the hood

StatefulSets give stable network identities (rs0-0.rs0.ns.svc) and stable volumes, which replica set configuration depends on. The operator usually runs a sidecar or init container for configuration and backups; the mongod container is the one to target for kubectl exec. When a node fails, the pod is rescheduled only where its volume can attach; zonal volumes pin members to zones.

Avoid / preferDo this, not that

kind: Deployment
spec:
  template:
    spec:
      volumes:
        - name: data
          emptyDir: {}

# Operator-managed StatefulSet: one PVC per member
volumeSpec:
  persistentVolumeClaim:
    storageClassName: fast-ssd   # volumeBindingMode: WaitForFirstConsumer
    resources:
      requests:
        storage: 100Gi

# no affinity: the scheduler may put all three members on one node

affinity:
  antiAffinityTopologyKey: "topology.kubernetes.io/zone"

PitfallsCommon mistakes

• Memory limit lower than what the default cache size assumes (OOM kills).
• StorageClass with Immediate binding creating volumes in zones where the pod cannot run.
• No PodDisruptionBudget: node maintenance evicts two members and the set loses its primary.
• Exposing a replica set externally through a single load balancer; drivers need each member addressable.
• kubectl exec into the first container of the pod (often a sidecar) instead of the mongod container.

LimitsWhen not to use it

Do not move MongoDB to Kubernetes without an operator and a team that can debug storage, networking and scheduling under pressure. If your volumes are slow network disks, or you run one small instance, a VM or a managed host is often simpler. Kubernetes helps when you run many clusters and want them declared, versioned and reconciled the same way.

SecuritySecurity notes

Operators keep system user passwords in Kubernetes Secrets, which are base64, not encrypted. Enable etcd encryption at rest or use an external secrets manager, restrict who can get secrets in the namespace, add a NetworkPolicy so only the application namespace reaches port 27017, and issue TLS certificates with cert-manager rather than self-signed defaults.

ChallengeTry it yourself

const voters = rs.conf().members.filter(m => m.votes > 0).length
const majority = Math.floor(voters / 2) + 1
printjson({ voters, majority, pdbMaxUnavailable: voters - majority })

InterviewQuestions you should be able to answer

QuizCheck your understanding

4 questions with instant feedback in the interactive lesson; a wrong answer goes into your review deck.