Production host tuning: OS, filesystem, memory and limits

conceptThe concept

whyWhy it matters

Many unexplained latency spikes come from the host: THP compaction stalls, NUMA imbalance, swapping, a 4 MB readahead pulling useless data into memory, or running out of file descriptors at 1,000 connections. These are cheap to get right before go-live and painful to discover in production.

Use caseReal-world scenario

A team sees p99 spikes of 2 s every few minutes. perf and kernel stats show THP compaction on a 7.0 deployment. Disabling THP through a systemd unit that runs before mongod removes the spikes. The fix is added to the base image and the pre-production checklist.

Try itExamples

Run and edit these in the interactive lesson.

# /etc/security/limits.d/mongod.conf
mongod soft nofile 64000
mongod hard nofile 64000
mongod soft nproc  64000
mongod hard nproc  64000

# /etc/sysctl.d/90-mongodb.conf
vm.swappiness = 1
vm.zone_reclaim_mode = 0
vm.max_map_count = 262144
net.ipv4.tcp_keepalive_time = 120

# fstab: XFS with noatime
/dev/nvme1n1  /var/lib/mongodb  xfs  defaults,noatime  0 0

# readahead for the data device
blockdev --setra 32 /dev/nvme1n1

# systemd: run with interleaved NUMA policy
ExecStart=/usr/bin/numactl --interleave=all /usr/bin/mongod --config /etc/mongod.conf

db.adminCommand({ getCmdLineOpts: 1 }).parsed
db.hostInfo()

internalsUnder the hood

mongod logs startup warnings for many of these (THP, NUMA, ulimits, access control); check them with db.adminCommand({ getLog: "startupWarnings" }). Cloud disks have IOPS and throughput caps per volume and per instance; size both, and remember that bursting volumes (e.g. gp2-style credits) run out under sustained load.

Avoid / preferDo this, not that

# /etc/security/limits.d/mongod.conf
mongod soft nofile 1024

# /etc/security/limits.d/mongod.conf (or LimitNOFILE in the systemd unit)
mongod soft nofile 64000
mongod hard nofile 64000
mongod soft nproc  64000

blockdev --setra 4096 /dev/nvme1n1   # 2 MB readahead

blockdev --setra 32 /dev/nvme1n1     # 16 KB readahead

PitfallsCommon mistakes

• Copying a THP rule from an old blog post without checking the production notes for your version.
• ext4 with default options, or network filesystems (NFS) for data files.
• Leaving ulimit nofile at 1024.
• Running other memory-hungry services on database hosts.

LimitsWhen not to use it

Do not paste a tuning checklist from an old blog post. Some advice changed between versions: transparent huge pages were "always disable" for years, and MongoDB 8.0 now recommends enabling them because of its updated TCMalloc. Check the production notes for your exact version and kernel, change one thing at a time and measure. In containers many of these settings belong to the host, not the pod.

SecuritySecurity notes

Run mongod as a dedicated unprivileged user, never root. The data directory should be owned by that user with mode 0700, and the keyfile or TLS private key readable only by it (0400). Keep SELinux or AppArmor in enforcing mode with the vendor policy instead of disabling it to make a permission error go away.

ChallengeTry it yourself

const o = db.adminCommand({ getCmdLineOpts: 1 }).parsed
const h = db.hostInfo().system
printjson({ dbPath: o.storage.dbPath, bindIp: o.net.bindIp, cacheGB: o.storage.wiredTiger.engineConfig.cacheSizeGB, cores: h.numCores, ramGB: Math.round(h.memSizeMB / 1024) })

InterviewQuestions you should be able to answer

QuizCheck your understanding

4 questions with instant feedback in the interactive lesson; a wrong answer goes into your review deck.