Running PostgreSQL on Linux: huge pages, OOM, kernel settings and memory contexts

conceptThe concept

• Huge pages: shared memory mapped in 2 MB pages instead of 4 kB shrinks page tables per backend (hundreds of MB with many connections) and reduces TLB misses. Set huge_pages = on (fail if unavailable) or try, and reserve vm.nr_hugepages from shared_memory_size_in_huge_pages. Disable transparent huge pages.
• Overcommit and the OOM killer: by default Linux promises more memory than exists, then kills a process when it runs out. If it kills a backend, the postmaster assumes shared memory may be corrupted, terminates every session and runs crash recovery. Use vm.overcommit_memory = 2 so allocations fail cleanly with "out of memory" in one session, and protect the postmaster with an OOM score.
• Kernel and limits: open file limit (LimitNOFILE, max_files_per_process), vm.swappiness low, dirty-page writeback in bytes rather than percentages, XFS or ext4 with noatime.
• systemd: Type=notify, KillMode=mixed, KillSignal=SIGINT (fast shutdown), and RemoveIPC=no in logind so logging out the postgres user does not delete shared memory.
• Shutdown modes: smart (wait for clients), fast (roll back sessions, checkpoint, exit; the default), immediate (no checkpoint; crash recovery on next start).
• Memory contexts: every backend allocates memory in a tree of contexts. pg_backend_memory_contexts shows your own session; pg_log_backend_memory_contexts(pid) writes another backend's tree to the server log.
• Progress views: pg_stat_progress_vacuum, _analyze, _create_index, _cluster, _copy and _basebackup show phase and blocks done for long operations.

whyWhy it matters

Many "PostgreSQL crashed" incidents are the OOM killer, and many "PostgreSQL is slow with 1000 connections" incidents are page-table memory without huge pages. Both are fixed in the OS, not in postgresql.conf. When a single query eats memory, memory contexts tell you which part of execution holds it.

Use caseReal-world scenario

A reporting query with a large hash join pushes a 64 GB server over its memory. With default overcommit the kernel kills a random backend, every application connection drops, and the server spends two minutes in crash recovery. After setting vm.overcommit_memory = 2, the same query fails alone with "out of memory" while every other session continues; pg_log_backend_memory_contexts on the offender shows a 9 GB HashTableContext, and the team lowers hash_mem_multiplier for the reporting role.

Try itExamples

Run and edit these in the interactive lesson.

SELECT name, setting, unit
FROM pg_settings
WHERE name IN ('shared_buffers', 'shared_memory_size', 'shared_memory_size_in_huge_pages',
               'huge_pages', 'huge_pages_status', 'huge_page_size', 'shared_memory_type')
ORDER BY name;

SELECT name, level, pg_size_pretty(total_bytes) AS total, pg_size_pretty(used_bytes) AS used
FROM pg_backend_memory_contexts
ORDER BY total_bytes DESC
LIMIT 5;
SELECT pg_size_pretty(sum(total_bytes)) AS backend_total
FROM pg_backend_memory_contexts;

SELECT relname AS progress_view
FROM pg_class
WHERE relname LIKE 'pg_stat_progress%' AND relnamespace = 'pg_catalog'::regnamespace
ORDER BY relname;
SELECT pid, relid::regclass AS index_on, phase, blocks_done, blocks_total
FROM pg_stat_progress_create_index;

SELECT name, setting
FROM pg_settings
WHERE name IN ('max_files_per_process', 'io_method', 'io_workers', 'effective_io_concurrency',
               'maintenance_io_concurrency', 'data_directory_mode')
ORDER BY name;

# /etc/sysctl.d/30-postgresql.conf
vm.overcommit_memory = 2
vm.overcommit_ratio = 90            # CommitLimit = swap + 90% of RAM; tune to the machine
vm.swappiness = 1
vm.nr_hugepages = 4300              # from: postgres -D $PGDATA -C shared_memory_size_in_huge_pages
vm.dirty_background_bytes = 67108864
vm.dirty_bytes = 536870912
# kernel command line: transparent_hugepage=never

# systemd unit override for the server
[Service]
Type=notify
User=postgres
OOMScoreAdjust=-1000
Environment=PG_OOM_ADJUST_FILE=/proc/self/oom_score_adj
Environment=PG_OOM_ADJUST_VALUE=0
LimitNOFILE=65536
KillMode=mixed
KillSignal=SIGINT
TimeoutSec=infinity

# /etc/systemd/logind.conf
RemoveIPC=no

# shutdown modes
pg_ctl stop -D "$PGDATA" -m smart       # wait for every client to disconnect
pg_ctl stop -D "$PGDATA" -m fast        # default: roll back sessions, checkpoint, exit
pg_ctl stop -D "$PGDATA" -m immediate   # no checkpoint, crash recovery on next start

-- write another backend's memory context tree to the server log
SELECT pg_log_backend_memory_contexts(12345);

internalsUnder the hood

Shared memory (shared buffers, WAL buffers, lock tables) is one mapping created by the postmaster and inherited by every forked backend; each process needs its own page-table entries for it, which is what huge pages shrink. Private memory (work_mem for each sort or hash node, maintenance_work_mem, catalog caches) comes from memory contexts that are freed in bulk when the context is reset, so leaks are bounded by query or transaction lifetime. On Linux, huge_pages_status (17+) reports whether huge pages were actually obtained. PostgreSQL 18 adds asynchronous I/O (io_method: worker by default, io_uring where supported, or sync).

Avoid / preferDo this, not that

# default vm.overcommit_memory = 0, no OOM protection

vm.overcommit_memory = 2
# plus OOMScoreAdjust=-1000 for the postmaster and PG_OOM_ADJUST_VALUE=0 for backends

kill -9 12345   # a stuck backend

SELECT pg_cancel_backend(12345);      -- cancel the query
SELECT pg_terminate_backend(12345);   -- end the session

huge_pages = on   # without reserving vm.nr_hugepages

# reserve first, then
huge_pages = on   # or try, and check huge_pages_status after start

PitfallsCommon mistakes

• Leaving transparent huge pages enabled, which causes latency spikes from background compaction.
• Setting overcommit_memory = 2 with a ratio too low for the machine, so the server cannot start or connections fail.
• Using immediate shutdown routinely; it skips the checkpoint and forces crash recovery.
• Forgetting RemoveIPC=no, so the server breaks when someone logs out of the postgres account.

LimitsWhen not to use it

On managed services (RDS, Cloud SQL, Azure) you cannot change kernel settings; huge pages and OOM behaviour are the provider's job. In Kubernetes, the container memory limit replaces overcommit settings: size the limit above shared_buffers plus max_connections times realistic work_mem, or the cgroup OOM killer triggers the same crash-restart cycle.

SecuritySecurity notes

The data directory must be owned by the postgres OS user with mode 0700 (or 0750 with group access); the server refuses to start otherwise. pg_log_backend_memory_contexts is superuser-only by default because the log can reveal query text; grant EXECUTE on it only to operations roles.

ChallengeTry it yourself

SELECT name, total_bytes / 1024 AS total_kb,
       round(100.0 * used_bytes / total_bytes) AS pct_used
FROM pg_backend_memory_contexts
WHERE used_bytes * 2 > total_bytes
ORDER BY total_bytes DESC
LIMIT 5;

InterviewQuestions you should be able to answer

QuizCheck your understanding

4 questions with instant feedback in the interactive lesson; a wrong answer goes into your review deck.

Go deeperOfficial documentation

• Resource consumption
• Query planning settings