How PostgreSQL Really Works: Postmaster, Buffer Pool, and WAL

Picture this: you click “run” on an app, a tiny connection request races across the network, and somewhere behind a firewall a database decides whether your query is even allowed to exist. Most people stop at “PostgreSQL accepts connections” and move on.

But PostgreSQL’s internals have a very physical feel to them: processes fork like buildings, memory pages act like parcels of cached reality, and a log file writes first so data can be trusted later. PGSimCity turns those mechanisms into a city you can watch. Even if you never run the simulator, that mental model makes the real engine click.

The very first heartbeat: TCP in, postmaster out

A connection starts with TCP (Transmission Control Protocol), the network rulebook that guarantees data arrives in order. PostgreSQL listens for TCP connections on a port. The process doing that listening is the postmaster.

The postmaster is special because it doesn’t handle your SQL directly. It serves as a supervisor: it checks the connection details and then creates a dedicated backend process to handle that one client session. PostgreSQL’s classic “process per connection” model means each client ends up with its own backend process.

This matters for two reasons:

1. If one backend crashes or misbehaves, others keep running.
2. Each backend can focus on its own transaction state and memory context without threads trampling one another.

In PGSimCity terms, the postmaster is the control tower, and the backend is the moving building that carries your request through the city.

Backends: where your query actually becomes work

A backend is one operating system process dedicated to one client connection. When your query runs, the backend reads pages, finds rows, applies filters, and produces results.

But the backend doesn’t read the disk directly every time. Instead, it consults the engine’s buffer pool, often controlled by the shared_buffers setting.

Define the buffer pool: cached pages shared by everyone

A buffer pool is a chunk of shared memory that stores recently used pages. A page is PostgreSQL’s fixed-size unit of I/O (commonly 8 KiB).

When a backend needs data, it asks the buffer manager for the page:

• If the page is already in memory, you get a buffer hit (fast).
• If not, PostgreSQL reads it from storage, bringing it into the buffer pool (slower).

Dirty pages, clean pages, and why disk writes “feel spiky”

PostgreSQL divides buffers into states:

• A clean page matches what’s on disk.
• A dirty page has been modified in memory but not yet written to disk.

Writing dirty pages is where latency spikes can appear. A user might not realize it, but commits, checkpoints, and background writers all compete for disk bandwidth.

The key concept tying this together is the difference between:

• logging changes (WAL)
• flushing modified pages to storage

PostgreSQL often logs first and flushes later.

The clock sweep: how PostgreSQL decides what to evict

The buffer pool has limited space. When it needs a free buffer, it must choose a victim. PostgreSQL uses a clock-sweep style algorithm (a classic “approximate least-recently-used” approach).

In plain terms, a background “hand” cycles through buffers, checking whether they’ve been used recently. Two pieces of metadata make this work:

• usage counters (a hint about recency)
• pinning (pages that cannot be evicted right now)

A pinned page is temporarily locked into the buffer pool because some backend is actively using it. If a page is pinned, the clock sweep skips it.

Why bother explaining eviction? Because it explains the shape of performance graphs: workloads with good locality keep useful pages hot; workloads that touch too many unique pages cause churn and more disk reads.

WAL: the durability contract that writes first

Now comes the most important part for understanding PostgreSQL correctness.

PostgreSQL uses a Write-Ahead Log (WAL). A WAL is an append-only log of changes. The central rule is:

Changes to data files must be described in WAL before those changes are considered durable.

So instead of forcing every modified page to disk at commit time, PostgreSQL can write WAL records quickly and then (depending on durability settings) flush them to permanent storage.

How does commit know it’s safe?

This question shows up everywhere in real incidents: How does PostgreSQL keep committed transactions durable without writing every page to disk every time?

The answer is: because WAL flushing is treated as the durability point. If the server crashes, PostgreSQL can replay WAL to bring data files forward to a consistent state.

The WAL behavior is influenced by settings like synchronous_commit, which controls how strongly PostgreSQL waits for WAL to be flushed to disk before telling the client “success.”

• When synchronous_commit is strict, PostgreSQL waits for WAL flush durability.
• When it’s relaxed, PostgreSQL may return success sooner, at the cost of potentially losing the newest transactions in a power-loss scenario.

Checkpointer and bgwriter: keeping the city writable

You now know why WAL helps durability. But someone still has to write dirty pages eventually.

PostgreSQL uses two key actors here:

• checkpointer: periodically flushes dirty buffers and coordinates checkpoint activity
• background writer (bgwriter): opportunistically writes dirty pages to keep the system from accumulating too many dirty buffers

A checkpoint is a point in time where PostgreSQL records progress so recovery can restart from a smaller WAL window. Checkpoints also force dirty buffers to reach disk.

In PGSimCity, this is the moment you feel in your frame rate: the checkpointer “cuts across” the city, writing out dirty pages and issuing the kind of disk synchronization that can create visible latency spikes.

A practical mental model:

• WAL logging keeps transactions safe.
• Checkpoints keep recovery efficient.
• bgwriter reduces surprise by smoothing out dirty page flushing.

If you ever watch pg_stat_bgwriter and pg_stat_checkpointer, the numbers tell a story about whether PostgreSQL is staying ahead of dirty page pressure or letting it build.

Example (run on a database you own):

SELECT *
FROM pg_stat_bgwriter;

SELECT *
FROM pg_stat_checkpointer;

Those views expose statistics about how many buffers were written and how often checkpoint work happened.

Storage and the data directory: the ground truth

In PGSimCity, storage is the “platters.” In real PostgreSQL terms, it’s the disk-backed files that live under the data directory.

A heap file stores table rows. Index files store search structures. If WAL and buffers are the planning and caching layer, the data directory is where reality is stored.

So when a page isn’t in the buffer pool, the backend reads it from storage into a buffer. When pages are dirty, some process must write them back.

Recovery and standby: rebuilding the world from WAL

PostgreSQL doesn’t rely on the disk files alone. It can rebuild forward using WAL.

Recovery: the “recovery ground”

During startup after a crash, PostgreSQL runs recovery. Recovery uses WAL replay to redo committed changes and reach a consistent point.

On a replica, recovery is ongoing. A standby server constantly replays WAL records it receives from the primary.

Standby and replication lag

In streaming replication, the standby receives WAL over the network and applies it in order. When the standby can’t keep up, replication lag grows.

In a performance sense, replication lag isn’t just “a number.” It determines how far behind the standby’s data is relative to the primary, affecting how safe it is to promote under failure.

Maintenance: vacuum as the city’s hidden cleaning crew

Dirty pages and checkpoints explain one kind of mess. PostgreSQL also creates another.

When rows are updated or deleted, PostgreSQL does not overwrite them in place in the way some databases do. Instead, it leaves behind dead tuples (rows that are no longer visible). Over time that can bloat tables and indexes.

The vacuum process (including autovacuum workers) cleans up dead tuples, recovers space, and helps prevent unbounded bloat.

That’s why “database maintenance” isn’t optional folklore. It’s part of how the engine stays fast.

The simulator’s heap page: why 8 KiB matters

PGSimCity zooms into a heap page (a data page) and shows how PostgreSQL lays out space:

• A small page header tracks metadata.
• Then comes an area of line pointers.
• Finally, tuple data is stored, growing from the end of the page backwards.

Even without memorizing every struct name, the important lesson is architectural:

• PostgreSQL works at page granularity for caching and I/O.
• Tuple placement inside a page affects space usage and the cost of scans.
• Dirty vs clean page state controls when disk writes happen.

That internal layout is why you can see the buffer pool as a sampler of page-cache reality, and why shared_buffers changes how many pages stay “lit” instead of requiring trips down into the storage pit.

Putting the city together: a timeline of what happens

When you send a SQL statement, the journey often looks like this:

1. Client connects via TCP.
2. Postmaster accepts, then forks/spawns a backend for that connection.
3. The backend reads needed pages from the buffer pool (shared_buffers).
4. If pages aren’t cached, it loads them from the data directory on storage.
5. For changes, it records them in WAL first (durability contract).
6. Dirty pages accumulate in the buffer pool.
7. bgwriter and checkpointer flush dirty pages to disk to support checkpointing and recovery.
8. If configured, a standby replays WAL and stays close via streaming replication.

Once you see that flow, PostgreSQL stops feeling like a black box and starts feeling like a system of cooperating processes with clear jobs.

Closing thought: why this mental model pays off

The biggest surprise for many newcomers is that PostgreSQL isn’t “one thing.” It’s a choreography between network processes, per-connection backends, shared cached pages, an append-only WAL, and maintenance workers that keep space from rotting.

PGSimCity’s city metaphor is fictional, but the mechanics are real. And when the real system starts behaving differently than expected—spiky latency, replication lag, or bloat—you can usually trace it back to one part of the city: what was pinned, what was dirty, what WAL guarantees were (or weren’t) enforced, and which maintenance crew was scheduled to clean up the consequences.