Chasing the SQLite WAL-Reset Bug: Forensics to Fix

When a “boring” database won’t stay boring

At first it looked like the kind of reliability wobble you learn to ignore: a few timeouts, a couple of retries, then an incident that warmed the office coffee machine with panic. Under the hood, though, the pattern had teeth.

SQLite corruption. Not “we lost some rows” corruption, but the scarier kind where SQLite’s own self-checks start reporting structural problems. Because SQLite is built to be durable and self-contained, a corrupted database file feels like a mystery novel where the culprit also wrote the alibi.

The turning point wasn’t a single clever log line. It was months of forensic reconstruction: repeatedly capturing evidence, trying to reproduce the failure in a lab (and failing), and then using that evidence to connect the corruption to a long-standing edge case inside SQLite itself—the WAL-Reset bug, a race between checkpointing and a write transaction.

If you’ve ever wondered how a database can be “single-writer safe” and still stumble into a race, this story is for you. Why does something like that even happen?

SQLite’s WAL and checkpoints, in human terms

SQLite runs in different journal modes. The one relevant to this bug is WAL mode.

• WAL (Write-Ahead Log): an append-only log file (the *.wal file) where SQLite records committed changes before they are copied into the main database file.
• Main database file: the *.db file that holds the “steady” view of data.
• Checkpointing: the process of copying pages from the WAL back into the main database file, so reads can stay fast and the WAL doesn’t grow forever.

Here’s the key idea: during normal operation in WAL mode, reads and writes can overlap. Readers can keep using a consistent view while a writer continues to append to the WAL. Checkpointing periodically “folds” the WAL back into the main file.

SQLite also needs a way to keep the WAL from growing without bound. Once the system has safely copied everything needed from the WAL into the main database and there are no readers that still depend on the old WAL content, SQLite can reset the WAL—effectively rewinding it so new transactions start overwriting from the beginning.

This is where things get subtle.

Checkpoints and resets are designed for concurrency. But concurrency plus an unlucky timing window is exactly how rare races are born.

The WAL-Reset bug: a data race in the checkpoint loop

The WAL-Reset bug is described as a rare data race in SQLite’s source code between a checkpoint operation and a write transaction.

A data race (in plain language) means two things are happening “at the same time” in a way that breaks an assumption about ordering. If the ordering flips, state can become internally inconsistent.

For WAL-Reset, the failure mechanism is brutal in its simplicity:

1. SQLite begins checkpointing and believes it has copied certain WAL pages into the main database file.
2. A write happens at a very specific point during the checkpoint.
3. The checkpointing process gets confused and ends up thinking some pages exist where they don’t.
4. Those pages never make it into the main database file.
5. Later, other structures (often indexes) are written that reference those missing pages.
6. The database becomes corrupt in a way SQLite can detect, even though the failure window was extremely narrow.

That’s why the symptom looks like “random corruption” when you only observe outcomes. The underlying trigger is timing-sensitive and rare.

And timing-sensitive bugs are hard to reproduce. In a controlled test, the machine is fast, slower, loaded differently, or simply unlucky in the way it isn’t unlucky enough.

Reliability at scale: why one bad shard hurts

In many systems, corruption is a self-contained problem. In Tailscale’s architecture (as an example of a real deployment pattern), each coordination server shard has a single-writer SQLite database: one long-running Go process owns writes and serves the coordination role.

That “single writer” strategy is exactly the model SQLite wants. It reduces locking complexity and makes application behavior easier to reason about.

But once corruption lands in a shard’s database file, the service impact can be surprisingly large:

• The shard’s control-plane process must be stopped while the database is repaired or restored.
• During that recovery window, tailnets on that shard lose access to control-plane-derived updates (like device discovery information).
• Existing peer-to-peer connections might keep working, but anything that depends on the control plane stalls.

So while other shards remain healthy, customers experience disruption that looks like outages.

There’s also a human factor: incident dashboards and global status pages tend to advertise “something is wrong” even when only a small fraction of shards are affected. Over repeated incidents, trust erodes even faster than uptime drops.

Forensics when you can’t reproduce the bug

The difficult part of the investigation wasn’t identifying that corruption can happen—it’s that it kept happening without a consistent trigger.

There weren’t clean “same shard, same customer, same time” correlations. Changes in the surrounding system didn’t line up with the corruption events. The low-level SQLite interaction code hadn’t changed recently, and it had lived in production for years.

That meant the team couldn’t confidently answer the most important reliability question:

If we can’t reproduce it, how do we prove what actually caused it?

The answer was to treat production like a witness stand and add passive telemetry—observability that doesn’t itself “fix” the bug but can capture enough evidence to reconstruct the sequence of events.

In SQLite terms, the team leaned into a powerful testing and debugging concept: the VFS (Virtual File System) layer.

A VFS is the abstraction SQLite uses to interact with the operating system’s file I/O and locking. Instead of guessing what SQLite thought happened, a VFS shim can trace file operations with more context than typical logs.

A key breakthrough came from instrumenting the VFS with a shim that recorded extra tracing information during WAL checkpoint activity. The goal was narrow and unglamorous: capture the exact overlap between checkpointing and WAL reset behavior.

Then, after another corruption incident, the traces finally pointed at a specific collision window: a checkpoint was progressing while a write transaction overlapped with WAL-reset-related behavior.

In other words, the “random corruption” stopped feeling random. It mapped cleanly onto the known internal class of failure: the WAL-Reset bug.

Fixes come in two flavors: upstream and downstream

Once the race was identified, the upstream fix in SQLite focused on adding an additional check inside checkpoint logic—detecting when the WAL has been reset by another thread so checkpointing doesn’t proceed with stale assumptions.

But rolling out a database fix in production never ends at “upgrade and celebrate.” There’s always a second wave of risk: compatibility problems, new false positives, or other correctness issues that only appear under the exact data patterns of a real system.

That’s what happened next.

A careful rollout of a SQLite version intended to fix the WAL-Reset bug initially produced alarming “corruption” reports during backup verification. The frightening part: it looked like the corruption problem returned.

The actual explanation was more nuanced:

• A different SQLite behavior change around expression indexes caused false corruption warnings.
• The root cause involved generated columns and high-precision timestamp handling stored in text form, then converted to floating-point through SQLite’s expression machinery.
• Floating-point rounding differences across versions made integrity checks complain about indexes whose computed values no longer matched the expected evaluation.

So the team had to manage a dual reality:

• The WAL-Reset bug fix was necessary.
• The integrity-check workflow needed to distinguish “true corruption” from “stale expression index math.”

Downstream, they reduced the timestamp precision to integer seconds so conversions were unambiguous. Upstream, SQLite later added automation to self-heal the problematic index category.

That sequence is worth absorbing: reliability engineering is often a choreography of upstream fixes, downstream mitigations, and operational validation.

Lessons you can apply to your own WAL-based systems

The WAL-Reset bug story is ultimately a reminder that “rare” doesn’t mean “never,” especially under aggressive operational patterns.

Here are the practical takeaways that survive contact with any specific vendor or framework:

1) Treat WAL state as part of the database’s true state

In WAL mode, a database isn’t fully represented by the *.db file alone. Any backup or snapshot process needs to ensure WAL and the main database are captured consistently (including any companion state needed for recovery).

If your backup process assumes “copy the database file and we’re good,” you’re setting yourself up for the kind of mystery corruption that shows up during restore.

2) Make checkpointing strategy a first-class operational decision

SQLite can auto-checkpoint, but some systems take manual control to coordinate performance or backups. Manual control can be totally reasonable—until it interacts with edge cases.

Aggressive checkpointing increases the number of opportunities for timing windows to line up with bugs.

3) Validate with integrity checks, but interpret the results correctly

Running PRAGMA integrity_check (or equivalent verification) is a strong signal. Still, integrity signals can be affected by version-dependent expression behavior and index evaluation.

A reliability process should be prepared for “false alarms” as well as true corruption.

4) Upgrade to the SQLite versions that include the fix

The WAL-Reset bug fix landed in SQLite releases after long periods of existence. If a system depends on WAL-mode SQLite correctness at scale, “staying current on SQLite patch releases” becomes a reliability requirement, not housekeeping.

The strange comfort of finally knowing

Months of outages made it easy to feel like the system was haunted by randomness. Then the evidence converged: checkpointing, WAL reset, and a specific overlap sequence.

Once that model matched production behavior, the path to remediation became clear. The long-standing bug got patched upstream. Downstream mitigations reduced false alarms and improved verification confidence. Finally, monitoring confirmed the repaired world was stable under real traffic.

In the end, the most technical win wasn’t a single patch—it was the method: when the failure won’t reproduce, build observational scaffolding until the failure’s timeline can’t hide.

And that’s the real lesson behind the WAL-Reset chase: reliability isn’t only about correctness. It’s about making correctness knowable.