Linux VRAM Overcommit, Explained: Why Low‑VRAM Games Fail (and How Linux Fixes It)

You know the feeling: the game launches, the first couple minutes are smooth, and then memory pressure creeps up like fog rolling in. Scenes stream slower. Stutter shows up in the “wrong” places. Then—sometimes—you get a hard failure logged by the kernel.

On AMD systems using the amdgpu driver (and related stacks like Mesa’s RADV), one common line looks like:

radv/amdgpu: Not enough memory for command submission.

That message is the clue to a deeper truth: running out of GPU memory isn’t only a performance problem. Under the hood, it can become a correctness problem at the exact moment the driver tries to get commands onto the GPU.

This is where Linux is improving the story. The work centers on VRAM management when the system is forced to “overcommit” device memory—allowing allocations to succeed even though, later, the GPU can’t actually keep everything in fast VRAM.

Let’s walk through what’s happening, why it turns ugly, and what the newer Linux kernel mechanisms aim to do better.

VRAM overcommit: “It fits… until it doesn’t”

GPU drivers often support overcommitting VRAM. In plain language, overcommit means the driver will allow applications to request more “device memory” than the GPU has physically available, trusting that the system can rearrange things when pressure rises.

The usual escape hatch is eviction: some allocations get moved out of VRAM and into slower, CPU-addressable memory. On many AMD GPUs, the relevant “slow but GPU-accessible” region is called GTT. GTT (Graphics Translation Table) is system RAM mapped so the GPU can access it—but doing so requires traffic over PCIe.

PCIe is the connection between CPU and GPU. It has both bandwidth limits (how much data can move per second) and latency limits (how long it takes to start moving data). Even if eviction only causes some allocations to land in GTT, the GPU’s workload can become dominated by PCIe traffic.

So far, this sounds like a performance story, not a stability story.

And for part of the timeline, that’s true.

Why “out of VRAM” can be survivable for a while

When some memory is evicted from VRAM to GTT, raw throughput drops. But not every access pays the full PCIe cost.

GPUs use caches: small, fast memory blocks that keep recently used data. When an access hits the cache, the latency is low no matter whether the backing storage was VRAM or GTT.

That means performance can degrade more gracefully than you’d expect, especially when:

• the evicted allocations are rarely touched,
• the GPU’s access pattern is cache-friendly,
• or the game allocates only a small subset of an allocation that later gets evicted.

A simple way to picture it: caches are like a pantry. You can store ingredients in a far-away warehouse (PCIe/GTT), but if the game keeps “opening the pantry drawer” (cache hits), you don’t feel the distance as often.

So why do so many people experience hard crashes and GPU hangs instead of just slowdowns?

The stability cliff: command submission needs accessible memory

The key difference is when the driver discovers the problem.

Allocations might succeed during setup, but command submission happens later—when the driver is about to hand command buffers to the GPU to execute.

At submission time, the amdgpu driver must ensure that memory referenced by the incoming commands is actually accessible in a form the GPU can use. In modern graphics APIs (think bindless resource models), the driver can’t assume only a small subset of allocations will matter; it often must assume that “any allocated resource could be referenced.”

Now combine that with VRAM pressure.

When VRAM is tight, eviction policies decide which allocations move out of VRAM, and where they are allowed to live. Under heavy contention, the driver may try to migrate/lock/prepare memory for the submission, and those attempts can fail.

When the kernel returns an error like -ENOMEM (“not enough memory”) at submission time, the application doesn’t just slow down—it can lose the Vulkan device, crash, or hang.

This is the “horror of late failure”: you don’t discover the mismatch during allocation; you discover it during execution.

The missing piece: eviction fairness between apps

On a typical desktop, a game is not alone. Web browsers, compositors, background streaming, capture tools, and desktop effects all allocate GPU memory.

So once VRAM pressure rises, the kernel is stuck in a fairness problem:

• If it evicts aggressively to keep allocations flowing, the game’s working set may end up in GTT.
• If it protects the game’s allocations, background apps may be forced into worse behavior—or have their allocations fail.

Historically, the kernel driver couldn’t reliably distinguish “foreground game memory” from “background tab memory.” From the driver’s point of view, both are GPU allocations competing under the same device-memory constraints.

That’s where the newer Linux work uses cgroups.

DMEM and dmemcg: making device memory priority-aware

A cgroup (control group) is a Linux mechanism for organizing processes into groups and then applying resource limits or policies per group.

DMEM refers to extending cgroup-based control into GPU device memory, not just CPU RAM. The corresponding controller is often referred to as dmemcg (device memory cgroup).

With dmemcg, the kernel can treat allocations belonging to different cgroup groups with different “protection” or “priority” rules.

This is where userspace helpers matter:

• dmemcg-booster helps ensure the right DMEM controller and protections are enabled for the running cgroups.
• plasma-foreground-booster (for KDE Plasma) tracks which application is in the foreground and adjusts which cgroup gets higher priority for VRAM.

The goal is straightforward: when VRAM starts running low, the kernel should evict background work first, and keep more of the game’s memory in VRAM.

And in practice, that’s what changes the “slow degradation over a session” into something closer to “stable gameplay until the game truly exceeds its own budget.”

Why Linux needs more than “foreground wins”

You might think the story ends there: give the foreground cgroup higher priority, done.

Reality is messier because contention is dynamic.

Consider a worst-case sequence:

1. Background apps are unprotected.
2. They fill VRAM quickly.
3. Then the protected game tries to allocate.
4. If the kernel’s eviction/charging logic doesn’t account for that timing, allocations intended to stay “protected” can still end up in GTT.

So even with foreground priority, you can get the ugly outcome: protected apps still see their critical memory placed into slower regions, and at command submission time, the driver may struggle to keep everything in an accessible/allowed state.

Newer kernel patch series (built around dmemcg + DRM/TTM memory handling) focus specifically on these contended cases: tightening how allocations and eviction behave when protected and unprotected workloads race for the same device-memory pool.

The effect you’re aiming for is subtle but important:

• fewer “surprise” spills into GTT,
• fewer late-stage failures during submission,
• and a smoother performance curve under VRAM pressure.

What you should take away

Running out of VRAM hurts because you eventually end up using slower memory paths across PCIe.

That part is fundamentally about bandwidth and latency.

But crashes and hard failures happen because command submission requires the driver to verify that referenced memory is accessible according to the kernel’s current placement rules—and under contention, those checks can fail.

The Linux improvements around dmemcg and DMEM priority attempt to prevent the worst placement outcomes in the first place, especially for low-VRAM systems and common gaming workflows where background apps are always “helpfully” allocating.

So the big mental model becomes this:

• Eviction timing and fairness determine whether VRAM pressure becomes “stutter” or becomes “broken submission.”
• Priority-aware device memory control (DMEM via dmemcg) helps the kernel decide whose data matters when the GPU is running out of fast space.
• Contended-case fixes reduce the chances that protected allocations still get shoved into the slowest fallback right as execution begins.

Closing: the goal isn’t immortality, it’s predictability

Even with the best scheduling, nothing can violate physics: PCIe will always be slower than dedicated VRAM, and if a game truly exceeds the device-memory budget it requires, something has to give.

What the newer Linux work changes is how it gives.

Instead of the system thrashing toward failure modes (including late command-submission errors), the intent is to keep games closer to a stable allocation regime for the duration of normal gameplay—at least until the game’s own VRAM usage genuinely grows beyond what your hardware can support.

That’s not a guarantee of miracles. But it is a meaningful shift from “VRAM pressure is catastrophic” to “VRAM pressure becomes a controlled performance event.”