One GPU, many LLMs: the VRAM guards that stopped a driver deadlock

At about 10:41 one October morning, my GPU driver deadlocked. The card had 33.9 of 34.2 GB of VRAM in use by language models, and then a music player on the desktop asked for a little more. The driver tried to make room by moving 24.7 GB of model memory out to system RAM, got stuck halfway, and every AI workload on the machine froze.

I got it back without a reboot. But it made me write down a rule I had been breaking for a week: on a single GPU that also drives your monitor, VRAM is a budget you have to enforce yourself. No inference server I used enforces it for you.

This post is the budget, the two small guards that enforce it, and the numbers that shaped them.

Contents · 11 sections

TL;DR #

• One 32 GB GPU runs my always-on 9B model, two embedders, a reranker, an on-demand 27B model, vision models for OCR, image generation, and the desktop.
• Filling VRAM to the last gigabyte triggered a driver eviction storm and a deadlock. The rule now: keep at least 4 GB free for the desktop.
• The llama.cpp router’s --models-max limits how many models run, not how much memory they use. So a guard enforces two rules every 15 seconds: at most one big model, and a free-VRAM floor.
• A second, faster guard handles the moment a big model is loading: the router waits a fixed 10 seconds before force-stopping a busy model, and the new model needs its memory after about 3.3 seconds.
• Batch workers must yield to interactive work on any model, because all models share one GPU.

The deadlock #

The card is an AMD Radeon with 32 GB of VRAM (the driver reports 34.2 GB). It is the only GPU in the machine, so it also renders the desktop.

That morning I was testing models. Loaded at once: a dense 27B model at 4-bit quantisation using 21.5 GB with its context, the always-on 9B pipeline model, and the bge-m3 embedder. Total: 33.9 GB.

Then the desktop music player needed GPU memory. The kernel’s memory manager (TTM) did what it is designed to do: evict buffers from VRAM to GTT, the GPU-visible part of system RAM. It moved 24.7 GB. During that move the player’s command submission got stuck in the page-pool allocator while holding a driver lock, and the kernel worker that restores compute memory sat in uninterruptible sleep. All compute froze: the 9B, the study-guide pipeline, my local AI agent, the embedder.

The recovery was to stop the llama.cpp service. That freed the VRAM, the stuck restore completed, and I started the service again. The music player crashed; nothing else was lost.

Later that day a smaller version happened with image generation: with the 27B idle in VRAM, an image job forced the driver to evict 19 GB to GTT, the image took 113 seconds instead of about 24, and the firmware logged MES failed to respond to INVALIDATE_TLBS, the same eviction path.

The lesson is that on a shared card, eviction is not a safety net. It is slow at best and dangerous at worst.

Why the inference server does not protect you #

llama.cpp’s router mode gives every model a preset and starts it on request. Its --models-max option sounds like protection, but it is a count of running instances. It does not know that one preset needs 1.2 GB and another 18 GB. Nothing in the router reserves memory for other users of the GPU.

So the budget is mine to write and mine to enforce.

The budget #

These are the measured footprints that made up the final layout, all on the same card:

Model	Role	VRAM	When
9B, 8 slots, 64k shared context, 4-bit KV cache	pipelines, chat, agent	about 10 to 11 GB	always
bge-m3 interactive + batch	embeddings	about 1.2 GB plus up to 1.8 GB	always / batch on demand
27B, 2 slots, 128k shared context, 4-bit KV cache	heavy reasoning	18.6 GB	on demand, sleeps after 300 s
vision 4B/8B	OCR	several GB	on demand, yields to the 27B
image generation	SDXL jobs	peak ~9 GB	on demand

With the 9B and the 27B awake under load, free VRAM sat at about 2.9 to 3.1 GB, tested under the study-guide load with long-context needle tests and no GTT eviction. That is slightly under the 4 GB floor, which is exactly why the guards below exist: the 27B is allowed in, but only alone, and it leaves when idle.

Three measurements changed how I size things:

1. KV cache per token depends on the architecture. The Qwen 3.5/3.6 models I use are hybrid: only every fourth layer has full attention and a KV cache. For the 9B that is about 32 KB per token (131k tokens = 4.3 GB at 16-bit); for the 35B mixture-of-experts model about 20 KB.
2. Compute buffers can dominate weights. A 4B model with 16 slots went from about 8 GB to 12.9 GB just by raising the micro-batch (-ub) from 512 to 2048, for 35% more throughput. Size memory with the exact flags you deploy.
3. More slots on a unified KV pool cost almost nothing. Going from 4 to 8 slots on the 9B with one shared 64k context added under 1 GB and raised batch throughput from 375 to 524 sections per hour.

Guard 1: one big model and a free-VRAM floor #

A shell script on a 15-second systemd timer. It reads the router’s model list and the card’s VRAM use, and enforces:

• Rule 1: at most one big model loaded. “Big” is everything except an allow-list of small, always-resident presets (the 9B and the embedders). If two big models are loaded, unload all but the most recently used.
• Rule 2: a free-VRAM floor. If free VRAM is under 4 GB, unload a big model that is loaded but idle. The batch embedding instance is the first thing given back.
• Rule 0: a big model is loading. Unload every other big model now, before the new one allocates.

“Idle” means no request in the last 180 seconds, judged from the router’s own log lines, and no slot currently processing. The 180 seconds exist because my agent sometimes waits minutes for a human approval in the middle of a task; cutting the model off then would waste the whole context.

The core, sanitised:

ROUTER=http://127.0.0.1:<port>
MIN_FREE_GB=4
SMALL_RE='^(<pipeline-model>|bge-m3|bge-m3-batch|<reranker>)$'

loaded=$(curl -s "$ROUTER/models" | python3 -c '
import sys, json, re
small = re.compile(sys.argv[1])
for m in json.load(sys.stdin)["data"]:
    st = (m.get("status") or {}).get("value")
    if st == "loaded" and not small.match(m["id"]):
        print(m["id"])' "$SMALL_RE")

used=$(rocm-smi --showmeminfo vram | grep -oE 'Used Memory \(B\): [0-9]+' | grep -oE '[0-9]+$')
total=$(rocm-smi --showmeminfo vram | grep -oE 'Total Memory \(B\): [0-9]+' | grep -oE '[0-9]+$')
free_gb=$(( (total - used) / 1000000000 ))
# ... rule 1 and rule 2 call POST $ROUTER/models/unload {"model": "<id>"}

Every action goes to syslog, so “why did my model disappear” always has an answer.

Guard 2: the loading race #

Rule 0 was not enough. After I moved the 27B to a RAM disk, it became fast to load, and the first request to it started failing with unable to allocate ROCm0 buffer (HTTP 500) whenever a vision model was busy reading a page.

The timing explained it:

• The 27B asked for its 13.3 GB buffer about 3.3 seconds after the request.
• The router asked the vision model to stop at once, and the OCR service on the other side yielded in 0.04 seconds.
• But a llama.cpp instance in the middle of a request ignores the graceful stop, and the router only force-kills it after a fixed 10 seconds, which is not configurable.

No amount of politeness can make 10 seconds shorter than 3.3. So the second guard runs as root every 0.5 seconds: if a big model is loading and another is still loaded, ask the router to unload it; if it is still there 1 second later, send SIGKILL to that instance’s llama-server process, found by its port. The router sees the process exit and marks the model unloaded; the killed model reloads on its next request, and the OCR service simply redoes the interrupted page.

After installing it, I requested the 27B while the vision model had a slot processing. The guard unloaded, waited 1 second, killed the vision instance, and the 27B answered the first request in 12.3 seconds.

Sharing one GPU between batch and interactive work #

Memory is one budget; compute time is the other. Two observations shaped the yield logic in my batch workers:

• Yield to every model, not just your own. All router models share one physical GPU. My study-guide workers yielded only to interactive use of their own 9B model, and their streams starved an essay on the 35B model down to about 1 token per second. Backoff now watches every chat model.
• Batch depth sets interactive latency. With 16 study-guide sections in flight, interactive generation on the big model fell to 3 to 6 tokens per second; with 8 in flight, about 15.

The same applies to embedders. A batch indexer sending 64-chunk requests to the same embedding instance as interactive search pushed a single search query to 3 to 7 seconds at the median, because llama.cpp spreads one request’s inputs over all slots. Two presets on the same weights fixed it: an interactive instance that is always loaded and never shared with batch jobs, and a batch instance that sleeps after 30 seconds and steps aside whenever the 27B is loaded.

The desktop is a GPU tenant too #

When the only GPU drives the monitor, desktop symptoms are GPU symptoms. I saw mouse judder at about 19% CPU load: GPU contention, not CPU. Once, the DisplayPort link itself dropped while the card was pinned at 100% by inference (DP_LANE_COUNT_SET failed in the kernel log). Draining the GPU first and then re-initialising the desktop session brought it back. The 4 GB floor is partly for these cases.

Checklist #

• Measure every model’s footprint with the exact flags you deploy: context, slots, KV type, micro-batch.
• Write the budget down, including the desktop and any non-LLM GPU users.
• Keep a free-VRAM floor (mine: 4 GB) and enforce it with a guard, not with hope.
• Allow one big model at a time; unload idle ones; never cut a model off mid-request.
• Handle the loading race: free memory before the new model allocates, not 10 seconds later.
• Make batch workers yield to any interactive model on the same GPU.
• Log every unload to syslog.

Frequently asked questions #

What happens when a GPU runs out of VRAM on Linux? #

With the amdgpu driver, buffers can be evicted from VRAM to system memory (GTT). That keeps things running, but it is slow, and in my case a 24.7 GB eviction under pressure deadlocked the driver. Treat eviction as a failure, not as headroom.

Does llama.cpp limit VRAM per model in router mode? #

No. --models-max limits the number of running instances. Each preset uses whatever its weights, context and buffers need. You have to budget and guard memory yourself.

How much VRAM should I leave free for the desktop? #

On my card, about 4 GB kept the desktop and its apps stable. The right number depends on your monitors and applications; measure what the desktop uses when idle and when a browser or video player is busy, and add margin.

Why not just unload models faster? #

Because a busy llama.cpp instance ignores a graceful stop until the router force-kills it after a fixed 10 seconds. If the new model allocates sooner than that, only a forced stop of the old process frees the memory in time.

The lesson #

On a shared GPU, the inference server optimises for its own models and knows nothing about the rest of the card. The safe setup is boring: a measured budget, a floor you never cross, one big model at a time, and two small guards that act in seconds. The driver gets the margin it needs, and an unload always has a log line that says why.