From Ollama to a llama.cpp router: 49 models down to a few presets

At the end of September my workstation’s Ollama store held 49 models and about 435 GB of weights. Two days later 47 of them were gone, about 400 GB of disk was free, and everything that used to call Ollama was talking to a single llama.cpp router with a handful of presets that load on demand and go to sleep when nobody needs them.

This post is how that migration went: why I left Ollama, how the llama.cpp router replaces its load/unload behaviour, the parity tests that decided which models could move, and the gotchas that cost me hours.

Contents · 10 sections

TL;DR #

• Ollama was convenient but, on one shared GPU with batch workers, its scheduler could wedge: a load that needed to evict a never-idle model waited forever, and every other load queued behind it.
• On the same prompts, a tuned llama.cpp server did 1,364 study-guide sections per hour where Ollama did 468.
• llama.cpp’s router mode gives you Ollama-like behaviour: one OpenAI-compatible port, many presets in one INI file, models loaded on request and put to sleep after an idle timeout.
• Embedding models only move if the vectors match. bge-m3 matched (cosine 0.99996 or better); nomic-embed-text did not, because the tokenizer differs.
• Retire the old server by measuring its clients, not by guessing: inventory the request log, wait for zero real clients, then delete.

Why I left Ollama #

I like Ollama. For one person and one model at a time, it is hard to beat. My workstation is not that: one GPU serves an interactive chat UI, my local AI agent, a library that writes study guides around the clock, embedders for search, and classifiers for a threat-intelligence pipeline. Several of those callers never stop.

Three things kept biting:

1. The scheduler wedge. If a new model had to evict a runner that was never idle (a batch caller sending request after request), Ollama waited for it to become idle. It never did. Every other new load queued behind that one, even after the original client had disconnected. The only way out was to pause the batch worker by hand.
2. Hidden keep-warm costs. A crontab warmed gemma3 every 20 minutes and four other models every 4 hours. Nobody needed them any more, but one of those warm-ups alone loaded 11.3 GB into VRAM.
3. Throughput. I benchmarked the study-guide prompt on 32 real sections, same prompt, worker paused. Ollama with gemma3 at 2 concurrent requests: 468 sections/h. llama.cpp with the same model family: 1,009/h at 16 concurrent with default batch settings, 1,364/h tuned. vLLM reached 2,004/h at 16 concurrent, but held 14.7 GiB of VRAM; llama.cpp used about 8 GB at default batch and about 12.9 GB tuned. I chose llama.cpp: about two thirds of vLLM’s speed when tuned, a much smaller footprint when not, and far simpler to run beside other models.

How the llama.cpp router replaces Ollama #

Recent llama-server builds have a router mode. You start one server with a preset file and it spawns a child llama-server per model when a request names it. The request’s model field is the preset’s section name.

A sanitised version of my preset file:

version = 1

[*]
ngl = 999        ; all layers on the GPU
fa = on

; pipeline model: always loaded, never sleeps
[qwen3.5-9b]
model = /models/Qwen_Qwen3.5-9B-Q6_K.gguf
spec-type = draft-mtp
jinja = true
c = 65536
parallel = 8
kv-unified = true
load-on-startup = true

; embeddings
[bge-m3]
model = /models/bge-m3-FP16.gguf
embedding = true
load-on-startup = true

; big model, on demand only
[big-model]
model = /models/<big-model>.gguf
c = 131072
parallel = 2
sleep-idle-seconds = 300

And the server:

llama-server --models-preset /config/models.ini --models-max 6 \
  --host 0.0.0.0 --port <port> --no-webui

The pieces map neatly onto what Ollama users know:

• sleep-idle-seconds is the equivalent of keep_alive. After the timeout, the model’s weights are freed; a sleeping instance keeps only about 0.24 to 0.32 GB. The next request wakes it; for bge-m3 that took about 2.5 seconds.
• load-on-startup pins a model, like a model that is always warm, without a cron job.
• POST /models/unload {"model": "<name>"} unloads one model by hand.
• GET /models lists every preset with its status (loaded, loading, sleeping).

To make loads and wake-ups fast, a small boot service copies the GGUF files I use most into a tmpfs ramdisk before the router starts. Copying the first set took 7 seconds; the router was up 42 seconds after start.

The parity tests that decided what could move #

Chat models are easy to move: if the answers are good, they are good. Embedding models are not. Every vector already stored in your search index was produced by the old server. If the new one produces even slightly different vectors for the same text, your existing index quietly gets worse.

So for each embedding model I embedded the same texts through both servers and compared:

• bge-m3: cosine similarity between Ollama’s and llama.cpp’s vectors was 0.99996 to 1.0 on 43 texts. Safe to move.
• nomic-embed-text: not safe. llama.cpp’s WordPiece tokenizer for this architecture splits text differently from Ollama’s. A Romanian test sentence became 9 tokens on one and 12 on the other. Stored vectors reproduced exactly (1.00000) only through Ollama; through llama.cpp, similarity fell to 0.86 to 0.98.

That second result changed the plan. Rather than re-embed a large English index, I checked whether I needed it at all. The multilingual bge-m3 index already covered 400 of 400 sampled documents from the English index, and an A/B on five English queries gave roughly equal quality. The English leg was redundant, so it stopped being a migration problem.

The general lesson: test exact-input parity before moving an embedder, including whatever prefix your indexer adds (mine embeds "{heading}: {chunk}", not the bare chunk).

Choosing the pipeline model #

I wanted speculative decoding with a multi-token-prediction (MTP) head for every generative model. Two findings matter for anyone doing the same:

• Not every GGUF conversion keeps the MTP head. For the Qwen models I tested, one community conversion kept the extra nextn tensors and two others dropped them. Check the tensor list before you download 20 GB.
• The 9B model with MTP, fully on the GPU, gave 67 tokens/s for one stream and 1,468 study-guide sections/h at 16 in flight, with 32 of 32 answers valid JSON. A 35B mixture-of-experts model wrote better summaries but managed only 236 sections/h at 8 in flight. The 9B became the pipeline model; the bigger model became on-demand only.

In production the first study-guide part took 5.7 seconds per section on the 9B, against 8.3 on the old gemma3 setup.

Gotchas that cost me hours #

The router routes on the section name, and only that #

An unknown model name returns HTTP 400. Aliases you set can be overwritten by the router. Clients that hard-code an old model name break loudly, which is good, but plan a rename: I later gave my big model a stable id (<big-model>) so that upgrading it only changes the model = line, not every consumer’s config.

--models-max is a count, not a memory check #

It limits how many instances run at once. It does not know how much VRAM they need. Never add a preset that cannot fit beside all the others, or guard VRAM yourself (that is a separate post).

A single-file bind mount keeps the old file #

I run the router in a container with the INI file bind-mounted. Tools like sed -i, mv or install replace the file with a new inode; the container keeps seeing the old one. Edit in place (cat new > models.ini), then call GET /models?reload=1, which reloads changed presets without restarting the always-loaded model.

Polling a per-model endpoint keeps the model awake #

The router proxies GET /slots?model=X and /props?model=X to the model process, which resets its idle timer and wakes it if it was asleep. A worker of mine polled /slots for the big model before each section, with 8 workers. Result: 1,205 proxied requests in 25 minutes, 0 sleep events, and 4.5 GB of VRAM permanently taken. Poll the router-level GET /models (the status field) instead, and only ask a model for /slots when the router says it is already loaded.

Deleting a GGUF from tmpfs does not free tmpfs #

If the router has the file mapped, rm removes the name but not the data, exactly like a deleted log file held open by a process. My model-switch script hit “No space left on device” this way. Copy the replacement after the router restart, or unload the model first. (And a failing cp inside an && list does not trip set -e.)

Flag syntax under systemd #

--chat-template-kwargs=VALUE was rejected; the value had to be a separate argument. Under systemd the container exited with code 1 and nothing obvious in the journal.

Retiring Ollama without breaking anything #

I did not trust my memory of who used Ollama, so I measured:

1. Inventory from the request log. Ollama’s access log records every client. Over 60 minutes it showed embedders, classifiers, a library search service, the chat UI, my agent, a monitoring agent and a keep-warm cron.
2. Attribute carefully. Cluster pod IPs are reused after short jobs end. One “client” I blamed turned out to be a different job’s pod that had held the same IP earlier. Attribute an IP to a pod only within that pod’s lifetime.
3. Move each client to the router, one at a time, and verify it.
4. Remove the keep-warm cron lines (6 of them, commented with a marker, backup kept).
5. Wait for zero real clients (my rule was 24 hours with no non-probe requests) before disabling the service.
6. Delete models with a manifest. I removed 47 of 49 models and saved a list of names, sizes and parameter counts so any of them can be pulled again. Disk use on that drive went from 61% to 39%.

Checklist #

• Benchmark the old and new server on your real prompts and concurrency, on a quiet GPU and on a busy one.
• For every embedding model, compare vectors for identical input through both servers before moving it.
• Check that your GGUF conversion keeps the features you rely on (MTP heads, vision projectors).
• Give consumers a stable model id, read from configuration, not hard-coded.
• Poll router-level status, never per-model endpoints, from monitors and workers.
• Edit bind-mounted config files in place; reload presets through the API.
• Retire the old server by log inventory, a waiting period with zero clients, and a deletion manifest.

Frequently asked questions #

Does llama.cpp have an equivalent of Ollama’s keep_alive? #

Yes. In router mode, each preset can set sleep-idle-seconds. After that many idle seconds the model’s weights are released and only a small stub stays; the next request wakes it. Presets with load-on-startup stay loaded.

Can I move my embedding model from Ollama to llama.cpp without re-indexing? #

Only if both produce the same vectors for the same input. Test it: embed a few dozen real chunks through both and compare cosine similarity. In my tests bge-m3 matched to 0.99996 or better; nomic-embed-text did not, because the tokenizers differ.

Is llama.cpp faster than Ollama? #

On my study-guide workload, tuned llama.cpp produced about 2.9 times Ollama’s throughput, mostly because it ran many requests in parallel on one model with a unified KV cache. vLLM was faster still but used much more VRAM. Measure on your own workload; single-user chat may show little difference.

How do I know when it is safe to switch Ollama off? #

When its log shows no real clients for a period you trust. Count requests by client, exclude your own health probes, and wait. A day of silence caught clients I had forgotten.

The lesson #

The migration itself was mostly configuration. The work that mattered was measurement: throughput on real prompts, vector parity for every embedder, a log inventory of every client, and a watch on what kept models awake. Each of those turned a guess into a number, and each number either made a move safe or showed me a problem before users did.