13 hours of question marks: a canary for LLMs that look healthy

For about 13 hours, the language model behind half of my home lab answered every request with question marks. Not an error page, not a timeout: HTTP 200 and a reply that read ????????. The server said the model was loaded. The health check said the server was up. Nothing alerted.

I found it the next morning only because a background pipeline that writes study guides had dropped from about 30 parts an hour to 5. This post is what went wrong, why ordinary health checks cannot catch it, and the small canary that now catches it within one 10-minute cycle.

Contents · 8 sections

TL;DR #

• A router restart reloaded every always-on model. One of them, a 9B model with a speculative-decoding (MTP) head, came back in a bad state and produced only token 30, which is ?.
• I had changed and verified a different model, an embedding preset. The restart also reloaded the chat model, and I never checked it.
• “Loaded” and “HTTP 200” say nothing about whether the output makes sense. You need a test that asks a question with a known answer.
• The fix for the incident was trivial: unload the model and let the next request reload it.
• The prevention is a canary: every 10 minutes, ask each loaded chat model a factual question and a JSON-schema question, classify the answer as healthy, degraded or garbage, export metrics, alert, and unload a garbage model once.

What happened #

On the evening of 1 October I restarted my llama.cpp router to fix the batch size of the bge-m3 embedding preset. I checked bge-m3 afterwards: it worked. I moved on.

The router, however, does not restart one model. It restarts the whole process and reloads every preset marked to load at startup. One of those was the 9B chat model that serves the pipelines, the chat UI, my local AI agent, a security-analysis agent and several classifiers.

After the reload, that model emitted only ?. Two symptoms, depending on the request type:

• plain chat completions returned HTTP 200 with a body of question marks;
• requests with a JSON schema returned HTTP 500 with Unexpected empty grammar stack after accepting piece: ? (30), because the grammar sampler could not accept a ? where JSON was required.

The speculative-decoding statistics were the clearest sign: draft acceptance was 0 of 891 draft tokens. A healthy run of the same model accepted most of them.

Every client of that model was broken. The study-guide pipeline, which writes structured JSON, mostly failed; its throughput fell from 27 to 35 parts per hour to about 5. That drop was the only thing anyone noticed, and it was noticed by a person checking the pipeline, not by a monitor.

The repair took seconds:

curl -X POST http://<router>/models/unload \
  -H 'Content-Type: application/json' -d '{"model":"<chat-model>"}'
# the next request loads it fresh

After the reload the model answered correctly and the MTP head worked again. The preset was unchanged, so the root cause was a bad load state at restart that I could not reproduce. I can live with not knowing why. I cannot live with not knowing that.

Why normal health checks miss this #

Most checks for an inference server look at the wrong layer:

• Process up / port open: the server was up.
• /health returns 200: the router answered /health happily.
• Model status “loaded”: true, the weights were in VRAM.
• Request succeeded: plain chat returned 200.

All four are about availability. A language model can be perfectly available and completely wrong. The only check that tests output quality is one that knows what the right answer looks like.

The same lesson had already reached me once, in a different shape. Earlier, after forcing flash attention and a quantised KV cache in Ollama, I tested the chat models and they were fine. The embedding model, which I had not tested, failed to allocate its compute buffer for about 20 minutes: 238 failed embedding requests, and library search degraded. Both times the mistake was the same: I verified the thing I changed, not everything the change touched.

The canary #

The canary is a small Python script on a systemd timer, every 10 minutes. For each loaded chat preset it sends two requests:

1. A factual question with one right answer: “What is the capital of France? Answer with one word.” Expect Paris.
2. A request with a tiny JSON schema, to exercise the grammar path that failed with HTTP 500.

It then classifies the result:

• healthy: both answers correct;
• degraded: an answer that is wrong but looks like language (alert, do nothing else);
• garbage: empty output, a flood of ?, one token repeated 12 or more times in a row, output with no letters at all, or the grammar 500 (alert and unload the model once);
• transient: timeouts and 503s while the model is busy or loading (skip, do not alert).

A simplified detector:

import re

def garbage_reason(text: str, http_status: int, body: str = "") -> str | None:
    if http_status == 500 and "empty grammar stack" in body:
        return "grammar-500"
    t = text.strip()
    if not t:
        return "empty"
    if t.count("?") >= 0.8 * len(t):
        return "question-mark-flood"
    if not re.search(r"[A-Za-zÀ-ɏ]", t):
        return "letter-free"
    if re.search(r"(\S+)(?:\s*\1){11,}", t):
        return "repeated-run"
    return None

Rules that make it safe to run unattended #

• Never wake a sleeping model. The routine run probes only presets the router reports as loaded. Probing a sleeping model would load it, take VRAM from other work, and keep it awake.
• Unload once, then stop. On garbage, the canary unloads the model at most once per preset per hour. If the model comes back as garbage again, a person needs to look; a loop of unloads would only hide the problem.
• Never act on timeouts. A busy or loading model is not a broken model.

Metrics and alerts #

The script writes a Prometheus file for node_exporter’s textfile collector, along these lines:

llm_canary_ok{model="<chat-model>"} 1
llm_canary_last_success_timestamp_seconds{model="<chat-model>"} 1.759e9

Two alert rules sit on top:

- alert: LLMModelGarbage
  expr: llm_canary_ok == 0
  for: 12m
  labels: {severity: critical}
- alert: LLMCanaryStale
  expr: time() - llm_canary_last_success_timestamp_seconds > 1800
  labels: {severity: warning}

The second rule matters as much as the first. A canary that silently stops running looks exactly like a healthy one.

The detector has 10 unit tests, one per garbage signature plus the normal cases, and the same definition of “garbage” is shared with the guards in other pipelines so that every part of the lab agrees on what a broken answer looks like.

After every restart: probe everything #

The routine run never wakes models. After a router restart I run the same script once with --all --wake: it wakes every chat preset and probes it. Exit code 0 is healthy, 1 degraded, 2 garbage. That one command is now a step in the restart runbook.

False alarms, and why each one taught something #

A canary that cries wolf gets ignored, so every false alarm got a fix.

The probe budget was too small #

A vision model started failing the JSON probe. It was not broken: the probe allowed 200 output tokens, and that model needed about 215 for the answer, so the JSON was cut off mid-object and reported as invalid. The fix: allow 512 tokens, and report a reply that ended because of the length limit (finish_reason: length) as truncated, not as invalid JSON.

The model was being unloaded at that moment #

The router unloaded a vision model to make room for a bigger one exactly while the canary probed it, and the probe got an HTTP error. A probe that fails only with an HTTP error, for a model that is no longer loaded when you check again, is now skipped instead of alerted.

Checklist #

• Health checks for LLM servers must include a known-answer request, not only /health and model status.
• Test every model class after any server change: chat, JSON/grammar, embeddings, reranker.
• After a restart, probe every model the restart reloaded, not just the one you changed.
• Watch speculative-decoding acceptance: zero acceptance is a strong garbage signal.
• Make the canary safe: never wake sleeping models, act at most once per hour, ignore transient errors.
• Alert on the canary itself going stale.
• Treat each false alarm as a bug in the canary and fix it.

Frequently asked questions #

How can an LLM server return HTTP 200 and still be broken? #

HTTP 200 only means the server produced a response. If the model is in a bad state, that response can be nonsense, such as a single token repeated. Only a check that knows the expected answer can tell.

What is a good canary prompt for a language model? #

A short question with one unambiguous answer that any model in your fleet knows, like the capital of France, plus one request that uses structured output (a JSON schema) to exercise the grammar path. Keep the prompts tiny so the probe costs almost nothing.

Should the canary restart a broken model automatically? #

It can unload it once, so that the next request reloads it, which fixed this incident. Limit it to once per model per hour and alert regardless; repeated automatic restarts hide a real fault.

How often should the canary run? #

Mine runs every 10 minutes, and the alert fires after 12 minutes of failure, so a garbage model is caught within about one cycle. After a planned restart, run it immediately with all models woken.

The lesson #

“Loaded” is not “working”, and “I tested my change” is not “I tested what my change touched”. A restart reloads more than the thing you meant to restart. The cheapest protection is a question whose answer you already know, asked every few minutes, with an alarm when the answer is wrong and another when nobody is asking.