Vision-model OCR that keeps the layout: what to trust it with

I asked a vision language model whether a paragraph on a book’s back cover was bold or italic. It said: not bold, not italic. The paragraph was bold italic, in a sans-serif face, with some words highlighted in yellow.

That answer shaped my whole OCR pipeline. A vision model is excellent at what the text says and decent at the structure of a page. It is unreliable at where small things are and at how they are printed. So my OCR service now splits the job: the model reads, classic OCR places, and plain pixel measurement judges the typography. Calibrated on 3,900 lines with known fonts, the pixel measurements got italic words right with 99% precision.

This post explains that split, the prompts, the measurements, and the operational lessons from running it on a shared GPU.

Contents · 9 sections

TL;DR #

• Classic OCR is “sometimes worse than nothing” on mixed scans with tables, columns and footnotes. A vision model reading the whole page handles layout far better.
• Ask the model for typed layout blocks with boxes (title, text, table, footnote, figure…) as JSON, in reading order, and forbid it from inventing text.
• Do not trust its boxes for small labels: on my test pages they were 5 to 14% off, with no consistent shift. Use Tesseract’s line boxes for position and the model for content.
• Do not trust it on typography. Measure bold, italic, size and colour from the pixels.
• Operationally, OCR on a shared GPU needs priorities, a pause that really frees the GPU, and deploys only at page boundaries.

The service #

My OCR service turns PDFs and photos into searchable text for a home research library, and into layout data for a translation service that rebuilds pages in another language. It runs as a small web service in my k3s cluster and calls a Qwen3-VL vision model (4B or 8B) through the llama.cpp router’s OpenAI-compatible API, so the engine itself is model-agnostic.

Per page:

1. Render the page (PDF) or load the image, at a resolution capped so a huge poster page does not become a gigantic bitmap.
2. Deskew small rotations. The model copes with slight rotation, but a straightened page measurably improves table and column fidelity. Never auto-flip 90 or 180 degrees on a guess.
3. Downscale to the model’s maximum input size.
4. Ask the model for either Markdown (plain transcription) or layout blocks (grounded mode).

The 4B model needed about 40 seconds per page on my card. A larger 35B model is not allowed for OCR: it is too big to share the GPU with everything else for this job.

The layout prompt #

Grounded mode asks for JSON blocks. The prompt, nearly verbatim:

Parse this document page into layout blocks in natural reading order. Include EVERY visible
element: titles, headers, paragraphs, lists, tables, figures, captions, footnotes and small print,
page numbers. Output ONLY a JSON array; each item: {"type": one of title|header|text|list|table|
figure|caption|footnote|page_number|footer, "bbox_2d": [x1,y1,x2,y2] in 0-1000 coordinates,
"font": serif|sans|mono, "content": exact transcribed text (tables as HTML <table>; figures:
printed caption or empty)}. Each footnote or endnote is its own footnote block. Mark superscript
note or citation marks as <sup>1</sup>. Do not translate, summarise, correct or invent text;
write [illegible] for unreadable spans.

Things that matter here:

• 0 to 1000 coordinates. Qwen-VL models return boxes in relative 0-1000 units. I verified it on a test page: a title at pixels 22-206 by 18-31 on a 900x520 page came back as [24,34,214,60]. Store boxes in the same units, independent of DPI.
• Fidelity clause. This text is indexed and attested as an archive’s searchable text, so “do not invent, write [illegible]” is part of the contract, not a nicety.
• Stream-friendly parsing. The model streams a JSON array. A small string-aware brace scanner yields each complete {...} object as it arrives, so the viewer can draw blocks while the page is still being read, and a truncated array still gives every complete block.

What the model got right reliably: block types, reading order, the serif/sans/mono call, one block per footnote, superscript citation marks, and tables as HTML.

Spot mode: text inside pictures #

Layout mode deliberately folds a figure into one empty figure block. For translation I also need the words inside figures: labels on diagrams, captions on photos, map names. A second prompt (“find every piece of visible text, including text inside pictures, diagrams, charts, maps, labels, signs and stamps; one item per line or label”) returns the same JSON shape.

Here the boxes became a problem. For small labels, the vision model’s boxes were 5 to 14% off, and not by a consistent shift you could correct. If you paint translated text at those coordinates, it lands next to the label, not on it.

The fix is a division of labour:

• the vision model says what text is there;
• Tesseract, run on the same image, gives precise line boxes for where;
• a matcher pairs each model item with the Tesseract line that reads the same text, using the model’s rough position only to break ties between look-alike lines;
• an item that cannot be located is skipped, not guessed.

Typography from pixels #

The rebuilt page needs to look like the original: footnotes smaller than body text, a bold heading still bold, highlighted words still highlighted. I tried asking the model. Asked per block, it called a bold italic paragraph “not bold, not italic”. Given several tasks at once, it dropped most inline marks.

So a separate module measures each block from the page image, with no model involved:

• ink, paper and accent colours: the paper colour from the median of a thin ring around the box, which hugs the glyphs; the ink colour of the text itself; an accent colour from pixels that sit off the line between ink and paper (yellow highlights, red initials);
• size: the x-height from the dense band of each line; font size is about x-height divided by 0.46 for typical book and sans faces;
• bold: stem width divided by x-height; regular book faces measured about 0.12 to 0.17, bold about 0.22 and up, so the threshold is 0.20;
• italic: the horizontal shear that makes the strokes most upright and sharpest; a line is italic when the best shear is at least 0.12, decided by a median vote across lines; a single word needs 0.18 alone, or 0.12 inside a phrase plus a clear sharpness gain;
• inline runs: per-word measurements are aligned to the model’s transcription tokens by width with dynamic programming, so a bold word in a regular sentence is marked on the right word.

A sketch of the bold test:

BOLD_RATIO = 0.20   # stem width / x-height

def is_bold(stem_px: float, x_height_px: float) -> bool:
    return x_height_px > 0 and stem_px / x_height_px >= BOLD_RATIO

Logos and pictures inside a text block are excluded from that block’s measurements, or a dark logo makes the whole paragraph “bold”.

I calibrated against pages that have a real text layer: render them at the OCR resolution, so the true fonts and sizes are known, and compare. On 3,900 lines:

• size estimate / true size: median 1.00;
• italic words: 99% precision, 96% recall;
• bold words: 81% precision, 86% recall (still the weakest part).

The back cover that started it now comes out bold italic sans, with the yellow-highlighted words highlighted and the logo intact.

Running OCR on a shared GPU #

The GPU also serves chat, an agent, embeddings, a 27B model and image generation. OCR is a background tenant, and the operational rules were all learned the hard way.

Priorities #

A single photo uploaded from my phone waited behind the whole library backlog, 15 documents. Now single uploads (web, API, chat UI, translation requests) get priority 10, and a bulk backlog run yields between pages to any higher-priority run.

A pause that really pauses #

The library’s “pause OCR” button only filtered the bulk queue. A running document kept going page after page: a 190-page request held the GPU while I needed it for a 3D render, and the vision model stayed in VRAM. Now a pause:

• makes the running job yield at once (the page in flight is cut off and redone on resume; finished pages are kept);
• starts nothing new while paused;
• unloads the vision model as soon as the worker is idle;
• on resume, continues with the person’s request first.

Yield to the big model #

When the 27B model is requested, OCR yields its vision model in under a tenth of a second. A busy vision instance can still hold its VRAM until the router force-stops it, which is why a separate guard on the host kills an instance that ignores the unload. The OCR service simply redoes the interrupted page.

Deploy at a page boundary #

I once rolled out a new version while a run was active. The pre-deploy check printed that a run was in progress, and I deployed anyway because the check did not block. Work in flight was lost. The rule now: poll the stats endpoint, wait until the running page number advances, then apply. A check that prints but does not block is not a check.

And a k3s detail: re-pushing the same image tag was not picked up, because the node had the old image cached. Always bump the tag.

A heat guard that read 0 °C #

A related batch OCR worker paused itself when GPU memory got hot. It read all hwmon files in one try block; an I/O error on a non-temperature file aborted the loop, so it saw 0 °C and never paused. Read only temp* sensors, each in its own try. After the fix it reported memory 68 °C, junction 68 °C, edge 60 °C while running 40 workers at about 162 pages per minute on small documents.

Checklist #

• Deskew before OCR; never auto-flip pages.
• Ask the vision model for typed blocks with boxes in reading order, as JSON, with an explicit no-invention clause.
• Parse streamed JSON incrementally so partial output is still useful.
• Use the model’s boxes for blocks, but Tesseract’s boxes for precise positions of small text.
• Measure typography from pixels; calibrate on pages with a known text layer.
• Give interactive requests priority; let bulk runs yield between pages.
• Make pause stop the running job and unload the model.
• Deploy only at a page boundary, with a check that blocks.

Frequently asked questions #

Is a vision language model better than Tesseract for OCR? #

For reading mixed layouts (tables, columns, footnotes, slightly rotated scans), the vision model was far better in my use. For exact positions of small text, Tesseract was better. Using both, each for what it does well, gave the best result.

Can a vision model detect bold and italic text? #

Not reliably in my tests: it called a bold italic paragraph “not bold, not italic”. Measuring stroke width and slant from the pixels was far more accurate, with 99% precision on italic words after calibration.

What coordinates does Qwen-VL use for bounding boxes? #

Relative coordinates from 0 to 1000 on each axis, independent of image size. Convert to pixels using the size of the image you sent, after any deskew or downscale.

How do I run OCR in the background without hurting interactive AI on the same GPU? #

Give single requests priority, let bulk jobs yield between pages, make pause release the model from VRAM, and have OCR give way to bigger interactive models. Redo an interrupted page rather than holding the GPU.

The lesson #

A vision model is a strong reader and a weak measurer. Let it do the reading, the structure and the hard layouts, then check everything that has a ground truth in the pixels with something that measures pixels. And on a shared GPU, a background job is only well-behaved if its pause, its priority and its deploys are all built to let go.