Translating whole books with local LLMs: the failure modes

A scanned 145-page book went into my translation service in English and was coming out in Romanian at about 68 seconds per page, with its pictures, columns, footnotes, captions and table of contents where they were in the original. Getting there took more than thirty versions in under a week, and most of what I fixed was not the translation model. It was everything around it: what text the model sees, which glossary it gets, which answers are allowed to land, and how the page is rebuilt.

This post is the design of a page-by-page, layout-preserving translator on local models, and the failure modes I hit, so you can skip them.

Contents · 12 sections

TL;DR #

• Translate segments with context: the previous and next text, across page breaks, plus the current section heading. Segments cut at page or line breaks otherwise come back as broken sentences.
• Send only the relevant glossary entries with each segment, not the whole glossary; keep a standing glossary per target language that overrides the automatic one.
• Use the best model where an error multiplies. The glossary is one call per book and its mistakes spread to every page, so it gets the 27B model even when pages use the 9B.
• Changing models mid-job created a race: answers already in flight from the old model landed after the reset and were saved as done. A per-segment generation number fixed it.
• Layout reconstruction has its own bugs: style is fixed at extraction, PDF rendering libraries have gaps, and a centred numbered line can be mistaken for a list item.

The pipeline #

The service accepts PDFs (with a text layer or scanned), photos, EPUB, DOCX, HTML and Markdown. For each document:

1. Extract units and segments. A unit is a page (PDF, scan) or a section (EPUB, DOCX). Segments are paragraphs, list items, table cells, captions and picture labels, each with its box and style.
2. For scans, call the OCR service in layout mode (typed blocks with boxes and measured typography) and, for pictures, in spot mode (text inside figures), with Tesseract for precise positions.
3. Build a glossary from a sample of the document, then merge my standing glossary for the target language on top.
4. Translate each segment with its context and its relevant glossary entries, four segments at a time.
5. Optionally review: a second pass proofreads the draft against the source.
6. Rebuild the page: paint over the original text with background pixels, set the translation in the original’s face, weight, slant, size and colour, and let paragraphs grow into empty paper rather than shrink.

Pause, resume, redo a page and per-task model choice are all in the web UI, and the result can be pushed back into my research library as a new edition.

Model choice per task #

My first real page test used the always-resident 9B model for everything. The layout was good. The Romanian had agreement errors and inconsistent terms. A 27B draft plus a 27B review plus a glossary gave the best result; 9B plus 9B review was usable with errors.

So each job picks a model per task (translate, review, glossary). The defaults reflect cost and blast radius:

DEFAULT_MODELS = {
    "translate": "<small-model>",   # many calls, shared GPU
    "review": "<small-model>",
    "glossary": "<big-model>",      # one call per book; its errors spread to every page
}

In a test the 9B’s glossary had invented words and missing diacritics. Paying a few seconds of the big model’s time once per book is cheap insurance.

Failure mode 1: segments without context #

The first versions translated each segment on its own. Three things went wrong:

• a sentence cut by a page break became two half-sentences, each translated as if complete;
• a line-broken PDF paragraph was sometimes split mid-sentence;
• short segments (a caption, a heading, a list item) lost the meaning that only the surrounding text gave them.

Now every request carries the previous and the following text, across page boundaries, and the current section heading, clearly marked as context:

Current section heading (context only): <heading>
Preceding text (context only, do NOT translate it):
<previous segment(s)>
Following text (context only, do NOT translate it):
<next segment(s)>

and the system prompt says the text may be a fragment: translate exactly that fragment so it joins the surrounding text naturally, and never translate, complete or repeat the context. Hyphenated line-wraps in PDFs are joined before translation.

Failure mode 2: the glossary #

Glossaries fixed term consistency and then caused new problems.

• The whole glossary with every segment distracted the small model and cost prompt time. Now only entries whose term (or its stem, for plurals and possessives) appears in the segment or its context are sent.
• The automatic glossary singularised plurals and ignored established target-language forms of names. A model asked to “list terms and their translations” will happily invent a translation for a name that already has a standard spelling.
• The fix is a standing glossary per target language, editable through the API and UI, that overrides the automatic one. For Romanian it started with 50 terms. There is also a “hold” option: the job stops after building the glossary so a human can check it before any page is translated.

The prompt also tells the model to inflect glossary terms as the grammar requires, otherwise a Romanian noun gets pasted in its dictionary form into every case.

Failure mode 3: the model-switch race #

Mid-book I switched a running job from the 9B to the 27B: pause, change models in settings, “redo all”, resume. Afterwards five segments on three pages were still the 9B’s translations.

The sequence:

1. Segments were in flight on the 9B when I paused.
2. “Redo all” reset every segment to pending.
3. The in-flight 9B answers arrived after the reset and were written back as done.
4. On resume, the worker saw them as finished and skipped them.

This is a classic lost-update race, and the fix is the classic one: compare-and-set. Every segment has a generation number. A reset or re-read increments it. A worker remembers the generation it started with, and its write only succeeds if the segment is still that generation:

def update_segment(job_id, unit, seq, _gen=None, cols):
    sql = "UPDATE segments SET ... WHERE job_id=? AND unit=? AND seq=?"
    if _gen is not None:
        sql += " AND COALESCE(gen, 0)=?"   # stale answer: zero rows updated
    ...
    return cursor.rowcount == 1

A late answer from the old model now updates zero rows and is dropped. Any background job that can be reset while work is in flight needs the same guard.

Failure mode 4: Markdown and structure #

Markdown lists and tables were flattened to one line each, because a list was translated as one segment. Now each list item and each table row is its own segment, and citations are kept as atomic tokens. Bibliographic references in square brackets stay exactly as written, and footnote marks, formulas and images are replaced by markers like ⟦1⟧ that the model must copy exactly once.

Failure mode 5: rebuilding the page #

Translation quality is only half of “looks like the original”. The other half produced its own list:

• Style is fixed at extraction. Fixing a layout bug and pressing “re-render” did nothing, because the style had been read when the page was extracted. Layout fixes need a re-read of the page. I ended up making this explicit in the UI.
• A rendering library gap. PyMuPDF’s insert_htmlbox rendered CSS font-variant: small-caps as garbage glyphs. Small caps are now faked with uppercase at 80% size.
• A centred “1.” is not a bullet. A centred numbered heading was treated as a list item with a 149-point hanging indent, squeezed to the right and shrunk. It affected 14 pages of one book.
• Text that shrinks. Scanned paragraphs without explicit line breaks were sized as if they were one line; scale factors fell far too low. After using the printed line count from OCR and letting paragraphs grow down into empty paper, every scan page scaled at 0.75 or more, against a low of 0.245 before.
• Picture labels. Captions split into lines by OCR are joined before translation and re-split after, so centred captions stay centred; label backgrounds are inpainted from surrounding pixels instead of a flat fill; a label Tesseract only partly read is widened over the missing words.
• Things that should not change. Unchanged translations (logos, names) are not repainted, so the original pixels stay. Music notation in one scanned book came through untouched.

Making room on a shared GPU #

The 9B model is shared with my library’s study-guide and enrichment pipelines. A “go faster” option pauses the OCR backlog and those pipelines while a translation runs, and resumes only what it paused. The 27B is a big model that takes the GPU from other big models, so a job using it waits its turn, the same way the OCR service does.

Checklist #

• Give every segment its previous and next text and its section heading, marked as context only.
• Join hyphenated line-wraps and treat page breaks as mid-sentence by default.
• Send only relevant glossary entries; keep a standing glossary that wins over the automatic one; offer a hold for human review.
• Use the strongest model for one-off, high-blast-radius steps (the glossary).
• Guard every write with a generation check** if work can be reset while in flight.
• Segment lists and tables per item and per row; protect references and markers.
• Separate re-read from re-render, and document which fixes need which.

Frequently asked questions #

Can a local 9B model translate books well? #

Usably, with errors in agreement and terminology in my Romanian tests. A 27B model for drafting and review, with a curated glossary, was clearly better. Context around each segment improved both.

How do I keep terms consistent across a long translated document? #

Use a glossary, but send only the entries relevant to each segment, and keep a human-curated standing glossary per language that overrides any automatic one. Automatic glossaries tend to invent translations for names that already have established forms.

Why did some segments keep the old model’s translation after I switched models? #

Because answers already in flight from the old model arrived after the reset and were saved. Store a generation number per segment, increment it on reset, and only accept a write whose generation still matches.

How do I preserve layout when translating a scanned PDF? #

Get typed blocks with boxes and measured typography from OCR, translate per block, paint over the original text with the surrounding background, and set the translation in the original’s style, letting text grow into free space instead of shrinking it.

The lesson #

The translation model was the easy part to change and the least of my problems. The quality came from the plumbing: the right context, the right subset of the glossary, the right model for each step, a write path that rejects stale answers, and a renderer that knows the difference between a list and a centred heading. Treat a translation job like any other distributed job, with races and all.