Local AI study guides for thousands of books, capped on purpose

The local study guide for Darwin’s On the Origin of Species in my library has 21 sections and stops in chapter II. The book’s text is about 950,000 characters; the guide reads only the opening. That is not a bug. It is a budget I chose, and this post is about that choice: generating study guides for thousands of books with a small local model, what it cost, what broke, and why I decided that “the first part of every book” beats “all of a few books”.

Contents · 11 sections

TL;DR #

• A hosted model wrote excellent study guides but cost about 80,000 tokens per 30-section part, with 11,600 parts waiting. I stopped after 53 documents and moved to a local 4B model on an 8 GB GPU.
• Small local models need mechanical guardrails, not better prompts: they ignore “don’t say X” rules and drift into the wrong language. Regex clean-up, a language vote and a temperature-0 retry fixed both.
• Two caps limit every guide: the search index keeps 100 chunks per book, and the guide builder stops at 60 sections of 7,000 characters (about 420,000 characters). Long books get a guide of their opening.
• On my Medicine & Health shelf, 86% of guided books were longer than the cap, and the median guide covered 44% of its book. I accepted that to give every book some guide first.
• Throughput depended on GPU contention, not worker count: 8 workers did 1,644 sections per hour; 16 workers did less.
• Guides became a third search channel, which improved retrieval where PDF text is noisy.

Why study guides at all #

My library holds tens of thousands of books in English and Romanian, many of them scanned PDFs with messy text. Keyword search over noisy OCR is poor, and semantic search over raw chunks retrieves fragments of headers and footnotes as happily as real content.

A study guide splits a book into sections and writes, for each one, a heading, a short summary and key points, in the book’s own language. It helps a reader decide whether a book is worth opening, and it gives search a clean paraphrase to match against.

Cost first: hosted versus local #

The first pass used a hosted model through batches of agent runs. Quality was very good. The cost was not: about 80k tokens per 30-section part, and there were 11.6k parts in the queue. After 53 documents I stopped and looked for a local option.

The GPU at the time was an 8 GB card shared with embedding models and other services. I tried three small models on one English and one Romanian part:

Model	Speed per section	Why
gemma3 4B (3.3 GB)	13 to 17 s	fits fully in VRAM beside the embedders
gemma4 e2b / e4b	40 to 53 s	per-layer embedding tables stay in host RAM; the runtime showed 74% CPU even with all layers offloaded

The 4B model won by being the only one that fitted. The first queue was 6,480 documents, 11,761 parts, at about 15 seconds per section: roughly two months on a single stream.

Small models need guardrails, not adjectives #

Two failures showed up immediately:

1. Negative rules are ignored. I told the model not to start summaries with “This section explains…”. It did so in 29 or 30 out of 30 summaries.
2. Language bleeds. If the prompt contained a single Romanian word, English books started getting Romanian summaries. And 3,130 section files had no language in their metadata, so without a fix Romanian books got English guides.

More prompting did not help. Mechanical checks did:

OPENERS = re.compile(r"^(This section (explains|describes|covers)|"
                     r"Această secțiune (explică|descrie))[^.]*?[,:]\s*", re.I)

def clean(summary: str) -> str:
    return OPENERS.sub("", summary).strip().capitalize()

def book_language(meta_lang, text):
    if meta_lang:
        return meta_lang
    return stopword_vote(text, ["en", "ro"])     # count common words per language

out = generate(section, lang=lang)
if detect_language(out) != lang:
    out = generate(section, lang=lang, temperature=0)   # one strict retry
if detect_language(out) != lang or not valid_json(out):
    reject(section)                                       # never store a wrong guide

The worker also yields the GPU when free VRAM drops below about 1.2 GB, keeps the model loaded for only a minute after its last request, and stops when a pause file exists. The library’s interactive search must never wait behind a batch job.

The caps, and why guides stop early #

Every guide is limited twice:

• The chunk cap. The guide reads the book’s search chunks, and the chunk index keeps only the first 100 chunks per book, about 150,000 characters (150 in the multilingual index).
• The section cap. The section builder stops at 60 sections of 7,000 characters, about 420,000 characters.

So the Darwin guide ends in chapter II, and the same is true for most long books. The same chunk cap means semantic search also sees only the opening of each book; keyword search still uses the full text.

When I checked the Medicine & Health shelf in late September, the picture was clear:

• 15,878 works, of which only 2,236 had a guide (14%);
• 86% of guided books were longer than the cap;
• the median guide covered 44% of its book.

I had two options: full guides for a few thousand books, or capped guides for everything. I chose capped guides for every unguided work first, and full books later, when a bigger GPU arrived. The deciding detail: sections are numbered globally, so continuing a guide from section 61 to the end redoes nothing. The cap is a pause, not a dead end.

For that shelf alone, the builder produced 572,432 sections (5.9 GB) for 11,040 new documents, skipping 1,316 non-prose files and 288 with garbled text.

One rule follows from this: the public site never claims that guides cover whole books.

Throughput: contention, not workers #

When the guide worker and interactive search shared the GPU, semantic search started timing out. The vector search itself took 0.6 seconds; the query embedding waited over 44 seconds for the GPU while the guide model served two parallel generations. Raising the model server’s parallel request limit from 2 to 4 brought semantic search back to 5.6 to 7.3 seconds under load, and guides ran at 6.7 seconds per section.

Later, with more GPU memory free, I measured worker counts over three hours:

Setup	Sections per hour
8 workers, with yield logic	1,644
16 workers, quiet benchmark	1,468
16 workers, contended	741

Eight workers saturated the card. More workers only added queueing and risked starving interactive requests. The old bottleneck had never been the worker count.

Guides as a search channel #

Guides are clean text in the document’s language, so they retrieve well where the PDF text is noisy. Each section is indexed with a bge-m3 vector and keyword fields (heading, summary, key points), and the search service fuses guide hits into hybrid and semantic results with reciprocal rank fusion. A hit then carries the matching section heading as its snippet.

Two traps worth knowing #

Re-chunking breaks chunk-keyed data. Guides are keyed by document and chunk index. Re-chunk a book and the guide silently points at the wrong text. Never re-chunk without migrating whatever is keyed to the chunks.

A forced rerun replaces, it does not merge. I re-ran facts for the Darwin showcase with “force”, and the stored facts from the earlier hosted-model pass vanished, replaced by thinner local output. They had to be restored from an export. Back up before any forced rerun.

Checklist: local LLM batch jobs on a shared GPU #

• Price the hosted run before starting: tokens per unit times units.
• Pick the model that fits fully in VRAM next to your interactive services, not the best one on paper.
• Enforce style and language in code: strip, detect, retry at temperature 0, reject.
• Make the batch worker yield to interactive load and support a pause switch.
• Measure throughput at several worker counts under real contention.
• Know your caps and state them honestly wherever the output is shown.
• Number work units so a capped job can resume later without redoing anything.
• Back up stored outputs before a forced regeneration.

Frequently asked questions #

Can a 4B local model write useful study guides? #

Yes, for section summaries and key points, if you add mechanical checks. Expect it to ignore negative instructions and to drift between languages; fix both with regex clean-up, a language check and a strict retry.

Why do the study guides cover only the beginning of long books? #

Because of two caps: the search index keeps 100 chunks per book and the guide builder stops at 60 sections of 7,000 characters. It was a deliberate trade-off to give every book a guide before giving any book a complete one.

How many parallel workers should a local LLM batch job use? #

As many as saturate the GPU and no more. Here 8 workers did 1,644 sections per hour, more than 16 workers did. Measure under real load, because contention with other models changes the answer.

Is it worth paying for a hosted model instead? #

For a few hundred documents, probably. At 80,000 tokens per part and over 11,000 parts, a local model running for weeks was the better deal for a personal library.

The lesson #

Coverage and depth compete for the same GPU hours. I chose breadth: a useful opening for every book now, the rest later, with the work numbered so nothing is wasted. What made it work was not the model but the plumbing around it: guardrails in code, a worker that steps aside for people, honest caps, and a queue that can resume.