Running 414 Kiwix ZIMs as an offline reference shelf

My home server holds 414 Kiwix ZIM files: offline Wikipedia in English and Romanian, Project Gutenberg, Stack Exchange, DevDocs and a few hundred more, 973 GB and about 86 million entries. Serving one of them is easy. Making all of them a useful reference shelf next to my own library, refreshing them without outages, and searching them from one box took a week of gotchas. This post collects them.

Contents · 10 sections

TL;DR #

• Decide what each collection is before importing anything. Book-shaped collections (Gutenberg, scanned PDF archives, textbooks) become browsable shelves. Encyclopedias and Q&A stay in Kiwix and are federated as a search leg, never copied in as works or embeddings.
• kiwix-serve provides undated book aliases only with -z / --nodatealiases, despite the name. Point every consumer at undated names, or the next refresh breaks them.
• kiwix-serve opens ZIMs lazily. Never move or delete a ZIM until the pod that maps it is gone. Doing so cost me 29 books returning HTTP 500.
• It only starts listening after every ZIM is opened (30 to 60 seconds for ~400 files over NFS). Add a readiness probe.
• About 240 of ~400 ZIMs ship without a full-text index; multi-book search fails on them, so fall back to the title index.
• Downloads of 100 to 250 GB stall. Resume with HTTP Range and keep the running SHA-256.

What is in the box, and what it is for #

An inventory pass over the OPDS catalogue and a directory-entry scan of each ZIM sorted the 414 files into two kinds.

Book-shaped, where each entry is a book or a document:

• Project Gutenberg: about 70,000 EPUBs, plus some PDFs;
• the Survivor Library: 14,366 PDFs, 249.5 GB, a median publication year of 1908;
• army publications, and a small set of Romanian-language school textbooks.

Not books, where entries are articles or answers:

• Wikipedia: 19 ZIMs, 8.7 million articles;
• Stack Exchange: 107 ZIMs, 65.9 million entries;
• DevDocs and other manuals, Wiktionary, video collections.

My own library at the time held 75,466 works and 928 GB, with zero files from these sources. The recommendation I wrote down, and followed, was: book-shaped subsets may become shelves; encyclopedias and Q&A are reference, searched live through Kiwix and shown beside library results. Embedding 8.7 million Wikipedia articles into my library’s vector index would have drowned my own books in encyclopaedia passages and cost weeks of GPU time for something Kiwix already searches.

Serving: aliases, readiness and lazy opening #

The ZIMs were 2021 to 2025 dumps, and every consumer, my metasearch engine, a dossier enricher, an MCP tool, referred to dated book names. Any refresh would break all of them.

kiwix-serve 3.8 can serve undated aliases (wikipedia_en_all_maxi pointing to the newest dated file), but only when started with -z. The long form is --nodatealiases, which sounds like the opposite. Without the flag, /content/<undated>/ returns 404 and searches by undated name return 400.

containers:
  - name: kiwix
    image: <registry>/kiwix-serve:<pinned-tag>
    args: ["-z", "-k", "-t", "8", "--library", "/data/library.xml"]
    readinessProbe:
      httpGet: { path: /catalog/v2/root.xml, port: http }
      periodSeconds: 10
      failureThreshold: 12
    livenessProbe:
      httpGet: { path: /catalog/v2/root.xml, port: http }
      initialDelaySeconds: 120

The readiness probe matters because the server only listens once every ZIM is open. With about 400 files on NFS that takes 30 to 60 seconds, and without a probe a rolling restart sends traffic to a pod that refuses connections.

The bigger trap is lazy opening. My updater originally moved each old ZIM to a .old/ folder right after swapping in the new one. The running pod still had the old file mapped. 29 books returned HTTP 500 until the next rolling restart. The rule now: leave old files in place, restart, confirm the new book is in the served catalogue, and only then delete. A second variant of the same problem: with two dated files of one book, the undated alias did not reliably point at the newer one, and deleting the old file broke the book. Always restart after deleting superseded files, too.

Refreshing 943 GB without an outage #

The refresh replaced 361 outdated ZIMs, 943 GB. The updater:

• downloads the smallest files first, throttled to 40 MB/s;
• computes SHA-256 while streaming and compares it with the mirror’s .sha256;
• swaps the file into place, and triggers a rolling restart after every 40 small files or any file of 5 GB or more;
• deletes old files only after the new book appears in the served catalogue.

Large files were the problem. Mirrors stalled on 100 to 250 GB downloads; a 120-second read timeout ended the transfer, and the next run re-hashed the whole partial file over NFS. The fix was to reconnect in-process with an HTTP Range request at the current byte, keeping the running hash:

h = hashlib.sha256()
pos = 0
while pos < size:
    try:
        with session.get(url, headers={"Range": f"bytes={pos}-"},
                         stream=True, timeout=(30, 120)) as r:
            for chunk in r.iter_content(1 << 20):
                out.write(chunk); h.update(chunk); pos += len(chunk)
    except (requests.ConnectionError, requests.Timeout):
        stalls += 1
        if stalls > 60:
            raise
if h.hexdigest() != expected_sha256:
    raise ValueError("checksum mismatch")

The first pass had 19 failures; a retry brought that to 6, all of them the giants (English Wikipedia, Stack Overflow, the Survivor Library, Gutenberg). They went into a third, stall-resilient pass using the Range resume above.

Searching ~400 books from one query #

kiwix-serve can search several books in one request, with limits that are easy to hit:

• books of different languages in one request return 400;
• books are searched sequentially, so two cold books took 19 seconds;
• one book without a full-text index fails the whole query.

That last one was silent at first. Roughly 240 of ~400 ZIMs ship without a full-text index, including Gutenberg, most DevDocs sets and Stack Overflow, and the catalogue’s index flag is unreliable. My reference search simply returned nothing for them. It now falls back to the title-suggestion endpoint for those books.

So the search service does the federation itself:

• books are grouped (encyclopedias, dictionaries, literature, programming, Q&A, science, health and so on), and topic words pick at most 24 books per group;
• programming docs and Q&A are added only when the query looks technical;
• each book is searched in parallel with an 8-second timeout, results are fused with reciprocal rank fusion, and every query is cached.

Cold searches on the 110 GB English Wikipedia took 8 to 11 seconds during a NAS scrub, and 60 ms when cached. The reference panel is a separate request, so it never delays the library’s own results.

Reading inside the library #

The public host sends X-Frame-Options: DENY, so I could not frame Kiwix pages. Articles are fetched, sanitised with lxml and nh3, and rendered natively in the library’s reader; media and PDFs are proxied raw. kiwix-serve ignores HTTP Range for ZIM entries, so PDFs download whole before the viewer shows them; fine at tens of megabytes.

The Gutenberg ZIM became a shelf of 69,918 books browsable by title, author, language and bookshelf. War and Peace, 573,000 words, opens in 2.4 seconds.

Linking authors to offline Wikipedia #

Author and entity pages now show the matching offline Wikipedia article. The matching rules are deliberately strict:

• a Wikidata ID on the page that matches the author: exact;
• otherwise the name in the title plus life years or one of the author’s own work titles;
• a lone birth year counts only with an exact title;
• never a disambiguation page.

An audit of 60 links found 60 correct, and curator decisions (manual or rejected) are never overwritten by the automatic matcher. The first backfill checked 3,941 authors and linked 1,475. It also found a bug worth generalising: when a name was shorter than three characters, the matcher exited early without reporting which books it had checked, so the item was never marked done and came back forever, 1,000 times. Every exit of a matcher must report what it considered.

Refreshes move media, too: 2024 Wikipedia dumps stored images under one path layout, 2026 dumps under another. Article paths survived; stored image paths did not, so a media refresh step now runs after every Wikipedia update.

Checklist: an offline reference shelf #

• Classify collections as books or reference before importing anything.
• Run kiwix-serve with -z and point every consumer at undated names.
• Add a readiness probe; startup scales with the number of ZIMs.
• Never move or delete a served ZIM before a restart; restart again after deleting.
• Verify downloads with SHA-256 and resume with Range.
• Expect many ZIMs without full-text indexes; fall back to titles.
• Route queries to a subset of books and search in parallel with timeouts and a cache.
• Make every matcher exit report what it checked.

Frequently asked questions #

How do I keep Kiwix links working after updating ZIM files? #

Start kiwix-serve with -z (--nodatealiases) so undated aliases exist, and use those names everywhere. Then a refreshed ZIM with a new date is picked up without changing any links.

Why does kiwix-serve return HTTP 500 after I replace a ZIM? #

Because it opens ZIM files lazily and the running process may still map the old file. Keep the old file until the server has restarted, then delete it and restart once more.

Why does Kiwix full-text search fail for some books? #

Many ZIMs ship without a full-text index, and a multi-book search that includes one of them fails entirely. Search those books by title suggestions instead, and search books of different languages in separate requests.

Should I import offline Wikipedia into my own vector search? #

Usually not. It is millions of short articles that will dominate your own documents in results, and Kiwix already searches it. Federate it as a separate search leg and show its results beside yours.

The lesson #

Offline knowledge is cheap to store and expensive to keep usable. Most of the work was not downloading 973 GB; it was respecting how the server actually opens, names and indexes its files, and deciding early that an encyclopedia is something you consult, not something you absorb.