From 96,216 messy files to a 70,000-work library catalogue

My library started as a folder tree of 96,216 files: ebooks, scans, course notes, manuals, gathered from many sources and named by whatever site or tool they came from. In 34 seconds, one script turned them into 62,786 editions of 60,675 works. Making those works trustworthy took weeks: about 12,700 fake authors had to go first, including “Accessibility help” and “Acknotfledgments xv”.

This post covers how I built the catalogue and the crawler behind a personal library that later grew to more than 70,000 works: the data model, the curation rules that worked, the ones that backfired, and the Elasticsearch and FSCrawler settings that cost me the most time.

Contents · 11 sections

TL;DR #

• Model the library as works, editions and files, not “one row per file”. One novel can have five files in three formats and two translations.
• Authors extracted from document metadata and page text are mostly noise. Judge them by form (page numbers, dot leaders, publisher words, mojibake), and let those rules override external authorities.
• An external authority such as Open Library is useful for enrichment and poor as an arbiter: it is thin on Romanian authors and contains garbage records of its own.
• Key your “needs review” markers on evidence quality, not on where a value came from. One badly keyed marker blocked 32,192 works.
• FSCrawler cannot extract “unlimited” text: the JVM caps arrays at 2^31-1. A dead crawler thread inside a running pod looks healthy to Kubernetes.
• For a datastore, benchmark fsync before you move it. SMART health is not performance.

From files to works #

The first version of the library was a flat table: one row per file, with a title and an author string. That breaks as soon as you ask normal library questions: “show me this book”, “which translations do I have?”, “is this the same text as that?”.

So the catalogue uses three levels:

• a work is the abstract book: War and Peace;
• an edition is one publication of it: a translation, a printing, a year;
• a file is one copy on disk: a PDF, an EPUB, a DOCX.

Contributors attach with roles (author, translator, editor). Series attach to works, with the index stored twice: a sortable number and the printed form, because real numbering includes Roman numerals, half volumes, “Partea a VIII-a” and Calibre’s “0” meaning “not set”.

Grouping files into works went in two passes: first by an Open Library work key where one existed, then by normalised title plus author surname. Splitting author strings needed one careful rule: always split on explicit separators, but split on a single comma only when the part before it contains a space. “Creanga, Ion” stays one person; “Ion Creanga, Mihai Eminescu” becomes two.

The first build produced 62,786 editions of 60,675 works from 96,216 files, with 34,845 people and organisations and 69,539 contributions. It also wrote 277,208 ambiguity markers, one per problem, indexed so they could be filtered and worked through.

The authors were mostly not authors #

When I looked at the shelves, the author folders were full of things like “ABREVIERI FOLOSITE”, “Accessibility help”, “Acknotfledgments xv” and OCR wreckage. They came from document metadata and page text taken at face value: tables of contents, front matter, Word interface strings, publishers.

My first instinct was to check each name against Open Library. That failed in both directions:

• Open Library is thin on Romanian writers, so “not found” proves nothing;
• it contains garbage author records itself: “A Blessing”, “action”, “Dune”, years like “1903”.

What worked was judging names by form, with rules that can be answered from the string alone: dot leaders, page numbers, publisher words, digits, articles, gerunds, document nouns, organisation markers, mojibake. Those rules override the authority. Names are cleaned before they are judged, so “Bessel van der Kolk MD” and “Mandy Morris 2024” survive as people, and “SKYE WEINTRAUB” is title-cased rather than deleted.

One detail cost me a rerun: my first pass wrote its verdicts only into the new model. The next rebuild recreated every deleted author from the old flat table. Write corrections to the source the rebuild reads, or they come back.

The result: about 12,700 non-person authors cleared, 9,007 titles cleaned of site names, upload ids and slugs, and 25,921 stale folders removed. One shelf went from 7,519 author folders to 5,532.

The marker that blocked a third of the library #

Ambiguity markers decide whether a work is safe to place on a shelf. One of mine, “title came only from the filename”, had high severity. It blocked 32,192 works, including 9,185 whose filename-derived metadata averaged 0.841 confidence, and it double-counted rows that already had a separate low-confidence marker.

The problem was not the data. It was that the marker was keyed on the source of a value instead of its quality. I re-keyed it on confidence below 0.80 and lowered its severity. Placeable works went from 27,938 to 39,030 overnight.

A similar lesson came from matching titles. Exact matching against Open Library found 115 of 32,737 works, because the dirt is asymmetric: my titles carry the printing (“Klinische Kardiologie 8th ed”), Open Library titles carry the work. The fix was to clean my side of the key only: strip edition, year, volume and scan markers for matching, keep them on the edition row. An author was accepted only when two or more records agreed, with confidence capped at 0.75.

The script that should only run once #

The build script begins with TRUNCATE ... RESTART IDENTITY on every catalogue table. Correct for a first build. Catastrophic afterwards: by then the catalogue held enrichment that existed nowhere else, 10,876 series names, 5,984 descriptions and 11,971 Open Library keys across 87,707 works, and every file id would be renumbered.

The note I wrote that day says it plainly: correct for a first build, catastrophic afterwards. Single-corpus fixes use a separate script that detaches only that corpus’s files and reuses surviving works. If a script is only safe once, make it refuse the second time:

DO $$
BEGIN
  IF (SELECT count(*) FROM works) > 0 THEN
    RAISE EXCEPTION 'catalogue not empty: refusing to truncate';
  END IF;
END $$;

Moves on a NAS: case collisions #

Reshelving 11,754 works in one run produced 158 Synology conflict folders and 175 phantom catalogue paths. Two works with titles that differed only by letter case, from different old shelves, were moved into the same author folder in the same batch. The planner had checked each destination against the disk case-insensitively, but never grouped the plan itself by lowercased destination.

Before any bulk move, bucket planned destinations and file names by lower(), give the losers the winner’s casing first, then move. Keep batches small (I use 20; a 50-work batch once pushed the service past its memory limit mid-move) and check for phantoms after any interruption.

The crawler: FSCrawler and Elasticsearch #

Full text comes from FSCrawler, which runs Apache Tika over each file and writes to Elasticsearch. Three settings mattered most.

There is no “unlimited” text extraction. I once set indexed_chars: -1. Tika buffers a document into one Java array, and the JVM cannot allocate an array longer than 2^31-1. The crawler thread died with “Required array length 2147483638 + 80 is too large”. The pod kept reporting Running with zero restarts, and the crawl sat at 3,284 documents for over an hour, looking healthy.

fs:
  indexed_chars: 50000000     # ~15,000 pages; bounded, 10x the old cap

A watchdog now samples the document count every 15 minutes and restarts the pod after three stalled samples. Liveness is measured by progress, not by process existence.

Create the index yourself. With template pushing off, FSCrawler will not create its mapping. Clone a working index’s settings and mappings, strip the generated fields, and create it explicitly. And if you delete and recreate an index, start the crawler once with --restart, or it resumes its checkpoint and skips every file it thinks it already indexed.

Pin the image. Mirror the image into your own registry under a dated, immutable tag. A deployment pinned to an upstream snapshot tag can lose its image permanently after node garbage collection.

Storage: health is not fitness #

I once moved 342 GB of Elasticsearch data to a drive that turned out to be 24.5 times slower on fsync, and moved it back the same day. I had checked SMART, which said the drive was healthy. It was. SMART answers “is it failing”, not “is it fast enough for a datastore”. For Elasticsearch, fsync latency is the floor under indexing latency. Benchmark both drives, p50 and p99 fsync, before you move.

Checklist: building a personal library catalogue #

• Use works, editions, files and role-tagged contributors.
• Split author strings with explicit, testable rules.
• Judge names by form first; use authorities to enrich, not to decide.
• Write corrections where the next rebuild reads.
• Key review markers on evidence quality.
• Make one-time scripts refuse to run twice.
• Check bulk moves for case collisions inside the plan.
• Bound indexed_chars; watch crawler progress, not pod status.
• Create indexes explicitly; use --restart after recreating one.
• Pin images by immutable tag in your own registry.
• Benchmark fsync before moving a datastore.

Frequently asked questions #

How do I catalogue a large personal ebook collection? #

Separate works from editions and files, group files into works by an authority key where one exists and by normalised title plus author surname otherwise, and record every ambiguity as a flag you can filter. Then fix the flags in batches, by class.

Why are the authors in my ebook metadata wrong? #

Because metadata and page text often carry tables of contents, publishers, software strings or OCR noise in the author field. Rules based on the form of the string catch most of them.

Can FSCrawler index very large documents? #

Yes, up to a bounded limit. Set indexed_chars to a large finite value, because text extraction runs inside the JVM and no array can exceed 2^31-1 elements. Watch progress, because a dead extraction thread does not stop the pod.

Is Open Library good enough to verify authors? #

It is good for enrichment: identifiers, subjects, dates. It is not a reliable judge of whether a name is a real author, especially for smaller languages, and it contains bad records itself.

The lesson #

Every big mistake here came from trusting a value because of where it came from: an author field because it was in the metadata, a name because an authority knew it, a marker because of its source, a pod because it said Running. The fixes all asked a better question about the value itself. Is this string shaped like a person? Is this evidence strong? Is the crawl actually moving?