From searching my own folders to a 72,000-work library

At 03:24 on 11 August 2026, a job that had been running for three hours and twenty-five minutes finished. It had pushed 2,965,788 pieces of text through sixteen parallel workers. When it was done, the number that mattered to me had moved from 9,270 to 2,568,138.

Those were chunks of Romanian. Books in my own language, which my search had been quietly skipping for months because the language model behind it only understood English. Overnight, a few million paragraphs that had been invisible became findable.

That night is not where this story starts, and it is not where it ends. But it is the night my files started to behave like a library.

Contents · 11 sections

In one minute #

• I began, in spring 2026, by pointing a search engine at the folders on my home storage, so I could find things in my own documents.
• By summer it held more than 175,000 documents and almost 8 million searchable chunks, and it was a mess: duplicates, missing authors, a whole language half-ignored, and a local AI that invented book citations when it could not find real ones.
• In August I rebuilt it as a real catalogue: every file identified, every work given an author, a shelf and a history. Today it holds 72,312 works in 123,123 files, plus an offline reference shelf of 414 books like Wikipedia.
• It all runs on one machine in one apartment in Iași. Nothing leaves the house.
• The lesson: search finds text. A library knows what the text is. You need both, and the second one is the hard part.

The before: a pile of folders with a search box #

The record I keep of my lab starts on 2 April 2026. On that very first day there is already an entry for a project called DocSearch: Elasticsearch, Kibana and FSCrawler, a crawler that walks a folder tree and feeds every document it finds into the search index. The same day I added a second, multilingual embedding model, so that Romanian questions would find Romanian text by meaning and not only by exact words. The note says the job was processing 125,000 documents, at about 3.8 documents per second on the CPU. A full re-embed took nine hours.

So the files were there before the record was. Years of them: e-books, scanned courses, technical manuals, PDFs of every kind, folders named by a younger me who never imagined anyone would need to understand them.

On 27 April I did a full inventory of what the lab actually stores. The numbers were strange to read in one place: 58 TB of network storage, 414 offline reference archives, about 28,000 books, 131,000 indexed documents. That same morning I built a page for my website about it and called it “The Vault”. I advertised only the open-licensed parts, and I framed it the way I saw it then: a resource for resilience. If the internet goes dark, the knowledge does not.

If you have ever kept a “Downloads” folder for ten years, you know the feeling. Everything is in there. Nothing is findable. And a search box on top of chaos gives you fast chaos.

The trigger: an AI that made up books #

In early August I wanted to ask questions of one shelf in particular, Religion & Spirituality. Before building anything new I surveyed what already existed, and the answer surprised me. That one shelf alone was 1,185,799 embedded chunks from 21,831 files, about 254 GB. The whole index held 175,977 documents and 7.84 million chunks in the multilingual index. The pipeline I was about to build already existed. It had been quietly running for months.

So on 8 August I built a bridge instead: a search tool and a filter so that a local chat model could pull passages from that index and answer with citations. I called the model Library Sage. An English question came back with real Romanian Orthodox sources, cited. That was the first real success.

Then the same session wrote down the other half: when the small local models were allowed to call tools, they fabricated library citations outright. Titles that did not exist. Authors who never wrote them. Delivered in the same confident tone as the real ones.

That line in my record changed the project. A library that an AI can lie about is worse than no library, because now the lie has my name on the shelf. If I wanted a machine to quote my books, the books had to be knowable: what each file is, who wrote it, whether it is a duplicate, whether the text inside is even readable. Retrieval was not enough. I needed a catalogue.

The plan, and what it found #

The next evening, 9 August, at 20:30, I wrote down an eight-phase plan with a title that sounds grander now than it felt then: “The perfect library”. It started with measured faults, not wishes. A few of them:

• 5,730 documents had no text at all. The embedder only selected documents that had a content field, so these were never processed and never marked as skipped. They sat in permanent limbo.
• The English search path asked its questions with a prefix that the index had been built without. I measured the mismatch as a cosine similarity of 0.9797. Close, but not the same, and “close” in vector search means the wrong neighbour wins on near-ties.
• The language-enrichment step ran an English-only model over a multilingual index. The largest language group after English was Romanian, 32,604 documents.
• Long books were sampled with a stride past a chunk cap, so the longest works, 983 of them, were only partly indexed. By definition the most substantial books were the least searchable.
• One rule I set myself that day: no OCR for now. The plan had to work with the text that existed.

Writing the faults as numbers turns “it’s messy” into a list, and each item on a list is a thing you can close. The invariant at the bottom of the plan was one line: documents in limbo, 5,730 today, must reach 0.

Two days later came the night I opened with. The full language re-enrichment, three hours twenty-five minutes, and Romanian went from almost nothing to 99.7% coverage. The next day I rebuilt the language of 174,428 documents from their content, reclassified 35,703 of them, and restricted the English index to text that was really English: from 8.27 million chunks down to 5.28 million. Fewer chunks, better answers. A theme that would repeat.

The build: from files to works #

On 13 August I created a new database for the curation itself and loaded a full inventory of the disk into it: 6.26 million rows, every file and folder. Then the real question: of 96,216 book files, which book is each one?

The identification pipeline resolved 69,834 of them at an average confidence of 0.951 on the first night, and queued 26,382 for adjudication. To get further without hammering a public service, I mirrored the Open Library bulk dumps into a local database, 128.4 million rows, and later a Wikidata mirror of 15.3 million rows, which turned out to be much weaker for Romanian books than I had predicted. Humility arrives on schedule.

On 15 August the first curated tree existed: 74,264 files, 394 GB, placed on shelves with a per-file audit trail. I removed language from the folder path after it proved impossible to get right in one pass, and kept it as metadata that can be corrected later. Then came weeks of the unglamorous part, which I covered in more detail in the catalogue post:

• clearing about 12,700 fabricated authors, invented by a metadata source that itself contained garbage records;
• giving 2,555 files a verified author from their container folders;
• naming files from their own first page, and recording “page-one evidence” for 89 files rather than inventing a title;
• emptying the staging area completely, 2,741 directories, after checking that every file’s content was on a shelf.

On 23 August the curated index was complete, 149,347 documents crawled with no size cap. The same day I started writing the services that turned it into something you can use: a catalogue service in Postgres with works, editions and files, a media service for covers, a gateway, a user interface, and an MCP adapter so AI agents can query it as a tool. Eight tasks in one day, each with its own tests, each deployed and checked live.

Then deduplication. On 24 August alone the catalogue went from 88,650 works to 79,720 to 77,464, each step a set of merges with full repointing, nothing deleted without a record of where it went. Later 2,392 dead work shells with no live files were retired, reversibly. The deduplicated copies went into a quarantine archive, verified restorable, with a log mapping every removed file to the copy that survived.

Not everything was done by software. On 4 September the record notes that the Literature shelf held 21,869 works, and that I had hand-cleaned A to G by title before asking for help with the rest. Seven letters of the alphabet, by hand. If you ever wondered whether the person behind a sovereign lab spends his time on architecture diagrams: sometimes the work is the letter C.

The technical heart: search is not a catalogue #

Here is the mechanism I wish I had understood in April. There are two different systems hiding inside the word “library”:

A search index answers “where does this text appear?” It stores documents and chunks of text, each with a vector for meaning and terms for keywords. It is fast and forgiving. It does not know that Boyce_Zoroastrians.pdf, zoroastrians (1).pdf and a Romanian translation are the same work, or that one of them is a 17-byte stub that will never render.

A catalogue answers “what is this, and how do I know?” It is a relational model, roughly:

work      (title, authors, subjects, series)      -- the idea of the book
edition   (language, publisher, year, cover)      -- one published form of it
file      (path, size, fingerprint, format)       -- the bytes on disk
mutation  (who, what, before, after, when)        -- every change, reversible

The two must be joined by identity, not by path. When a file moves to a better shelf, the catalogue changes one row, and the search index must keep that file’s existing text and vectors instead of re-crawling and re-embedding it. I proved this “identity carry” on 12 August through a full crawl cycle: the crawler reported zero new documents after a reorganisation, which is exactly what you want to see.

A few rules came out of that summer that I would give anyone building a personal library:

1. Measure limbo. Count the documents that are neither processed nor marked as skipped. Any pipeline that selects on “has field X” has a limbo.
2. Embed queries exactly the way you embedded documents. Same model, same prefix, same endpoint. Test it with a cosine similarity between the two paths; it should be 1.0, not 0.98.
3. Never let a cleanup infer “orphan” from absence. A sweep that deletes what it cannot see will one day delete what a slow query failed to return. I learned that one the hard way, and it has its own post: 239,000 live chunks gone in one run.
4. Every mutation gets an audit row. My catalogue has logged 460,114 of them. That number is why an AI agent may edit a title at all: every edit can be undone.
5. Prefer “unknown” to invented. An empty author field is honest. A plausible fake one poisons everything downstream.

The after: what the library became #

Once the catalogue existed, the rest grew on top of it faster than I expected.

On 29 August the language layer (summary, subjects, people, places, organisations) went from 8,048 works to 59,072 in one session. Entity pages became walkable: you can stand on a person and see which works connect them to another. An “ask the library” page answers questions from retrieved passages, with citations. Study guides, generated by a local model, cover the openings of thousands of books, a limit I chose deliberately and wrote about in the study guides post. The offline reference shelf, 414 Kiwix archives including Wikipedia in English and Romanian, sits inside the same search; it is how a single author’s claim gets checked against a mainstream baseline, and it has its own story. In October every file got a fingerprint in a log anchored in Bitcoin, so that anyone can verify a quoted file is the one I held.

Today the live counters on my site say: 72,312 works, 79,277 editions, 35,717 authors, 53 shelves, 68 file formats, 365,968 named entities, and almost 13 million multilingual chunks. Each number is checked every half hour against the running system.

And in October the library did something I had not planned when I pointed a crawler at my folders in April: it became the source of a book of more than 1,500 pages, written from its shelves with every claim cited by catalogue number. But that is the next story.

What it cost, and what is still wrong #

It cost a summer. Most of August, honestly, and many nights of it. It cost disk: an index of nearly 97 GB on a disk that was at 81% when I planned the work. It cost a good amount of GPU time and a lot of patience with tools that fail silently. FSCrawler skipped files whose modification time was older than its last scan, with no error. Its exclude patterns treated the dot in *.zip as “any character”, and silently dropped seventeen files whose names merely ended in “zip” or “map”.

And it is not finished. On 24 August there were still 29,050 files with an unknown author. Some shelves are cleaner than others. The duplicate detector has an irreducible core of groups that only a human reading the books could settle. Music scores, which look like books to a crawler and like noise to a text model, still need their own approach. I would rather tell you that than pretend the counters mean perfection.

What changed #

• I stopped trusting “it’s indexed” as a measure of anything. Now I ask: identified, deduplicated, attributed, verifiable?
• Every AI that answers from my books must cite a catalogue number, and the citation can be checked.
• My own language is a first-class citizen in my own library.
• The record of every change is part of the library, not an afterthought.

Questions people ask me #

Why not just use a cloud service or a commercial library app? #

Because the point is that nothing leaves the house: not the files, not the questions I ask, not the reading habits an index reveals. A hosted service would also decide for me what counts as a duplicate, what language a book is in, and what I may search. On my own machine those are my decisions, recorded and reversible.

Do you need special hardware to search your own documents? #

No. Elasticsearch and a crawler run on modest hardware, and keyword search alone is a huge step up from a file browser. Embedding models for search by meaning run on a CPU too; mine processed about 3.8 documents per second before a GPU was involved. Start small, on one folder, and grow.

What is the difference between a search index and a library catalogue? #

A search index tells you where text appears. A catalogue tells you what a file is: which work, which edition, which author, whether it is a duplicate. You can build search in a weekend. A trustworthy catalogue takes months, because identity is hard and every mistake multiplies.

How do you stop a local AI from inventing citations? #

You give it a tool that returns real passages with real identifiers, you make it cite those identifiers, and you check them. Small models will still try to fill gaps with fiction, so the catalogue number is the safeguard: a citation that does not resolve to a work is visibly fake.

Closing #

In April I wanted to find things in my own folders. What I built instead is a place where a sentence can be traced back to a page, the page to a file, the file to a work, and the work to a record of every hand that touched it, mine and the machines’.

One apartment, one machine, a few hundred thousand decisions. The search box was the easy part. The library is what you get when you refuse to let anything, human or AI, pretend to know what it does not.