-
From searching my own folders to a 72,000-work library
In April 2026 I pointed a search engine at my own folders. By October it was a catalogue of 72,312 works in 123,123 files. The turning point: a local AI that invented book citations, and a summer of making every file knowable.
-
The cleanup job that deleted 239,000 live search chunks
A nightly orphan-chunk sweep in my Elasticsearch library deleted 239k live chunks from 19.4k documents for four weeks. Why its liveness check was wrong, how I recovered, and the count-first circuit breaker that followed.
-
A shared memory for me and my AI agents: from 9 to 12 out of 15
I moved my lab's record of tasks, mistakes and runbooks into a context database shared by me, AI assistant sessions and my local agent. Two secrets slipped past redaction on day one; one fact per file plus a reranker lifted retrieval to 12 of 15.
-
Hybrid search with bge-m3 and Elasticsearch: six lessons
What I learned building BM25 + kNN hybrid search with reciprocal rank fusion and bge-m3 on one shared GPU: an analyser mismatch (38 to 2,631 hits), RRF pitfalls, embedding timeouts and a highlighting limit.
-
From 96,216 messy files to a 70,000-work library catalogue
How I turned 96,216 ebook and scan files into a works/editions/files catalogue on Elasticsearch and FSCrawler: curation rules that worked, markers that backfired, and crawler settings that cost me the most time.
-
Running 414 Kiwix ZIMs as an offline reference shelf
What I learned serving 414 Kiwix ZIM files (973 GB) beside my own library: undated aliases need -z, lazily opened ZIMs break when moved, 240 lack full-text indexes, and encyclopedias should be federated, not embedded.
-
Six million files, one OSINT shelf: crawl, OCR, embed, NLP
Growing an OSINT records shelf from 35,733 to 185,000+ indexed documents: separate pipelines from the library, files an incremental crawler never sees, 70k image-only PDFs, and OCR measured in page-years.