Six million files, one OSINT shelf: crawl, OCR, embed, NLP

My OSINT shelf, a collection of public records next to my research library, grew from 35,733 documents to more than 185,000 indexed in a few days at the end of October 2026. Then a presence audit walked the shelf and counted 6,029,923 files on disk. Most were deliberately excluded. A few hundred were silently never going to be indexed at all. And roughly 70,000 of the new PDFs had no text layer, so search could not see them.

This post is about building the crawl, OCR, embedding and NLP pipelines for that shelf, and about one rule I kept even when it would have been easier to break: the OSINT pipelines stay separate from the library’s, sharing code but never queues, indexes or configuration.

Contents · 10 sections

TL;DR #

• The OSINT shelf has its own index, crawler, OCR, embedder, NLP enrichment and study guides. They reuse the library’s code, but run as separate instances. The only shared surface is the status and pipelines pages.
• The NLP is different on purpose: the OSINT enrichment keeps organisations, people, dates and names that the library’s enrichment filters as noise.
• An incremental crawler never picks up a file whose modification time is older than its last scan. 331 PDFs were invisible forever until I touched them.
• A crawler configured to run every 7 days instead of every hour indexed 0 new files for a day while downloads piled up.
• About 70,000 of 115,000 new PDFs were image-only. Full vision-model OCR at 56 seconds per page would have taken years; the plan is CPU OCR for the bulk and the vision model only for low-confidence pages.
• Never clone hundreds of thousands of tiny files onto a spinning disk. Stream archives and pack them into shards.

Why keep the pipelines separate #

It is tempting to treat OSINT documents as “more books” and point the library’s pipelines at them. I decided against it, and the reasons are worth spelling out because they apply to any mixed corpus.

Different NLP goals. The library’s entity extraction is tuned for books: it drops most person names that appear in passing, and a stoplist removed thousands of noisy person entities. For court records, inquiries and declassified files, the names, organisations and dates are the content. The OSINT enrichment keeps them.

Different text quality. Library books are mostly born-digital or decent scans. OSINT folders hold faxed memos, typewritten cables and redacted pages. They need OCR settings, confidence thresholds and chunk limits of their own.

Different failure blast radius. A bug in one pipeline should not damage the other. When the library’s nightly orphan sweep did not know about OSINT chunks living in a shared index, it deleted about 239,000 of them. Separate queues and explicit producer lists are how you keep that from happening twice.

Different access rules. Some OSINT collections are restricted to specific accounts. Mixing them into library indexes would make every library query a potential leak.

So the rule is: shared code is fine; shared instances are not. Each OSINT stage has its own deployment, its own queue, its own pause switch (named osint-ocr, osint-embed, osint-nlp and osint-guide) and its own heartbeat. They follow the library’s conventions, pause and resume, auto-yield to interactive GPU work, requeue and redo, but they never point at the library’s indexes, and the library’s pipelines never point at the OSINT index.

Downloading: polite, resumable, verified #

The shelf is filled by a downloader that works from manifests (one row per document, with source URL and target folder). Each fetch is resumed on failure, retried a few times, validated by its magic bytes (a “PDF” that is actually an HTML error page is rejected), hashed with SHA-256 into a ledger, and only then promoted into the shelf with a fresh modification time.

Rate limits shaped the design more than bandwidth did. A large public archive started answering with HTTP 429. The fix was a single global slot for that host, small chunks of 25 rows and a 20-second rest between chunks, with a separate lane for its web-archive service. Sources that blocked automated access were not worked around; they were recorded as blocked.

Some sources were too fine-grained to ingest as files: hundreds of thousands of tiny records. Those were packed into batch text files the crawler can index: the 314,616 documents of one official diplomatic history series became 1,695 files, and a war-logs archive of 76,911 reports became 2,185 daily files. One lesson there: large CSVs were indexed truncated at 5 million characters, which is why they were regrouped into smaller files.

Crawling: the files an incremental crawler will never see #

The crawler is FSCrawler, as for the library, with its own job. Two surprises:

The schedule was a week. The OSINT crawler’s configuration said update_rate: 7d; I had assumed it was hourly because another job’s configuration was. Its run on 30 October indexed nothing, while downloads kept landing. Restarting it (it keeps a persistent checkpoint, so the restart is incremental) took the shelf from 35,733 to 41,882 documents in three minutes. The rate is now one hour.

# FSCrawler job settings (excerpt)
fs:
  url: "/docs/<shelf>"
  update_rate: "1h"
  excludes: ["*/@eaDir/*", "*/<attachment-dirs>/*"]

Incremental means “newer than the last scan”. The presence audit walked all 6,029,923 files in 34 minutes and compared them with the index:

• 185,444 indexed;
• 5,790,024 excluded by patterns, on purpose: mostly raw attachments of one huge email archive and NAS thumbnail folders;
• 54,455 not indexed: 41,914 in folders the current crawl had not reached yet, 10,047 newer than the pass over their folder (lag, not loss), and 331 PDFs dated June in folders the crawler had already finished.

Those 331 would never be indexed. An incremental crawler skips any file whose modification time is older than the scan date of its folder; these files carried June modification times. Touching them fixed it:

# make files older than the last scan visible to an incremental crawler
find /docs/<shelf>/<folder> -name '*.pdf' ! -newermt '<last-scan-date>' \
  -print0 | xargs -0 touch

The same audit found 57 index entries whose file had gone, and 4,137 contents with two or more copies (42.5 GB redundant). Duplicates are reported, not deleted: in a records collection, the same file in two folders often means two different contexts.

OCR: when the math says “years” #

After the crawl caught up, about 70,000 of 115,000 new PDFs turned out to be image-only. Some collections were 100% scans. The crawler indexed them with empty content, the embedder skipped them, and search was blind. One collection of 60,192 documents had zero OCR done, so a keyword search for its most famous name returned nothing inside it.

My best OCR is a vision-language model that handles tables, footnotes and rotated pages well. At about 56 seconds per page and roughly 1.34 million image-only pages in the backlog, it would have taken years.

The plan instead:

1. Tesseract on CPU for the bulk. With 40 workers it ran at about 180 to 280 pages per minute, keeping CPU temperatures around 66 to 70 °C.
2. The vision model only for low-confidence pages, where Tesseract’s own confidence says the result is poor.
3. Documents requested by a person jump the queue. A request once sat behind about 250 bulk documents before that rule existed.

After OCR, the OSINT embedder and NLP enrichment pick documents up on their own queues: around 430 chunks per minute for embedding and 800 for NLP at the time I measured.

Storage: tiny files and spinning disks #

The very first OSINT build, months earlier, cloned two huge repositories of tiny files (161,000 and 300,000 files) straight onto a cold spinning disk. The disk sat at 100% utilisation with about 2.4 seconds of latency, load climbed to 19 and the host hung.

Never materialise hundreds of thousands of small files on a hard disk. Stream the archive as a tarball and pack entries into shards of about 2 MB, and run any bulk disk job under ionice -c3 nice -n19. Today staging lives on NVMe and the shelf on the NAS; nothing touches the cold disk.

Trust the text, not the filename #

A name check over a sample of files (a first-page read of 107 files, automated token overlap on 1,180, and a case-number check of court dockets) found one set of seven files named after a well-known leak case that were really an unrelated bank-fraud case. They were renamed with a logged, reversible script. A filename is a claim; the first page is evidence.

Checklist: an OSINT document pipeline #

• Give OSINT its own index, queues and configs; share code, not instances.
• Decide what the NLP must keep for records, not just what it drops.
• Validate downloads by magic bytes and hash them into a ledger.
• Respect rate limits with per-host lanes; record blocked sources instead of evading them.
• Pack tiny records into batch files; avoid giant single files the extractor truncates.
• Check the crawler’s schedule and run a presence audit that compares disk with index.
• Remember that incremental crawlers skip old timestamps.
• Estimate OCR in page-years before choosing an engine; reserve the expensive one for low-confidence pages.
• Never put tiny-file workloads on spinning disks.
• Verify names against document text.

Frequently asked questions #

Should OSINT documents share a search index with a personal library? #

Not if you can avoid it. The NLP goals, text quality and access rules differ, and a bug in one pipeline can damage the other. Share code and dashboards, keep indexes and queues separate.

Why are some files never indexed by FSCrawler? #

Its incremental mode skips files whose modification time is older than the last scan of their folder. Files copied in with preserved timestamps can stay invisible. Touch them, or rescan with a restart.

How do I OCR over a million scanned pages at home? #

Use a fast CPU engine such as Tesseract for the bulk and reserve a slower vision model for pages where the CPU engine reports low confidence. Calculate total pages times seconds per page first; it decides the design.

What should an OSINT collection leave out? #

Mine excludes credential dumps, hacked customer databases, malware and any victim imagery, regardless of where they were published. Public records are the goal, not stolen data.

The lesson #

The shelf grew fivefold in days, and almost nothing about that was hard in the obvious way. The hard parts were the quiet ones: a crawler that ran weekly, files with old timestamps, scans with no text, a disk that could not take small files. Keeping the OSINT pipelines separate did not prevent those, but it kept every one of them contained to the shelf where it happened.