shuffle-on
The cleanup job that deleted 239,000 live search chunks
Blog · search

The cleanup job that deleted 239,000 live search chunks

An orphan sweep, a third producer it never knew about, and the circuit breaker every destructive cron job needs

8 min read

A nightly cleanup job in my research library deleted about 239,000 search chunks from 19,400 documents, and kept doing it for four weeks. Nothing alerted. Every run reported success. The job was doing exactly what I had written it to do.

This post is about that job: the “orphan sweep” that removes leftover search chunks, why it decided that a whole shelf of live documents was garbage, how I found it, and the circuit breaker every cleanup job of this kind should have from day one.

Contents · 10 sections

TL;DR #

  • My library splits every document into chunks with embeddings for semantic search. When a document moves or disappears, its old chunks become orphans, and a nightly job sweeps them away.
  • The sweep decided “is this chunk’s parent still alive?” by checking two places. A third producer wrote chunks into the same index, with its parents in a third place the sweep never looked at.
  • For four weeks in September 2026, every night, the sweep deleted those chunks as orphans. Their “already embedded” markers stayed set, so nothing re-created them.
  • The fix was not just “check the third place”. The sweep now counts first and refuses when the number is implausible, a rescue step runs before it, and a failed rescue blocks the sweep.
  • The general lesson: a cleanup that decides liveness must know every producer that writes into the thing it cleans, and must have a ceiling on how much it may delete in one run.

The setup: chunks, parents and orphans #

My library runs on Elasticsearch. Each book or document lives in a “parent” index with its full text. For semantic search, a separate embedder splits the text into passages, computes a vector for each, and writes them into a chunk index. Each chunk carries a parent_doc_id that points back to its document.

The document id is derived from the file path. That is convenient for the crawler, and painful for everything else: move a file and its id changes. The old chunks still point at the old id. Nobody owns them any more. They are orphans.

Orphans are not harmless. They make semantic search return passages from documents that no longer exist under that id, they inflate the index, and in my case they once made the chunk index cover 110% of the library. So in August I added a third layer of healing:

  1. at move time, re-key the moved file’s chunks to the new id;
  2. an hourly job that re-keys orphans when it can find the move record;
  3. a daily sweep that deletes any chunk group whose parent is gone, “from any cause”.

The sweep’s liveness test was simple: a chunk group is alive if its parent id exists in the library index or in the catalogue’s file table. Otherwise, delete. A dry run on 2026-08-31 found 14,939 historical orphan groups, and I scheduled it nightly.

What it actually deleted #

The library is not the only thing that writes into that multilingual chunk index. My OSINT shelf, a separate collection of public records with its own crawler and its own embedder, writes its chunks into the same index. Its parents live in a different Elasticsearch index, and those documents have no row in the library’s file table.

So for the sweep, every OSINT chunk looked like this:

  • parent not in the library index: true;
  • parent not in the catalogue files: true;
  • verdict: orphan.

Every night it deleted them. Because the OSINT embedder marks a document as “embedded” when it writes the chunks, and the sweep never touched that marker, the embedder believed its work was done. The shelf’s semantic search quietly emptied while keyword search, which uses the parent text, kept working.

I found it on 2026-09-29, during a test pass of every tool my AI assistants use to read the library. “Ask this document” on an OSINT record returned nothing. The chunks were simply gone: about 239k chunks across 19.4k documents.

Root cause #

The sweep was correct for the world I had in mind when I wrote it, and wrong for the world it ran in. The liveness check knew two producers. There were three.

This is the same shape as an earlier mistake in the same project. In August, a “retire plan rows whose source file no longer exists” step deleted about 26,000 rows twice in one session. Once the staging folder had been emptied on purpose, “source missing” had become the normal state for every book. The predicate was written for an older world and never re-derived.

Both times the cleanup was not buggy in the usual sense. It was confidently applying a rule whose premise had stopped being true.

Recovery #

Recovery was cheap because the source files and parent documents were intact. Only derived data was lost:

# 1. find parents marked "embedded" that have zero chunks
#    (composite aggregation over parent_doc_id in the chunk index,
#     compared with the parent index)

# 2. clear the marker so the embedder picks them up again
POST /<parent-index>/_update_by_query?conflicts=proceed&wait_for_completion=false
{
  "script": { "source": "ctx._source.remove('chunks_embedded_ml'); ctx._source.remove('chunks_embedded_ml_at')" },
  "query": { "bool": {
    "filter": [ { "prefix": { "path.real": "/docs/<collection>/" } } ],
    "must_not": [ { "ids": { "values": ["<ids that still have chunks>"] } } ]
  } }
}

I reset the marker on 19,044 documents, and the embedder rebuilt them on its normal five-minute watch. The embedder deletes a document’s old chunks before writing new ones, so the reset was safe to repeat. The cost was GPU time and slower answers while it ran: the library’s question-answering latency went from about 7 to 15 seconds during the rebuild.

The same day, the sweep learned to check the OSINT parent index too. That was the obvious fix. It was not enough.

The circuit breaker #

A month later, an integrity audit found two more problems with the same job:

  • its limit counted parents across both chunk indexes together (212,053 that day), so part of the multilingual index was never swept at all;
  • the rescue step, which re-attaches orphans to their new ids, filled its batch limit with OSINT groups before reaching real library orphans.

More importantly, the sweep still deleted page by page with no plausibility check. If any future producer appeared, it would repeat the incident. So the sweep changed shape:

  1. Collect first, delete later. The sweep builds the complete list of orphan groups before deleting anything.
  2. Refuse implausible numbers. If the list is larger than 5,000 groups or 5% of all parents, the job returns HTTP 409 and deletes nothing. A human looks.
  3. Rescue before sweep. An init container runs the rescue step first, so anything that can be mapped to its new id is carried over instead of deleted.
  4. A failed rescue blocks the sweep. If the rescue hits any Elasticsearch error, it now fails loudly (HTTP 502), the init container fails, and the sweep never starts. Before, the rescue returned 200 after a failed copy, and the sweep could delete the group it had just failed to save.
  5. No retries. The CronJob has backoffLimit: 0. A retry pod must not sweep while the first rescue is still copying.

In code, the guard is a few lines:

SWEEP_MAX_ORPHANS = 5000
SWEEP_MAX_FRACTION = 0.05

orphans = collect_orphan_groups(indexes)          # full scan, no deletes
parents = count_live_parents(indexes)

if not dry_run and (len(orphans) > SWEEP_MAX_ORPHANS
                    or len(orphans) > SWEEP_MAX_FRACTION * parents):
    raise HTTPException(409, f"refusing: {len(orphans)} orphans of {parents} parents")

And the CronJob chains the two steps so the order cannot be forgotten:

spec:
  jobTemplate:
    spec:
      backoffLimit: 0
      template:
        spec:
          initContainers:
            - name: rescue        # exits non-zero on any copy error
              command: ["sh", "-c", "curl -fsS -X POST http://<catalogue>/rescue-orphan-chunks?limit=2000"]
          containers:
            - name: sweep         # only runs if rescue succeeded
              command: ["sh", "-c", "curl -fsS -X POST http://<catalogue>/sweep-orphan-chunks?limit=2000000"]

After the change, the rescue re-keyed 103 legacy groups that the old sweep would eventually have deleted, and the nightly dry runs report 0 orphans over 212,045 parents. That zero is the number I watch.

Why the breaker matters more than the fix #

Adding the OSINT index to the liveness check fixed this incident. The breaker fixes the class. Next time someone adds a producer, a new index or a new shelf, and forgets to register it, the sweep will see thousands of “orphans” at once and stop.

Once the historical backlog of 14,939 groups had been cleared, real orphans could only come from the day’s moves, and the move path now carries chunks to the new id itself. A sudden jump to tens of thousands is never routine. It is either a real catastrophe upstream or a bug in the sweep. In both cases, deleting is the wrong reaction.

Checklist: before you schedule a cleanup job #

  • List every producer that writes into the store you clean, and where each keeps its parents. Write the list into the code, next to the liveness check.
  • When you add a producer, add it to the liveness check in the same change.
  • Collect, then delete. Never delete inside the scan loop.
  • Put a ceiling on one run: an absolute number and a fraction. Above it, refuse and alert.
  • Run rescue before sweep, and make a failed rescue stop the sweep.
  • Disable automatic retries for destructive jobs.
  • Keep a dry-run mode and look at its count before the first real run, and after every change upstream.
  • Do not leave “already processed” markers set when you delete what they describe.

Frequently asked questions #

What is an orphaned chunk in a vector search index? #

It is a passage with an embedding whose parent document no longer exists under the id it points to. It usually appears when a document’s id changes, for example because the id is derived from the file path and the file moved. Orphans waste space and can return passages from documents that are gone.

Why not just delete every chunk whose parent is missing? #

Because “missing” depends on where you look. If any process writes chunks whose parents live somewhere your check does not cover, those live chunks look like orphans. That is exactly what deleted 239k chunks here.

What threshold should a cleanup circuit breaker use? #

Base it on your normal churn. I use the lower of 5,000 groups or 5% of all parents, because normal nights are far below either. The exact number matters less than having one, and failing safe when it is crossed.

How do you recover chunks that were deleted by mistake? #

If the parent documents are intact, chunks are derived data: reset the “embedded” marker on the affected parents and let the embedder rebuild them. Make sure the embedder deletes old chunks before writing new ones, so the reset is idempotent.

The lesson #

The sweep never crashed, never logged an error, and never deleted anything it was not told to delete. That is what made it dangerous. A cleanup job encodes a belief about the world, “these are all the things that should exist”, and that belief goes stale silently as the system grows. Write the belief down next to the code, and give the job a ceiling so that when the belief is wrong, it stops instead of succeeding.