Entity resolution lessons from a library and a threat-intel store

Before applying an automatic author-matching rule to my library, I sampled five of its decisions. One would have credited a book about the Chinese Zen master Ma Tzu to Ma Tzu, who died more than a thousand years before it was written. The rule was sound on paper: if the text does not say who the author is, keep the name that has a real identity in Open Library or Wikidata. The subject of a book is often more famous than its author, and the rule could not tell them apart.

That sample of five stopped the rule before it ran; the affected groups went to human review instead. It is also the shortest summary I have of entity resolution: an authority can confirm that a name is real, never that it wrote these bytes. This post collects what I learned resolving entities in two very different systems: a personal research library (authors, people, places, concepts) and a threat-intelligence store (IP addresses, networks, organisations, domains).

Contents · 11 sections

TL;DR #

• Keep every source’s claim. Derive a “golden” view from them; never overwrite the claims themselves. Disagreements between sources are signal.
• Use deterministic ids (a hash of type plus natural key) so any pipeline can reference an entity without a lookup.
• Record who decided each merge or alias (automatic, local model, hosted model, human) and never let a lower tier overwrite a higher one.
• Co-occurrence is only evidence above the base rate. Two always-on sources always co-occur. Gate on lift, not counts.
• Exclude bulk lists from link extraction. 213 false links came from two of them.
• Sample the weakest decisions before every bulk apply.

Two systems, one problem #

In the library, entity resolution means deciding that “Eminescu”, “M. Eminescu” and “Mihai Eminescu” are one person, that “Tesla” in a book about engineering is Nikola Tesla, and that an author folder called “Accessibility help” is not a person at all.

In the threat-intelligence store, it means deciding that an IP address, its network, the organisation that holds it and the abuse contact on its registration are connected, and how strongly. The store holds about 3 million fact rows; one weekly audit checked 967,938 values.

The mechanics differ. The principles turned out to be the same.

Principle 1: keep every claim, derive the answer #

The threat-intel resolver stores each source’s assertion as a separate fact row: source, attribute, value, first seen, last seen. It never overwrites one. A “golden” record is derived from those facts by precedence rules, and can be rebuilt at any time.

CREATE TABLE entity_fact (
  entity_id   uuid NOT NULL,
  attr        text NOT NULL,
  value       text NOT NULL,
  source      text NOT NULL REFERENCES fact_source(name),
  first_seen  timestamptz NOT NULL,   -- never updated
  last_seen   timestamptz NOT NULL,   -- bumped on every observation
  PRIMARY KEY (entity_id, attr, value, source)
);

Two benefits follow.

First, disagreements become data. When the registry says a network is in one country and its operation is in another, or an organisation’s name differs between sources, that is often the most interesting fact about it: a rebrand, a front, a reallocation. A conflicts view surfaces every attribute with divergent values.

Second, every answer can be audited. A weekly job proves that every golden value traces back to a fact, and every projected relationship traces back to an edge. The baseline run checked 967,938 values with 0 violations. Because the store is cumulative, taken-down or renamed entities persist with their history instead of disappearing.

A lesson inside this one: I once had two definitions of “conflict” in the same codebase. One counted distinct values (100,409 flagged), the other required distinct sources and excluded membership attributes (63,155). Both looked reasonable. Now there is one definition, and the list of excluded attributes lives in one place.

Principle 2: deterministic ids #

Entities get ids computed from their type and natural key:

import uuid
NS = uuid.UUID("<your fixed namespace uuid>")

def entity_id(etype: str, key: str) -> uuid.UUID:
    return uuid.uuid5(NS, f"{etype}:{key}")

entity_id("asn", "64500")
entity_id("domain", "example.com")

Any pipeline can compute the id of “network 64500” without a database round trip, bulk upserts scale to hundreds of thousands of rows, and two systems that never talk to each other agree on ids. The natural key must be normalised the same way everywhere, which surfaced a small but real gotcha: the same IP column was inet in one database and text in another, so a cross-database query had to cast differently on each side.

Principle 3: record who decided, and respect it #

In the library, every canonical entity and alias carries a tier: automatic, local model, hosted model, human. The rule is enforced on the server: a lower tier can never replace a higher one, and a rebuild never overwrites anything that is not automatic. Judged aliases pin their canonical entity.

That made it safe to replace a paid hosted model with local models for judging entities. I tested a 9B and a 27B local model against entities that humans had already corrected:

• the 9B caught 13 of 15 junk entities and kept 26 of 30 real ones;
• the 27B caught all 15 junk entities but dropped common nouns the hosted model had kept;
• combined, “drop only if both say junk”, they got 11 of 11 human-corrected mistakes right, with about 16% of decisions sent to a review queue.

Disagreements go to people. Agreements are stored with the models’ names as the judge.

A related trap: when one work has several files, only one of them may carry the extracted facts. The work page used to pick the first file, and 16,241 works showed empty insights even though they had been enriched. When one entity has several ids, resolve by where the data is, not by position.

Principle 4: co-occurrence is not a relationship #

The threat-intel linker looked for addresses that were active in the same time windows across different layers, for example an SSH brute-forcer and a web client. Its strongest pair was a brute-forcer and an RSS reader legitimately polling my own blog’s feed.

Two always-on, periodic sources always co-occur. Raw window counts measure how busy something is, not whether two things are related. The fix had three parts:

• filter the inputs to real threats first (a threat-score threshold);
• require lift of at least 2: observed co-occurrence divided by what chance would give;
• make confidence multiplicative in lift and support.

expected = (windows_a / total) * (windows_b / total) * total
lift = shared / expected if expected else 0.0
if lift < 2.0 or shared < 3:
    continue            # not evidence of anything

A later change grew the input pool 16 times (from 89 to 1,414 addresses), and the same code went from 6 pairs to 201. My earlier verification no longer held. A check is only valid for the population it ran against; when an upstream filter changes, re-verify everything downstream.

Principle 5: bulk lists poison links #

Cross-corpus links once suggested that my OSINT document collection was connected to addresses seen by my honeypots. All 213 links came from two documents: a list of open DNS resolvers and a public blocklist. Bulk lists mention thousands of unrelated addresses, so every match is a false mention, and a human reading the graph over-reads it as a hidden connection.

Bulk-list documents are now excluded from link extraction, only addresses actually observed are linked, and confidence depends on the document type. The same idea appears in contact data: an abuse address shared by more than 100 entities is a registrar catch-all, not a relationship, and is skipped.

Principle 6: sample before you apply #

Every bulk resolution now runs as a dry run first, and I read a sample of the weakest-evidence class before applying. That is how the Ma Tzu rule was caught. Other catches from the same habit:

• a hosted model “identified” about 44 mislabelled PDFs by inventing titles from plausible filenames, and a second model checking its rows did not notice. A deterministic check now requires title tokens to appear in the document text;
• a confidence cap ran after an evidence rule and overwrote it, dropping 80 works from one archive below the shelving threshold. Rule order is now fixed: caps first, evidence last, and known sub-collections are counted before every apply.

Checklist: entity resolution you can trust #

• Store claims, derive answers; never overwrite a source’s assertion.
• Treat cross-source conflicts as a feature and surface them.
• Use deterministic ids from normalised natural keys.
• Tag every decision with its tier and enforce precedence in the server.
• Gate co-occurrence on lift, not raw counts; check the strongest edge against known benign sources.
• Exclude bulk lists and catch-all contacts from link extraction.
• Re-verify downstream after any change to an upstream population.
• Sample the weakest class before every bulk apply.
• Check identifications against the document text, not filenames.

Frequently asked questions #

What is entity resolution? #

It is deciding which records refer to the same real-world thing, a person, organisation, network or domain, and linking or merging them. It covers matching name variants, splitting look-alikes and assigning stable ids.

Can I use Wikidata or Open Library to verify authors? #

They confirm that a name belongs to a real person. They cannot confirm that this person wrote this document; books about famous people are a classic false match. Use them for enrichment and require evidence from the document itself for authorship.

How do I avoid false relationships from co-occurrence? #

Compare observed co-occurrence with what chance predicts (lift), filter inputs to relevant entities first, and exclude documents that list thousands of entities at once. Then look at the strongest edge yourself.

Should a merge overwrite the original records? #

No. Keep each source’s original assertion and derive the merged view. Then you can audit every value, undo bad merges and see where sources disagree.

The lesson #

Every bad link in both systems came from treating a weak signal as a strong one: an authority as proof of authorship, a coincidence as a relationship, a bulk list as a mention, a filename as a title. Keep the raw claims, make the strength of every decision explicit, and read the weakest five before you trust the next five thousand.