docs/detection_unit_protocol.md

# Detection unit protocol

**Pre-registered. Written and committed before the actor-level label was built,
the stacker retrained, or any of the numbers below were seen.**

## Why this exists

Every number this project has quoted so far has been *per wallet*: what share of
illicit-labelled wallets did we alert on. That is not what the product does and
not what an investigator asks for. A ransomware operation is one case. It owns a
collector address and a dozen change addresses in a peel chain; a layering
scheme owns a source wallet, tens of hop wallets and a sink. Wallet recall
scores a system for how many of those it enumerates, which rewards the
uninteresting half of the job (the hops are near-identical one-transaction
wallets) and says nothing about the interesting half: did we find the operation
at all, and once found, can we walk out to the rest of it.

Choosing the unit after seeing which unit flatters the numbers is how a metric
becomes marketing. So the unit is fixed here, in advance.

## The unit of detection: the actor

**An actor is one ground-truth illicit operation** — the criminal enterprise
behind one typology instance, not each address it touched.

Derived from `ground_truth.json` (`eval/actors.py`), never from anything the
pipeline produces:

- **ransomware_collector** — one actor per instance: the collector actor. Its
  wallets are the collector address plus every change address in its peel chain.
  Victims are not part of the actor (they are the complainants) and neither are
  the `cashout` counterparties, which are separate actors receiving the peel.
- **layering** — one actor per instance, identified by the originating cluster
  of that instance's `layering_split` / `layering_merge` transactions. Its
  wallets are the source, every intermediate hop wallet and the sink. The
  generator models each hop as its own `Actor` object because each needs an
  owner; in the world being simulated they are one launderer's wallets, and
  they are grouped back together here.

Patterns that are *not* actors, and why: `normal` and `exchange` are not
illicit; `same_actor_cluster` is a clustering test, not a crime; `coinjoin` is
mixing, which is suspicious but not by itself illegal (consistent with
`engines.gnn.illicit_typologies`); `ransomware_victim` and `cashout` are
counterparties of an actor, not the actor.

An **entity** (our clustering's output) is illicit **iff at least one of its
wallets belongs to an illicit actor**. That is the label the stacker is
retrained on. A wallet is **alerted** iff the entity holding it is in the final
alert list at the configured `fusion.alert_threshold`.

## Metrics

Reported per typology, on the standard and the shifted set, both at the
canonical observation rate.

1. **`case_detection_rate`** — fraction of illicit actors with at least one
   alerted wallet. *Did we find the operation?* This is the headline number.
2. **`alert_precision`** — fraction of alerts whose entity holds at least one
   wallet of any illicit actor. *Of what we put in front of an investigator, how
   much was a real case?* Alerts on victim or cash-out wallets count as
   imprecise here even though they are adjacent to crime; adjacency is what
   taint reports, not what an alert claims.
3. **`trace_coverage@N`**, N = 2 and 4 — for each *detected* actor, start from
   its alerted wallets and walk the entity graph outward as taint does
   (`fusion.taint`'s direction and hop budget), N hops. Score the fraction of
   that actor's *other* wallets that are reached. *Once we have one thread, how
   much of the operation can we pull in?* Undetected actors are excluded — this
   measures expansion, not detection, and averaging a zero over cases we never
   found would confound the two.
4. **`wallet_recall_broad`** — the old metric, kept as **secondary**: recall over
   every wallet of every illicit-patterned cluster including pass-through hops,
   under the broad label. Reported so this pass is comparable with the previous
   one, not because it is what the product optimises.

Precision has no actor-level denominator by design: an alert is a unit of an
investigator's attention, so it is counted per alert.

## Origin attribution: outcomes and costs

Accuracy treats every origin error alike. It is not alike: naming a public relay
operator's address as a suspect's is a different kind of wrong from having no
answer. Each non-degraded origin estimate falls into exactly one outcome, and
the score is the mean weight (`engines.propagation.origin_filter.cost_weights`,
defaults below):

| outcome | when | weight |
| --- | --- | --- |
| `correct_actionable` | not abstained, correct, and the named address is residential/unclassified — a lead an ISP request can act on | +1 |
| `correct_infrastructure` | not abstained, correct, but the address is a Tor exit, hosting/VPN or relay — a true *anonymized entry point*, worth recording, not directly actionable | +0.3 |
| `wrong_uninvolved_third_party` | not abstained and wrong — we named an address that did not send the transaction | -3 |
| `abstained` | `low_confidence_origin` is set: we declined to name anyone | 0 |

The asymmetry is the point: abstaining is free, being wrong out loud is three
times as costly as being right is valuable.

Three filter configurations are reported side by side, on the same data:
`off` (no class weighting), `combined` (the old filter: relay, Tor and hosting
all penalised in the ranking) and `split` (relay penalised in the ranking; Tor
and hosting not penalised, but reported as an anonymized entry point with a
reduced *attribution* confidence handed to the correlation engine). The split
exists because a masked broadcast genuinely does originate at the Tor exit or
hosting address — penalising it in the ranking pushes the estimator off the
right answer — whereas a Bitnodes-listed relay forwards other people's traffic
and is never the sender.

## `low_confidence_origin`

The flag formerly named `origin_likely_unobserved`. It was renamed because the
name asserted something it cannot know — that the true origin is absent from the
data — while what it actually measures is that this estimate is weak. It is now
called what it is, everywhere: API, alerts, config, docs.

**Cutoff selection rule, fixed here in advance:**

> The confidence cutoff is chosen on **seed A** (`eval.seed`, 41) as the value in
> {0.10, 0.15, … 0.90} maximising the cost-weighted origin score above, under
> the split filter, at the canonical observation rate. Ties go to the lower
> cutoff, which abstains less. The chosen value is then reported **unchanged**
> on **seed B** (`eval.seed_b`, 43), together with the flag's precision, recall
> and the accuracy split (accuracy with the flag clear vs. raised).

Seed B is never used to choose anything. If the seed B numbers are worse than
seed A's, that difference is the honest estimate of the selection bias, and it
gets written down rather than tuned away.

## Datasets

Unchanged from `docs/origin_eval_protocol.md`: the canonical `eval:` block in
`config.yaml` — 200 actors, 1200 transactions, observation rates 0.1 / 0.3 /
0.6, decisions at 0.3. Every table states its dataset (standard or shifted),
its seed and its label definition. The shifted set is the headline wherever
both are shown.

## What would invalidate this

If the actor-level numbers come out worse than the wallet-level ones, that is
the answer, and both go in the report. Any deviation from this document must be
written into `eval/results.md` with its reason.