File size: 2,951 Bytes
824aa03 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 | # Origin estimator selection protocol
**Pre-registered. Written and committed before the topology fix was made or any
post-fix numbers were seen.**
## Why this exists
The first comparison of origin estimators was run, the result was read, and the
default was then changed to match it. That is defensible once, but it is also
exactly how a result gets chosen rather than measured — especially now that we
are about to change the generator's topology in a way we expect to help the
centrality estimators specifically. Deciding the rule in advance removes the
opportunity to rationalise whichever number comes out.
## The rule
> After fixing topology realism, the default estimator is whichever has the best
> top-1 accuracy at observation rate 0.3. Ties go to `first_timestamp`.
"Tie" means a difference of less than 1 percentage point in top-1 accuracy on
the canonical evaluation dataset. `first_timestamp` wins ties because it is the
simplest estimator and the cheapest to explain to an investigator.
## What is fixed in advance
- **Dataset**: the canonical one from `config.yaml`'s `eval:` block — seed 41,
200 actors, 1200 transactions. Stated in `eval/results.md` with every table.
- **Decision rate**: observation rate 0.3. Rates 0.1 and 0.6 are reported for
context but do not decide the default.
- **Metric**: top-1 accuracy — the estimated origin equals the generator's
recorded `observed_origin_ip`. Not top-3, which is nearly saturated at the
ceiling for every estimator and therefore does not discriminate.
- **Candidates**: `first_timestamp`, `rumor_centrality`,
`timestamp_weighted_centrality`, each with the relay/Tor/hosting class
weighting applied, as configured.
- **Ceiling**: reported alongside, as the fraction of multi-hop transactions
whose true origin appears anywhere in the observed tree. No estimator can
exceed it; an estimator's share of the ceiling is the honest comparison.
## What is also reported, but does not decide
- Conditional accuracy: accuracy restricted to transactions where the origin
was observed at all. This separates "the estimator is bad" from "the evidence
is not there".
- Calibration: reliability curve of stated confidence against correctness, plus
a Brier score. An estimator that is wrong but honest about it is more useful
than one that is wrong and confident.
- Ablation with and without the relay/Tor/hosting class weighting, to show what
that filter is worth.
- The `low_confidence_origin` flag's precision and recall (renamed from
`origin_likely_unobserved`; its cutoff is now chosen and reported per
`docs/detection_unit_protocol.md`).
## What would invalidate this
If the topology fix changes the ranking, that is the answer, not a reason to
re-open the rule. If a later change to the generator alters the result again,
the rule still applies: re-run, re-read, re-default. Any deviation must be
written into `eval/results.md` with its reason.
|