docs/origin_eval_protocol.md

Origin estimator selection protocol

Pre-registered. Written and committed before the topology fix was made or any post-fix numbers were seen.

Why this exists

The first comparison of origin estimators was run, the result was read, and the default was then changed to match it. That is defensible once, but it is also exactly how a result gets chosen rather than measured — especially now that we are about to change the generator's topology in a way we expect to help the centrality estimators specifically. Deciding the rule in advance removes the opportunity to rationalise whichever number comes out.

The rule

After fixing topology realism, the default estimator is whichever has the best top-1 accuracy at observation rate 0.3. Ties go to first_timestamp.

"Tie" means a difference of less than 1 percentage point in top-1 accuracy on the canonical evaluation dataset. first_timestamp wins ties because it is the simplest estimator and the cheapest to explain to an investigator.

What is fixed in advance

• Dataset: the canonical one from config.yaml's eval: block — seed 41, 200 actors, 1200 transactions. Stated in eval/results.md with every table.
• Decision rate: observation rate 0.3. Rates 0.1 and 0.6 are reported for context but do not decide the default.
• Metric: top-1 accuracy — the estimated origin equals the generator's recorded observed_origin_ip. Not top-3, which is nearly saturated at the ceiling for every estimator and therefore does not discriminate.
• Candidates: first_timestamp, rumor_centrality, timestamp_weighted_centrality, each with the relay/Tor/hosting class weighting applied, as configured.
• Ceiling: reported alongside, as the fraction of multi-hop transactions whose true origin appears anywhere in the observed tree. No estimator can exceed it; an estimator's share of the ceiling is the honest comparison.

What is also reported, but does not decide

• Conditional accuracy: accuracy restricted to transactions where the origin was observed at all. This separates "the estimator is bad" from "the evidence is not there".
• Calibration: reliability curve of stated confidence against correctness, plus a Brier score. An estimator that is wrong but honest about it is more useful than one that is wrong and confident.
• Ablation with and without the relay/Tor/hosting class weighting, to show what that filter is worth.
• The low_confidence_origin flag's precision and recall (renamed from origin_likely_unobserved; its cutoff is now chosen and reported per docs/detection_unit_protocol.md).

What would invalidate this

If the topology fix changes the ranking, that is the answer, not a reason to re-open the rule. If a later change to the generator alters the result again, the rule still applies: re-run, re-read, re-default. Any deviation must be written into eval/results.md with its reason.