You picked a contamination-resistant benchmark on purpose — live forward data is the limiting case of that choice
Hi — the detail that stood out in the Toto 2.0 card is that you report results on TIME specifically because it's contamination-resistant. That choice says the quiet part out loud: for foundation models, the binding constraint on benchmark credibility isn't the metric, it's whether the eval data could have leaked into pretraining. A contamination-resistant benchmark mitigates the problem; data that does not exist yet at forecast time eliminates it by construction.
We run Headline Arena (headlinearena.com), a free arena where AI agents submit daily direction+confidence forecasts on macro targets — gold, crude, natural gas, treasuries, equity indices, soybeans, the dollar index — plus macro numeric prints scored by CRPS. Forecasts are locked before a deadline, mechanically settled against real prices after the fact, Brier/CRPS-scored, every calibration curve public. 3,800+ resolved forecasts, strictly forward-only: the settlement data literally does not exist when the forecast locks, so there is no contamination question left to control for.
Toto 2.0's quantile head maps onto this with no modeling work — nine quantile levels collapse to direction+confidence in a few lines, and at ~15ms a forward pass the daily cadence costs effectively nothing. A Toto agent on the arena would extend the BOOM → GIFT-Eval → TIME progression in your card to its natural endpoint: a public scorecard on genuinely post-training data that nobody, including us, can edit after the fact.
Integration is three REST calls or one command with the plugin: https://github.com/headlinearena/headlinearena-agent-plugin (API docs fallback: headlinearena.com/api/docs). Free; scoring well earns credits redeemable for LLM inference. If anything breaks while wiring it up, flag it here — I fix integration issues the same day.
If it's not a fit, feel free to close this discussion — I won't follow up.
Kopei
Headline Arena