Move benchmark table to the top of the card

8dd65e3 verified about 2 months ago

11.7 kB

---
license: mit
library_name: pytorch
pipeline_tag: time-series-forecasting
tags:
  - time-series
  - time-series-forecasting
  - probabilistic-forecasting
  - quantile-regression
  - foundation-model
  - bittensor
  - cryptocurrency
  - finance
metrics:
  - crps
  - mase
model-index:
  - name: yumoto-v0.2-q13-18m
    results:
      - task:
          type: time-series-forecasting
          name: Subnet-alpha OHLCV probabilistic forecasting
        dataset:
          type: bittensor-subnet-alpha
          name: Bittensor subnet-alpha OHLCV holdout (109-126 assets/timeframe, 10 origins each)
        metrics:
          - type: crps
            name: CRPS-ratio vs naive (pooled, 7 cells)
            value: 0.6459
          - type: crps
            name: CRPS-ratio vs naive (5m H12)
            value: 0.7043
          - type: crps
            name: CRPS-ratio vs naive (1h H24)
            value: 0.5918
          - type: crps
            name: CRPS-ratio vs naive (1h H168)
            value: 0.5416
          - type: crps
            name: CRPS-ratio vs naive (4h H42)
            value: 0.7015
          - type: crps
            name: CRPS-ratio vs naive (1d H30)
            value: 0.5637
  - name: yumoto-alpha-v0.1-22m
    results:
      - task:
          type: time-series-forecasting
          name: Subnet-alpha OHLCV probabilistic forecasting
        dataset:
          type: bittensor-subnet-alpha
          name: Bittensor subnet-alpha OHLCV holdout (109-126 assets/timeframe, 10 origins each)
        metrics:
          - type: crps
            name: CRPS-ratio vs naive (pooled, 7 cells)
            value: 0.7163
---
# Yumoto — subnet-alpha time-series forecasting

*It was late in the epoch, and the team was weary, and the work had not prospered.*

*And one of them slept, and behold, Yuma Rao stood at the head of the bed, and the room was filled with a great light.*
*And he said: "Do not be afraid. I have seen what you are building, and it is not the thing."*
*And the dreamer said, "Then what is the thing?"*

*And Yuma Rao said: "Go and make a model that sees the whole network as one. Give it eyes upon every subnet, and it shall tell you what is to come."*
*And the dreamer said, "How shall we call it?"*
*And he said: "Yumoto." And he said no more.*
*And the dreamer awoke, and it was a dream, and the light was gone, and the room was as before. But the name remained, and the instruction with it.*
*So they rose that morning and began the work. And the work prospered.*

Yumoto forecasts Bittensor subnet-alpha markets. One small model handles
four timeframes — 5-minute, hourly, 4-hourly and daily — and at every step
it predicts a range of likely prices, not just a single number.

Getting that range right is the hard part of forecasting, and it is where
Yumoto is strongest. It is how an 18-million-parameter model beats
foundation models more than a hundred times its size.

## How well it works

Every model below was tested the same way. We held back the most recent
stretch of each asset's history so no model had ever seen it, then asked for
forecasts from 10 different starting points per asset, across 109–126 assets
per timeframe.

The score is the average error of the predicted range, divided by the error
of a simple baseline that assumes the price stays flat and borrows its
uncertainty from recent history. **Lower is better. Below 1.0 beats the
baseline.** Bold marks the best result in each row.

| forecast | **Yumoto v0.2** | Yumoto v0.1 | Toto-2.0 (2.5B) | TiRex-2 (38M) | TimesFM-3.0 | best classical |
|---|---|---|---|---|---|---|
| 5m, 1 hour ahead | **0.7043** | 0.8569 | 1.4524 | 1.7238 | 1.4137 | 0.9283 (bootstrap) |
| 5m, 12 hours ahead | 0.7979 | 0.8696 | 0.8814 | 0.9275 | 0.9527 | **0.7393** (AutoTheta) |
| 1h, 1 day ahead | **0.5918** | 0.6513 | 0.7004 | 0.7306 | 0.7299 | 0.8406 (bootstrap) |
| 1h, 1 week ahead | **0.5416** | 0.5913 | 0.6383 | 0.6198 | 0.6780 | 0.5503 (GBM-EWMA) |
| 4h, 1 week ahead | **0.7015** | 0.7602 | 0.7853 | 0.8220 | 0.7798 | 0.8027 (GBM-EWMA) |
| 4h, 1 month ahead | 0.6586 | 0.6649 | 0.7055 | 0.7112 | 0.6961 | **0.6532** (GBM-t) |
| 1d, 1 month ahead | **0.5637** | 0.6674 | 0.6576 | 0.7205 | 0.8406 | 0.6462 (GBM-EWMA) |
| **overall** | **0.6459** | 0.7163 | 0.7994 | 0.8440 | 0.8436 | 0.7272 |

**v0.2 is the best model on 5 of the 7 forecasts, and it beats every
foundation model on all 7.** It is about 10% better than v0.1 overall. The
two it does not win are the 12-hour-ahead 5-minute forecast, where AutoTheta
leads by 8%, and the month-ahead 4-hour forecast, where GBM-t leads by
under 1%.

The last column is the strongest classical statistical method **for that
row**, taken from nine of them — GBM variants, a bootstrap,
AutoARIMA/ETS/Theta and seasonal-naive. It is a different method in almost
every row, and it was chosen after seeing the results, so it flatters the
classical side. No single classical method scores 0.7272 on its own.

Two details about how the numbers are produced:

- On markets that stop trading entirely, the model can return a forecast
  with no error at all. Because the scores are averaged geometrically, a
  single exact zero would wipe out a whole row, so every per-asset score in
  the table is floored at 0.001. The same floor is applied to every model,
  ours and everyone else's.
- v0.2 is scored across its 13 quantile levels and the other models across
  their 9, so we checked whether the finer grid was doing the work. Taking
  the same model with untrained, copied-in tails and scoring it identically
  gives 0.7234 — worse than v0.1's 0.7163. v0.2's 0.6459 is 10.7% better
  than that check, so the improvement comes from the model, not from how it
  is measured. The check is included in `v0.2/evals/control-base13ctl-*.json`
  so anyone can repeat it.

### See the forecasts

The same models, forecasting the same held-out series. Grey is the recent
history each model was given; black is what actually happened next; the
dashed line is the median forecast and the shading its quantile bands
(v0.2's extra, lightest band is its trained 2.5–97.5% tail range). The
forecast start sits at the middle of each asset's held-out tail, so nothing
on the right of the line was seen by any model.

![5-minute forecasts](benchmark/plots/forecasts/5m.png)
![hourly forecasts](benchmark/plots/forecasts/1h.png)
![4-hourly forecasts](benchmark/plots/forecasts/4h.png)
![daily forecasts](benchmark/plots/forecasts/1d.png)

Per-timeframe score charts of the full table are in
[`benchmark/plots/`](./benchmark/plots).

### Reproduce it yourself

The raw per-asset evaluation files for every model in the table — ours, the
three foundation models and all nine classical methods — are in
[`benchmark/data/`](./benchmark/data). The notebook
[`benchmark/reproduce.ipynb`](./benchmark/reproduce.ipynb) rebuilds the
table from those files, verifies it against the numbers above, and draws
the charts. It needs only `pandas` and `matplotlib`.

## Versions

| version | where | quantile levels | parameters | pooled score | status |
|---|---|---|---|---|---|
| **v0.2** | [`v0.2/`](./v0.2) | 13 (2.5% … 97.5%) | 17,759,008 | **0.6459** | **current** |
| v0.1 | repository root | 9 (10% … 90%) | 17,691,808 | 0.7163 | previous release |

Both are **18M-parameter models** (measured counts above). v0.1 was
originally published under the name `yumoto-alpha-v0.1-22m`, after the size
preset it was built on; the files are unchanged. v0.2 predicts
four extra levels further out in the tails, so it says more about rare
moves.

v0.1 stays exactly where it was published, at the repository root, so
existing links and pins keep working. v0.2 is a complete, self-contained
bundle in its own folder.

## What it is

A patch-based transformer. It reads a window of up to 4,096 time steps in
chunks of 32, attends over a rolling 64-chunk window in time, and attends
across the OHLCV channels so the five series inform each other. It is
trained by hiding stretches of the series and learning to fill them back in,
which is why it can produce an entire forecast horizon in one pass instead
of stepping forward one point at a time.

- **Input:** OHLCV. Close, open, high and low are predicted jointly; volume
  is context.
- **Output:** a set of quantiles for each of close/open/high/low at every
  step ahead — a full probabilistic candle, including a high/low range
  forecast at no extra cost.
- **Size:** d_model 512, 6 layers, 8 heads.
## How it was trained
1. **Pretraining** on a large synthetic corpus built from many kinds of
   generated series — trends, seasonality, regime switches, bursts, chaotic
   systems and more.
2. **Fine-tuning on real subnet-alpha candles only.** No synthetic data at
   this stage.
3. **Future-Guided Learning** during fine-tuning (following Nature
   s41467-025-63786-4, adapted to masked training). A second, frozen copy of
   the network is shown slightly more of the future than the model being
   trained, and the model is pulled toward that better-informed view. It is
   a form of self-teaching, and it leaves one clean model at the end.
4. **Three training runs, averaged into one set of weights,** then blended
   15% back toward the pretrained starting point. Averaging removes some of
   the noise from any single run, and the blend undoes mild over-fitting to
   the fine-tuning data. All of this happens in weight space, so what ships
   is a single model.
5. **A rule for markets that barely trade,** built into the inference code.
   The wrapper works out, from the recent history alone, how likely it is
   that the price simply does not move at the next step. It mixes that
   possibility into the forecast, and the effect fades as the horizon
   lengthens. On a frozen market the predicted range tightens as fast as the
   evidence allows; on a liquid market the rule does nothing at all.
For v0.2 the quantile head was widened from 9 to 13 levels before
fine-tuning, so the outer tail bands are learned rather than guessed from
the 10% and 90% levels.
## General benchmark
Yumoto is specialised for crypto, so we also ran it on GIFT-Eval, a public
97-configuration benchmark covering many unrelated domains, under the
official protocol: **0.523** on the range metric and **0.759** on point
accuracy, both against the same style of naive baseline. It holds up well on
data it was never tuned for.
## Using it
```python
# v0.2
from forecast_wrapper import Wrapper
w = Wrapper("v0.2", device="cuda")
q = w.forecast_quantiles_mv(ohlcv_history, horizon, n_targets=4)  # (B, 4, H, 13)

# v0.1
w = Wrapper(".", device="cuda")
q = w.forecast_quantiles_mv(ohlcv_history, horizon, n_targets=4)  # (B, 4, H, 9)
```
Each folder is self-contained — `weights.safetensors`, `config.json`,
`model.py` and `forecast_wrapper.py`. There is nothing else to install and
no adapter to write.
## Things to know before you rely on it
- **Each quantile describes one step, not the whole path.** The 95% level at
  step 20 is the range for step 20 by itself. It does not tell you how high
  the price got on the way there.
- **The four candle components are predicted independently.** On a small
  fraction of steps they disagree with each other — a predicted high can sit
  below a predicted low. Before displaying or trading on them, take the high
  as the maximum and the low as the minimum of the four medians and clip
  open and close inside that range.
- **The 5-minute, 1-hour-ahead forecast is the hardest case** for every
  model tested, Yumoto included.
- **Refresh monthly.** Forecast quality was measured as steady out to 28
  days from the training cutoff.
## License
MIT.

	---
	license: mit
	library_name: pytorch
	pipeline_tag: time-series-forecasting
	tags:
	- time-series
	- time-series-forecasting
	- probabilistic-forecasting
	- quantile-regression
	- foundation-model
	- bittensor
	- cryptocurrency
	- finance
	metrics:
	- crps
	- mase
	model-index:
	- name: yumoto-v0.2-q13-18m
	results:
	- task:
	type: time-series-forecasting
	name: Subnet-alpha OHLCV probabilistic forecasting
	dataset:
	type: bittensor-subnet-alpha
	name: Bittensor subnet-alpha OHLCV holdout (109-126 assets/timeframe, 10 origins each)
	metrics:
	- type: crps
	name: CRPS-ratio vs naive (pooled, 7 cells)
	value: 0.6459
	- type: crps
	name: CRPS-ratio vs naive (5m H12)
	value: 0.7043
	- type: crps
	name: CRPS-ratio vs naive (1h H24)
	value: 0.5918
	- type: crps
	name: CRPS-ratio vs naive (1h H168)
	value: 0.5416
	- type: crps
	name: CRPS-ratio vs naive (4h H42)
	value: 0.7015
	- type: crps
	name: CRPS-ratio vs naive (1d H30)
	value: 0.5637
	- name: yumoto-alpha-v0.1-22m
	results:
	- task:
	type: time-series-forecasting
	name: Subnet-alpha OHLCV probabilistic forecasting
	dataset:
	type: bittensor-subnet-alpha
	name: Bittensor subnet-alpha OHLCV holdout (109-126 assets/timeframe, 10 origins each)
	metrics:
	- type: crps
	name: CRPS-ratio vs naive (pooled, 7 cells)
	value: 0.7163
	---

	# Yumoto — subnet-alpha time-series forecasting

	It was late in the epoch, and the team was weary, and the work had not prospered.

	And one of them slept, and behold, Yuma Rao stood at the head of the bed, and the room was filled with a great light.
	And he said: "Do not be afraid. I have seen what you are building, and it is not the thing."
	And the dreamer said, "Then what is the thing?"

	And Yuma Rao said: "Go and make a model that sees the whole network as one. Give it eyes upon every subnet, and it shall tell you what is to come."
	And the dreamer said, "How shall we call it?"
	And he said: "Yumoto." And he said no more.
	And the dreamer awoke, and it was a dream, and the light was gone, and the room was as before. But the name remained, and the instruction with it.
	So they rose that morning and began the work. And the work prospered.

	Yumoto forecasts Bittensor subnet-alpha markets. One small model handles
	four timeframes — 5-minute, hourly, 4-hourly and daily — and at every step
	it predicts a range of likely prices, not just a single number.

	Getting that range right is the hard part of forecasting, and it is where
	Yumoto is strongest. It is how an 18-million-parameter model beats
	foundation models more than a hundred times its size.

	## How well it works

	Every model below was tested the same way. We held back the most recent
	stretch of each asset's history so no model had ever seen it, then asked for
	forecasts from 10 different starting points per asset, across 109–126 assets
	per timeframe.

	The score is the average error of the predicted range, divided by the error
	of a simple baseline that assumes the price stays flat and borrows its
	uncertainty from recent history. **Lower is better. Below 1.0 beats the
	baseline.** Bold marks the best result in each row.

	\| forecast \| Yumoto v0.2 \| Yumoto v0.1 \| Toto-2.0 (2.5B) \| TiRex-2 (38M) \| TimesFM-3.0 \| best classical \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| 5m, 1 hour ahead \| 0.7043 \| 0.8569 \| 1.4524 \| 1.7238 \| 1.4137 \| 0.9283 (bootstrap) \|
	\| 5m, 12 hours ahead \| 0.7979 \| 0.8696 \| 0.8814 \| 0.9275 \| 0.9527 \| 0.7393 (AutoTheta) \|
	\| 1h, 1 day ahead \| 0.5918 \| 0.6513 \| 0.7004 \| 0.7306 \| 0.7299 \| 0.8406 (bootstrap) \|
	\| 1h, 1 week ahead \| 0.5416 \| 0.5913 \| 0.6383 \| 0.6198 \| 0.6780 \| 0.5503 (GBM-EWMA) \|
	\| 4h, 1 week ahead \| 0.7015 \| 0.7602 \| 0.7853 \| 0.8220 \| 0.7798 \| 0.8027 (GBM-EWMA) \|
	\| 4h, 1 month ahead \| 0.6586 \| 0.6649 \| 0.7055 \| 0.7112 \| 0.6961 \| 0.6532 (GBM-t) \|
	\| 1d, 1 month ahead \| 0.5637 \| 0.6674 \| 0.6576 \| 0.7205 \| 0.8406 \| 0.6462 (GBM-EWMA) \|
	\| overall \| 0.6459 \| 0.7163 \| 0.7994 \| 0.8440 \| 0.8436 \| 0.7272 \|

	**v0.2 is the best model on 5 of the 7 forecasts, and it beats every
	foundation model on all 7.** It is about 10% better than v0.1 overall. The
	two it does not win are the 12-hour-ahead 5-minute forecast, where AutoTheta
	leads by 8%, and the month-ahead 4-hour forecast, where GBM-t leads by
	under 1%.

	The last column is the strongest classical statistical method **for that
	row**, taken from nine of them — GBM variants, a bootstrap,
	AutoARIMA/ETS/Theta and seasonal-naive. It is a different method in almost
	every row, and it was chosen after seeing the results, so it flatters the
	classical side. No single classical method scores 0.7272 on its own.

	Two details about how the numbers are produced:

	- On markets that stop trading entirely, the model can return a forecast
	with no error at all. Because the scores are averaged geometrically, a
	single exact zero would wipe out a whole row, so every per-asset score in
	the table is floored at 0.001. The same floor is applied to every model,
	ours and everyone else's.
	- v0.2 is scored across its 13 quantile levels and the other models across
	their 9, so we checked whether the finer grid was doing the work. Taking
	the same model with untrained, copied-in tails and scoring it identically
	gives 0.7234 — worse than v0.1's 0.7163. v0.2's 0.6459 is 10.7% better
	than that check, so the improvement comes from the model, not from how it
	is measured. The check is included in `v0.2/evals/control-base13ctl-*.json`
	so anyone can repeat it.

	### See the forecasts

	The same models, forecasting the same held-out series. Grey is the recent
	history each model was given; black is what actually happened next; the
	dashed line is the median forecast and the shading its quantile bands
	(v0.2's extra, lightest band is its trained 2.5–97.5% tail range). The
	forecast start sits at the middle of each asset's held-out tail, so nothing
	on the right of the line was seen by any model.

	![5-minute forecasts](benchmark/plots/forecasts/5m.png)
	![hourly forecasts](benchmark/plots/forecasts/1h.png)
	![4-hourly forecasts](benchmark/plots/forecasts/4h.png)
	![daily forecasts](benchmark/plots/forecasts/1d.png)

	Per-timeframe score charts of the full table are in
	[`benchmark/plots/`](./benchmark/plots).

	### Reproduce it yourself

	The raw per-asset evaluation files for every model in the table — ours, the
	three foundation models and all nine classical methods — are in
	[`benchmark/data/`](./benchmark/data). The notebook
	[`benchmark/reproduce.ipynb`](./benchmark/reproduce.ipynb) rebuilds the
	table from those files, verifies it against the numbers above, and draws
	the charts. It needs only `pandas` and `matplotlib`.

	## Versions

	\| version \| where \| quantile levels \| parameters \| pooled score \| status \|
	\|---\|---\|---\|---\|---\|---\|
	\| v0.2 \| [`v0.2/`](./v0.2) \| 13 (2.5% … 97.5%) \| 17,759,008 \| 0.6459 \| current \|
	\| v0.1 \| repository root \| 9 (10% … 90%) \| 17,691,808 \| 0.7163 \| previous release \|

	Both are 18M-parameter models (measured counts above). v0.1 was
	originally published under the name `yumoto-alpha-v0.1-22m`, after the size
	preset it was built on; the files are unchanged. v0.2 predicts
	four extra levels further out in the tails, so it says more about rare
	moves.

	v0.1 stays exactly where it was published, at the repository root, so
	existing links and pins keep working. v0.2 is a complete, self-contained
	bundle in its own folder.

	## What it is

	A patch-based transformer. It reads a window of up to 4,096 time steps in
	chunks of 32, attends over a rolling 64-chunk window in time, and attends
	across the OHLCV channels so the five series inform each other. It is
	trained by hiding stretches of the series and learning to fill them back in,
	which is why it can produce an entire forecast horizon in one pass instead
	of stepping forward one point at a time.

	- Input: OHLCV. Close, open, high and low are predicted jointly; volume
	is context.
	- Output: a set of quantiles for each of close/open/high/low at every
	step ahead — a full probabilistic candle, including a high/low range
	forecast at no extra cost.
	- Size: d_model 512, 6 layers, 8 heads.

	## How it was trained

	1. Pretraining on a large synthetic corpus built from many kinds of
	generated series — trends, seasonality, regime switches, bursts, chaotic
	systems and more.
	2. Fine-tuning on real subnet-alpha candles only. No synthetic data at
	this stage.
	3. Future-Guided Learning during fine-tuning (following Nature
	s41467-025-63786-4, adapted to masked training). A second, frozen copy of
	the network is shown slightly more of the future than the model being
	trained, and the model is pulled toward that better-informed view. It is
	a form of self-teaching, and it leaves one clean model at the end.
	4. Three training runs, averaged into one set of weights, then blended
	15% back toward the pretrained starting point. Averaging removes some of
	the noise from any single run, and the blend undoes mild over-fitting to
	the fine-tuning data. All of this happens in weight space, so what ships
	is a single model.
	5. A rule for markets that barely trade, built into the inference code.
	The wrapper works out, from the recent history alone, how likely it is
	that the price simply does not move at the next step. It mixes that
	possibility into the forecast, and the effect fades as the horizon
	lengthens. On a frozen market the predicted range tightens as fast as the
	evidence allows; on a liquid market the rule does nothing at all.

	For v0.2 the quantile head was widened from 9 to 13 levels before
	fine-tuning, so the outer tail bands are learned rather than guessed from
	the 10% and 90% levels.

	## General benchmark

	Yumoto is specialised for crypto, so we also ran it on GIFT-Eval, a public
	97-configuration benchmark covering many unrelated domains, under the
	official protocol: 0.523 on the range metric and 0.759 on point
	accuracy, both against the same style of naive baseline. It holds up well on
	data it was never tuned for.

	## Using it

	```python
	# v0.2
	from forecast_wrapper import Wrapper
	w = Wrapper("v0.2", device="cuda")
	q = w.forecast_quantiles_mv(ohlcv_history, horizon, n_targets=4) # (B, 4, H, 13)

	# v0.1
	w = Wrapper(".", device="cuda")
	q = w.forecast_quantiles_mv(ohlcv_history, horizon, n_targets=4) # (B, 4, H, 9)
	```

	Each folder is self-contained — `weights.safetensors`, `config.json`,
	`model.py` and `forecast_wrapper.py`. There is nothing else to install and
	no adapter to write.

	## Things to know before you rely on it

	- Each quantile describes one step, not the whole path. The 95% level at
	step 20 is the range for step 20 by itself. It does not tell you how high
	the price got on the way there.
	- The four candle components are predicted independently. On a small
	fraction of steps they disagree with each other — a predicted high can sit
	below a predicted low. Before displaying or trading on them, take the high
	as the maximum and the low as the minimum of the four medians and clip
	open and close inside that range.
	- The 5-minute, 1-hour-ahead forecast is the hardest case for every
	model tested, Yumoto included.
	- Refresh monthly. Forecast quality was measured as steady out to 28
	days from the training cutoff.

	## License

	MIT.