RetCast-JEPA (2M–9M RetNet2.0 + JEPA Time-Series Forecasters)

Zero-shot time-series foundation models (2.05M, 5.11M, and 9.19M parameters) built from scratch with RetNet2.0 (delta-rule overwritable memory + causal depthwise short convolution + multi-scale retention) and Yann LeCun's JEPA ("predict what the future means, not its raw values").

Code, training scripts, tests, and full benchmark logs live at https://github.com/prateekstark/retcast.


Zero-Shot vs. Zero-Shot Benchmark (12 Held-Out ETT Tasks)

Pretrained strictly on NonETTPretrainCorpus (1,345 real series across 9 non-ETT domains from autogluon/chronos_datasets + UCI solar_AL + multi-rate temporal resampling + 25% synthetic trajectories; 0 ETT files seen during pretraining), and evaluated head-to-head against Google's timesfm-1.0-200m-pytorch on ETTh1, ETTh2, ETTm1, and ETTm2 at horizons $H \in {96, 192, 336}$ (context_len=512, stride=pred_len):

Model Params Mean MSE ↓ Mean MAE ↓ Mean WAPE ↓ Mean NRMSE ↓ Latency (B=16, L=512) Peak VRAM
Google TimesFM 200M 203.57M 0.3878 0.3807 0.3878 0.6273 413.49 ms 796.6 MB
RetCast-JEPA 2M (Zero-Shot) 2.05M 0.3706 0.3800 0.3869 0.6150 17.44 ms 22.0 MB
RetCast-JEPA 5M (Zero-Shot) 5.11M 0.3831 0.3845 0.3913 0.6251 32.17 ms 36.7 MB
RetCast-JEPA 9M (Zero-Shot) 9.19M 0.3799 0.3836 0.3887 0.6190 50.31 ms 55.3 MB
Seasonal Naive 0 0.4638 0.4089 0.4166 0.6892 β€” β€”

Zero-shot comparison vs Google TimesFM 200M

Controlled 2M Zero-Shot Ablations

Variant (2M Zero-Shot) Params Mean MSE ↓ Mean MAE ↓ JEPA Cosine ↑ Eff. Rank ↑ Retrieval R@1 ↑
RetNet2.0 + JEPA (2M) 2,052,180 0.3706 0.3800 0.809 17.9 0.070
RetNet2.0 Raw (No JEPA) 2,052,180 0.3792 0.3819 0.030 18.8 0.013
Plain RetNet + JEPA 2,047,044 0.3985 0.3927 0.798 17.5 0.080
Transformer + JEPA 2,049,604 0.4209 0.4003 0.721 16.4 0.062

Available Variants in This Repository

Variant Regime Backbone Online Params Step SafeTensors Size
zs_retcast_2m_jepa zero_shot retnet (delta) 2,052,180 2,400 8.09 MB
zs_retcast_5m_jepa zero_shot retnet (delta) 5,105,304 2,100 20.01 MB
zs_retcast_9m_jepa zero_shot retnet (delta) 9,187,292 2,100 35.87 MB
zs_ablation_2m_raw zero_shot retnet (delta) 2,052,180 1,800 8.09 MB
zs_ablation_2m_nodelta zero_shot retnet (none) 2,047,044 2,400 8.07 MB
zs_ablation_2m_transformer zero_shot transformer (delta) 2,049,604 2,400 8.08 MB
retcast_2m_jepa in_domain retnet (delta) 2,049,905 1,600 8.08 MB
retcast_5m_jepa in_domain retnet (delta) 5,101,909 1,600 20.00 MB
retcast_9m_jepa in_domain retnet (delta) 9,183,001 1,800 35.86 MB

Convenient aliases supported by RetCastJEPA.from_pretrained:

  • "2m" / "zs_2m" β†’ "zs_retcast_2m_jepa"
  • "5m" / "zs_5m" β†’ "zs_retcast_5m_jepa"
  • "9m" / "zs_9m" β†’ "zs_retcast_9m_jepa"
  • "indomain_2m" / "indomain_5m" / "indomain_9m" β†’ "retcast_2m_jepa" / "retcast_5m_jepa" / "retcast_9m_jepa"

Quickstart

Clone https://github.com/prateekstark/retcast (or install with pip install -e .), then load any variant directly from HuggingFace Hub:

import torch
from retcast import RetCastJEPA

# Load zero-shot RetCast-JEPA 2M directly from HuggingFace Hub
model = RetCastJEPA.from_pretrained("starkprateek1/retcast-jepa", variant="2m", device="cuda")

# Univariate context batch: shape (B, L)
context = torch.randn(4, 512, device="cuda")
with torch.no_grad():
    out = model(context, horizon_len=192, mode="chunkwise")

point = out["point"]          # (4, 192)
quantiles = out["quantiles"]  # (4, 192, 10) -> [point, q10, q20, ..., q90]

O(1)-Memory Recurrent Latent Rollout

with torch.no_grad():
    ctx_norm, stats = model.revin.normalize(context)
    patches, pad = model._patchify(ctx_norm)
    z_ctx = model.encoder(patches, pad) + model.freq_emb(torch.zeros(4, dtype=torch.long, device="cuda")).unsqueeze(1)
    h_ctx, state = model.forward_backbone(z_ctx, mode="chunkwise", return_state=True)
    z_rollout, next_state = model.rollout_latent_recurrent(h_ctx, state, offset=patches.shape[1], n_patches=6)

Why RetNet2.0 + JEPA?

  1. Parallel training + chunkwise prefill + $O(1)$ recurrent state (830 KB fixed at $L=16,384$ vs 30,750 KB Transformer KV cache): 1.85Γ— faster and 2.72Γ— lower VRAM than a parameter-matched Transformer at $L=16,384$.
  2. Delta-rule overwritable memory ($\beta_{\max}=2.0$) + causal short_conv=4: Erases stale regime projections before writing new observations and enables negative eigenvalues in $(-1, 1)$ for cycle/phase tracking, reducing zero-shot MSE by -7.0% over plain RetNet (0.3706 vs 0.3985) and -11.9% over Causal Transformer (0.3706 vs 0.4209).
  3. JEPA + VICReg: Predicts stop-gradient EMA time-frequency patch semantics rather than raw point noise alone, yielding 0.809–0.842 future latent cosine alignment and 5.4Γ— higher future-horizon retrieval R@1.

License

Code and weights are released under the MIT License. Pretraining datasets (autogluon/chronos_datasets, UCI solar_AL) retain their respective upstream licenses.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Dataset used to train starkprateek1/retcast-jepa