RetCast-JEPA (2Mβ9M RetNet2.0 + JEPA Time-Series Forecasters)
Zero-shot time-series foundation models (2.05M, 5.11M, and 9.19M parameters) built from scratch with RetNet2.0 (delta-rule overwritable memory + causal depthwise short convolution + multi-scale retention) and Yann LeCun's JEPA ("predict what the future means, not its raw values").
Code, training scripts, tests, and full benchmark logs live at https://github.com/prateekstark/retcast.
Zero-Shot vs. Zero-Shot Benchmark (12 Held-Out ETT Tasks)
Pretrained strictly on NonETTPretrainCorpus (1,345 real series across 9 non-ETT domains from autogluon/chronos_datasets + UCI solar_AL + multi-rate temporal resampling + 25% synthetic trajectories; 0 ETT files seen during pretraining), and evaluated head-to-head against Google's timesfm-1.0-200m-pytorch on ETTh1, ETTh2, ETTm1, and ETTm2 at horizons $H \in {96, 192, 336}$ (context_len=512, stride=pred_len):
| Model | Params | Mean MSE β | Mean MAE β | Mean WAPE β | Mean NRMSE β | Latency (B=16, L=512) | Peak VRAM |
|---|---|---|---|---|---|---|---|
| Google TimesFM 200M | 203.57M | 0.3878 | 0.3807 | 0.3878 | 0.6273 | 413.49 ms | 796.6 MB |
| RetCast-JEPA 2M (Zero-Shot) | 2.05M | 0.3706 | 0.3800 | 0.3869 | 0.6150 | 17.44 ms | 22.0 MB |
| RetCast-JEPA 5M (Zero-Shot) | 5.11M | 0.3831 | 0.3845 | 0.3913 | 0.6251 | 32.17 ms | 36.7 MB |
| RetCast-JEPA 9M (Zero-Shot) | 9.19M | 0.3799 | 0.3836 | 0.3887 | 0.6190 | 50.31 ms | 55.3 MB |
| Seasonal Naive | 0 | 0.4638 | 0.4089 | 0.4166 | 0.6892 | β | β |
Controlled 2M Zero-Shot Ablations
| Variant (2M Zero-Shot) | Params | Mean MSE β | Mean MAE β | JEPA Cosine β | Eff. Rank β | Retrieval R@1 β |
|---|---|---|---|---|---|---|
| RetNet2.0 + JEPA (2M) | 2,052,180 | 0.3706 | 0.3800 | 0.809 | 17.9 | 0.070 |
| RetNet2.0 Raw (No JEPA) | 2,052,180 | 0.3792 | 0.3819 | 0.030 | 18.8 | 0.013 |
| Plain RetNet + JEPA | 2,047,044 | 0.3985 | 0.3927 | 0.798 | 17.5 | 0.080 |
| Transformer + JEPA | 2,049,604 | 0.4209 | 0.4003 | 0.721 | 16.4 | 0.062 |
Available Variants in This Repository
| Variant | Regime | Backbone | Online Params | Step | SafeTensors Size |
|---|---|---|---|---|---|
zs_retcast_2m_jepa |
zero_shot | retnet (delta) |
2,052,180 | 2,400 | 8.09 MB |
zs_retcast_5m_jepa |
zero_shot | retnet (delta) |
5,105,304 | 2,100 | 20.01 MB |
zs_retcast_9m_jepa |
zero_shot | retnet (delta) |
9,187,292 | 2,100 | 35.87 MB |
zs_ablation_2m_raw |
zero_shot | retnet (delta) |
2,052,180 | 1,800 | 8.09 MB |
zs_ablation_2m_nodelta |
zero_shot | retnet (none) |
2,047,044 | 2,400 | 8.07 MB |
zs_ablation_2m_transformer |
zero_shot | transformer (delta) |
2,049,604 | 2,400 | 8.08 MB |
retcast_2m_jepa |
in_domain | retnet (delta) |
2,049,905 | 1,600 | 8.08 MB |
retcast_5m_jepa |
in_domain | retnet (delta) |
5,101,909 | 1,600 | 20.00 MB |
retcast_9m_jepa |
in_domain | retnet (delta) |
9,183,001 | 1,800 | 35.86 MB |
Convenient aliases supported by RetCastJEPA.from_pretrained:
"2m"/"zs_2m"β"zs_retcast_2m_jepa""5m"/"zs_5m"β"zs_retcast_5m_jepa""9m"/"zs_9m"β"zs_retcast_9m_jepa""indomain_2m"/"indomain_5m"/"indomain_9m"β"retcast_2m_jepa"/"retcast_5m_jepa"/"retcast_9m_jepa"
Quickstart
Clone https://github.com/prateekstark/retcast (or install with pip install -e .), then load any variant directly from HuggingFace Hub:
import torch
from retcast import RetCastJEPA
# Load zero-shot RetCast-JEPA 2M directly from HuggingFace Hub
model = RetCastJEPA.from_pretrained("starkprateek1/retcast-jepa", variant="2m", device="cuda")
# Univariate context batch: shape (B, L)
context = torch.randn(4, 512, device="cuda")
with torch.no_grad():
out = model(context, horizon_len=192, mode="chunkwise")
point = out["point"] # (4, 192)
quantiles = out["quantiles"] # (4, 192, 10) -> [point, q10, q20, ..., q90]
O(1)-Memory Recurrent Latent Rollout
with torch.no_grad():
ctx_norm, stats = model.revin.normalize(context)
patches, pad = model._patchify(ctx_norm)
z_ctx = model.encoder(patches, pad) + model.freq_emb(torch.zeros(4, dtype=torch.long, device="cuda")).unsqueeze(1)
h_ctx, state = model.forward_backbone(z_ctx, mode="chunkwise", return_state=True)
z_rollout, next_state = model.rollout_latent_recurrent(h_ctx, state, offset=patches.shape[1], n_patches=6)
Why RetNet2.0 + JEPA?
- Parallel training + chunkwise prefill + $O(1)$ recurrent state (
830 KBfixed at $L=16,384$ vs30,750 KBTransformer KV cache): 1.85Γ faster and 2.72Γ lower VRAM than a parameter-matched Transformer at $L=16,384$. - Delta-rule overwritable memory ($\beta_{\max}=2.0$) + causal
short_conv=4: Erases stale regime projections before writing new observations and enables negative eigenvalues in $(-1, 1)$ for cycle/phase tracking, reducing zero-shot MSE by -7.0% over plain RetNet (0.3706vs0.3985) and -11.9% over Causal Transformer (0.3706vs0.4209). - JEPA + VICReg: Predicts stop-gradient EMA time-frequency patch semantics rather than raw point noise alone, yielding 0.809β0.842 future latent cosine alignment and 5.4Γ higher future-horizon retrieval R@1.
License
Code and weights are released under the MIT License. Pretraining datasets (autogluon/chronos_datasets, UCI solar_AL) retain their respective upstream licenses.
