Tabby-Pretrain

The pretrained PatchTST backbone behind Tabby. It is a encoder only patch transformer trained with a masked-patch objective and a 99-quantile pinball loss, so a single forward pass returns a full predictive distribution.

This repository holds the frozen backbone only — use it for zero-shot forecasting, or as a base to adapt.

Looking for the full model? The post-trained model, backbone and prompt merged into one checkpoint that runs on its own, is published as Tabby. The prompt module on its own is Tabby-Prompt.

Architecture

Parameters	145.84 M (253 tensors)
Layers	20
d_model	768
Attention heads	12 (head dim 64)
MLP hidden	3072 (4× d_model)
Patch size	16
Context window	8192 steps (512 patches)
Quantile levels	99 (0.01 … 0.99)
Dropout	0.1
Additional pretraining	165,000 steps after a 10,000-step initialization checkpoint
Precision	released in fp32; trained in bf16

The quantile head is monotone by construction: it predicts a base level plus cumulative softplus increments, so the 99 levels can never cross.

Files

File	What it is
model.safetensors	backbone weights, 583 MB, fp32
config.json	architecture config
training_args.json	the recorded pretraining hyperparameters (dataset paths redacted)

The published weights are bit-for-bit the model state dict of the original training checkpoint. Only the optimizer state was dropped, which is what takes the file from 1750 MB down to 583 MB. If you need to resume pretraining rather than run inference, the optimizer state is not in this repository.

Usage

The loader in the Tabby code recognises a config.json + model.safetensors directory and loads it with no key renaming:

from huggingface_hub import snapshot_download
from tabby_prompt.model import load_patchtstfm_backbone

path = snapshot_download("paris-noah/Tabby-Pretrain")
backbone, cfg, step = load_patchtstfm_backbone(path, device="cuda", code_dir=PRETRAIN_CODE)

The loader reads a cumulative checkpoint counter from model.safetensors metadata. That counter includes the 10,000-step initialization checkpoint; the training duration reported here is the subsequent 165,000 steps.

To use it zero-shot through the same predict path Tabby uses, wrap it with prompt_len=0:

from tabby_prompt.model import PatchTSTFMPromptCFG, PromptedPatchTSTFM

cfg = PatchTSTFMPromptCFG(prompt_len=0, context_aware=False,
                          prediction_length=96, context_length=8096)
model = PromptedPatchTSTFM(backbone, cfg).to("cuda").eval()
out = model(context=x, prediction_length=96)   # out["quantile_preds"]: (B, 99, H)

Zero-shot reference

Measured through the Tabby evaluation harness at context_length=8096, each variate forecast independently:

Benchmark	MASE	CRPS
GIFT-Eval, 97 configs, max(H, 128) inference	0.7040	0.4827

These are Seasonal-Naive normalised geometric means from the supplied GIFT-Eval evaluation CSV. Updated TIME window-level outputs are being submitted for official leaderboard aggregation; no TIME aggregate is claimed here.

Training

Initialized from a 10,000-step checkpoint, then trained for 165,000 steps in bf16 on a mixture of real and synthetic series:

• real series from BLAST (ratio 0.3) and a GIFT-Eval pretraining pool (ratio 0.3), both domain-balanced;
• synthetic series from a Chronos KernelSynth corpus (ratio 0.3), plus CauKer-V2 (ratio 0.1).

Objective: masked patch modelling with mask_ratio=0.4 in contiguous blocks of 8 patches, plus terminal masking of 0–2 patches, scored with a 99-quantile pinball loss. Intermediate supervision is applied at layer exits [0, 5, 10, 15, 20] (lambda_ds=0.5, lambda_fm=0.1).

Optimizer: AdamW, β=(0.9, 0.95), weight decay 0.1, gradient clip 1.0, WSD schedule with 10,000 warmup and 20,000 decay steps, peak and floor LR both 1e-5, effective batch 1200 (200 × 6 accumulation steps).

Full values are in training_args.json.

License

CC BY-NC 4.0 — non-commercial use only.