---
library_name: timestral
base_model:
- mistralai/Timestral-209M
---
This tiny model is intended for debugging. It is randomly initialized using the configuration adapted from [mistralai/Timestral-209M](https://huggingface.co/mistralai/Timestral-209M).
Structural comparison with `mistralai/Timestral-209M`:
| Feature | Source | Tiny |
|---|---:|---:|
| Forecasting interface | univariate, multivariate, past/known covariates | same |
| Stored precision | F32 | F32 |
| Patch size / block size | 32 / 32 | same |
| Transformer layers | 24 | 8; two 3-time + 1-time/variate cycles |
| Width / attention heads / head dimension | 768 / 12 / 64 | 64 / 1 / 64 |
| SwiGLU intermediate width | 2,048 | 128 |
| Quantile knots | 21 | same |
| DHF two-stream decoder and monotone quantile head | present | same |
This checkpoint uses Mistral's public `timestral` inference package (release
commit `a534bdbd244d64f3d9d743f46c7bf6fc330ed4ff`): install it with
`pip install git+https://github.com/mistralai/timestral.git` before running the
example. It is a forecasting model, not a Transformers causal-LM or SGLang/vLLM
serving model.
| File path | Size |
|------|------|
| model.safetensors | 2.7MB |
### Example usage:
- Transformers (✅ verified)
```python
import torch
from timestral import TimestralForecaster
model_id = "tiny-random/timestral"
device = "cuda" if torch.cuda.is_available() else "cpu"
forecaster = TimestralForecaster.from_pretrained(model_id, device=device, per_shot_horizon_patches=32)
context = torch.sin(torch.arange(96, dtype=torch.float32) / 6).reshape(1, 1, -1)
past = torch.cos(torch.arange(96, dtype=torch.float32) / 5).reshape(1, 1, -1)
known = torch.linspace(0, 1, 128, dtype=torch.float32).reshape(1, 1, -1)
forecast = forecaster.forecast(context, prediction_length=32, past_covariates=past, known_covariates=known)
print(forecast.quantiles.shape)
```
### Codes to create this repo (✅ verified):
Click to expand
```python
import json
from pathlib import Path
from shutil import copyfile
import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import save_file
from timestral import TimestralConfig, TimestralModel
source_model_id = "mistralai/Timestral-209M"
save_folder = "/tmp/tiny-random/timestral"
source_revision = "9874d252ab0aae16f7f7c999f98f98d6ad7d8365"
torch.manual_seed(42)
folder = Path(save_folder)
folder.mkdir(parents=True, exist_ok=True)
raw = json.loads(Path(hf_hub_download(source_model_id, "config.json", revision=source_revision)).read_text())
# Keep two source 3-time + 1-time/variate cycles. A 64-wide model with one
# head preserves Timestral's 64-wide attention head exactly.
raw.update(
d_model=64, num_heads=1, mlp_hidden_dim=128, context_length=1024,
block_schedule=["time", "time", "time", "time_variate"] * 2,
)
(folder / "config.json").write_text(json.dumps(raw, indent=2) + "\n", encoding="utf-8")
config = TimestralConfig.load(folder / "config.json")
model = TimestralModel(config).float()
with torch.no_grad():
for name, parameter in sorted(model.named_parameters()):
if name.endswith("scale"):
parameter.fill_(1)
else:
torch.nn.init.normal_(parameter, mean=0, std=0.02)
save_file(model.state_dict(), folder / "model.safetensors", metadata={"format": "pt"})
for filename in ("LICENSE", "timestral-hourglass.png"):
try:
copyfile(hf_hub_download(source_model_id, filename, revision=source_revision), folder / filename)
except Exception:
# The executable checkpoint consists solely of config.json and model.safetensors.
# Preserve optional presentation assets when their source revision supplies them.
pass
```
Click to expand
```text
TimestralModel(
(representation): PatchRepresentation(
(patch_proj): ResidualBlock(
(hidden_layer): Linear(in_features=64, out_features=256, bias=True)
(output_layer): Linear(in_features=256, out_features=64, bias=True)
(residual_layer): Linear(in_features=64, out_features=64, bias=True)
(activation): SiLU()
)
)
(transformer): Transformer(
(layers): ModuleList(
(0-2): 3 x TransformerBlock(
(attn_norms): ModuleList(
(0): RMSNorm()
)
(attns): ModuleList(
(0): Attention(
(wQKV): Linear(in_features=64, out_features=192, bias=False)
(wO): Linear(in_features=64, out_features=64, bias=False)
(_gate): Linear(in_features=64, out_features=64, bias=False)
)
)
(mlp_norm): RMSNorm()
(mlp): FusedSwiGLU(
(w12): Linear(in_features=64, out_features=256, bias=False)
(w3): Linear(in_features=128, out_features=64, bias=False)
)
)
(3): TransformerBlock(
(attn_norms): ModuleList(
(0-1): 2 x RMSNorm()
)
(attns): ModuleList(
(0-1): 2 x Attention(
(wQKV): Linear(in_features=64, out_features=192, bias=False)
(wO): Linear(in_features=64, out_features=64, bias=False)
(_gate): Linear(in_features=64, out_features=64, bias=False)
)
)
(mlp_norm): RMSNorm()
(mlp): FusedSwiGLU(
(w12): Linear(in_features=64, out_features=256, bias=False)
(w3): Linear(in_features=128, out_features=64, bias=False)
)
)
(4-6): 3 x TransformerBlock(
(attn_norms): ModuleList(
(0): RMSNorm()
)
(attns): ModuleList(
(0): Attention(
(wQKV): Linear(in_features=64, out_features=192, bias=False)
(wO): Linear(in_features=64, out_features=64, bias=False)
(_gate): Linear(in_features=64, out_features=64, bias=False)
)
)
(mlp_norm): RMSNorm()
(mlp): FusedSwiGLU(
(w12): Linear(in_features=64, out_features=256, bias=False)
(w3): Linear(in_features=128, out_features=64, bias=False)
)
)
(7): TransformerBlock(
(attn_norms): ModuleList(
(0-1): 2 x RMSNorm()
)
(attns): ModuleList(
(0-1): 2 x Attention(
(wQKV): Linear(in_features=64, out_features=192, bias=False)
(wO): Linear(in_features=64, out_features=64, bias=False)
(_gate): Linear(in_features=64, out_features=64, bias=False)
)
)
(mlp_norm): RMSNorm()
(mlp): FusedSwiGLU(
(w12): Linear(in_features=64, out_features=256, bias=False)
(w3): Linear(in_features=128, out_features=64, bias=False)
)
)
)
(rope): SinCosRoPE()
(out_norm): RMSNorm()
)
(head): QuantileHead(
(projection): ResidualBlock(
(hidden_layer): Linear(in_features=64, out_features=256, bias=False)
(output_layer): Linear(in_features=256, out_features=672, bias=False)
(residual_layer): Linear(in_features=64, out_features=672, bias=False)
(activation): SiLU()
)
)
)
```