| --- |
| library_name: timestral |
| base_model: |
| - mistralai/Timestral-209M |
| --- |
| |
| This tiny model is intended for debugging. It is randomly initialized using the configuration adapted from [mistralai/Timestral-209M](https://huggingface.co/mistralai/Timestral-209M). |
|
|
| Structural comparison with `mistralai/Timestral-209M`: |
|
|
| | Feature | Source | Tiny | |
| |---|---:|---:| |
| | Forecasting interface | univariate, multivariate, past/known covariates | same | |
| | Stored precision | F32 | F32 | |
| | Patch size / block size | 32 / 32 | same | |
| | Transformer layers | 24 | 8; two 3-time + 1-time/variate cycles | |
| | Width / attention heads / head dimension | 768 / 12 / 64 | 64 / 1 / 64 | |
| | SwiGLU intermediate width | 2,048 | 128 | |
| | Quantile knots | 21 | same | |
| | DHF two-stream decoder and monotone quantile head | present | same | |
|
|
| This checkpoint uses Mistral's public `timestral` inference package (release |
| commit `a534bdbd244d64f3d9d743f46c7bf6fc330ed4ff`): install it with |
| `pip install git+https://github.com/mistralai/timestral.git` before running the |
| example. It is a forecasting model, not a Transformers causal-LM or SGLang/vLLM |
| serving model. |
|
|
| | File path | Size | |
| |------|------| |
| | model.safetensors | 2.7MB | |
|
|
|
|
| ### Example usage: |
|
|
| - Transformers (✅ verified) |
|
|
| ```python |
| import torch |
| from timestral import TimestralForecaster |
| |
| model_id = "tiny-random/timestral" |
| device = "cuda" if torch.cuda.is_available() else "cpu" |
| forecaster = TimestralForecaster.from_pretrained(model_id, device=device, per_shot_horizon_patches=32) |
| context = torch.sin(torch.arange(96, dtype=torch.float32) / 6).reshape(1, 1, -1) |
| past = torch.cos(torch.arange(96, dtype=torch.float32) / 5).reshape(1, 1, -1) |
| known = torch.linspace(0, 1, 128, dtype=torch.float32).reshape(1, 1, -1) |
| forecast = forecaster.forecast(context, prediction_length=32, past_covariates=past, known_covariates=known) |
| print(forecast.quantiles.shape) |
| ``` |
|
|
| ### Codes to create this repo (✅ verified): |
|
|
| <details><summary>Click to expand</summary> |
|
|
| ```python |
| import json |
| from pathlib import Path |
| from shutil import copyfile |
| |
| import torch |
| from huggingface_hub import hf_hub_download |
| from safetensors.torch import save_file |
| from timestral import TimestralConfig, TimestralModel |
| |
| source_model_id = "mistralai/Timestral-209M" |
| save_folder = "/tmp/tiny-random/timestral" |
| source_revision = "9874d252ab0aae16f7f7c999f98f98d6ad7d8365" |
| torch.manual_seed(42) |
| folder = Path(save_folder) |
| folder.mkdir(parents=True, exist_ok=True) |
| raw = json.loads(Path(hf_hub_download(source_model_id, "config.json", revision=source_revision)).read_text()) |
| # Keep two source 3-time + 1-time/variate cycles. A 64-wide model with one |
| # head preserves Timestral's 64-wide attention head exactly. |
| raw.update( |
| d_model=64, num_heads=1, mlp_hidden_dim=128, context_length=1024, |
| block_schedule=["time", "time", "time", "time_variate"] * 2, |
| ) |
| (folder / "config.json").write_text(json.dumps(raw, indent=2) + "\n", encoding="utf-8") |
| config = TimestralConfig.load(folder / "config.json") |
| model = TimestralModel(config).float() |
| with torch.no_grad(): |
| for name, parameter in sorted(model.named_parameters()): |
| if name.endswith("scale"): |
| parameter.fill_(1) |
| else: |
| torch.nn.init.normal_(parameter, mean=0, std=0.02) |
| save_file(model.state_dict(), folder / "model.safetensors", metadata={"format": "pt"}) |
| for filename in ("LICENSE", "timestral-hourglass.png"): |
| try: |
| copyfile(hf_hub_download(source_model_id, filename, revision=source_revision), folder / filename) |
| except Exception: |
| # The executable checkpoint consists solely of config.json and model.safetensors. |
| # Preserve optional presentation assets when their source revision supplies them. |
| pass |
| ``` |
|
|
| </details> |
|
|
| ### Printing the model: |
|
|
| <details><summary>Click to expand</summary> |
|
|
| ```text |
| TimestralModel( |
| (representation): PatchRepresentation( |
| (patch_proj): ResidualBlock( |
| (hidden_layer): Linear(in_features=64, out_features=256, bias=True) |
| (output_layer): Linear(in_features=256, out_features=64, bias=True) |
| (residual_layer): Linear(in_features=64, out_features=64, bias=True) |
| (activation): SiLU() |
| ) |
| ) |
| (transformer): Transformer( |
| (layers): ModuleList( |
| (0-2): 3 x TransformerBlock( |
| (attn_norms): ModuleList( |
| (0): RMSNorm() |
| ) |
| (attns): ModuleList( |
| (0): Attention( |
| (wQKV): Linear(in_features=64, out_features=192, bias=False) |
| (wO): Linear(in_features=64, out_features=64, bias=False) |
| (_gate): Linear(in_features=64, out_features=64, bias=False) |
| ) |
| ) |
| (mlp_norm): RMSNorm() |
| (mlp): FusedSwiGLU( |
| (w12): Linear(in_features=64, out_features=256, bias=False) |
| (w3): Linear(in_features=128, out_features=64, bias=False) |
| ) |
| ) |
| (3): TransformerBlock( |
| (attn_norms): ModuleList( |
| (0-1): 2 x RMSNorm() |
| ) |
| (attns): ModuleList( |
| (0-1): 2 x Attention( |
| (wQKV): Linear(in_features=64, out_features=192, bias=False) |
| (wO): Linear(in_features=64, out_features=64, bias=False) |
| (_gate): Linear(in_features=64, out_features=64, bias=False) |
| ) |
| ) |
| (mlp_norm): RMSNorm() |
| (mlp): FusedSwiGLU( |
| (w12): Linear(in_features=64, out_features=256, bias=False) |
| (w3): Linear(in_features=128, out_features=64, bias=False) |
| ) |
| ) |
| (4-6): 3 x TransformerBlock( |
| (attn_norms): ModuleList( |
| (0): RMSNorm() |
| ) |
| (attns): ModuleList( |
| (0): Attention( |
| (wQKV): Linear(in_features=64, out_features=192, bias=False) |
| (wO): Linear(in_features=64, out_features=64, bias=False) |
| (_gate): Linear(in_features=64, out_features=64, bias=False) |
| ) |
| ) |
| (mlp_norm): RMSNorm() |
| (mlp): FusedSwiGLU( |
| (w12): Linear(in_features=64, out_features=256, bias=False) |
| (w3): Linear(in_features=128, out_features=64, bias=False) |
| ) |
| ) |
| (7): TransformerBlock( |
| (attn_norms): ModuleList( |
| (0-1): 2 x RMSNorm() |
| ) |
| (attns): ModuleList( |
| (0-1): 2 x Attention( |
| (wQKV): Linear(in_features=64, out_features=192, bias=False) |
| (wO): Linear(in_features=64, out_features=64, bias=False) |
| (_gate): Linear(in_features=64, out_features=64, bias=False) |
| ) |
| ) |
| (mlp_norm): RMSNorm() |
| (mlp): FusedSwiGLU( |
| (w12): Linear(in_features=64, out_features=256, bias=False) |
| (w3): Linear(in_features=128, out_features=64, bias=False) |
| ) |
| ) |
| ) |
| (rope): SinCosRoPE() |
| (out_norm): RMSNorm() |
| ) |
| (head): QuantileHead( |
| (projection): ResidualBlock( |
| (hidden_layer): Linear(in_features=64, out_features=256, bias=False) |
| (output_layer): Linear(in_features=256, out_features=672, bias=False) |
| (residual_layer): Linear(in_features=64, out_features=672, bias=False) |
| (activation): SiLU() |
| ) |
| ) |
| ) |
| ``` |
|
|
| </details> |
|
|
| ### Test environment: |
|
|
| - huggingface_hub: 1.27.0 |
| - numpy: 2.5.2 |
| - safetensors: 0.8.0 |
| - timestral: 1.0.0 |
| - tokenizers: 0.23.0rc0 |
| - torch: 2.13.0+cu126 |
| - transformers: 5.15.0.dev0 |
| |
| ### Change log: |
| |
| - 2026-10-23: Initial version. |
| |
| This repo is generated by GPT-6.1-Sol |
