TITLE = """ TSFM Realworld Bench Leaderboard """ INTRODUCTION_TEXT = """ **TSFM Realworld Bench** evaluates time series foundation models on **live TS-Bench real-world data** with **zero-shot API inference** via [TSFM.ai](https://tsfm.ai/). Following [GIFT-Eval](https://huggingface.co/spaces/Salesforce/GIFT-Eval), the leaderboard reports **absolute metric values** and **per-dataset ranks**. The Overall tab also reports the cumulative **up-to-now live result** with MSE, RMSE, MAPE, quantile CRPS, RTG, temporal Stability, Kendall Improvement, Average Rank, Win Rate, and Elo. The **GIFT-style Aggregates** tab provides Seasonal-Naive-normalized MSE, CRPS, and mean CRPS rank grouped by actual prediction length, domain, and frequency. Each subsequent domain tab retains the original absolute per-dataset results. """ LLM_BENCHMARKS_TEXT = """ ## How to participate (Submit a New Model) To evaluate your model on this leaderboard (run securely on our servers to prevent test data leakage while we absorb the compute costs), you can submit a Pull Request to our repository. Please prepare and submit the following files: 1. **Model Source Code Wrapper**: - Create a Python file under `space/user_models/.py` implementing a predictor class that inherits from `RepresentablePredictor`. - Implement the `predict(dataset: GluonDataset) -> Iterator[Forecast]` method conforming to our evaluation input-output interface. 2. **Model Weights / Checkpoint**: - Provide your pre-trained model weights by either specifying a public Hugging Face model ID, or adding a checkpoint file under `space/user_models/_checkpoint.pt`. 3. **Dependency Environment**: - Add any custom libraries your model depends on to the optional packages list in `pyproject.toml` or `space/requirements.txt`. 4. **Model Configuration**: - Register your model specification in [configs/models/online_tsfm.yaml](file:///Users/bytedance/Desktop/TSFM_Bench/configs/models/online_tsfm.yaml): ```yaml - model_id: local/ display_name: My-Custom-TSFM model_type: local org: My Organization model_class: user_models.. model_path: space/user_models/_checkpoint.pt ``` Once your Pull Request is opened, our automated sandbox pipeline will load your model, run zero-shot evaluations on the live test streams under the exact same conditions, compute the metrics fairly, and update the leaderboard automatically. ## Metrics - **MSE** — Mean Squared Error on the mean forecast (absolute) - **RMSE** — Root Mean Squared Error on the mean forecast - **MAPE** — Mean Absolute Percentage Error, reported only away from zero - **CRPS** — quantile approximation of the Continuous Ranked Probability Score - **RTG** — normalized real-time MSE gain over causal Seasonal-Naive (higher is better) - **Stability** — standard deviation of release-level MSE (lower is better) - **Improvement** — Kendall trend statistic over release-level MSE (more negative is better) - **Average Rank / Win Rate / Elo** — paired summaries over shared future releases - **RankScore** — Elo-style aggregate from per-dataset MSE and CRPS ranks (higher is better) - **MSE_Rank** / **CRPS_Rank** — per-dataset rank (lower is better) - **Grouped MSE / CRPS** — geometric mean after per-configuration normalization against Seasonal-Naive (lower is better; 1.0 equals the baseline) - **Grouped Rank** — mean per-configuration CRPS rank (lower is better) """ CITATION_BUTTON_LABEL = "Copy citation" CITATION_BUTTON_TEXT = r""" @misc{tsfm_realworld_bench, title={TSFM Realworld Bench: A Benchmark for Time Series Foundation Models}, author={TSFM Realworld Bench Team}, year={2026}, howpublished={\url{https://huggingface.co/spaces/CityMindDev/TSFM-Realworld-Bench}} } """