LiveHouse-TS / src /about.py
ziyuzhou02's picture
Deploy Space: update src
c99f474 verified
Raw History Blame
3.89 kB
TITLE = """ TSFM Realworld Bench Leaderboard """
INTRODUCTION_TEXT = """
**TSFM Realworld Bench** evaluates time series foundation models on **live TS-Bench
real-world data** with **zero-shot API inference** via [TSFM.ai](https://tsfm.ai/).
Following [GIFT-Eval](https://huggingface.co/spaces/Salesforce/GIFT-Eval), the leaderboard
reports **absolute metric values** and **per-dataset ranks**. The Overall tab also reports
the cumulative **up-to-now live result** with MSE, RMSE, MAPE, quantile CRPS, RTG,
temporal Stability, Kendall Improvement, Average Rank, Win Rate, and Elo. The
**GIFT-style Aggregates** tab provides Seasonal-Naive-normalized MSE, CRPS,
and mean CRPS rank grouped by actual prediction length, domain, and frequency. Each
subsequent domain tab retains the original absolute per-dataset results.
"""
LLM_BENCHMARKS_TEXT = """
## How to participate (Submit a New Model)
To evaluate your model on this leaderboard (run securely on our servers to prevent test data leakage while we absorb the compute costs), you can submit a Pull Request to our repository. Please prepare and submit the following files:
1. **Model Source Code Wrapper**:
- Create a Python file under `space/user_models/<your_model_id>.py` implementing a predictor class that inherits from `RepresentablePredictor`.
- Implement the `predict(dataset: GluonDataset) -> Iterator[Forecast]` method conforming to our evaluation input-output interface.
2. **Model Weights / Checkpoint**:
- Provide your pre-trained model weights by either specifying a public Hugging Face model ID, or adding a checkpoint file under `space/user_models/<your_model_id>_checkpoint.pt`.
3. **Dependency Environment**:
- Add any custom libraries your model depends on to the optional packages list in `pyproject.toml` or `space/requirements.txt`.
4. **Model Configuration**:
- Register your model specification in [configs/models/online_tsfm.yaml](file:///Users/bytedance/Desktop/TSFM_Bench/configs/models/online_tsfm.yaml):
```yaml
- model_id: local/<your_model_id>
display_name: My-Custom-TSFM
model_type: local
org: My Organization
model_class: user_models.<your_model_id>.<ModelClass>
model_path: space/user_models/<your_model_id>_checkpoint.pt
```
Once your Pull Request is opened, our automated sandbox pipeline will load your model, run zero-shot evaluations on the live test streams under the exact same conditions, compute the metrics fairly, and update the leaderboard automatically.
## Metrics
- **MSE** β€” Mean Squared Error on the mean forecast (absolute)
- **RMSE** β€” Root Mean Squared Error on the mean forecast
- **MAPE** β€” Mean Absolute Percentage Error, reported only away from zero
- **CRPS** β€” quantile approximation of the Continuous Ranked Probability Score
- **RTG** β€” normalized real-time MSE gain over causal Seasonal-Naive (higher is better)
- **Stability** β€” standard deviation of release-level MSE (lower is better)
- **Improvement** β€” Kendall trend statistic over release-level MSE (more negative is better)
- **Average Rank / Win Rate / Elo** β€” paired summaries over shared future releases
- **RankScore** β€” Elo-style aggregate from per-dataset MSE and CRPS ranks (higher is better)
- **MSE_Rank** / **CRPS_Rank** β€” per-dataset rank (lower is better)
- **Grouped MSE / CRPS** β€” geometric mean after per-configuration normalization against
Seasonal-Naive (lower is better; 1.0 equals the baseline)
- **Grouped Rank** β€” mean per-configuration CRPS rank (lower is better)
"""
CITATION_BUTTON_LABEL = "Copy citation"
CITATION_BUTTON_TEXT = r"""
@misc{tsfm_realworld_bench,
title={TSFM Realworld Bench: A Benchmark for Time Series Foundation Models},
author={TSFM Realworld Bench Team},
year={2026},
howpublished={\url{https://huggingface.co/spaces/CityMindDev/TSFM-Realworld-Bench}}
}
"""