Spaces:
Runtime error
Runtime error
Download src/about.py from ThinkcatLab/LiveHouse-TS: direct link, hf CLI and curl.
- Browser
- Download file 3.89 kB
-
https://huggingface.co/spaces/ThinkcatLab/LiveHouse-TS/resolve/bbcfda6992f05f2d8660b3fa2139dd2a5208409e/src/about.py
- Command line
-
hf download hf://spaces/ThinkcatLab/LiveHouse-TS@bbcfda6992f05f2d8660b3fa2139dd2a5208409e/src/about.py
-
curl -L -o about.py https://huggingface.co/spaces/ThinkcatLab/LiveHouse-TS/resolve/bbcfda6992f05f2d8660b3fa2139dd2a5208409e/src/about.py
3.89 kB
| TITLE = """ TSFM Realworld Bench Leaderboard """ | |
| INTRODUCTION_TEXT = """ | |
| **TSFM Realworld Bench** evaluates time series foundation models on **live TS-Bench | |
| real-world data** with **zero-shot API inference** via [TSFM.ai](https://tsfm.ai/). | |
| Following [GIFT-Eval](https://huggingface.co/spaces/Salesforce/GIFT-Eval), the leaderboard | |
| reports **absolute metric values** and **per-dataset ranks**. The Overall tab also reports | |
| the cumulative **up-to-now live result** with MSE, RMSE, MAPE, quantile CRPS, RTG, | |
| temporal Stability, Kendall Improvement, Average Rank, Win Rate, and Elo. The | |
| **GIFT-style Aggregates** tab provides Seasonal-Naive-normalized MSE, CRPS, | |
| and mean CRPS rank grouped by actual prediction length, domain, and frequency. Each | |
| subsequent domain tab retains the original absolute per-dataset results. | |
| """ | |
| LLM_BENCHMARKS_TEXT = """ | |
| ## How to participate (Submit a New Model) | |
| To evaluate your model on this leaderboard (run securely on our servers to prevent test data leakage while we absorb the compute costs), you can submit a Pull Request to our repository. Please prepare and submit the following files: | |
| 1. **Model Source Code Wrapper**: | |
| - Create a Python file under `space/user_models/<your_model_id>.py` implementing a predictor class that inherits from `RepresentablePredictor`. | |
| - Implement the `predict(dataset: GluonDataset) -> Iterator[Forecast]` method conforming to our evaluation input-output interface. | |
| 2. **Model Weights / Checkpoint**: | |
| - Provide your pre-trained model weights by either specifying a public Hugging Face model ID, or adding a checkpoint file under `space/user_models/<your_model_id>_checkpoint.pt`. | |
| 3. **Dependency Environment**: | |
| - Add any custom libraries your model depends on to the optional packages list in `pyproject.toml` or `space/requirements.txt`. | |
| 4. **Model Configuration**: | |
| - Register your model specification in [configs/models/online_tsfm.yaml](file:///Users/bytedance/Desktop/TSFM_Bench/configs/models/online_tsfm.yaml): | |
| ```yaml | |
| - model_id: local/<your_model_id> | |
| display_name: My-Custom-TSFM | |
| model_type: local | |
| org: My Organization | |
| model_class: user_models.<your_model_id>.<ModelClass> | |
| model_path: space/user_models/<your_model_id>_checkpoint.pt | |
| ``` | |
| Once your Pull Request is opened, our automated sandbox pipeline will load your model, run zero-shot evaluations on the live test streams under the exact same conditions, compute the metrics fairly, and update the leaderboard automatically. | |
| ## Metrics | |
| - **MSE** β Mean Squared Error on the mean forecast (absolute) | |
| - **RMSE** β Root Mean Squared Error on the mean forecast | |
| - **MAPE** β Mean Absolute Percentage Error, reported only away from zero | |
| - **CRPS** β quantile approximation of the Continuous Ranked Probability Score | |
| - **RTG** β normalized real-time MSE gain over causal Seasonal-Naive (higher is better) | |
| - **Stability** β standard deviation of release-level MSE (lower is better) | |
| - **Improvement** β Kendall trend statistic over release-level MSE (more negative is better) | |
| - **Average Rank / Win Rate / Elo** β paired summaries over shared future releases | |
| - **RankScore** β Elo-style aggregate from per-dataset MSE and CRPS ranks (higher is better) | |
| - **MSE_Rank** / **CRPS_Rank** β per-dataset rank (lower is better) | |
| - **Grouped MSE / CRPS** β geometric mean after per-configuration normalization against | |
| Seasonal-Naive (lower is better; 1.0 equals the baseline) | |
| - **Grouped Rank** β mean per-configuration CRPS rank (lower is better) | |
| """ | |
| CITATION_BUTTON_LABEL = "Copy citation" | |
| CITATION_BUTTON_TEXT = r""" | |
| @misc{tsfm_realworld_bench, | |
| title={TSFM Realworld Bench: A Benchmark for Time Series Foundation Models}, | |
| author={TSFM Realworld Bench Team}, | |
| year={2026}, | |
| howpublished={\url{https://huggingface.co/spaces/CityMindDev/TSFM-Realworld-Bench}} | |
| } | |
| """ | |