Spaces:
Running
Running
Deploy Space: update src
Browse files- src/about.py +23 -15
- src/utils.py +2 -1
src/about.py
CHANGED
|
@@ -10,27 +10,35 @@ ranks**. Use the **Overall** tab for aggregate scores; each subsequent tab is on
|
|
| 10 |
"""
|
| 11 |
|
| 12 |
LLM_BENCHMARKS_TEXT = """
|
| 13 |
-
## How to participate
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
|
| 20 |
## Metrics
|
| 21 |
|
| 22 |
- **MASE** — Mean Absolute Scaled Error (absolute)
|
| 23 |
- **CRPS** — Mean Weighted Sum Quantile Loss (probabilistic forecast quality)
|
| 24 |
- **MASE_Rank** / **CRPS_Rank** — per-dataset rank (lower is better)
|
| 25 |
-
|
| 26 |
-
## Automated evaluation
|
| 27 |
-
|
| 28 |
-
```bash
|
| 29 |
-
export TSFM_API_KEY=your_key
|
| 30 |
-
export HF_TOKEN=your_hf_token
|
| 31 |
-
python scripts/run_online_daemon.py --once
|
| 32 |
-
python scripts/run_online_daemon.py # every 5 min by default
|
| 33 |
-
```
|
| 34 |
"""
|
| 35 |
|
| 36 |
CITATION_BUTTON_LABEL = "Copy citation"
|
|
|
|
| 10 |
"""
|
| 11 |
|
| 12 |
LLM_BENCHMARKS_TEXT = """
|
| 13 |
+
## How to participate (Submit a New Model)
|
| 14 |
+
|
| 15 |
+
To evaluate your model on this leaderboard (run securely on our servers to prevent test data leakage while we absorb the compute costs), you can submit a Pull Request to our repository. Please prepare and submit the following files:
|
| 16 |
+
|
| 17 |
+
1. **Model Source Code Wrapper**:
|
| 18 |
+
- Create a Python file under `space/user_models/<your_model_id>.py` implementing a predictor class that inherits from `RepresentablePredictor`.
|
| 19 |
+
- Implement the `predict(dataset: GluonDataset) -> Iterator[Forecast]` method conforming to our evaluation input-output interface.
|
| 20 |
+
2. **Model Weights / Checkpoint**:
|
| 21 |
+
- Provide your pre-trained model weights by either specifying a public Hugging Face model ID, or adding a checkpoint file under `space/user_models/<your_model_id>_checkpoint.pt`.
|
| 22 |
+
3. **Dependency Environment**:
|
| 23 |
+
- Add any custom libraries your model depends on to the optional packages list in `pyproject.toml` or `space/requirements.txt`.
|
| 24 |
+
4. **Model Configuration**:
|
| 25 |
+
- Register your model specification in [configs/models/online_tsfm.yaml](file:///Users/bytedance/Desktop/TSFM_Bench/configs/models/online_tsfm.yaml):
|
| 26 |
+
```yaml
|
| 27 |
+
- model_id: local/<your_model_id>
|
| 28 |
+
display_name: My-Custom-TSFM
|
| 29 |
+
model_type: local
|
| 30 |
+
org: My Organization
|
| 31 |
+
model_class: user_models.<your_model_id>.<ModelClass>
|
| 32 |
+
model_path: space/user_models/<your_model_id>_checkpoint.pt
|
| 33 |
+
```
|
| 34 |
+
|
| 35 |
+
Once your Pull Request is opened, our automated sandbox pipeline will load your model, run zero-shot evaluations on the live test streams under the exact same conditions, compute the metrics fairly, and update the leaderboard automatically.
|
| 36 |
|
| 37 |
## Metrics
|
| 38 |
|
| 39 |
- **MASE** — Mean Absolute Scaled Error (absolute)
|
| 40 |
- **CRPS** — Mean Weighted Sum Quantile Loss (probabilistic forecast quality)
|
| 41 |
- **MASE_Rank** / **CRPS_Rank** — per-dataset rank (lower is better)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 42 |
"""
|
| 43 |
|
| 44 |
CITATION_BUTTON_LABEL = "Copy citation"
|
src/utils.py
CHANGED
|
@@ -752,9 +752,10 @@ def make_domain_pie_chart(root_dir: str = "results") -> object | None:
|
|
| 752 |
marker=dict(colors=colors),
|
| 753 |
hole=0.25,
|
| 754 |
)])
|
|
|
|
| 755 |
fig.update_layout(
|
| 756 |
title=dict(
|
| 757 |
-
text="Evaluated timestamps by domain (
|
| 758 |
font=dict(size=16, weight="bold"),
|
| 759 |
),
|
| 760 |
margin=dict(l=20, r=20, t=60, b=20),
|
|
|
|
| 752 |
marker=dict(colors=colors),
|
| 753 |
hole=0.25,
|
| 754 |
)])
|
| 755 |
+
date_str = today_start.strftime("%Y-%m-%d")
|
| 756 |
fig.update_layout(
|
| 757 |
title=dict(
|
| 758 |
+
text=f"Evaluated timestamps by domain ({date_str}, UTC+8)",
|
| 759 |
font=dict(size=16, weight="bold"),
|
| 760 |
),
|
| 761 |
margin=dict(l=20, r=20, t=60, b=20),
|