ziyuzhou02 commited on
Commit
9ebdf13
·
verified ·
1 Parent(s): 6c9b2a8

Deploy Space: update src

Browse files
Files changed (2) hide show
  1. src/about.py +23 -15
  2. src/utils.py +2 -1
src/about.py CHANGED
@@ -10,27 +10,35 @@ ranks**. Use the **Overall** tab for aggregate scores; each subsequent tab is on
10
  """
11
 
12
  LLM_BENCHMARKS_TEXT = """
13
- ## How to participate
14
-
15
- 1. Clone [TSFM_Bench](https://github.com/zhouziyu02/TSFM_Bench)
16
- 2. Implement your model against the streaming data interface (`tsfm_bench.data`)
17
- 3. Run evaluation and write `results/<MODEL_NAME>/all_results.csv`
18
- 4. Open a pull request; results sync to this leaderboard
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
19
 
20
  ## Metrics
21
 
22
  - **MASE** — Mean Absolute Scaled Error (absolute)
23
  - **CRPS** — Mean Weighted Sum Quantile Loss (probabilistic forecast quality)
24
  - **MASE_Rank** / **CRPS_Rank** — per-dataset rank (lower is better)
25
-
26
- ## Automated evaluation
27
-
28
- ```bash
29
- export TSFM_API_KEY=your_key
30
- export HF_TOKEN=your_hf_token
31
- python scripts/run_online_daemon.py --once
32
- python scripts/run_online_daemon.py # every 5 min by default
33
- ```
34
  """
35
 
36
  CITATION_BUTTON_LABEL = "Copy citation"
 
10
  """
11
 
12
  LLM_BENCHMARKS_TEXT = """
13
+ ## How to participate (Submit a New Model)
14
+
15
+ To evaluate your model on this leaderboard (run securely on our servers to prevent test data leakage while we absorb the compute costs), you can submit a Pull Request to our repository. Please prepare and submit the following files:
16
+
17
+ 1. **Model Source Code Wrapper**:
18
+ - Create a Python file under `space/user_models/<your_model_id>.py` implementing a predictor class that inherits from `RepresentablePredictor`.
19
+ - Implement the `predict(dataset: GluonDataset) -> Iterator[Forecast]` method conforming to our evaluation input-output interface.
20
+ 2. **Model Weights / Checkpoint**:
21
+ - Provide your pre-trained model weights by either specifying a public Hugging Face model ID, or adding a checkpoint file under `space/user_models/<your_model_id>_checkpoint.pt`.
22
+ 3. **Dependency Environment**:
23
+ - Add any custom libraries your model depends on to the optional packages list in `pyproject.toml` or `space/requirements.txt`.
24
+ 4. **Model Configuration**:
25
+ - Register your model specification in [configs/models/online_tsfm.yaml](file:///Users/bytedance/Desktop/TSFM_Bench/configs/models/online_tsfm.yaml):
26
+ ```yaml
27
+ - model_id: local/<your_model_id>
28
+ display_name: My-Custom-TSFM
29
+ model_type: local
30
+ org: My Organization
31
+ model_class: user_models.<your_model_id>.<ModelClass>
32
+ model_path: space/user_models/<your_model_id>_checkpoint.pt
33
+ ```
34
+
35
+ Once your Pull Request is opened, our automated sandbox pipeline will load your model, run zero-shot evaluations on the live test streams under the exact same conditions, compute the metrics fairly, and update the leaderboard automatically.
36
 
37
  ## Metrics
38
 
39
  - **MASE** — Mean Absolute Scaled Error (absolute)
40
  - **CRPS** — Mean Weighted Sum Quantile Loss (probabilistic forecast quality)
41
  - **MASE_Rank** / **CRPS_Rank** — per-dataset rank (lower is better)
 
 
 
 
 
 
 
 
 
42
  """
43
 
44
  CITATION_BUTTON_LABEL = "Copy citation"
src/utils.py CHANGED
@@ -752,9 +752,10 @@ def make_domain_pie_chart(root_dir: str = "results") -> object | None:
752
  marker=dict(colors=colors),
753
  hole=0.25,
754
  )])
 
755
  fig.update_layout(
756
  title=dict(
757
- text="Evaluated timestamps by domain (today, UTC+8)",
758
  font=dict(size=16, weight="bold"),
759
  ),
760
  margin=dict(l=20, r=20, t=60, b=20),
 
752
  marker=dict(colors=colors),
753
  hole=0.25,
754
  )])
755
+ date_str = today_start.strftime("%Y-%m-%d")
756
  fig.update_layout(
757
  title=dict(
758
+ text=f"Evaluated timestamps by domain ({date_str}, UTC+8)",
759
  font=dict(size=16, weight="bold"),
760
  ),
761
  margin=dict(l=20, r=20, t=60, b=20),