ziyuzhou02 commited on
Commit
66928b3
Β·
verified Β·
1 Parent(s): 89292ae

Fix Space startup: sync full app source

Browse files
Files changed (1) hide show
  1. src/about.py +22 -50
src/about.py CHANGED
@@ -1,66 +1,38 @@
1
  TITLE = """ TSFM Realworld Bench Leaderboard """
2
 
3
  INTRODUCTION_TEXT = """
4
- **TSFM Realworld Bench** evaluates time series foundation models on **live TS-Bench
5
- data** with **zero-shot API inference** via [TSFM.ai](https://tsfm.ai/).
6
-
7
- This Space is **UI-only**: it displays leaderboard results pushed from a local runner
8
- (or GitHub Actions). Evaluations are **not** run inside the container, so the page loads
9
- instantly.
10
  """
11
 
12
  LLM_BENCHMARKS_TEXT = """
13
- ## Architecture
14
 
15
- | Component | Role |
16
- |---|---|
17
- | **HF Space** | Gradio leaderboard UI (reads `results/` CSVs) |
18
- | **Local runner** | Every hour: TS-Bench collect β†’ TSFM.ai eval β†’ push to HF |
19
- | **GitHub Actions** | Optional hourly CI eval + upload (same as local runner) |
20
 
21
- ### Local runner (recommended)
22
 
23
- ```bash
24
- # .env needs TSFM_API_KEY and HF_TOKEN
25
- cd /path/to/TSFM_Bench
26
- source .venv/bin/activate
27
- python scripts/run_online_daemon.py # loop: eval + push every 1 h
28
- python scripts/run_online_daemon.py --once # single cycle
29
- ```
30
 
31
- Or from `space/`:
 
 
 
 
32
 
33
  ```bash
34
- python scripts/run_online_daemon.py --once
 
 
35
  ```
36
-
37
- ### HF Space settings
38
-
39
- No secrets required for display-only mode. Optionally set:
40
-
41
- | Variable | Default | Purpose |
42
- |---|---|---|
43
- | `TSFM_LIVE_EVAL` | `0` | Set `1` only if you want eval inside the Space (not recommended) |
44
- | `TSFM_BENCH_INTERVAL_SECONDS` | `3600` | Countdown interval shown in UI |
45
-
46
- ### Required for local runner (.env)
47
-
48
- | Variable | Purpose |
49
- |---|---|
50
- | `TSFM_API_KEY` | TSFM.ai API key |
51
- | `HF_TOKEN` | Hugging Face write token for pushing results |
52
-
53
- ## Metrics
54
-
55
- - **MASE** β€” Mean Absolute Scaled Error
56
- - **CRPS** β€” Mean Weighted Sum Quantile Loss
57
- - Ranks are averaged per-dataset on raw metrics (lower is better)
58
-
59
- ## Aggregation (GIFT-Eval style)
60
-
61
- 1. Compute evaluation metrics on each dataset (raw values).
62
- 2. Rank models per dataset.
63
- 3. Aggregate with **geometric mean** (metrics) and **arithmetic mean** (ranks).
64
  """
65
 
66
  CITATION_BUTTON_LABEL = "Copy citation"
 
1
  TITLE = """ TSFM Realworld Bench Leaderboard """
2
 
3
  INTRODUCTION_TEXT = """
4
+ **TSFM Realworld Bench** evaluates time series foundation models on **live Open-Meteo
5
+ weather data** with **zero-shot API inference** via [TSFM.ai](https://tsfm.ai/).
6
+ Scores are normalized against **Chronos-Bolt-Base**. The leaderboard shows
7
+ **metric values** and **ranks** in separate tables (MASE, CRPS, MAE, RMSE, sMAPE,
8
+ MSIS, ND, NRMSE, MAPE).
 
9
  """
10
 
11
  LLM_BENCHMARKS_TEXT = """
12
+ ## How to participate
13
 
14
+ 1. Clone [TSFM_Bench](https://github.com/zhouziyu02/TSFM_Bench)
15
+ 2. Implement your model against the streaming data interface (`tsfm_bench.data`)
16
+ 3. Run evaluation and write `results/<MODEL_NAME>/all_results.csv`
17
+ 4. Open a pull request; results sync to this leaderboard
 
18
 
19
+ ## Metrics
20
 
21
+ - **MASE** β€” Mean Absolute Scaled Error (normalized)
22
+ - **CRPS** β€” Mean Weighted Sum Quantile Loss (probabilistic forecast quality)
23
+ - **CRPS_Rank** / **MASE_Rank** β€” average per-dataset rank (lower is better)
 
 
 
 
24
 
25
+ ## Data interface
26
+
27
+ Live weather series are streamed from **Open-Meteo** (`OpenMeteoDataSource`). Each evaluation
28
+ holds out the most recent forecast horizon as ground truth and calls TSFM models through
29
+ the **TSFM.ai** API for zero-shot forecasts.
30
 
31
  ```bash
32
+ export TSFM_API_KEY=your_key
33
+ python scripts/run_online_eval.py
34
+ python scripts/run_online_daemon.py --interval-minutes 60
35
  ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
36
  """
37
 
38
  CITATION_BUTTON_LABEL = "Copy citation"