shchuro commited on
Commit
a7553dc
·
1 Parent(s): 7924574

Add add-model skill for onboarding new leaderboard models

Browse files
Files changed (1) hide show
  1. .claude/skills/add-model/SKILL.md +64 -0
.claude/skills/add-model/SKILL.md ADDED
@@ -0,0 +1,64 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ name: add-model
3
+ description: Add a new forecasting model to the fev-bench leaderboard. Use when asked to add/register a model, regenerate tables for a new model, or when given a model name plus its HuggingFace/org/type metadata. Regenerates tables from the latest autogluon/fev commit, adds the MODEL_CONFIG entry, commits, and previews the app.
4
+ ---
5
+
6
+ # Add a new model to the leaderboard
7
+
8
+ Adds a model to the fev-bench leaderboard. The model's results must already be
9
+ merged into `autogluon/fev` (its rows appear in the benchmark result CSVs) —
10
+ this skill pulls those in and wires up the display metadata.
11
+
12
+ See commit `7924574` ("Add TimesFM-3.0") for a canonical example.
13
+
14
+ ## Inputs (collect from the user)
15
+
16
+ Required:
17
+ - **model_name**: the key used in the benchmark summaries (matches `model_name`
18
+ in the fev result CSVs), e.g. `timesfm-3`. This is the dict key in `MODEL_CONFIG`.
19
+ - **url**: HuggingFace repo id (e.g. `google/timesfm-3.0-pytorch`) for pretrained
20
+ models, or a project URL for task-specific/statistical models.
21
+ - **organization**: e.g. `Google`. Use `—` for library-provided baselines.
22
+
23
+ Optional (with sensible defaults):
24
+ - **is_zero_shot**: `True` for pretrained/zero-shot models, `False` otherwise. Default `True`.
25
+ - **model_type**: one of `pretrained`, `task-specific`, `statistical`. Default `pretrained`.
26
+ - **display_name**: optional 5th tuple element, only when the auto-derived
27
+ display name from the key is wrong (see `citras-fm` → `CITRAS-FM`).
28
+ - **commit**: `autogluon/fev` commit SHA or branch to regenerate from. Default `main`.
29
+
30
+ If the user hasn't given the required fields, ask for them before proceeding.
31
+
32
+ ## Steps
33
+
34
+ 1. **Regenerate tables** from the target commit:
35
+ ```bash
36
+ uv run python save_tables.py [commit]
37
+ ```
38
+ (omit `[commit]` to use `main`). This rewrites `tables/**` including the new
39
+ model's rows. Confirm the run reported the expected number of tasks and no errors.
40
+
41
+ 2. **Add the MODEL_CONFIG entry** in `src/utils.py`. Add one line to the
42
+ `MODEL_CONFIG` dict under the appropriate section comment (create a new
43
+ comment group if the org is new). The tuple is
44
+ `(url, organization, is_zero_shot, model_type[, display_name])`:
45
+ ```python
46
+ "timesfm-3": ("google/timesfm-3.0-pytorch", "Google", True, "pretrained"),
47
+ ```
48
+ Keep the 4-tuple form unless a `display_name` override is needed.
49
+
50
+ 3. **Verify** the model shows up: `git status` should show `src/utils.py` plus
51
+ many `tables/**` CSVs changed. The new `model_name` should be present in
52
+ `tables/full/leaderboard_SQL.csv`.
53
+
54
+ 4. **Commit** with a message matching the existing convention:
55
+ ```bash
56
+ git add -A && git commit -m "Add <ModelDisplayName>"
57
+ ```
58
+ Branch off `main` first if the repo policy requires it.
59
+
60
+ 5. **Preview** the app locally so the user can eyeball the new entry:
61
+ ```bash
62
+ uv run streamlit run fev-leaderboard-app.py --server.port=8501 --server.address=0.0.0.0
63
+ ```
64
+ Run it in the background and share the URL, or use the `run` skill.