Spaces:
Running
Running
Add add-model skill for onboarding new leaderboard models
Browse files
.claude/skills/add-model/SKILL.md
ADDED
|
@@ -0,0 +1,64 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
name: add-model
|
| 3 |
+
description: Add a new forecasting model to the fev-bench leaderboard. Use when asked to add/register a model, regenerate tables for a new model, or when given a model name plus its HuggingFace/org/type metadata. Regenerates tables from the latest autogluon/fev commit, adds the MODEL_CONFIG entry, commits, and previews the app.
|
| 4 |
+
---
|
| 5 |
+
|
| 6 |
+
# Add a new model to the leaderboard
|
| 7 |
+
|
| 8 |
+
Adds a model to the fev-bench leaderboard. The model's results must already be
|
| 9 |
+
merged into `autogluon/fev` (its rows appear in the benchmark result CSVs) —
|
| 10 |
+
this skill pulls those in and wires up the display metadata.
|
| 11 |
+
|
| 12 |
+
See commit `7924574` ("Add TimesFM-3.0") for a canonical example.
|
| 13 |
+
|
| 14 |
+
## Inputs (collect from the user)
|
| 15 |
+
|
| 16 |
+
Required:
|
| 17 |
+
- **model_name**: the key used in the benchmark summaries (matches `model_name`
|
| 18 |
+
in the fev result CSVs), e.g. `timesfm-3`. This is the dict key in `MODEL_CONFIG`.
|
| 19 |
+
- **url**: HuggingFace repo id (e.g. `google/timesfm-3.0-pytorch`) for pretrained
|
| 20 |
+
models, or a project URL for task-specific/statistical models.
|
| 21 |
+
- **organization**: e.g. `Google`. Use `—` for library-provided baselines.
|
| 22 |
+
|
| 23 |
+
Optional (with sensible defaults):
|
| 24 |
+
- **is_zero_shot**: `True` for pretrained/zero-shot models, `False` otherwise. Default `True`.
|
| 25 |
+
- **model_type**: one of `pretrained`, `task-specific`, `statistical`. Default `pretrained`.
|
| 26 |
+
- **display_name**: optional 5th tuple element, only when the auto-derived
|
| 27 |
+
display name from the key is wrong (see `citras-fm` → `CITRAS-FM`).
|
| 28 |
+
- **commit**: `autogluon/fev` commit SHA or branch to regenerate from. Default `main`.
|
| 29 |
+
|
| 30 |
+
If the user hasn't given the required fields, ask for them before proceeding.
|
| 31 |
+
|
| 32 |
+
## Steps
|
| 33 |
+
|
| 34 |
+
1. **Regenerate tables** from the target commit:
|
| 35 |
+
```bash
|
| 36 |
+
uv run python save_tables.py [commit]
|
| 37 |
+
```
|
| 38 |
+
(omit `[commit]` to use `main`). This rewrites `tables/**` including the new
|
| 39 |
+
model's rows. Confirm the run reported the expected number of tasks and no errors.
|
| 40 |
+
|
| 41 |
+
2. **Add the MODEL_CONFIG entry** in `src/utils.py`. Add one line to the
|
| 42 |
+
`MODEL_CONFIG` dict under the appropriate section comment (create a new
|
| 43 |
+
comment group if the org is new). The tuple is
|
| 44 |
+
`(url, organization, is_zero_shot, model_type[, display_name])`:
|
| 45 |
+
```python
|
| 46 |
+
"timesfm-3": ("google/timesfm-3.0-pytorch", "Google", True, "pretrained"),
|
| 47 |
+
```
|
| 48 |
+
Keep the 4-tuple form unless a `display_name` override is needed.
|
| 49 |
+
|
| 50 |
+
3. **Verify** the model shows up: `git status` should show `src/utils.py` plus
|
| 51 |
+
many `tables/**` CSVs changed. The new `model_name` should be present in
|
| 52 |
+
`tables/full/leaderboard_SQL.csv`.
|
| 53 |
+
|
| 54 |
+
4. **Commit** with a message matching the existing convention:
|
| 55 |
+
```bash
|
| 56 |
+
git add -A && git commit -m "Add <ModelDisplayName>"
|
| 57 |
+
```
|
| 58 |
+
Branch off `main` first if the repo policy requires it.
|
| 59 |
+
|
| 60 |
+
5. **Preview** the app locally so the user can eyeball the new entry:
|
| 61 |
+
```bash
|
| 62 |
+
uv run streamlit run fev-leaderboard-app.py --server.port=8501 --server.address=0.0.0.0
|
| 63 |
+
```
|
| 64 |
+
Run it in the background and share the URL, or use the `run` skill.
|