Solves 500 Errors For Some Users

.claude/skills/add-model/SKILL.md

@@ -1,27 +0,0 @@
----
-name: add-model
-description: Get a newly merged model onto the fev-bench leaderboard. Use when asked to add/register a model, pull in new results, or refresh the leaderboard.
----
-
-# Add a new model to the leaderboard
-
-Models reach the leaderboard by being merged into `autogluon/fev`: their results CSV lands in
-`benchmarks/fev_bench/results/` and their display metadata in `benchmarks/fev_bench/models.yaml`.
-This repo holds no model data or metadata. The Docker build runs `save_tables.py main`, so the Space
-shows whatever is on fev `main` when it was last built.
-
-If the user supplies metadata (org, URL, type), it belongs in `models.yaml` in the fev repo, not
-here; point them at `models/README.md` there.
-
-## Steps
-
-1. **Preview locally** (optional) to check the new model before it goes live:
-   ```bash
-   uv run python save_tables.py            # latest fev main
-   uv run streamlit run fev-leaderboard-app.py --server.port=8501 --server.address=0.0.0.0
-   ```
-   `save_tables.py` fails loudly if a results file has no `models.yaml` entry. A `system` or
-   `closed-api` model only appears after selecting its type in the sidebar.
-
-2. **Publish**: rebuild the Space (Settings → Factory rebuild on huggingface.co/spaces/autogluon/fev-bench).
-   No commit to this repo is needed. Afterwards, the About page should show the expected fev commit.

.gitattributes

@@ -25,7 +25,6 @@
 *.safetensors filter=lfs diff=lfs merge=lfs -text
 saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
 *.tflite filter=lfs diff=lfs merge=lfs -text
 *.tgz filter=lfs diff=lfs merge=lfs -text
 *.wasm filter=lfs diff=lfs merge=lfs -text
@@ -33,3 +32,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+scale-hf-logo.png filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -1,4 +1,13 @@
-__pycache__
-# Generated at build time by save_tables.py (see Dockerfile)
-tables/
-src/vendor/
+auto_evals/
+venv/
+__pycache__/
+.env
+.ipynb_checkpoints
+*ipynb
+.vscode/
+
+eval-queue/
+eval-results/
+eval-queue-bk/
+eval-results-bk/
+logs/

.pre-commit-config.yaml

@@ -0,0 +1,53 @@
+# Copyright (c) 2022, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+default_language_version:
+  python: python3
+
+ci:
+  autofix_prs: true
+  autoupdate_commit_msg: '[pre-commit.ci] pre-commit suggestions'
+  autoupdate_schedule: quarterly
+
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.3.0
+    hooks:
+      - id: check-yaml
+      - id: check-case-conflict
+      - id: detect-private-key
+      - id: check-added-large-files
+        args: ['--maxkb=1000']
+      - id: requirements-txt-fixer
+      - id: end-of-file-fixer
+      - id: trailing-whitespace
+
+  - repo: https://github.com/PyCQA/isort
+    rev: 5.12.0
+    hooks:
+      - id: isort
+        name: Format imports
+
+  - repo: https://github.com/psf/black
+    rev: 22.12.0
+    hooks:
+      - id: black
+        name: Format code
+        additional_dependencies: ['click==8.0.2']
+
+  - repo: https://github.com/charliermarsh/ruff-pre-commit
+    # Ruff version.
+    rev: 'v0.0.267'
+    hooks:
+      - id: ruff

.streamlit/config.toml

@@ -1,4 +0,0 @@
-[theme]
-base = "light"
-# Muted blue for sidebar controls and links, instead of Streamlit's red
-primaryColor = "#6093c4"

CLAUDE.md

@@ -1,97 +0,0 @@
-# CLAUDE.md
-
-This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
-
-## Project Overview
-
-fev-bench Leaderboard is a Streamlit web application displaying time series forecasting model evaluation results from the fev-bench benchmark. It evaluates 30+ forecasting models using multiple metrics (SQL, MASE, WQL, WAPE) across 100 benchmark tasks.
-
-## Common Commands
-
-```bash
-# Run the Streamlit app locally
-uv run streamlit run fev-leaderboard-app.py --server.port=8501 --server.address=0.0.0.0
-
-# Generate data from the autogluon/fev repo (required once before running locally)
-uv run python save_tables.py [commit]        # defaults to latest main
-uv run python save_tables.py --fev-repo ../fev   # use an existing local checkout
-
-# Docker build and run
-docker build -t fev-leaderboard .
-docker run -p 8501:8501 fev-leaderboard
-```
-
-Note: Use `uv run` prefix for all Python commands in this project.
-
-No test or lint frameworks are configured.
-
-## Architecture
-
-```
-fev-leaderboard-app.py     # Main entry point (Streamlit multi-page router)
-save_tables.py             # Generates tables/ and src/vendor/ from a single fev commit (run at Docker build)
-pages/
-├── fev_bench.py           # Main leaderboard: computes leaderboard/pairwise, renders them
-└── about.py               # Help page with links
-src/
-├── utils.py               # Visualization, formatting, model metadata lookups, color palette
-├── strings.py             # UI text, metric descriptions, paper citations
-├── task_groups.py         # Task groupings by frequency and domain
-└── vendor/                # GENERATED, gitignored: compute logic copied from the fev repo
-    └── fev_bench_compute.py
-tables/                    # GENERATED by save_tables.py, gitignored
-├── summaries.csv          # Raw evaluation summaries (leaderboards are computed from these)
-├── models.csv             # Display metadata, from the fev repo's benchmarks/fev_bench/models.yaml
-├── pivot_*.csv            # Per-task error matrices (+ `_baseline_imputed` / `_leakage_imputed`)
-└── source.json            # Which fev commit the tables came from
-```
-
-**Data flow**: GitHub (autogluon/fev) → `save_tables.py` → `tables/` + vendored compute → `fev_bench.py`
-
-`tables/` and `src/vendor/` are not committed. The Dockerfile runs `save_tables.py main` at build time,
-so rebuilding the Space (Settings → Factory rebuild) picks up whatever is on fev `main`. The About
-page shows which fev commit the running Space was built from.
-
-Leaderboard and pairwise tables are **not** precomputed. Win rate and skill score depend on which
-models are included, and the sidebar lets readers include/exclude model types, so `fev_bench.py`
-computes them per (task group, metric, included types) and caches with `st.cache_data`.
-
-## Key Modules
-
-**`src/utils.py`**: Core module containing:
-- `get_model_metadata()`: Model attributes loaded from `tables/models.csv` (organization, url, model_type, zero_shot, commercial_use, display_name)
-- `ALL_METRICS`: Dict with SQL, MASE, WQL, WAPE definitions
-- `format_leaderboard()`, `construct_bar_chart()`, `construct_pairwise_chart()`, `construct_pivot_table()`: Styling functions
-- `COLORS`: Custom palette (purple, gold, silver, bronze)
-
-**`src/strings.py`**: Documentation strings for metric formulas, win rate/skill score calculations, imputation strategies
-
-## Metrics
-
-| Metric | Type | Description |
-|--------|------|-------------|
-| SQL | Probabilistic | Scaled Quantile Loss (scale-invariant) |
-| MASE | Point | Mean Absolute Scaled Error (scale-invariant) |
-| WQL | Probabilistic | Weighted Quantile Loss (scale-dependent) |
-| WAPE | Point | Weighted Absolute Percentage Error (scale-dependent) |
-
-## Model Types
-
-Each submission declares a `model_type` in `benchmarks/fev_bench/models.yaml` in the fev repo:
-`pretrained`, `task-specific`, `statistical`, `system`, or `closed-api`. This drives the name color
-on the leaderboard. `system` and `closed-api` models are hidden until the reader opts in from the
-sidebar, because their results cannot be reproduced independently or compared to single models.
-
-Model metadata is never edited here: it is declared by submitters in the fev repo and pulled in by
-`save_tables.py`. Models flagged `hidden: true` there are dropped before any metric is computed.
-
-## Imputation Strategy
-
-- **Failed tasks**: Replaced with Seasonal Naive scores
-- **Leaky tasks** (training corpus overlap for zero-shot models): Replaced with Chronos-Bolt scores
-
-## External References
-
-- fev-bench paper: https://arxiv.org/abs/2509.26468
-- fev library docs: https://autogluon.github.io/fev/latest/
-- GitHub: https://github.com/autogluon/fev

Dockerfile

@@ -1,27 +0,0 @@
-FROM python:3.13.5-slim
-
-RUN useradd -m -u 1000 user
-WORKDIR /app
-
-RUN apt-get update && apt-get install -y \
-    build-essential \
-    curl \
-    git \
-    && rm -rf /var/lib/apt/lists/*
-
-COPY --chown=user ./requirements.txt requirements.txt
-COPY --chown=user . /app
-
-RUN pip3 install -r requirements.txt
-
-# Results, metadata and leaderboard compute all come from the latest autogluon/fev main at build time.
-# The ADD changes whenever main moves, so Docker re-runs save_tables.py instead of reusing a cached
-# layer: rebuilding the Space picks up newly merged models.
-ADD https://api.github.com/repos/autogluon/fev/commits/main /tmp/fev-main.json
-RUN python save_tables.py main
-
-EXPOSE 8501
-
-HEALTHCHECK CMD curl --fail http://localhost:8501/_stcore/health
-
-ENTRYPOINT ["streamlit", "run", "fev-leaderboard-app.py", "--server.port=8501", "--server.address=0.0.0.0"]

Makefile

@@ -0,0 +1,13 @@
+.PHONY: style format
+
+
+style:
+	python -m black --line-length 119 .
+	python -m isort .
+	ruff check --fix .
+
+
+quality:
+	python -m black --check --line-length 119 .
+	python -m isort --check-only .
+	ruff check .

README.md

@@ -1,20 +1,44 @@
 ---
-title: fev-bench
-emoji: 🏆
+title: Fev Leaderboard
+emoji: 🥇
 colorFrom: green
 colorTo: indigo
-sdk: docker
-app_port: 8501
-tags:
-- streamlit
-pinned: false
-short_description: Forecast evaluation benchmark
+sdk: gradio
+app_file: app.py
+pinned: true
 license: apache-2.0
 ---
 
-# Welcome to Streamlit!
+# Start the configuration
 
-Edit `/src/streamlit_app.py` to customize this app to your heart's desire. :heart:
+Most of the variables to change for a default leaderboard are in `src/env.py` (replace the path for your leaderboard) and `src/about.py` (for tasks).
 
-If you have any questions, checkout our [documentation](https://docs.streamlit.io) and [community
-forums](https://discuss.streamlit.io).
+Results files should have the following format and be stored as json files:
+```json
+{
+    "config": {
+        "model_dtype": "torch.float16", # or torch.bfloat16 or 8bit or 4bit
+        "model_name": "path of the model on the hub: org/model",
+        "model_sha": "revision on the hub",
+    },
+    "results": {
+        "task_name": {
+            "metric_name": score,
+        },
+        "task_name2": {
+            "metric_name": score,
+        }
+    }
+}
+```
+
+Request files are created automatically by this tool.
+
+If you encounter problem on the space, don't hesitate to restart it to remove the create eval-queue, eval-queue-bk, eval-results and eval-results-bk created folder.
+
+# Code logic for more complex edits
+
+You'll find 
+- the main table' columns names and properties in `src/display/utils.py`
+- the logic to read all results and request files, then convert them in dataframe lines, in `src/leaderboard/read_evals.py`, and `src/populate.py`
+- the logic to allow or filter submissions in `src/submission/submit.py` and `src/submission/check_validity.py`

app.py

@@ -0,0 +1,97 @@
+import fev
+import gradio as gr
+import pandas as pd
+
+from src import about
+from src.custom_html_js import custom_css
+from src.formatting import make_clickable_model
+
+# Load the CSV data into a pandas DataFrame
+df = pd.read_csv(
+    "https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/chronos_zeroshot/results/seasonal_naive.csv"
+)
+
+
+summary_urls = [
+    "https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/chronos_zeroshot/results/auto_arima.csv",
+    "https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/chronos_zeroshot/results/auto_ets.csv",
+    "https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/chronos_zeroshot/results/auto_theta.csv",
+    "https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/chronos_zeroshot/results/chronos_base.csv",
+    "https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/chronos_zeroshot/results/chronos_large.csv",
+    "https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/chronos_zeroshot/results/chronos_mini.csv",
+    "https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/chronos_zeroshot/results/chronos_small.csv",
+    "https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/chronos_zeroshot/results/chronos_tiny.csv",
+    "https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/chronos_zeroshot/results/chronos_bolt_base.csv",
+    "https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/chronos_zeroshot/results/chronos_bolt_mini.csv",
+    "https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/chronos_zeroshot/results/chronos_bolt_small.csv",
+    "https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/chronos_zeroshot/results/chronos_bolt_tiny.csv",
+    "https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/chronos_zeroshot/results/moirai_base.csv",
+    "https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/chronos_zeroshot/results/moirai_large.csv",
+    "https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/chronos_zeroshot/results/moirai_small.csv",
+    "https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/chronos_zeroshot/results/seasonal_naive.csv",
+    "https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/chronos_zeroshot/results/timesfm.csv",
+    "https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/chronos_zeroshot/results/timesfm-2.0.csv",
+    "https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/chronos_zeroshot/results/ttm-r2.csv",
+    "https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/chronos_zeroshot/results/tirex.csv",
+]
+
+rename_cols = {
+    "gmean_relative_error": "Average relative error",
+    "avg_rank": "Average rank",
+    "median_inference_time_s": "Median inference time (s)",
+    "training_corpus_overlap": "Training corpus overlap (%)",
+}
+selected_cols = list(rename_cols.keys())
+
+
+def highlight_zeroshot(styler):
+    """Highlight training overlap for zero-shot models with bold green."""
+
+    def style_func(val):
+        if val == 0:
+            return "color: green; font-weight: bold"
+        else:
+            return "color: black"
+
+    return styler.map(style_func, subset=["Training corpus overlap (%)"])
+
+
+leaderboards = {}
+for metric in ["WQL", "MASE"]:
+    lb = fev.leaderboard(summary_urls, metric_column=metric)[selected_cols].rename(columns=rename_cols)
+    lb = lb.astype("float64").round(3).reset_index()
+    lb["Training corpus overlap (%)"] = (lb["Training corpus overlap (%)"] * 100).round(1)
+    lb["model_name"] = lb["model_name"].apply(make_clickable_model)
+    leaderboards[metric] = highlight_zeroshot(lb.style).format(precision=3)
+
+
+with gr.Blocks(css=custom_css) as demo:
+    gr.HTML(about.TITLE)
+    gr.Markdown(about.INTRODUCTION_TEXT, elem_classes="markdown-text")
+
+    with gr.Tabs(elem_classes="tab-buttons"):
+        with gr.Tab("🏅 Chronos Benchmark II", id=0):
+            with gr.Column():
+                gr.Markdown(about.CHRONOS_BENCHMARK, elem_classes="markdown-text")
+            with gr.Tabs():
+                with gr.Tab("📊 Probabilistic forecast (WQL)"):
+                    gr.Markdown("""Forecast accuracy measured by Weighted Quantile Loss.""")
+                    gr.Dataframe(
+                        value=leaderboards["WQL"],
+                        datatype=["markdown", "number", "number", "number"],
+                        interactive=False,
+                    )
+
+                with gr.Tab("📈 Point forecast (MASE)"):
+                    gr.Markdown("""Forecast accuracy measured by Mean Absolute Scaled Error.""")
+                    gr.Dataframe(
+                        value=leaderboards["MASE"],
+                        datatype=["markdown", "number", "number", "number"],
+                        interactive=False,
+                    )
+
+        with gr.Tab("📝 About", id=1):
+            gr.Markdown(about.ABOUT_LEADERBOARD)
+
+if __name__ == "__main__":
+    demo.launch(ssr_mode=False)

fev-leaderboard-app.py

@@ -1,9 +0,0 @@
-import streamlit as st
-
-pages = [
-    st.Page("pages/fev_bench.py", title="fev-bench", icon=":material/trophy:"),
-    st.Page("pages/about.py", title="About", icon=":material/info:"),
-]
-
-page = st.navigation(pages)
-page.run()

pages/about.py

@@ -1,34 +0,0 @@
-import json
-from pathlib import Path
-
-import streamlit as st
-
-# Written by save_tables.py, which the Docker build runs against the latest autogluon/fev main
-SOURCE_JSON = Path(__file__).resolve().parent.parent / "tables" / "source.json"
-
-ABOUT_LEADERBOARD = """
-## About
-
-[**fev**](https://github.com/autogluon/fev) is a lightweight wrapper around the 🤗 [datasets](https://huggingface.co/docs/datasets/en/index) library designed to streamline
- benchmarking of time series forecasting models.
-
-### 📚 Resources
-- **Documentation**: [Official docs](https://autogluon.github.io/fev/latest/)
-- **Publication**: ["fev-bench: A Realistic Benchmark for Time Series Forecasting"](https://arxiv.org/abs/2509.26468)
-- **Source Code**: [GitHub repository](https://github.com/autogluon/fev)
-- **Issues & Questions**: [GitHub Issues](https://github.com/autogluon/fev/issues)
-
-### 🚀 Submit Your Model
-Ready to add your model to the leaderboard? Follow this [tutorial](https://autogluon.github.io/fev/latest/tutorials/05-add-your-model/) to evaluate your model with fev and contribute your results.
-"""
-st.set_page_config(layout="wide", page_title="About FEV", page_icon=":material/info:")
-st.markdown(ABOUT_LEADERBOARD)
-
-source = json.loads(SOURCE_JSON.read_text())
-commit_url = f"https://github.com/autogluon/fev/tree/{source['commit']}"
-st.markdown(
-    f"""
-### 🔁 Reproducibility
-Results shown are from [autogluon/fev@{source['commit']}]({commit_url}), generated on {source['generated_at'][:10]}.
-"""
-)

pages/fev_bench.py

@@ -1,353 +0,0 @@
-import sys
-from pathlib import Path
-
-sys.path.append(str(Path(__file__).parent))
-
-import pandas as pd
-import streamlit as st
-from src.strings import (
-    CITATION_FEV,
-    CITATION_HEADER,
-    FEV_BENCHMARK_DETAILS,
-    PAIRWISE_BENCHMARK_DETAILS,
-    get_pivot_legend,
-)
-from src.task_groups import (
-    ALL_TASKS,
-    DOMAIN_GROUPS,
-    FREQUENCY_GROUPS,
-    MINI_TASKS,
-    TASK_TYPE_GROUPS,
-)
-from src.utils import (
-    COLORS,
-    DEFAULT_VISIBLE_MODEL_TYPES,
-    MODEL_TYPE_LABELS,
-    TAG_COLORS,
-    construct_pairwise_chart,
-    format_leaderboard,
-    format_metric_name,
-    get_metric_description,
-    get_model_display_name,
-    select_model_names,
-    validate_model_metadata,
-)
-
-# Leaderboard computation vendored from the fev repo by save_tables.py, so the numbers shown here are
-# produced by the same code as the paper figures.
-from src.vendor.fev_bench_compute import (
-    AVAILABLE_METRICS,
-    BASELINE_MODEL,
-    LEAKAGE_IMPUTATION_MODEL,
-    SORT_COL,
-    TOP_K_MODELS_TO_PLOT,
-    compute_leaderboard,
-    compute_pairwise,
-)
-from streamlit.elements.lib.column_types import ColumnConfig
-
-st.set_page_config(
-    layout="wide", page_title="fev leaderboard", page_icon=":material/trophy:"
-)
-
-TITLE = "<h1 style='text-align: center; font-size: 350%;'>fev-bench</h1>"
-
-MODEL_TYPES_HELP = (
-    "Win rates are computed among the included models. "
-    f"{BASELINE_MODEL} and {LEAKAGE_IMPUTATION_MODEL} are always included, as they are used for imputation. "
-    'Click "See details" below the leaderboard for a description of each model type.'
-)
-NON_COMMERCIAL_HELP = "Include models whose license does not permit commercial use."
-
-# Group type options
-GROUP_TYPES = [
-    "Full (100 tasks)",
-    "Mini (20 tasks)",
-    "By frequency",
-    "By domain",
-    "By task type",
-]
-FREQUENCY_OPTIONS = list(FREQUENCY_GROUPS.keys())
-DOMAIN_OPTIONS = list(DOMAIN_GROUPS.keys())
-TASK_TYPE_OPTIONS = list(TASK_TYPE_GROUPS.keys())
-
-
-def get_subset_description(
-    group_type: str, subgroup: str | None, num_tasks: int, metric_name: str
-) -> str:
-    """Generate a description of the current subset."""
-    base = f"Results for various forecasting models, measured using the **{metric_name}** metric, on **{num_tasks} tasks**"
-    if group_type == "Full (100 tasks)":
-        subset_desc = "from the full **fev-bench** benchmark"
-    elif group_type == "Mini (20 tasks)":
-        subset_desc = "from the **fev-bench-mini** subset"
-    elif group_type == "By frequency":
-        subset_desc = f"with **{subgroup.lower()}** frequency"
-    elif group_type == "By task type":
-        subset_desc = f"of type **{subgroup.lower()}**"
-    else:  # By domain
-        subset_desc = f"from the **{subgroup}** domain"
-    paper_link = "[fev-bench: A Realistic Benchmark for Time Series Forecasting](https://arxiv.org/abs/2509.26468)"
-    return f"{base} {subset_desc}, as described in {paper_link}."
-
-
-@st.cache_data()
-def get_summaries() -> pd.DataFrame:
-    return pd.read_csv("tables/summaries.csv")
-
-
-@st.cache_data()
-def get_leaderboard(
-    metric_name: str, task_list: tuple[str, ...], model_names: tuple[str, ...]
-) -> pd.DataFrame:
-    """Leaderboard for the selected tasks and models.
-
-    Win rate and skill score depend on which models are included, so these are computed here rather
-    than precomputed: including an extra model type changes every row.
-    """
-    return compute_leaderboard(_filter_summaries(metric_name, task_list, model_names), metric_name)
-
-
-@st.cache_data()
-def get_pairwise(
-    metric_name: str, task_list: tuple[str, ...], model_names: tuple[str, ...]
-) -> pd.DataFrame:
-    summaries = _filter_summaries(metric_name, task_list, model_names)
-    leaderboard_df = get_leaderboard(metric_name, task_list, model_names)
-    top_k_models = (
-        leaderboard_df.sort_values(by=SORT_COL, ascending=False)
-        .head(TOP_K_MODELS_TO_PLOT)["model_name"]
-        .tolist()
-    )
-    return compute_pairwise(summaries, metric_name, top_k_models)
-
-
-def _filter_summaries(
-    metric_name: str, task_list: tuple[str, ...], model_names: tuple[str, ...]
-) -> pd.DataFrame:
-    summaries = get_summaries()
-    # The baseline and leakage-imputation models must stay in the data for imputation to work, even
-    # if the reader excluded their type.
-    included = set(model_names) | {BASELINE_MODEL, LEAKAGE_IMPUTATION_MODEL}
-    return summaries[
-        summaries["task_name"].isin(task_list) & summaries["model_name"].isin(included)
-    ]
-
-
-@st.cache_data()
-def get_pivot_table(
-    metric_name: str,
-) -> tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame]:
-    pivot_df = pd.read_csv(f"tables/pivot_{metric_name}.csv")
-    baseline_imputed = pd.read_csv(f"tables/pivot_{metric_name}_baseline_imputed.csv")
-    leakage_imputed = pd.read_csv(f"tables/pivot_{metric_name}_leakage_imputed.csv")
-    return pivot_df, baseline_imputed, leakage_imputed
-
-
-with st.sidebar:
-    # Task group selection
-    selected_group_type = st.selectbox("Subset", options=GROUP_TYPES)
-
-    # Conditional sub-selection for frequency/domain
-    selected_subgroup = None
-    if selected_group_type == "By frequency":
-        selected_subgroup = st.selectbox("Frequency", options=FREQUENCY_OPTIONS)
-    elif selected_group_type == "By domain":
-        selected_subgroup = st.selectbox("Domain", options=DOMAIN_OPTIONS)
-    elif selected_group_type == "By task type":
-        selected_subgroup = st.selectbox("Task type", options=TASK_TYPE_OPTIONS)
-
-    # Determine the tasks making up the selected subset
-    if selected_group_type in ["Full (100 tasks)", "Mini (20 tasks)"]:
-        task_list = (
-            ALL_TASKS if selected_group_type == "Full (100 tasks)" else MINI_TASKS
-        )
-    else:
-        if selected_group_type == "By frequency":
-            task_list = FREQUENCY_GROUPS[selected_subgroup]
-        elif selected_group_type == "By task type":
-            task_list = TASK_TYPE_GROUPS[selected_subgroup]
-        else:
-            task_list = DOMAIN_GROUPS[selected_subgroup]
-
-    st.caption(f"{len(task_list)} tasks")
-
-    st.divider()
-
-    selected_metric = st.selectbox(
-        "Evaluation Metric", options=AVAILABLE_METRICS, format_func=format_metric_name
-    )
-    st.caption(get_metric_description(selected_metric))
-
-    st.divider()
-
-    included_model_types = st.pills(
-        "Model types",
-        options=list(MODEL_TYPE_LABELS),
-        format_func=MODEL_TYPE_LABELS.get,
-        selection_mode="multi",
-        default=list(DEFAULT_VISIBLE_MODEL_TYPES),
-        help=MODEL_TYPES_HELP,
-    )
-    include_non_commercial = st.toggle(
-        "Include non-commercial models", value=True, help=NON_COMMERCIAL_HELP
-    )
-    included_models = tuple(
-        select_model_names(included_model_types, include_non_commercial)
-    )
-    if included_models:
-        # The imputation models are always part of the computation, so count them too
-        num_models = len(
-            set(included_models) | {BASELINE_MODEL, LEAKAGE_IMPUTATION_MODEL}
-        )
-        st.caption(f"{num_models} models")
-
-cols = st.columns(spec=[0.025, 0.95, 0.025])
-
-with cols[1] as main_container:
-    st.markdown(TITLE, unsafe_allow_html=True)
-
-    if not included_models:
-        st.warning("No models match the current selection. Select at least one model type in the sidebar.")
-        st.stop()
-
-    task_names = tuple(task_list)
-    metric_df = get_leaderboard(selected_metric, task_names, included_models).sort_values(
-        by=SORT_COL, ascending=False
-    )
-    validate_model_metadata(metric_df["model_name"])
-
-    pairwise_df = get_pairwise(selected_metric, task_names, included_models)
-
-    st.markdown("## :material/trophy: Leaderboard", unsafe_allow_html=True)
-    st.markdown(
-        get_subset_description(
-            selected_group_type, selected_subgroup, len(task_list), selected_metric
-        ),
-        unsafe_allow_html=True,
-    )
-    df_styled = format_leaderboard(metric_df)
-    st.dataframe(
-        df_styled,
-        width="stretch",
-        hide_index=True,
-        column_config={
-            "model_name": st.column_config.LinkColumn(
-                label="Model", display_text=r"#(.*)$"
-            ),
-            "win_rate": st.column_config.NumberColumn(
-                label="Win rate (%)", format="%.1f"
-            ),
-            "skill_score": st.column_config.NumberColumn(
-                label="Skill score (%)", format="%.1f"
-            ),
-            "median_inference_time_s_per100": st.column_config.NumberColumn(
-                label="Runtime (s / 100 series)",
-                format="%.1f",
-                help="Median end-to-end runtime per 100 time series.",
-            ),
-            "training_corpus_overlap": st.column_config.NumberColumn(
-                label="Leakage (%)", format="%d"
-            ),
-            "num_failures": st.column_config.NumberColumn(
-                label="Failed (%)",
-                format="%.0f",
-                help="Percentage of tasks where the model failed to produce a forecast.",
-            ),
-            "tags": st.column_config.MultiselectColumn(
-                label="Tags",
-                options=list(TAG_COLORS),
-                color=list(TAG_COLORS.values()),
-                help="Model type; Zero-shot = no training on the task; Non-commercial = license does not permit commercial use.",
-            ),
-            "org": ColumnConfig(label="Organization", alignment="left"),
-        },
-    )
-
-    with st.expander("See details"):
-        st.markdown(FEV_BENCHMARK_DETAILS, unsafe_allow_html=True)
-
-    st.markdown("## :material/bar_chart: Pairwise comparison", unsafe_allow_html=True)
-    chart_col_1, _, chart_col_2 = st.columns(spec=[0.45, 0.1, 0.45])
-
-    with chart_col_1:
-        st.altair_chart(
-            construct_pairwise_chart(
-                pairwise_df, col="win_rate", metric_name=selected_metric
-            ),
-            use_container_width=True,
-        )
-
-    with chart_col_2:
-        st.altair_chart(
-            construct_pairwise_chart(
-                pairwise_df, col="skill_score", metric_name=selected_metric
-            ),
-            use_container_width=True,
-        )
-
-    with st.expander("See details"):
-        st.markdown(PAIRWISE_BENCHMARK_DETAILS, unsafe_allow_html=True)
-
-    st.markdown(
-        "## :material/table_chart: Results for individual tasks", unsafe_allow_html=True
-    )
-    with st.expander("Show detailed results"):
-        st.markdown(
-            get_pivot_legend("Seasonal Naive", "Chronos-Bolt"), unsafe_allow_html=True
-        )
-        pivot_df, baseline_imputed, leakage_imputed = get_pivot_table(selected_metric)
-        pivot_df = pivot_df.set_index("Task name")
-        baseline_imputed = baseline_imputed.set_index("Task name")
-        leakage_imputed = leakage_imputed.set_index("Task name")
-
-        # Keep only the models included by the sidebar toggles
-        excluded_cols = [c for c in pivot_df.columns if c not in included_models]
-        pivot_df = pivot_df.drop(columns=excluded_cols)
-        baseline_imputed = baseline_imputed.drop(columns=excluded_cols)
-        leakage_imputed = leakage_imputed.drop(columns=excluded_cols)
-
-        # Filter pivot table to only show tasks in the selected group
-        available_tasks = [t for t in task_list if t in pivot_df.index]
-        pivot_df = pivot_df.loc[available_tasks]
-        baseline_imputed = baseline_imputed.loc[available_tasks]
-        leakage_imputed = leakage_imputed.loc[available_tasks]
-
-        # Apply display-name overrides to model columns
-        col_rename = {c: get_model_display_name(c) for c in pivot_df.columns}
-        pivot_df = pivot_df.rename(columns=col_rename)
-        baseline_imputed = baseline_imputed.rename(columns=col_rename)
-        leakage_imputed = leakage_imputed.rename(columns=col_rename)
-
-        def style_pivot_table(errors, is_baseline_imputed, is_leakage_imputed):
-            rank_colors = {1: COLORS["gold"], 2: COLORS["silver"], 3: COLORS["bronze"]}
-
-            def highlight_by_position(styler):
-                for row_idx in errors.index:
-                    row_ranks = errors.loc[row_idx].rank(method="min")
-                    for col_idx in errors.columns:
-                        rank = row_ranks[col_idx]
-                        style_parts = []
-                        if rank <= 3:
-                            style_parts.append(f"background-color: {rank_colors[rank]}")
-                        if is_leakage_imputed.loc[row_idx, col_idx]:
-                            style_parts.append(f"color: {COLORS['leakage_impute']}")
-                        elif is_baseline_imputed.loc[row_idx, col_idx]:
-                            style_parts.append(f"color: {COLORS['failure_impute']}")
-                        elif not style_parts:
-                            style_parts.append(f"color: {COLORS['text_default']}")
-                        if style_parts:
-                            styler = styler.map(
-                                lambda x, s="; ".join(style_parts): s,
-                                subset=pd.IndexSlice[row_idx:row_idx, col_idx:col_idx],
-                            )
-                return styler
-
-            return highlight_by_position(errors.style).format(precision=3)
-
-        st.dataframe(style_pivot_table(pivot_df, baseline_imputed, leakage_imputed))
-
-    st.divider()
-    st.markdown("### :material/format_quote: Citation", unsafe_allow_html=True)
-    st.markdown(CITATION_HEADER)
-    st.markdown(CITATION_FEV)

pyproject.toml

@@ -1,12 +1,13 @@
-[project]
-name = "hf-leaderboard"
-version = "0.1.0"
-requires-python = ">=3.11"
-dependencies = [
-    "altair>=6.0.0",
-    "fev>=0.7.0",
-    "numpy<2.2",
-    "pyarrow<21",
-    "scipy<1.15",
-    "streamlit>=1.53.1",
-]
+[tool.ruff]
+# Enable pycodestyle (`E`) and Pyflakes (`F`) codes by default.
+select = ["E", "F"]
+ignore = ["E501"] # line too long (black is taking care of this)
+line-length = 119
+fixable = ["A", "B", "C", "D", "E", "F", "G", "I", "N", "Q", "S", "T", "W", "ANN", "ARG", "BLE", "COM", "DJ", "DTZ", "EM", "ERA", "EXE", "FBT", "ICN", "INP", "ISC", "NPY", "PD", "PGH", "PIE", "PL", "PT", "PTH", "PYI", "RET", "RSE", "RUF", "SIM", "SLF", "TCH", "TID", "TRY", "UP", "YTT"]
+
+[tool.isort]
+profile = "black"
+line_length = 119
+
+[tool.black]
+line-length = 119

requirements.txt

@@ -1,7 +1,11 @@
+APScheduler
+black
+datasets
+gradio
+gradio[oauth]
+gradio_client
+huggingface-hub>=0.18.0
 matplotlib
 numpy
 pandas
-requests
-streamlit==1.53.1
-fev>=0.6.0
-altair>=5.5.0
+fev==0.4.0

save_tables.py

@@ -1,240 +0,0 @@
-#!/usr/bin/env python3
-"""Refresh the leaderboard's data from the autogluon/fev repo.
-
-One command regenerates everything the app needs from a single fev commit:
-
-    uv run python save_tables.py            # latest main
-    uv run python save_tables.py abc123     # a specific commit
-    uv run python save_tables.py --fev-repo ../fev   # an existing local checkout
-
-Writes into ``tables/``:
-
-    summaries.csv              raw evaluation summaries (the app computes leaderboards from these)
-    models.csv                 display metadata, from the fev repo's benchmarks/fev_bench/models.yaml
-    pivot_<metric>.csv         per-task error matrix (+ `_baseline_imputed` / `_leakage_imputed` masks)
-    source.json               which fev commit the tables came from
-
-and vendors the fev repo's leaderboard computation into ``src/vendor/`` so the app's win rates and
-pairwise comparisons are produced by exactly the code at that commit (the same code that generates
-the paper figures). Leaderboard and pairwise tables are *not* precomputed: they depend on which
-model types the reader includes, so the app computes them on the fly and caches the result.
-"""
-
-from __future__ import annotations
-
-import argparse
-import importlib.util
-import json
-import shutil
-import subprocess
-import sys
-import tempfile
-from datetime import datetime, timezone
-from pathlib import Path
-from types import ModuleType
-
-import pandas as pd
-
-GITHUB_REPO = "https://github.com/autogluon/fev.git"
-TABLES_DIR = Path(__file__).parent / "tables"
-# Modules copied out of the fev repo so the app runs the same compute logic as the pinned commit.
-VENDOR_DIR = Path(__file__).parent / "src" / "vendor"
-VENDORED_MODULES = {"fev_bench_compute.py": "scripts/generate_fev_bench_figures.py"}
-VENDOR_HEADER = """# Generated file - do not edit.
-# Vendored from {source} in autogluon/fev at commit {commit} by save_tables.py.
-# Regenerate with: uv run python save_tables.py [commit]
-"""
-
-
-def clone_fev_repo(commit: str, dest: Path) -> None:
-    """Shallow-fetch a single fev commit (works for both branch names and SHAs)."""
-    print(f"Fetching {GITHUB_REPO} at {commit}...")
-    run = lambda *args: subprocess.run(args, cwd=dest, check=True, stdout=subprocess.DEVNULL)  # noqa: E731
-    run("git", "init", "--quiet")
-    run("git", "remote", "add", "origin", GITHUB_REPO)
-    run("git", "fetch", "--depth", "1", "--quiet", "origin", commit)
-    run("git", "checkout", "--quiet", "FETCH_HEAD")
-
-
-def resolved_commit(fev_repo: Path) -> str:
-    """Short SHA of the checkout, so the generated tables record their exact source."""
-    result = subprocess.run(
-        ["git", "rev-parse", "--short", "HEAD"], cwd=fev_repo, capture_output=True, text=True, check=True
-    )
-    return result.stdout.strip()
-
-
-def load_module_from(path: Path, name: str) -> ModuleType:
-    """Import a module from a file path, with its directory importable (for sibling imports)."""
-    sys.path.insert(0, str(path.parent))
-    spec = importlib.util.spec_from_file_location(name, path)
-    module = importlib.util.module_from_spec(spec)
-    # Registered before executing so that pydantic models in the module can resolve their annotations
-    sys.modules[name] = module
-    spec.loader.exec_module(module)
-    return module
-
-
-def vendor_compute_modules(fev_repo: Path, commit: str) -> None:
-    """Copy the fev repo's compute logic into src/vendor/ with a generated-file header."""
-    VENDOR_DIR.mkdir(parents=True, exist_ok=True)
-    (VENDOR_DIR / "__init__.py").write_text("")
-    for target, source in VENDORED_MODULES.items():
-        source_path = fev_repo / source
-        if not source_path.exists():
-            raise FileNotFoundError(f"Expected {source} in the fev checkout at {fev_repo}")
-        header = VENDOR_HEADER.format(source=source, commit=commit)
-        (VENDOR_DIR / target).write_text(header + "\n" + source_path.read_text())
-        print(f"Vendored: src/vendor/{target} (from {source})")
-
-
-def load_model_metadata(fev_repo: Path) -> pd.DataFrame:
-    """Load benchmarks/fev_bench/models.yaml, validated against the results files."""
-    metadata_module = load_module_from(fev_repo / "scripts" / "fev_bench_metadata.py", "fev_bench_metadata")
-    metadata = metadata_module.load_model_metadata(
-        fev_repo / "benchmarks" / "fev_bench" / "results",
-        fev_repo / "benchmarks" / "fev_bench" / "models.yaml",
-    )
-    columns = [
-        "model_name",
-        "display_name",
-        "organization",
-        "url",
-        "model_type",
-        "zero_shot",
-        "commercial_use",
-        "hidden",
-        "results_file",
-    ]
-    return pd.DataFrame(metadata.values())[columns].sort_values("model_name")
-
-
-def load_summaries(results_dir: Path) -> pd.DataFrame:
-    csv_files = sorted(results_dir.glob("*.csv"))
-    if not csv_files:
-        raise FileNotFoundError(f"No CSV files found in {results_dir}")
-    print(f"Loading {len(csv_files)} result files from {results_dir}")
-    return pd.concat([pd.read_csv(f) for f in csv_files], ignore_index=True)
-
-
-def compute_pivot_table(
-    summaries: pd.DataFrame,
-    metric_name: str,
-    baseline_model: str,
-    leakage_imputation_model: str,
-) -> tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame]:
-    import fev
-
-    errors = fev.pivot_table(summaries=summaries, metric_column=metric_name, task_columns=["task_name"])
-    train_overlap = (
-        fev.pivot_table(summaries=summaries, metric_column="trained_on_this_dataset", task_columns=["task_name"])
-        .fillna(False)
-        .astype(bool)
-    )
-
-    is_imputed_baseline = errors.isna()
-    is_leakage_imputed = train_overlap
-
-    # Handle imputations
-    errors = errors.mask(train_overlap, errors[leakage_imputation_model], axis=0)
-    for col in errors.columns:
-        if col != baseline_model:
-            errors[col] = errors[col].fillna(errors[baseline_model])
-
-    errors = errors[errors.rank(axis=1).mean().sort_values().index]
-    is_imputed_baseline = is_imputed_baseline[errors.columns]
-    is_leakage_imputed = is_leakage_imputed[errors.columns]
-
-    for df in (errors, is_imputed_baseline, is_leakage_imputed):
-        df.index.rename("Task name", inplace=True)
-
-    return errors.reset_index(), is_imputed_baseline.reset_index(), is_leakage_imputed.reset_index()
-
-
-def main() -> int:
-    parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
-    parser.add_argument(
-        "commit",
-        nargs="?",
-        default="main",
-        help="Commit SHA or branch in autogluon/fev to generate the tables from (default: main)",
-    )
-    parser.add_argument(
-        "--fev-repo",
-        type=Path,
-        default=None,
-        help="Use an existing local fev checkout instead of fetching from GitHub (ignores `commit`)",
-    )
-    args = parser.parse_args()
-
-    with tempfile.TemporaryDirectory() as tmp_dir:
-        if args.fev_repo is not None:
-            fev_repo = args.fev_repo.resolve()
-            print(f"Using local fev checkout: {fev_repo}")
-        else:
-            fev_repo = Path(tmp_dir)
-            clone_fev_repo(args.commit, fev_repo)
-        commit = resolved_commit(fev_repo)
-
-        summaries = load_summaries(fev_repo / "benchmarks" / "fev_bench" / "results")
-        models = load_model_metadata(fev_repo)
-        vendor_compute_modules(fev_repo, commit)
-        compute = load_module_from(VENDOR_DIR / "fev_bench_compute.py", "fev_bench_compute")
-
-    # Models hidden from the leaderboard are dropped before any metric is computed, so that they do
-    # not influence other models' win rates (same as the paper figures).
-    hidden_models = models.loc[models["hidden"], "model_name"].tolist()
-    if hidden_models:
-        print(f"Dropping {len(hidden_models)} hidden models: {hidden_models}")
-        summaries = summaries[~summaries["model_name"].isin(hidden_models)]
-    # `hidden` is not written out: the app only ever sees models it is meant to display.
-    models = models[~models["hidden"]].drop(columns="hidden")
-
-    missing_metadata = sorted(set(summaries["model_name"]) - set(models["model_name"]))
-    if missing_metadata:
-        raise ValueError(f"Summaries contain models without metadata: {missing_metadata}")
-
-    TABLES_DIR.mkdir(exist_ok=True)
-    summaries.to_csv(TABLES_DIR / "summaries.csv", index=False)
-    print(f"Saved: summaries.csv ({summaries['model_name'].nunique()} models, {summaries['task_name'].nunique()} tasks)")
-    models.to_csv(TABLES_DIR / "models.csv", index=False)
-    print(f"Saved: models.csv ({len(models)} models)")
-
-    for metric in compute.AVAILABLE_METRICS:
-        pivot_df, baseline_imputed, leakage_imputed = compute_pivot_table(
-            summaries, metric, compute.BASELINE_MODEL, compute.LEAKAGE_IMPUTATION_MODEL
-        )
-        pivot_df.to_csv(TABLES_DIR / f"pivot_{metric}.csv", index=False)
-        baseline_imputed.to_csv(TABLES_DIR / f"pivot_{metric}_baseline_imputed.csv", index=False)
-        leakage_imputed.to_csv(TABLES_DIR / f"pivot_{metric}_leakage_imputed.csv", index=False)
-        print(f"Saved: pivot_{metric}.csv")
-
-    (TABLES_DIR / "source.json").write_text(
-        json.dumps(
-            {
-                "repo": GITHUB_REPO,
-                "commit": commit,
-                "requested": args.commit if args.fev_repo is None else str(args.fev_repo),
-                "generated_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
-            },
-            indent=2,
-        )
-        + "\n"
-    )
-    print(f"Saved: source.json (fev commit {commit})")
-
-    # Leaderboard and pairwise tables used to be precomputed per task group; the app now computes
-    # them so the reader can include/exclude model types. Clean up anything left from older runs.
-    for stale in sorted(p for p in TABLES_DIR.iterdir() if p.is_dir()):
-        shutil.rmtree(stale)
-        print(f"Removed stale table directory: {stale.name}/")
-    for stale in sorted(TABLES_DIR.glob("leaderboard_*.csv")) + sorted(TABLES_DIR.glob("pairwise_*.csv")):
-        stale.unlink()
-        print(f"Removed stale table: {stale.name}")
-
-    print(f"\nAll tables written to {TABLES_DIR}/")
-    return 0
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())

src/about.py

@@ -0,0 +1,50 @@
+TITLE = """<h1 align="center" id="space-title">Forecast evaluation leaderboard</h1>"""
+
+# What does your leaderboard evaluate?
+INTRODUCTION_TEXT = """
+This space hosts evaluation results for time series forecasting models.
+
+The results are obtained using [fev](https://github.com/autogluon/fev) - a lightweight library for evaluating time series forecasting models.
+"""
+
+ABOUT_LEADERBOARD = """
+## What is `fev`?
+
+[`fev`](https://github.com/autogluon/fev) is a lightweight wrapper around the 🤗 [`datasets`](https://huggingface.co/docs/datasets/en/index) library that makes it easy to benchmark time series forecasting models.
+
+For more information about `fev`, please check out [github.com/autogluon/fev](https://github.com/autogluon/fev).
+
+Currently, the results in this space are a minimal proof of concept. We plan to add new benchmark datasets and tasks in the future.
+
+## How is `fev` different from other benchmarking tools?
+Existing forecasting benchmarks usually fall into one of two categories:
+
+- Standalone datasets without any supporting infrastructure. These provide no guarantees that the results obtained by different users are comparable. For example, changing the start date or duration of the forecast horizon totally changes the meaning of the scores.
+- Bespoke end-to-end systems that combine models, datasets and forecasting tasks. Such packages usually come with lots of dependencies and assumptions, which makes extending or integrating these libraries into existing systems difficult.
+
+`fev` aims for the middle ground - it provides the core benchmarking functionality without introducing unnecessary constraints or bloated dependencies. The library supports point & probabilistic forecasting, different types of covariates, as well as all popular forecasting metrics.
+
+
+## Submitting your model
+For instructions on how to evaluate your model using `fev` and contribute your results to the leaderboard, please follow the [instructions in the GitHub repo](https://github.com/autogluon/fev/blob/main/docs/04-models.ipynb).
+"""
+
+CHRONOS_BENCHMARK = """
+## Chronos Benchmark II results
+
+This tab contains results for various forecasting models on the 27 datasets used in Benchmark II in the publication [Chronos: Learning the Language of Time Series](https://arxiv.org/abs/2403.07815).
+
+These datasets were used for zero-shot evaluation of Chronos models (i.e., Chronos models were not trained on these datasets), but some other models did include certain datasets in their training corpus.
+
+Each table contains the following information:
+
+* **Average relative error**: Geometric mean of the relative errors for each task. The relative error for each task is computed as `model_error / baseline_error`.
+* **Average rank**: Arithmetic mean of the ranks achieved by each model on each task.
+* **Median inference time (s)**: Median of the times required to make predictions for the entire dataset (in seconds).
+* **Training corpus overlap (%)**: Percentage of the datasets used in the benchmark that were included in the model's training corpus. Zero-shot models are highlighted in <span style="color:green; font-weight:bold;">green</span>.
+
+Lower values are better for all of the above metrics.
+
+Task definitions and the detailed results are available on [GitHub](https://github.com/autogluon/fev/tree/main/benchmarks/chronos_zeroshot). More information for the datasets is available in [Table 3 of the paper](https://arxiv.org/abs/2403.07815).
+
+"""

src/colors.py

@@ -1,6 +0,0 @@
-# Legacy colors - kept for backward compatibility if needed elsewhere
-VERY_PALE_PURPLE = "#e8d9f3"
-VERY_PALE_GREEN = "#cffdbc"
-VERY_PALE_BLUE = "#d6fffe"
-DEEP_LAVENDER = "#8d5eb7"
-GRASS_GREEN = "#3f9b0b"

src/custom_html_js.py

@@ -0,0 +1,99 @@
+custom_css = """
+
+.markdown-text {
+    font-size: 20px !important;
+}
+
+"""
+
+
+# .tab-buttons button {
+#     font-size: 20px;
+# }
+
+# #citation-button span {
+#     font-size: 16px !important;
+# }
+
+# #citation-button textarea {
+#     font-size: 16px !important;
+# }
+
+# #citation-button > label > button {
+#     margin: 6px;
+#     transform: scale(1.3);
+# }
+
+
+# #leaderboard-table-lite {
+#     margin-top: 15px
+# }
+
+# #search-bar-table-box > div:first-child {
+#     background: none;
+#     border: none;
+# }
+
+# #search-bar {
+#     padding: 0px;
+# }
+
+# /* Hides the final AutoEvalColumn */
+# #llm-benchmark-tab-table table td:last-child,
+# #llm-benchmark-tab-table table th:last-child {
+#     display: none;
+# }
+
+# /* Limit the width of the first AutoEvalColumn so that names don't expand too much */
+# table td:first-child,
+# table th:first-child {
+#     max-width: 400px;
+#     overflow: auto;
+#     white-space: nowrap;
+# }
+
+
+# #scale-logo {
+#     border-style: none !important;
+#     box-shadow: none;
+#     display: block;
+#     margin-left: auto;
+#     margin-right: auto;
+#     max-width: 600px;
+# }
+
+# #scale-logo .download {
+#     display: none;
+# }
+# #filter_type{
+#     border: 0;
+#     padding-left: 0;
+#     padding-top: 0;
+# }
+# #filter_type label {
+#     display: flex;
+# }
+# #filter_type label > span{
+#     margin-top: var(--spacing-lg);
+#     margin-right: 0.5em;
+# }
+# #filter_type label > .wrap{
+#     width: 103px;
+# }
+# #filter_type label > .wrap .wrap-inner{
+#     padding: 2px;
+# }
+# #filter_type label > .wrap .wrap-inner input{
+#     width: 1px
+# }
+# #filter-columns-type{
+#     border:0;
+#     padding:0.5;
+# }
+# #filter-columns-size{
+#     border:0;
+#     padding:0.5;
+# }
+# #box-filter > .form{
+#     border: 0
+# }

src/formatting.py

@@ -0,0 +1,31 @@
+def model_hyperlink(link, model_name):
+    return f'<a target="_blank" href="{link}" style="color: var(--link-text-color); text-decoration: underline;text-decoration-style: dotted;">{model_name}</a>'
+
+
+MODEL_URLS = {
+    "chronos_tiny": "amazon/chronos-t5-tiny",
+    "chronos_mini": "amazon/chronos-t5-mini",
+    "chronos_small": "amazon/chronos-t5-small",
+    "chronos_base": "amazon/chronos-t5-base",
+    "chronos_large": "amazon/chronos-t5-large",
+    "chronos_bolt_tiny": "amazon/chronos-bolt-tiny",
+    "chronos_bolt_mini": "amazon/chronos-bolt-mini",
+    "chronos_bolt_small": "amazon/chronos-bolt-small",
+    "chronos_bolt_base": "amazon/chronos-bolt-base",
+    "moirai_large": "Salesforce/moirai-1.1-R-large",
+    "moirai_base": "Salesforce/moirai-1.1-R-base",
+    "moirai_small": "Salesforce/moirai-1.1-R-small",
+    "timesfm": "google/timesfm-1.0-200m-pytorch",
+    "timesfm-2.0": "google/timesfm-2.0-500m-pytorch",
+    "ttm-r2": "ibm-granite/granite-timeseries-ttm-r2",
+    "tirex": "NX-AI/TiRex",
+}
+
+
+def make_clickable_model(model_name):
+    if model_name in MODEL_URLS:
+        model_path = MODEL_URLS.get(model_name)
+        link = f"https://huggingface.co/{model_path}"
+        return model_hyperlink(link, model_name)
+    else:
+        return model_name

src/streamlit_app.py

@@ -1,9 +0,0 @@
-import streamlit as st
-
-pages = [
-    st.Page("../pages/fev_bench.py", title="fev-bench", icon=":material/trophy:"),
-    st.Page("../pages/about.py", title="About", icon=":material/info:"),
-]
-
-page = st.navigation(pages)
-page.run()

src/strings.py

@@ -1,135 +0,0 @@
-from src.utils import COLORS, MODEL_TYPE_LABELS, NON_COMMERCIAL_TAG, TAG_COLORS, ZERO_SHOT_TAG
-
-
-def tag_badge(tag: str) -> str:
-    """Inline HTML badge that matches the chip a tag gets in the leaderboard's Tags column."""
-    return (
-        f"<span style='background: {TAG_COLORS[tag]}; color: {COLORS['text_default']}; "
-        "padding: 1px 8px; border-radius: 999px; font-size: 0.85em; white-space: nowrap;'>"
-        f"{tag}</span>"
-    )
-
-
-INTRODUCTION_TEXT = """
-This space hosts evaluation results for time series forecasting models. The results are obtained using [fev](https://github.com/autogluon/fev) - a lightweight library for evaluating time series forecasting models.
-"""
-
-LEGEND = """
-"""
-
-TABLE_INFO = f"""
-The leaderboard summarizes the performance of all models evaluated on the 100 tasks comprising **fev-bench**. More details available in the [paper](https://arxiv.org/abs/2509.26468).
-
-Click a model name to open its page.
-
-The full matrix $E_{{rj}}$ with the error of each model $j$ on task $r$ is available at the bottom of the page.
-
-#### Performance
-
-* **Win rate (%)**: Fraction of all possible model pairs and tasks where this model achieves lower error than the competing model. For model $j$, defined as $W_j = \\frac{{1}}{{R(M-1)}} \\sum_{{r=1}}^{{R}} \\sum_{{k \\neq j}} (\\mathbf{{1}}(E_{{rj}} < E_{{rk}}) + 0.5 \\cdot \\mathbf{{1}}(E_{{rj}} = E_{{rk}}))$ where $R$ is number of tasks, $M$ is number of models. Ties count as half-wins.
-
-    Ranges from 0% (worst) to 100% (best). Higher values are better. This value changes as new models are added to the benchmark.
-
-* **Skill score (%)**: Measures how much the model reduces forecasting error compared to the Seasonal Naive baseline. Computed as $S_j = 100 \\times (1 - \\sqrt[R]{{\\prod_{{r=1}}^{{R}} E_{{rj}}/E_{{r\\beta}}}})$, where $E_{{r\\beta}}$ is baseline error on task $r$. Relative errors are clipped between 0.01 and 100 before aggregation to avoid extreme outliers. Positive values indicate better-than-baseline performance, negative values indicate worse-than-baseline performance.
-
-    Higher values are better. This value does not change as new models are added to the benchmark.
-
-* **Runtime (s / 100 series)**: Median end-to-end time (training + prediction across all evaluation windows) in seconds per 100 time series. Note that inference times depend on hardware, batch sizes, and implementation details, so these serve as a rough guide rather than definitive performance benchmarks.
-
-#### Tags
-
-* **Model type**: {tag_badge(MODEL_TYPE_LABELS["pretrained"])} open-weights foundation model pretrained on large amounts of data, {tag_badge(MODEL_TYPE_LABELS["task-specific"])} traditional ML model trained from scratch on each task, {tag_badge(MODEL_TYPE_LABELS["statistical"])} classical method fit per task, {tag_badge(MODEL_TYPE_LABELS["system"])} pipeline combining or selecting several models, {tag_badge(MODEL_TYPE_LABELS["closed-api"])} served behind a closed API, or weights unavailable.
-* **Zero-shot**: {tag_badge(ZERO_SHOT_TAG)} the model makes predictions without task-specific training.
-* **Non-commercial**: {tag_badge(NON_COMMERCIAL_TAG)} the model's license does not permit commercial use. Follow the model's link to check its license.
-
-Tags are defined in [`benchmarks/fev_bench/models.yaml`](https://github.com/autogluon/fev/blob/main/benchmarks/fev_bench/models.yaml).
-
-#### Imputation
-
-* **Leakage (%)**: For zero-shot models, percentage of benchmark datasets included in the model's training corpus. Results for tasks with reported overlap are replaced with Chronos-Bolt (Base) performance to prevent data leakage.
-
-* **Failed (%)**: Percentage of tasks where the model failed to produce a forecast. Results for failed tasks are replaced with Seasonal Naive performance.
-
-"""
-
-CHRONOS_BENCHMARK_BASIC_INFO = f"""
-**Chronos Benchmark II** contains results for various forecasting models on the 27 datasets used in Benchmark II in the paper [Chronos: Learning the Language of Time Series](https://arxiv.org/abs/2403.07815). {LEGEND}
-"""
-
-CHRONOS_BENCHMARK_DETAILS = f"""
-{TABLE_INFO}
-
-Task definitions and the detailed results are available on [GitHub](https://github.com/autogluon/fev/tree/main/benchmarks/chronos_zeroshot). More information for the datasets is available in [Table 3 of the paper](https://arxiv.org/abs/2403.07815).
-"""
-
-FEV_BENCHMARK_BASIC_INFO = f"""
-Results for various forecasting models on 100 tasks of the **fev-bench** benchmark, as described in [fev-bench: A Realistic Benchmark for Time Series Forecasting](https://arxiv.org/abs/2509.26468). {LEGEND}
-"""
-
-FEV_BENCHMARK_DETAILS = f"""
-{TABLE_INFO}
-
-Task definitions and the detailed results are available on [GitHub](https://github.com/autogluon/fev/tree/main/benchmarks/fev_bench/). For model wrappers, see [`models/`](https://github.com/autogluon/fev/tree/main/models). Datasets used for evaluation are available on [Hugging Face](https://huggingface.co/datasets/autogluon/fev_datasets).
-"""
-
-CITATION_HEADER = """
-If you find this leaderboard useful for your research, please consider citing the associated paper(s):
-
-"""
-CITATION_FEV = """
-```
-@article{shchur2025fev,
-  title={{fev-bench}: A Realistic Benchmark for Time Series Forecasting},
-  author={Shchur, Oleksandr and Ansari, Abdul Fatir and Turkmen, Caner and Stella, Lorenzo and Erickson, Nick and Guerron, Pablo and Bohlke-Schneider, Michael and Wang, Yuyang},
-  year={2025},
-  eprint={2509.26468},
-  archivePrefix={arXiv},
-  primaryClass={cs.LG}
-}
-```
-"""
-
-
-def get_pivot_legend(baseline_model: str, leakage_imputation_model: str) -> str:
-    return f"""
-Task definitions and raw results in CSV format are available on [GitHub](https://github.com/autogluon/fev/tree/main/benchmarks/fev_bench).
-
-Best results for each task are marked with
-<span style='background: {COLORS["gold"]}; color: {COLORS["text_default"]}; padding: 3px; border-radius: 5px;'>🥇 1st</span>
-<span style='background: {COLORS["silver"]}; color: {COLORS["text_default"]}; padding: 3px; border-radius: 5px;'>🥈 2nd</span>
-<span style='background: {COLORS["bronze"]}; color: {COLORS["text_default"]}; padding: 3px; border-radius: 5px;'>🥉 3rd</span>
-<br><br>
-**Imputation:**
-- <span style='color: {COLORS["failure_impute"]}; font-weight: bold;'>Failed tasks</span> imputed by {baseline_model}
-- <span style='color: {COLORS["leakage_impute"]}; font-weight: bold;'>Leaky tasks</span> imputed by {leakage_imputation_model}
-"""
-
-
-PAIRWISE_BENCHMARK_DETAILS = """
-The pairwise charts show head-to-head results between models:
-
-* **Win rate**: Percentage of tasks where Model 1 achieves lower error than Model 2 (ties count as half-wins).
-  A value above 50% means Model 1 is more accurate than Model 2 on average.
-
-* **Skill score**: Average relative error reduction of Model 1 with respect to Model 2.
-  A positive value means Model 1 reduces forecasting error compared to Model 2 on average.
-
-**Confidence Intervals**: 95% intervals are estimated using 1000 bootstrap samples over tasks.
-For each bootstrap sample, tasks are resampled with replacement and the pairwise win rate / skill score are recomputed.
-The intervals correspond to the 2.5th and 97.5th percentiles of these bootstrap distributions,
-capturing how model comparisons vary under alternative benchmark compositions.
-"""
-
-
-CITATION_CHRONOS = """
-```
-@article{ansari2024chronos,
-  title={Chronos: Learning the Language of Time Series},
-  author={Ansari, Abdul Fatir and Stella, Lorenzo and Turkmen, Caner and Zhang, Xiyuan, and Mercado, Pedro and Shen, Huibin and Shchur, Oleksandr and Rangapuram, Syama Syndar and Pineda Arango, Sebastian and Kapoor, Shubham and Zschiegner, Jasper and Maddix, Danielle C. and Wang, Hao and Mahoney, Michael W. and Torkkola, Kari and Gordon Wilson, Andrew and Bohlke-Schneider, Michael and Wang, Yuyang},
-  journal={Transactions on Machine Learning Research},
-  issn={2835-8856},
-  year={2024},
-  url={https://openreview.net/forum?id=gerNCVqqtR}
-}
-```
-"""

src/task_groups.py

@@ -1,266 +0,0 @@
-"""Task groupings for filtering the leaderboard by subsets."""
-
-# All tasks in the benchmark (100 tasks)
-ALL_TASKS = [
-    "ETT_15T", "ETT_1D", "ETT_1H", "ETT_1W",
-    "LOOP_SEATTLE_1D", "LOOP_SEATTLE_1H", "LOOP_SEATTLE_5T",
-    "M_DENSE_1D", "M_DENSE_1H",
-    "SZ_TAXI_15T", "SZ_TAXI_1H",
-    "australian_tourism",
-    "bizitobs_l2c_1H", "bizitobs_l2c_5T",
-    "boomlet_1062", "boomlet_1209", "boomlet_1225", "boomlet_1230", "boomlet_1282",
-    "boomlet_1487", "boomlet_1631", "boomlet_1676", "boomlet_1855", "boomlet_1975",
-    "boomlet_2187", "boomlet_285", "boomlet_619", "boomlet_772", "boomlet_963",
-    "ecdc_ili",
-    "entsoe_15T", "entsoe_1H", "entsoe_30T",
-    "epf_be", "epf_de", "epf_fr", "epf_np", "epf_pjm",
-    "ercot_1D", "ercot_1H", "ercot_1M", "ercot_1W",
-    "favorita_stores_1D", "favorita_stores_1M", "favorita_stores_1W",
-    "favorita_transactions_1D", "favorita_transactions_1M", "favorita_transactions_1W",
-    "fred_md_2025/cee", "fred_md_2025/macro",
-    "fred_qd_2025/cee", "fred_qd_2025/macro",
-    "gvar",
-    "hermes",
-    "hierarchical_sales_1D", "hierarchical_sales_1W",
-    "hospital", "hospital_admissions_1D", "hospital_admissions_1W",
-    "jena_weather_10T", "jena_weather_1D", "jena_weather_1H",
-    "kdd_cup_2022_10T", "kdd_cup_2022_1D", "kdd_cup_2022_30T",
-    "m5_1D", "m5_1M", "m5_1W",
-    "proenfo_gfc12", "proenfo_gfc14", "proenfo_gfc17",
-    "redset_15T", "redset_1H", "redset_5T",
-    "restaurant",
-    "rohlik_orders_1D", "rohlik_orders_1W", "rohlik_sales_1D", "rohlik_sales_1W",
-    "rossmann_1D", "rossmann_1W",
-    "solar_1D", "solar_1W", "solar_with_weather_15T", "solar_with_weather_1H",
-    "uci_air_quality_1D", "uci_air_quality_1H",
-    "uk_covid_nation_1D/cumulative", "uk_covid_nation_1D/new",
-    "uk_covid_nation_1W/cumulative", "uk_covid_nation_1W/new",
-    "uk_covid_utla_1D/new", "uk_covid_utla_1W/cumulative",
-    "us_consumption_1M", "us_consumption_1Q", "us_consumption_1Y",
-    "walmart",
-    "world_co2_emissions", "world_life_expectancy", "world_tourism",
-]
-
-# Mini benchmark - representative subset (20 tasks)
-MINI_TASKS = [
-    "jena_weather_1H",
-    "M_DENSE_1D",
-    "bizitobs_l2c_5T",
-    "rohlik_orders_1D",
-    "boomlet_1282",
-    "rossmann_1D",
-    "rossmann_1W",
-    "boomlet_1676",
-    "solar_with_weather_1H",
-    "boomlet_619",
-    "uci_air_quality_1H",
-    "uk_covid_nation_1D/cumulative",
-    "us_consumption_1Y",
-    "epf_np",
-    "world_co2_emissions",
-    "ETT_15T",
-    "ETT_1H",
-    "proenfo_gfc14",
-    "hospital_admissions_1D",
-    "hospital_admissions_1W",
-]
-
-# Frequency-based groupings
-FREQUENCY_GROUPS = {
-    "Sub-hourly": [
-        # T (1 minute)
-        "boomlet_1225", "boomlet_1282", "boomlet_285", "boomlet_619", "boomlet_772", "boomlet_963",
-        # 5T (5 minutes)
-        "LOOP_SEATTLE_5T", "bizitobs_l2c_5T", "redset_5T",
-        "boomlet_1062", "boomlet_1209", "boomlet_1230", "boomlet_1487",
-        # 10T (10 minutes)
-        "jena_weather_10T", "kdd_cup_2022_10T",
-        # 15T (15 minutes)
-        "ETT_15T", "SZ_TAXI_15T", "entsoe_15T", "redset_15T", "solar_with_weather_15T",
-        # 30T (30 minutes)
-        "entsoe_30T", "kdd_cup_2022_30T", "boomlet_1631", "boomlet_1676",
-    ],
-    "Hourly": [
-        "ETT_1H", "LOOP_SEATTLE_1H", "M_DENSE_1H", "SZ_TAXI_1H",
-        "bizitobs_l2c_1H", "entsoe_1H", "ercot_1H",
-        "epf_be", "epf_de", "epf_fr", "epf_np", "epf_pjm",
-        "jena_weather_1H",
-        "proenfo_gfc12", "proenfo_gfc14", "proenfo_gfc17",
-        "redset_1H", "solar_with_weather_1H", "uci_air_quality_1H",
-        "boomlet_1855", "boomlet_1975", "boomlet_2187",
-    ],
-    "Daily": [
-        "ETT_1D", "LOOP_SEATTLE_1D", "M_DENSE_1D",
-        "ercot_1D", "kdd_cup_2022_1D", "solar_1D",
-        "favorita_stores_1D", "favorita_transactions_1D",
-        "hierarchical_sales_1D", "m5_1D",
-        "restaurant",
-        "rohlik_orders_1D", "rohlik_sales_1D", "rossmann_1D",
-        "jena_weather_1D", "uci_air_quality_1D",
-        "hospital_admissions_1D",
-        "uk_covid_nation_1D/cumulative", "uk_covid_nation_1D/new", "uk_covid_utla_1D/new",
-    ],
-    "Weekly": [
-        "ETT_1W", "ercot_1W", "solar_1W",
-        "favorita_stores_1W", "favorita_transactions_1W",
-        "hierarchical_sales_1W", "m5_1W",
-        "hermes", "walmart",
-        "rohlik_orders_1W", "rohlik_sales_1W", "rossmann_1W",
-        "ecdc_ili",
-        "hospital_admissions_1W",
-        "uk_covid_nation_1W/cumulative", "uk_covid_nation_1W/new", "uk_covid_utla_1W/cumulative",
-    ],
-    "Monthly+": [
-        # Monthly
-        "ercot_1M",
-        "favorita_stores_1M", "favorita_transactions_1M", "m5_1M",
-        "fred_md_2025/cee", "fred_md_2025/macro",
-        "hospital",
-        "us_consumption_1M",
-        # Quarterly
-        "australian_tourism", "gvar",
-        "fred_qd_2025/cee", "fred_qd_2025/macro",
-        "us_consumption_1Q",
-        # Yearly
-        "us_consumption_1Y",
-        "world_co2_emissions", "world_life_expectancy", "world_tourism",
-    ],
-}
-
-# Domain-based groupings
-DOMAIN_GROUPS = {
-    "Energy": [
-        "ETT_15T", "ETT_1D", "ETT_1H", "ETT_1W",
-        "entsoe_15T", "entsoe_1H", "entsoe_30T",
-        "epf_be", "epf_de", "epf_fr", "epf_np", "epf_pjm",
-        "ercot_1D", "ercot_1H", "ercot_1M", "ercot_1W",
-        "kdd_cup_2022_10T", "kdd_cup_2022_1D", "kdd_cup_2022_30T",
-        "proenfo_gfc12", "proenfo_gfc14", "proenfo_gfc17",
-        "solar_1D", "solar_1W", "solar_with_weather_15T", "solar_with_weather_1H",
-    ],
-    "Retail": [
-        "favorita_stores_1D", "favorita_stores_1M", "favorita_stores_1W",
-        "favorita_transactions_1D", "favorita_transactions_1M", "favorita_transactions_1W",
-        "hermes",
-        "hierarchical_sales_1D", "hierarchical_sales_1W",
-        "m5_1D", "m5_1M", "m5_1W",
-        "restaurant",
-        "rohlik_orders_1D", "rohlik_orders_1W", "rohlik_sales_1D", "rohlik_sales_1W",
-        "rossmann_1D", "rossmann_1W",
-        "walmart",
-    ],
-    "Nature": [
-        "jena_weather_10T", "jena_weather_1D", "jena_weather_1H",
-        "uci_air_quality_1D", "uci_air_quality_1H",
-    ],
-    "Cloud": [
-        "bizitobs_l2c_1H", "bizitobs_l2c_5T",
-        "boomlet_1062", "boomlet_1209", "boomlet_1225", "boomlet_1230", "boomlet_1282",
-        "boomlet_1487", "boomlet_1631", "boomlet_1676", "boomlet_1855", "boomlet_1975",
-        "boomlet_2187", "boomlet_285", "boomlet_619", "boomlet_772", "boomlet_963",
-        "redset_15T", "redset_1H", "redset_5T",
-    ],
-    "Health": [
-        "ecdc_ili",
-        "hospital", "hospital_admissions_1D", "hospital_admissions_1W",
-        "uk_covid_nation_1D/cumulative", "uk_covid_nation_1D/new",
-        "uk_covid_nation_1W/cumulative", "uk_covid_nation_1W/new",
-        "uk_covid_utla_1D/new", "uk_covid_utla_1W/cumulative",
-    ],
-    "Econ": [
-        "australian_tourism",
-        "fred_md_2025/cee", "fred_md_2025/macro",
-        "fred_qd_2025/cee", "fred_qd_2025/macro",
-        "gvar",
-        "us_consumption_1M", "us_consumption_1Q", "us_consumption_1Y",
-        "world_co2_emissions", "world_life_expectancy", "world_tourism",
-    ],
-    "Mobility": [
-        "LOOP_SEATTLE_1D", "LOOP_SEATTLE_1H", "LOOP_SEATTLE_5T",
-        "M_DENSE_1D", "M_DENSE_1H",
-        "SZ_TAXI_15T", "SZ_TAXI_1H",
-    ],
-}
-# Task type groupings (overlapping: a multivariate task with covariates
-# belongs to both "Covariate-informed" and "Multivariate").
-# - Multivariate: target is a list of columns
-# - Covariate-informed: known_dynamic_columns or past_dynamic_columns is non-empty
-# - Univariate: neither of the above
-TASK_TYPE_GROUPS = {
-    "Univariate": [
-        "LOOP_SEATTLE_1D", "LOOP_SEATTLE_1H", "LOOP_SEATTLE_5T",
-        "M_DENSE_1D", "M_DENSE_1H",
-        "SZ_TAXI_15T", "SZ_TAXI_1H",
-        "australian_tourism", "ecdc_ili",
-        "ercot_1D", "ercot_1H", "ercot_1M", "ercot_1W",
-        "hierarchical_sales_1D", "hierarchical_sales_1W",
-        "hospital", "hospital_admissions_1D", "hospital_admissions_1W",
-        "redset_15T", "redset_1H", "redset_5T",
-        "restaurant",
-        "solar_1D", "solar_1W",
-        "uk_covid_utla_1D/new", "uk_covid_utla_1W/cumulative",
-        "us_consumption_1M", "us_consumption_1Q", "us_consumption_1Y",
-        "world_co2_emissions", "world_life_expectancy", "world_tourism",
-    ],
-    "Covariate-informed": [
-        "entsoe_15T", "entsoe_1H", "entsoe_30T",
-        "epf_be", "epf_de", "epf_fr", "epf_np", "epf_pjm",
-        "favorita_stores_1D", "favorita_stores_1M", "favorita_stores_1W",
-        "favorita_transactions_1D", "favorita_transactions_1M", "favorita_transactions_1W",
-        "fred_md_2025/cee", "fred_qd_2025/cee",
-        "gvar", "hermes",
-        "kdd_cup_2022_10T", "kdd_cup_2022_1D", "kdd_cup_2022_30T",
-        "m5_1D", "m5_1M", "m5_1W",
-        "proenfo_gfc12", "proenfo_gfc14", "proenfo_gfc17",
-        "rohlik_orders_1D", "rohlik_orders_1W", "rohlik_sales_1D", "rohlik_sales_1W",
-        "rossmann_1D", "rossmann_1W",
-        "solar_with_weather_15T", "solar_with_weather_1H",
-        "uci_air_quality_1D", "uci_air_quality_1H",
-        "uk_covid_nation_1D/cumulative", "uk_covid_nation_1D/new",
-        "uk_covid_nation_1W/cumulative", "uk_covid_nation_1W/new",
-        "walmart",
-    ],
-    "Multivariate": [
-        "ETT_15T", "ETT_1D", "ETT_1H", "ETT_1W",
-        "bizitobs_l2c_1H", "bizitobs_l2c_5T",
-        "boomlet_1062", "boomlet_1209", "boomlet_1225", "boomlet_1230", "boomlet_1282",
-        "boomlet_1487", "boomlet_1631", "boomlet_1676", "boomlet_1855", "boomlet_1975",
-        "boomlet_2187", "boomlet_285", "boomlet_619", "boomlet_772", "boomlet_963",
-        "fred_md_2025/cee", "fred_md_2025/macro", "fred_qd_2025/cee", "fred_qd_2025/macro",
-        "gvar",
-        "jena_weather_10T", "jena_weather_1D", "jena_weather_1H",
-        "uci_air_quality_1D", "uci_air_quality_1H",
-        "uk_covid_nation_1D/cumulative", "uk_covid_nation_1D/new",
-        "uk_covid_nation_1W/cumulative", "uk_covid_nation_1W/new",
-    ],
-}
-
-
-def get_task_group(group_type: str, group_value: str | None = None) -> list[str]:
-    """Get the list of tasks for a given group type and value.
-
-    Args:
-        group_type: One of "full", "mini", "frequency", "domain", "task_type"
-        group_value: Required for "frequency", "domain", and "task_type" types
-
-    Returns:
-        List of task names belonging to the group
-    """
-    if group_type == "full":
-        return ALL_TASKS
-    elif group_type == "mini":
-        return MINI_TASKS
-    elif group_type == "frequency":
-        if group_value is None:
-            raise ValueError("group_value required for frequency grouping")
-        return FREQUENCY_GROUPS[group_value]
-    elif group_type == "domain":
-        if group_value is None:
-            raise ValueError("group_value required for domain grouping")
-        return DOMAIN_GROUPS[group_value]
-    elif group_type == "task_type":
-        if group_value is None:
-            raise ValueError("group_value required for task_type grouping")
-        return TASK_TYPE_GROUPS[group_value]
-    else:
-        raise ValueError(f"Unknown group_type: {group_type}")

src/utils.py

@@ -1,401 +0,0 @@
-import functools
-from pathlib import Path
-
-import altair as alt
-import fev
-import pandas as pd
-import pandas.io.formats.style
-
-# Color constants - all colors defined in one place
-
-COLORS = {
-    "bar_fill": "#8d5eb7",
-    "error_bar": "#222222",
-    "point": "#111111",
-    "text_white": "white",
-    "text_black": "black",
-    "text_default": "#111",
-    "gold": "#F7D36B",
-    "silver": "#E5E7EB",
-    "bronze": "#E6B089",
-    "leakage_impute": "#3B82A0",
-    "failure_impute": "#E07B39",
-}
-HEATMAP_COLOR_SCHEME = "purplegreen"
-
-# Display metadata for every model on the leaderboard. Generated by save_tables.py from
-# benchmarks/fev_bench/models.yaml in the fev repo, where submitters declare it themselves.
-MODELS_CSV = Path(__file__).resolve().parent.parent / "tables" / "models.csv"
-
-# Human-readable labels for the model types.
-MODEL_TYPE_LABELS = {
-    "pretrained": "Pretrained",
-    "task-specific": "Task-specific",
-    "statistical": "Statistical",
-    "system": "System",
-    "closed-api": "Closed API",
-}
-# Tags shown in the leaderboard's Tags column, in display order, with the chip color for each.
-# Pale tints drawn from the page palette (the purple bars, the purple-green heatmap and the medal
-# colors), so the tags stay readable without competing with the numbers. Non-commercial gets the warmest tint,
-# as the one tag readers usually scan for.
-ZERO_SHOT_TAG = "Zero-shot"
-NON_COMMERCIAL_TAG = "Non-commercial"
-TAG_COLORS = {
-    "Pretrained": "#E9DDF5",  # pale lavender
-    "Task-specific": "#DCE6F2",  # pale blue
-    "Statistical": "#ECE6DD",  # pale sand
-    "System": "#DCEEE4",  # pale sage
-    "Closed API": "#F3DEDE",  # pale rose
-    ZERO_SHOT_TAG: "#ECEDEF",  # neutral gray
-    NON_COMMERCIAL_TAG: "#F7E2B0",  # pale gold
-}
-# Types shown unless the reader opts into more. Results for the others either cannot be reproduced
-# independently (closed-api) or are not comparable to single models (system).
-DEFAULT_VISIBLE_MODEL_TYPES = ("pretrained", "task-specific", "statistical")
-
-
-@functools.lru_cache(maxsize=1)
-def get_model_metadata() -> dict[str, dict]:
-    """Display metadata for each model, keyed by the `model_name` used in the results."""
-    df = pd.read_csv(MODELS_CSV)
-    return {row["model_name"]: row for row in df.to_dict("records")}
-
-
-def validate_model_metadata(model_names) -> None:
-    """Raise if any displayed model lacks metadata, so the app never shows blank attributes."""
-    metadata = get_model_metadata()
-    missing = sorted({m for m in model_names if m not in metadata})
-    if missing:
-        raise ValueError(
-            f"Missing metadata (organization, link, type) for models: {missing}. "
-            "Add an entry to benchmarks/fev_bench/models.yaml in the fev repo, "
-            "then regenerate the tables with `uv run python save_tables.py`."
-        )
-
-
-def select_model_names(model_types, include_non_commercial: bool = True) -> list[str]:
-    """Names of the models with a type in `model_types`, optionally excluding non-commercial ones."""
-    model_types = set(model_types)
-    return sorted(
-        name
-        for name, entry in get_model_metadata().items()
-        if entry["model_type"] in model_types and (include_non_commercial or entry["commercial_use"])
-    )
-
-
-ALL_METRICS = {
-    "SQL": (
-        "SQL: Scaled Quantile Loss",
-        "The [Scaled Quantile Loss (SQL)](https://auto.gluon.ai/dev/tutorials/timeseries/forecasting-metrics.html#autogluon.timeseries.metrics.SQL) is a **scale-invariant** metric for evaluating **probabilistic** forecasts.",
-    ),
-    "MASE": (
-        "MASE: Mean Absolute Scaled Error",
-        "The [Mean Absolute Scaled Error (MASE)](https://auto.gluon.ai/dev/tutorials/timeseries/forecasting-metrics.html#autogluon.timeseries.metrics.MASE) is a **scale-invariant** metric for evaluating **point** forecasts.",
-    ),
-    "WQL": (
-        "WQL: Weighted Quantile Loss",
-        "The [Weighted Quantile Loss (WQL)](https://auto.gluon.ai/dev/tutorials/timeseries/forecasting-metrics.html#autogluon.timeseries.metrics.WQL), is a **scale-dependent** metric for evaluating **probabilistic** forecasts.",
-    ),
-    "WAPE": (
-        "WAPE: Weighted Absolute Percentage Error",
-        "The [Weighted Absolute Percentage Error (WAPE)](https://auto.gluon.ai/dev/tutorials/timeseries/forecasting-metrics.html#autogluon.timeseries.metrics.WAPE) is a **scale-dependent** metric for evaluating **point** forecasts.",
-    ),
-}
-
-
-def format_metric_name(metric_name: str):
-    return ALL_METRICS[metric_name][0]
-
-
-def get_metric_description(metric_name: str):
-    return ALL_METRICS[metric_name][1]
-
-
-def get_model_link(model_name):
-    return get_model_metadata()[model_name]["url"]
-
-
-def get_model_organization(model_name):
-    return get_model_metadata()[model_name]["organization"]
-
-
-def get_model_type(model_name):
-    return get_model_metadata()[model_name]["model_type"]
-
-
-def get_model_type_label(model_name):
-    return MODEL_TYPE_LABELS[get_model_type(model_name)]
-
-
-def get_model_display_name(model_name):
-    """Display name declared in models.yaml, falling back to the name used in the results."""
-    entry = get_model_metadata().get(model_name)
-    return entry["display_name"] if entry else model_name
-
-
-def get_model_name_link(model_name):
-    """Model URL with the display name as fragment, rendered as a named link by the page's LinkColumn."""
-    return f"{get_model_link(model_name)}#{get_model_display_name(model_name)}"
-
-
-def get_model_tags(model_name) -> list[str]:
-    """Tags describing a model: its type, whether it is zero-shot, and whether its license is non-commercial."""
-    entry = get_model_metadata()[model_name]
-    tags = [MODEL_TYPE_LABELS[entry["model_type"]]]
-    if entry["zero_shot"]:
-        tags.append(ZERO_SHOT_TAG)
-    if not entry["commercial_use"]:
-        tags.append(NON_COMMERCIAL_TAG)
-    return tags
-
-
-def format_leaderboard(df: pd.DataFrame):
-    df = df.copy()
-    df["skill_score"] = df["skill_score"].round(1)
-    df["win_rate"] = df["win_rate"].round(1)
-    # Leakage only applies to zero-shot models; report 0 for models trained on each task
-    is_zero_shot = df["model_name"].map(lambda m: bool(get_model_metadata()[m]["zero_shot"]))
-    df["training_corpus_overlap"] = (
-        (df["training_corpus_overlap"] * 100).round().astype(int).where(is_zero_shot, 0)
-    )
-    # Model type, zero-shot and licensing are folded into a single column of tags to keep the table
-    # narrow. Non-commercial is only shown as the exception.
-    df["tags"] = df["model_name"].map(get_model_tags)
-    df["org"] = df["model_name"].apply(get_model_organization)
-    df["model_name"] = df["model_name"].apply(get_model_name_link)
-    df = df[
-        [
-            "model_name",
-            "win_rate",
-            "skill_score",
-            "median_inference_time_s_per100",
-            "training_corpus_overlap",
-            "num_failures",
-            "org",
-            "tags",
-        ]
-    ]
-    return df.style.apply(
-        lambda x: [
-            "background-color: #f8f9fa" if i % 2 == 1 else "" for i in range(len(x))
-        ],
-        axis=0,
-    )
-
-
-def construct_bar_chart(df: pd.DataFrame, col: str, metric_name: str):
-    label = "Skill Score" if col == "skill_score" else "Win Rate"
-
-    tooltip = [
-        alt.Tooltip("model_name:N"),
-        alt.Tooltip(f"{col}:Q", format=".2f"),
-        alt.Tooltip(f"{col}_lower:Q", title="95% CI Lower", format=".2f"),
-        alt.Tooltip(f"{col}_upper:Q", title="95% CI Upper", format=".2f"),
-    ]
-
-    base_encode = {
-        "y": alt.Y("model_name:N", title="Forecasting Model", sort=None),
-        "tooltip": tooltip,
-    }
-
-    bars = (
-        alt.Chart(df)
-        .mark_bar(color=COLORS["bar_fill"], cornerRadius=4)
-        .encode(
-            x=alt.X(f"{col}:Q", title=f"{label} (%)", scale=alt.Scale(zero=False)),
-            **base_encode,
-        )
-    )
-
-    error_bars = (
-        alt.Chart(df)
-        .mark_errorbar(ticks={"height": 5}, color=COLORS["error_bar"])
-        .encode(
-            y=alt.Y("model_name:N", title=None, sort=None),
-            x=alt.X(f"{col}_lower:Q", title=f"{label} (%)"),
-            x2=alt.X2(f"{col}_upper:Q"),
-            tooltip=tooltip,
-        )
-    )
-
-    points = (
-        alt.Chart(df)
-        .mark_point(filled=True, color=COLORS["point"])
-        .encode(x=alt.X(f"{col}:Q", title=f"{label} (%)"), **base_encode)
-    )
-
-    return (
-        (bars + error_bars + points)
-        .properties(height=500, title=f"{label} ({metric_name}) with 95% CIs")
-        .configure_title(fontSize=16)
-    )
-
-
-def construct_pairwise_chart(df: pd.DataFrame, col: str, metric_name: str):
-    # Color ranges kept in sync with the fev repo's generate_fev_bench_figures.py
-    config = {
-        "win_rate": ("Win Rate", [0, 100], 50, f"abs(datum.{col} - 50) > 30"),
-        "skill_score": ("Skill Score", [-15, 15], 0, f"abs(datum.{col}) > 10"),
-    }
-    cbar_label, domain, domain_mid, text_condition = config[col]
-
-    df = df.copy()
-    for c in [col, f"{col}_lower", f"{col}_upper"]:
-        df[c] *= 100
-
-    df["model_1"] = df["model_1"].apply(get_model_display_name)
-    df["model_2"] = df["model_2"].apply(get_model_display_name)
-
-    model_order = (
-        df.groupby("model_1")[col].mean().sort_values(ascending=False).index.tolist()
-    )
-
-    tooltip = [
-        alt.Tooltip("model_1:N", title="Model 1"),
-        alt.Tooltip("model_2:N", title="Model 2"),
-        alt.Tooltip(f"{col}:Q", title=cbar_label.split(" ")[0], format=".1f"),
-        alt.Tooltip(f"{col}_lower:Q", title="95% CI Lower", format=".1f"),
-        alt.Tooltip(f"{col}_upper:Q", title="95% CI Upper", format=".1f"),
-    ]
-
-    base = alt.Chart(df).encode(
-        x=alt.X(
-            "model_2:N",
-            sort=model_order,
-            title="Model 2",
-            axis=alt.Axis(orient="top", labelAngle=-90),
-        ),
-        y=alt.Y("model_1:N", sort=model_order, title="Model 1"),
-    )
-
-    heatmap = base.mark_rect().encode(
-        color=alt.Color(
-            f"{col}:Q",
-            legend=None,
-            scale=alt.Scale(
-                scheme=HEATMAP_COLOR_SCHEME,
-                domain=domain,
-                domainMid=domain_mid,
-                clamp=True,
-            ),
-        ),
-        tooltip=tooltip,
-    )
-
-    text_main = base.mark_text(dy=-8, fontSize=8, baseline="top", yOffset=5).encode(
-        text=alt.Text(f"{col}:Q", format=".1f"),
-        color=alt.condition(
-            text_condition,
-            alt.value(COLORS["text_white"]),
-            alt.value(COLORS["text_black"]),
-        ),
-        tooltip=tooltip,
-    )
-
-    return (
-        (heatmap + text_main)
-        .properties(
-            height=550,
-            title={
-                "text": f"Pairwise {cbar_label} ({metric_name}) with 95% CIs",
-                "fontSize": 16,
-            },
-        )
-        .configure_axis(labelFontSize=11, titleFontSize=13, titleFontWeight="bold")
-        .resolve_scale(color="independent")
-    )
-
-
-def construct_pivot_table_from_df(
-    errors: pd.DataFrame, metric_name: str
-) -> pd.io.formats.style.Styler:
-    """Construct styled pivot table from precomputed DataFrame."""
-
-    def highlight_by_position(styler):
-        rank_colors = {1: COLORS["gold"], 2: COLORS["silver"], 3: COLORS["bronze"]}
-
-        for row_idx in errors.index:
-            row_ranks = errors.loc[row_idx].rank(method="min")
-            for col_idx in errors.columns:
-                rank = row_ranks[col_idx]
-                style_parts = []
-
-                # Rank background colors
-                if rank <= 3:
-                    style_parts.append(f"background-color: {rank_colors[rank]}")
-                else:
-                    style_parts.append(f"color: {COLORS['text_default']}")
-
-                if style_parts:
-                    styler = styler.map(
-                        lambda x, s="; ".join(style_parts): s,
-                        subset=pd.IndexSlice[row_idx:row_idx, col_idx:col_idx],
-                    )
-        return styler
-
-    return highlight_by_position(errors.style).format(precision=3)
-
-
-def construct_pivot_table(
-    summaries: pd.DataFrame,
-    metric_name: str,
-    baseline_model: str,
-    leakage_imputation_model: str,
-) -> pd.io.formats.style.Styler:
-    errors = fev.pivot_table(
-        summaries=summaries, metric_column=metric_name, task_columns=["task_name"]
-    )
-    train_overlap = (
-        fev.pivot_table(
-            summaries=summaries,
-            metric_column="trained_on_this_dataset",
-            task_columns=["task_name"],
-        )
-        .fillna(False)
-        .astype(bool)
-    )
-
-    is_imputed_baseline = errors.isna()
-    is_leakage_imputed = train_overlap
-
-    # Handle imputations
-    errors = errors.mask(train_overlap, errors[leakage_imputation_model], axis=0)
-    for col in errors.columns:
-        if col != baseline_model:
-            errors[col] = errors[col].fillna(errors[baseline_model])
-
-    errors = errors[errors.rank(axis=1).mean().sort_values().index]
-    errors.index.rename("Task name", inplace=True)
-
-    def highlight_by_position(styler):
-        rank_colors = {1: COLORS["gold"], 2: COLORS["silver"], 3: COLORS["bronze"]}
-
-        for row_idx in errors.index:
-            row_ranks = errors.loc[row_idx].rank(method="min")
-            for col_idx in errors.columns:
-                rank = row_ranks[col_idx]
-                style_parts = []
-
-                # Rank background colors
-                if rank <= 3:
-                    style_parts.append(f"background-color: {rank_colors[rank]}")
-
-                # Imputation text colors
-                if is_leakage_imputed.loc[row_idx, col_idx]:
-                    style_parts.append(f"color: {COLORS['leakage_impute']}")
-                elif is_imputed_baseline.loc[row_idx, col_idx]:
-                    style_parts.append(f"color: {COLORS['failure_impute']}")
-                elif not style_parts or (
-                    len(style_parts) == 1 and "font-weight" in style_parts[0]
-                ):
-                    style_parts.append(f"color: {COLORS['text_default']}")
-
-                if style_parts:
-                    styler = styler.map(
-                        lambda x, s="; ".join(style_parts): s,
-                        subset=pd.IndexSlice[row_idx:row_idx, col_idx:col_idx],
-                    )
-        return styler
-
-    return highlight_by_position(errors.style).format(precision=3)