feat: run the eval worker inside the Space (CPU, isolated venv)

Make the Space self-contained — it now runs the eval engine itself on CPU,
no external machine needed:

- Dockerfile: build an isolated /opt/worker-venv (CPU torch + open-eeg-bench),
kept OUT of the web app's pinned stack to avoid version conflicts; launch the
worker as a background '--watch' loop alongside uvicorn + the frontend, gated
by RUN_EVAL_WORKER (set =0 in Space settings to disable without a rebuild).
Built before the code COPY (cached across code rebuilds) and non-fatal (a
failed worker-deps install never breaks the web app).
- run_eval_worker.py: import-light (only huggingface_hub + pure app modules, so
it runs in the minimal venv), default --device cpu, add process_queue() and a
--watch/--interval service loop.
- app/services/__init__.py: stop eagerly importing the heavy submodules so the
pure results_mapping can be imported without dragging in datasets/transformers/
aiohttp. All call sites use direct submodule imports, so nothing breaks.

Verified: the worker runs from a venv containing only huggingface_hub+pandas+
pyarrow (no backend stack, no oeb); web app + 16 tests still green.

Dockerfile

@@ -26,6 +26,19 @@ COPY backend/pyproject.toml backend/poetry.lock* ./
 RUN poetry config virtualenvs.create false \
     && poetry install --no-interaction --no-ansi --no-root --only main
 
+# Isolated venv for the eval worker (heavy ML deps: torch + braindecode from git),
+# kept OUT of the web app's pinned stack to avoid version conflicts. Built before
+# the code COPY so it stays cached across code-only rebuilds. Non-fatal: if the
+# install fails, the web app still builds and the worker is simply disabled.
+RUN apt-get update && apt-get install -y --no-install-recommends git \
+    && python -m venv /opt/worker-venv \
+    && /opt/worker-venv/bin/pip install --no-cache-dir -U pip \
+    && (/opt/worker-venv/bin/pip install --no-cache-dir torch --index-url https://download.pytorch.org/whl/cpu \
+        && /opt/worker-venv/bin/pip install --no-cache-dir huggingface_hub pandas pyarrow open-eeg-bench \
+        || echo "WARN: eval-worker deps failed to install; worker disabled at runtime") \
+    && chown -R user:user /opt/worker-venv \
+    && rm -rf /var/lib/apt/lists/*
+
 # Copy backend code
 COPY backend/ .
 
@@ -52,11 +65,14 @@ ENV HF_HOME=/app/.cache \
     HF_DATASETS_CACHE=/app/.cache \
     INTERNAL_API_PORT=7861 \
     PORT=7860 \
-    NODE_ENV=production
+    NODE_ENV=production \
+    RUN_EVAL_WORKER=1
 
 # Note: HF_TOKEN should be provided at runtime, not build time
 USER user
 EXPOSE 7860
 
-# Start both servers with wait-for
-CMD ["sh", "-c", "uvicorn app.asgi:app --host 0.0.0.0 --port 7861 & while ! nc -z localhost 7861; do sleep 1; done && cd frontend && npm run serve"]
+# Start the API, the (optional) CPU eval worker, and the frontend. The worker
+# runs in its own venv as a background --watch loop. Disable it without a rebuild
+# by setting RUN_EVAL_WORKER=0 in the Space settings, then restarting.
+CMD ["sh", "-c", "uvicorn app.asgi:app --host 0.0.0.0 --port 7861 & if [ \"$RUN_EVAL_WORKER\" = \"1\" ]; then PYTHONPATH=/app /opt/worker-venv/bin/python -m scripts.run_eval_worker --watch --limit 1 --interval 300 --device cpu & fi; while ! nc -z localhost 7861; do sleep 1; done && cd frontend && npm run serve"]

backend/app/services/__init__.py

@@ -1,3 +1,8 @@
-from . import hf_service, leaderboard, votes, models
+"""Service layer.
 
-__all__ = ["hf_service", "leaderboard", "votes", "models"]
+Submodules are imported directly (``from app.services.models import ...``) rather
+than eagerly here. This keeps light, dependency-free modules like
+``results_mapping`` importable without dragging in the whole backend stack
+(datasets, transformers, aiohttp, ...), so the eval worker can import it inside a
+minimal isolated venv.
+"""

backend/scripts/run_eval_worker.py

@@ -1,14 +1,20 @@
 """OpenEEGBench eval worker — the engine behind the arena.
 
-This is NOT run by the Space (a CPU web server). Run it on a machine with a GPU
-(or SLURM access) that has ``open-eeg-bench`` installed and an ``HF_TOKEN`` with
-write access to the ``braindecode`` org:
+The eval engine. By default the **Space runs this itself** in a background
+process (see the Dockerfile): a CPU ``--watch`` loop in an *isolated venv*
+(open-eeg-bench + its heavy deps, kept out of the web app's pinned stack) that
+polls the queue and publishes results. **A GPU is optional** — linear/ridge
+probing and inference run fine on CPU (the default); heavier strategies like
+``full_finetune`` are just slower. It can also run on any other machine that has
+``open-eeg-bench`` installed and an ``HF_TOKEN`` with ``braindecode`` write:
 
     pip install -r scripts/worker-requirements.txt   # open-eeg-bench (+ torch, braindecode)
     export HF_TOKEN=hf_xxx
-    python -m scripts.run_eval_worker --dry-run                  # list pending, do nothing
-    python -m scripts.run_eval_worker --device cuda              # process the queue once
-    python -m scripts.run_eval_worker --slurm --infra-folder ./oeb-cache --device cuda
+    python -m scripts.run_eval_worker --dry-run                 # list pending, do nothing
+    python -m scripts.run_eval_worker --limit 1                 # process the queue once, CPU
+    python -m scripts.run_eval_worker --watch --interval 300    # run as a service (what the Space does)
+    python -m scripts.run_eval_worker --device cuda             # ... use a GPU if you have one
+    python -m scripts.run_eval_worker --slurm --infra-folder ./oeb-cache   # ... or fan out via SLURM
 
 For each PENDING request it:
   1. marks the request RUNNING (in braindecode/requests),
@@ -27,10 +33,18 @@ import logging
 from datetime import datetime, timezone
 from pathlib import Path
 
-from app.config.base import HF_TOKEN
-from app.config.hf_config import API as hf_api, QUEUE_REPO, AGGREGATED_REPO
+from huggingface_hub import HfApi
+
+# Import only import-light, dependency-free app modules so this worker can run in
+# its own isolated venv (open-eeg-bench + its heavy deps) without pulling the web
+# app's pinned stack. base.py is pure stdlib; results_mapping is pure Python.
+from app.config.base import HF_TOKEN, HF_ORGANIZATION
 from app.services.results_mapping import map_oeb_results_to_contents
 
+QUEUE_REPO = f"{HF_ORGANIZATION}/requests"
+AGGREGATED_REPO = f"{HF_ORGANIZATION}/contents"
+hf_api = HfApi(token=HF_TOKEN)
+
 logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
 logger = logging.getLogger("eval_worker")
 
@@ -133,22 +147,44 @@ def publish(rows):
     return len(combined)
 
 
+def process_queue(device="cpu", limit=None, infra=None):
+    """Run every PENDING submission once. Returns how many were processed."""
+    pending = load_pending(limit)
+    logger.info("Found %d pending submission(s).", len(pending))
+    for path_in_repo, entry in pending:
+        name = entry.get("model_name", "?")
+        logger.info("RUNNING %s", name)
+        set_status(path_in_repo, entry, "RUNNING", started_time=_now())
+        try:
+            rows = run_benchmark(entry, device, infra)
+            if not rows:
+                raise RuntimeError("benchmark produced no completed results")
+            total = publish(rows)
+            set_status(path_in_repo, entry, "FINISHED", finished_time=_now(), n_rows=len(rows))
+            logger.info("FINISHED %s — published %d row(s); contents now has %d.", name, len(rows), total)
+        except Exception as e:  # noqa: BLE001 — record the failure on the request and move on
+            logger.exception("FAILED %s", name)
+            set_status(path_in_repo, entry, "FAILED", finished_time=_now(), error=str(e)[:500])
+    return len(pending)
+
+
 def main():
+    import time
+
     ap = argparse.ArgumentParser(
         description="Run queued OpenEEGBench submissions and publish results."
     )
     ap.add_argument("--dry-run", action="store_true", help="List pending submissions and exit.")
-    ap.add_argument("--device", default="cuda", help="Torch device for benchmark() (default: cuda).")
-    ap.add_argument("--limit", type=int, default=None, help="Max submissions to process this run.")
+    ap.add_argument("--watch", action="store_true", help="Keep polling the queue (run as a long-lived service).")
+    ap.add_argument("--interval", type=int, default=300, help="Seconds between polls in --watch mode (default 300).")
+    ap.add_argument("--device", default="cpu", help="Torch device for benchmark() (default: cpu; use cuda if a GPU is available).")
+    ap.add_argument("--limit", type=int, default=None, help="Max submissions to process per cycle.")
     ap.add_argument("--infra-folder", default=None, help="oeb cache/results folder (enables caching + SLURM).")
     ap.add_argument("--slurm", action="store_true", help="Submit experiments via SLURM (oeb infra cluster=slurm).")
     args = ap.parse_args()
 
-    pending = load_pending(args.limit)
-    logger.info("Found %d pending submission(s).", len(pending))
-
     if args.dry_run:
-        for _, e in pending:
+        for _, e in load_pending(args.limit):
             logger.info("PENDING %s — %s", e.get("model_name"), (e.get("benchmark_kwargs") or {}).get("model_cls"))
         return
 
@@ -159,20 +195,19 @@ def main():
         infra["cluster"] = "slurm"
     infra = infra or None
 
-    for path_in_repo, entry in pending:
-        name = entry.get("model_name", "?")
-        logger.info("RUNNING %s", name)
-        set_status(path_in_repo, entry, "RUNNING", started_time=_now())
-        try:
-            rows = run_benchmark(entry, args.device, infra)
-            if not rows:
-                raise RuntimeError("benchmark produced no completed results")
-            total = publish(rows)
-            set_status(path_in_repo, entry, "FINISHED", finished_time=_now(), n_rows=len(rows))
-            logger.info("FINISHED %s — published %d row(s); contents now has %d.", name, len(rows), total)
-        except Exception as e:  # noqa: BLE001 — record the failure on the request and move on
-            logger.exception("FAILED %s", name)
-            set_status(path_in_repo, entry, "FAILED", finished_time=_now(), error=str(e)[:500])
+    if args.watch:
+        logger.info(
+            "Worker watching %s every %ds (device=%s, limit=%s).",
+            QUEUE_REPO, args.interval, args.device, args.limit,
+        )
+        while True:
+            try:
+                process_queue(args.device, args.limit, infra)
+            except Exception:  # noqa: BLE001 — never let one cycle kill the worker
+                logger.exception("worker cycle error")
+            time.sleep(args.interval)
+    else:
+        process_queue(args.device, args.limit, infra)
 
 
 if __name__ == "__main__":

backend/scripts/worker-requirements.txt

@@ -1,6 +1,6 @@
 # Extra dependency for the eval worker (scripts/run_eval_worker.py).
-# The worker runs OUTSIDE the Space, on a GPU/SLURM machine. Install it on top
-# of the backend deps (the worker imports app.config.* / app.services.*):
+# The worker runs OUTSIDE the Space, on ANY machine (CPU is fine; GPU optional).
+# Install it on top of the backend deps (the worker imports app.config.* / app.services.*):
 #
 #   cd backend && pip install -e . -r scripts/worker-requirements.txt
 #   # or: poetry install && pip install -r scripts/worker-requirements.txt