Spaces:
Running
Running
Commit History
-
Upload from GitHub Actions: Merge pull request #32 from datenlabor-bmz/eval-crash-safe-incremental-runs Running
Verified signatureRecover interrupted eval runs; cap CI to incremental work55e7c3b
-
Upload from GitHub Actions: blocklist: drop the grace for slow failing models, not just egregious ones
Verified signaturePer feedback: the 2-run grace is only worth it when retrying is cheap. main.py now measures per-model seconds/sample during the run and flags rate-limited models (> AUTO_BLOCKLIST_SLOW_SEC_PER_SAMPLE = 0.3s/sample); update_blocklist_ strikes excludes a failing model after ONE run if it's egregious (>=80%) OR slow. The grace now only protects a fast, moderately-failing model — i.e. retry only when the time investment is reasonable.608b646 -
Upload from GitHub Actions: models: drop kimi-k2.6; exclude egregiously-failing models after one run
Verified signature- Remove moonshotai/kimi-k2.6 from important_models: ~58% errors and so heavily rate-limited that a single model ran >50min (it ate a run's 6h budget). kimi-k2 represents Moonshot. - Tiered strikes: a model failing >= AUTO_BLOCKLIST_IMMEDIATE_FAIL_PCT (80%) is excluded after ONE bad run instead of waiting out the 2-run grace — it's broken, not transiently rate-limited. The grace still protects the borderline 50-80% band where a provider outage is plausible.4c601bb
-
Upload from GitHub Actions: eval: check runtime budget per-batch so a slow model can't blow the 6h cap
Verified signatureThe graceful budget was only checked between models, so a single rate-limited model (kimi-k2.6 ran >50min) blew past 5.5h and the run got hard-cancelled at 6h. Now check the budget (a) before starting each model and (b) after every batch — a slow model is interrupted at a batch boundary, its partial work saved to results-detailed (skipped next run) but NOT published (not coverage-complete). Lower the workflow budget to 5h for headroom. Checkpointing already worked: the cancelled run published 42 clean fully-covered models before the cap.594d28a -
Upload from GitHub Actions: eval: fix per-combo resilience (tqdm_asyncio.gather has no return_exceptions)
Verified signatureThe previous resilience commit passed return_exceptions=True to tqdm_asyncio.gather, which it doesn't accept (TqdmKeyError) — crashing the loop immediately. Wrap each task in a _safe() coroutine that returns non-fatal exceptions as values (combo skipped, re-attempted next run) and re-raises FatalAPIError. Verified locally: normal results pass, non-fatal skipped, fatal propagates.18acfdb -
Upload from GitHub Actions: eval: don't let one bad (task,language) combo crash the whole run
Verified signatureA transient HF dataset download (e.g. arc-easy-autotranslated config 'rn' failing to build on the runner) raised uncaught and killed a multi-hour run. Two layers of resilience: - _load_dataset retries the 'Couldn't find cache for config X' ValueError (flaky build) with backoff, bounded so a genuinely-missing config still surfaces. - the eval loop gathers with return_exceptions and SKIPS a combo that still throws (logged, no row) so it's simply re-attempted next run; FatalAPIError still aborts. Transient failures no longer count against a model's strikes.a92221e -
Upload from GitHub Actions: models: migrate catalog to /api/v1/models; enforce privacy per-request
Verified signatureOpenRouter removed the undocumented /api/frontend/models endpoint (404), which crashed load_models at import on every fresh run/backend start. Rebuild the catalog from the official /api/v1/models and normalize it to the shape the rest of the module expects (slug/permaslug/created_at/endpoint.pricing/hf_slug from hugging_face_id), with retry/backoff on the fetch. That catalog no longer exposes per-provider data policy, so privacy moves to a STRONGER per-request guarantee: complete() now sends provider.data_collection= 'deny', so OpenRouter refuses to route any prompt to a training/retaining provider (verified: gpt-5->OpenAI, deepseek->DeepInfra, qwen->Parasail, all training=false). The catalog-level dataPolicy check is kept as a no-op for shape compatibility. Also drop 'openrouter' from the discovery provider allowlist (its namespace is routing meta-models / cloaked test models, not real LLMs).f502bec -
Upload from GitHub Actions: discovery: surface newer flagships from curated families; blocklist: require 2 consecutive bad runs
Verified signaturediscovery: skip a candidate only if we already curate something in its family that is at least as NEW (version-aware), so a future Claude Opus 5 / GPT-6 is still auto-surfaced — the coarse family key alone would have permanently blocked every model from a lab we track. auto-blocklist: replace the single-snapshot threshold with a persisted strike counter (HF dataset 'model-health-strikes'). A model past the failure threshold earns one strike but is NOT excluded; it must stay bad for AUTO_BLOCKLIST_MIN_RUNS (2) consecutive runs before exclusion, and any recovering run resets it. This stops a provider rate-limiting us during one long run from wrongly excluding healthy open models (they get re-attempted next run). load_auto_blocklist is now a cheap read of the strike table (also speeds up backend startup); main.py updates strikes once per run.f28fed1
-
Upload from GitHub Actions: discovery: one flagship per product line; eval: graceful 6h-safe runtime budget
Verified signature- _family_key now strips size-tier/variant suffixes and parameter-size tokens in addition to versions/dates, collapsing size sweeps so auto-discovery keeps a single flagship per product line (highest cost wins). Cuts a recent discovery run from 50 candidates to 14. Specific variants still go in important_models by hand. - main.py honors MAX_RUNTIME_SECONDS: after a model's checkpoint, if the wall- clock budget is exceeded it stops gracefully (exit 0) with remaining models deferred to the next run. The weekly workflow sets 5.5h so a run never hits the GitHub 6h hard-cancel — it exits clean, restarts the Space, and resumes.4047210
-
Upload from GitHub Actions: main: gate publishing on coverage-completeness, not just error rate
Verified signatureThe previous publish gate only filtered high-error models, so a model with sparse-but-successful history (e.g. 5 of ~200 languages) passed and showed an inflated mean — the exact failure that put a small model at rank #1. Now a model is publishable only once it is coverage-complete (zero pending combis: either fully evaluated before this run, or finished in it) AND healthy. Verified against live results-detailed: publishable set drops from 88 (incl. 5-language models) to 19, all with full 221-language coverage.c1041db -
Upload from GitHub Actions: util: retry HF push with backoff; write local snapshot before push
Verified signaturePer-model checkpointing pushes to HF frequently; HF intermittently 429s or drops connections. Retry the push 5x with exponential backoff so a transient failure doesn't kill a multi-hour run, and persist the local JSON before the push so on-disk progress survives a push failure.2a1f0a5
-
Upload from GitHub Actions: main: checkpoint per fully-evaluated model instead of once at the end
Verified signatureThe eval loop previously saved only after ALL batches finished, so a run killed by the GitHub 6h runner cap (or any interruption) persisted nothing while still being billed for the API calls. Now evaluate one model at a time and push to HF after each model completes its entire matrix. Two guarantees: - progress survives interruption (next run skips fully-logged models), so a large onboarding completes across multiple runs; - a model only enters the published aggregate once fully covered AND healthy (publishable_models excludes models past the auto-blocklist failure threshold), so a sparse/broken model can't show an inflated mean. Lifts publishable_models/checkpoint to module level; unit-tested the gate.eaa7534
-
Upload from GitHub Actions: models: replace claude-opus-4.5/4.6 with 4.8 in curated list
Verified signatureKeep one current Opus flagship lineage: 4.8 (newest) + 4.7. Drop the superseded 4.5/4.6 from important_models.15e8f68 -
Upload from GitHub Actions: discovery: filter out voice/ASR/vision/build endpoints
Verified signatureAuto-discovery was surfacing non-text endpoints (microsoft/mai-voice-2, nvidia/parakeet ASR, glm-4.5v / glm-5v-turbo vision, x-ai/grok-build). Add voice/parakeet/canary/build skip rules and a regex for the bare-"v" vision suffix that the -vision/-vl tags miss.f2add9e
-
Upload from GitHub Actions: backend: handle null creation_date in three apply() calls
Verified signatureThe aggregated results may contain rows for models that don't have metadata in models.json (e.g. when results-detailed includes historical models that have since rotated out of important_models). The results+models left-join leaves those rows with NaN/NaT creation_date, which the old "x.isoformat() if x else None" lambdas tripped over because pandas NaT is truthy and a float-NaN doesn't have .isoformat. Replace with "pd.notna(x) and hasattr(x, 'isoformat')" in all three spots: make_model_table, make_language_tier_history, make_license_history. Surfaced after this morning's results-restore widened the aggregate to historical models that aren't in the current cohort; the HF Space returned 500 for every /api/data call ("Network response was not ok" in the frontend).f6a28ed -
Upload from GitHub Actions: fast-fail on account-level API errors; refuse to ship runs with >80% errors
Verified signatureToday's workflow run hit a 403 'Key limit exceeded' from OpenRouter on the very first request and would have kept burning through the entire combinatorial matrix tagging every row with status=error. We caught it manually and cancelled before save() fired, but the fragility is real: any future account-level outage (auth, credit cap, key revoke) would flood results-detailed with bogus errors and then auto_blocklist would mass-exclude the auto-discovered models from the *next* run, even after the underlying issue is fixed. Two layers of protection. Layer 1 - propagate account-level failures (evals/models.py + tasks.py): - Add FatalAPIError. Raised by complete() when the OpenRouter error message matches markers like "key limit exceeded", "insufficient credits", "unauthorized", "payment required", "invalid api key". - query() in tasks.py re-raises FatalAPIError instead of swallowing it into status=error. asyncio.gather then surfaces it, the eval loop aborts, and save() …c2afc16 -
Upload from GitHub Actions: unblock workflow: materialize gcloud creds on runner; lazy-init translate client
Verified signatureThe nightly workflow has been silently failing since at least Dec 2025 because the eval pipeline depends on gcloud_credentials.json for the Google Translate baseline but the file is gitignored and there was no GHA secret/step to materialize it on the runner. Also, evals/models.py called translate.Client() at module import time, so any import of the module (including backend.py, smoke tests, any contributor without GCP locally) blew up with DefaultCredentialsError. .github/workflows/nightly-evals.yml - Add a "Write Google Cloud credentials" step that decodes the GCLOUD_CREDENTIALS_JSON secret into gcloud_credentials.json at the repo root + validates the file parses as JSON (fail-fast). - Set GOOGLE_APPLICATION_CREDENTIALS=gcloud_credentials.json in the "Run evaluations" env so the google-auth lookup finds it. evals/models.py - Replace module-level `translate_client = translate.Client()` with a lazy factory `_get_translate_client()` that defers instantiation u…bec2f46 -
Upload from GitHub Actions: guard main.py against partial-scale HF pushes; restore aggregated results
Verified signatureA smoke run earlier today (N_LANGUAGES=5 N_MODELS=150) silently overwrote fair-forward/evals-for-every-language-results with a filtered subset: the aggregation step at main.py:97 filters results_agg to current_models x current_languages, then save() pushes regardless of scale, so a 5-language smoke truncated the dashboard from ~1000 langs to 5. results-detailed was untouched (it merges full history), which made recovery possible. Restoration: re-aggregated the entire results-detailed table (701k rows, 135 models, 221 langs -> 70.9k aggregate rows) and pushed only the `results` dataset back. Local results/*.json snapshots now reflect the restored state. Footgun patch: - evals/main.py: * Add CANONICAL_N_LANGUAGES_FOR_PUSH (1000) and CANONICAL_N_MODELS_FOR_PUSH (100) constants. * Compute ALLOW_HF_PUSH_RESULTS at module load. When either env scale is below canonical, route results/models/languages through save_local_only() instead of save(), keeping the …691e6c2
-
Upload from GitHub Actions: refresh pyproject metadata + README HF frontmatter
Verified signaturepyproject.toml - description: real one (was "Add your description here") - drop bert-score and tiktoken from the dev group; neither is imported anywhere. Kept elevenlabs + jiwer because they pair with the dormant FLEURS/CommonVoice ASR pipeline. README.md (HF Spaces frontmatter) - models: refreshed to 8 representative current frontier models (was 6 stale entries including openai/whisper-large-v3 which is ASR and not active). Added a comment noting evals/models.py is canonical (the full curated list is ~42). - datasets: dropped mozilla-foundation/common_voice_1_0 and google/fleurs (ASR dormant); added the Masakhane benchmarks that actually drive multilingual coverage. - language: tags expanded from English+German only to a small cross-tier sample (CJK, Indic, Romance, Semitic, Bantu, Niger- Congo) for HF discoverability. uv.lock: regenerated.1eccc3f
-
Upload from GitHub Actions: new model
Verified signature78be468
-
Upload from GitHub Actions: added new models
Verified signature83d2972
-
Upload from GitHub Actions: added new models
Verified signature93a8617
-
Upload from GitHub Actions: Merge pull request #28 from datenlabor-bmz/jn-dev
Verified signatureJn dev55b63ea
-
Upload from GitHub Actions: cleaned up code
Verified signature2586cfe -
Upload from GitHub Actions: added opus 4.5
Verified signature0a17acf
-
Upload from GitHub Actions: add gpt-5.1, gemini-3
Verified signature9ea2dd3
-
Upload from GitHub Actions: flores filter for available dev split
Verified signature34b05c6
-
Upload from GitHub Actions: model name no bracket stuff
Verified signatureaa92add -
Upload from GitHub Actions: drop normalization
Verified signature972026c -
Upload from GitHub Actions: improve norwegian fix
Verified signature6f0e312
-
Upload from GitHub Actions: fix norwegian
Verified signature0cbac6c -
Upload from GitHub Actions: Merge pull request #22 from datenlabor-bmz/dev
Verified signatureDev2cdada4
-
Upload from GitHub Actions: Add auto-translated datasets
Verified signature68a93b5
-
Upload from GitHub Actions: Merge pull request #18 from datenlabor-bmz/pr-17
Verified signaturePr 17a0d1624
-
Upload from GitHub Actions: Add auto-translated datasets
Verified signaturec790fdb
-
Upload from GitHub Actions: ran full evaluation locally
Verified signature088f96f
-
Upload from GitHub Actions: minor chashing change
Verified signatureb39df3c -
Upload from GitHub Actions: updated and cleaned up scripts for new eval runs
Verified signature963cb78
-
Upload from GitHub Actions: Update models.py, models.json, and results.json with latest evaluation data and model additions
Verified signature8eebb41
-
Upload from GitHub Actions: Add Todos for using existing machine-translated datasets rather than our own ones
Verified signature56adaa2
-
Upload from GitHub Actions: updated translation functions
Verified signature8f5ce26 -
Upload from GitHub Actions: import flexibility on backend
Verified signatureb8cbeff -
Upload from GitHub Actions: fixed import error
Verified signature0a30811 -
Upload from GitHub Actions: updated frontend and backend to fix bugs
Verified signature4e8cb1a
-
Upload from GitHub Actions: Merge pull request #13 from datenlabor-bmz/jn-dev
Verified signatureJn dev80d21cb
-
Upload from GitHub Actions: Merge pull request #10 from datenlabor-bmz/jn-dev
Verified signatureImprove evaluation checkpointing and increased to 24h Actions runningc2eeeac -
Upload from GitHub Actions: updated batch size and delay
Verified signature02f927b -
Upload from GitHub Actions: updated workflow settings
Verified signaturee51c770 -
Upload from GitHub Actions: Merge pull request #9 from datenlabor-bmz/jn-dev
Verified signatureMajor Update - see description7c06aef
-
Upload from GitHub Actions: Merge pull request #7 from datenlabor-bmz/jn-dev
Verified signatureupdated scraping and added visualisation6878a71