Spaces:
Running
Running
Commit History
-
Upload from GitHub Actions: Merge pull request #32 from datenlabor-bmz/eval-crash-safe-incremental-runs Running
Verified signatureRecover interrupted eval runs; cap CI to incremental work55e7c3b
-
Upload from GitHub Actions: eval: don't let one bad (task,language) combo crash the whole run
Verified signatureA transient HF dataset download (e.g. arc-easy-autotranslated config 'rn' failing to build on the runner) raised uncaught and killed a multi-hour run. Two layers of resilience: - _load_dataset retries the 'Couldn't find cache for config X' ValueError (flaky build) with backoff, bounded so a genuinely-missing config still surfaces. - the eval loop gathers with return_exceptions and SKIPS a combo that still throws (logged, no row) so it's simply re-attempted next run; FatalAPIError still aborts. Transient failures no longer count against a model's strikes.a92221e
-
Upload from GitHub Actions: util: retry HF push with backoff; write local snapshot before push
Verified signaturePer-model checkpointing pushes to HF frequently; HF intermittently 429s or drops connections. Retry the push 5x with exponential backoff so a transient failure doesn't kill a multi-hour run, and persist the local JSON before the push so on-disk progress survives a push failure.2a1f0a5
-
Upload from GitHub Actions: guard main.py against partial-scale HF pushes; restore aggregated results
Verified signatureA smoke run earlier today (N_LANGUAGES=5 N_MODELS=150) silently overwrote fair-forward/evals-for-every-language-results with a filtered subset: the aggregation step at main.py:97 filters results_agg to current_models x current_languages, then save() pushes regardless of scale, so a 5-language smoke truncated the dashboard from ~1000 langs to 5. results-detailed was untouched (it merges full history), which made recovery possible. Restoration: re-aggregated the entire results-detailed table (701k rows, 135 models, 221 langs -> 70.9k aggregate rows) and pushed only the `results` dataset back. Local results/*.json snapshots now reflect the restored state. Footgun patch: - evals/main.py: * Add CANONICAL_N_LANGUAGES_FOR_PUSH (1000) and CANONICAL_N_MODELS_FOR_PUSH (100) constants. * Compute ALLOW_HF_PUSH_RESULTS at module load. When either env scale is below canonical, route results/models/languages through save_local_only() instead of save(), keeping the …691e6c2
-
Upload from GitHub Actions: refresh pyproject metadata + README HF frontmatter
Verified signaturepyproject.toml - description: real one (was "Add your description here") - drop bert-score and tiktoken from the dev group; neither is imported anywhere. Kept elevenlabs + jiwer because they pair with the dormant FLEURS/CommonVoice ASR pipeline. README.md (HF Spaces frontmatter) - models: refreshed to 8 representative current frontier models (was 6 stale entries including openai/whisper-large-v3 which is ASR and not active). Added a comment noting evals/models.py is canonical (the full curated list is ~42). - datasets: dropped mozilla-foundation/common_voice_1_0 and google/fleurs (ASR dormant); added the Masakhane benchmarks that actually drive multilingual coverage. - language: tags expanded from English+German only to a small cross-tier sample (CJK, Indic, Romance, Semitic, Bantu, Niger- Congo) for HF discoverability. uv.lock: regenerated.1eccc3f
-
Upload from GitHub Actions: flores filter for available dev split
Verified signature34b05c6
-
Upload from GitHub Actions: improve norwegian fix
Verified signature6f0e312
-
Upload from GitHub Actions: fix norwegian
Verified signature0cbac6c -
Upload from GitHub Actions: Merge pull request #22 from datenlabor-bmz/dev
Verified signatureDev2cdada4
-
Upload from GitHub Actions: Add auto-translated datasets
Verified signature68a93b5
-
Upload from GitHub Actions: Merge pull request #18 from datenlabor-bmz/pr-17
Verified signaturePr 17a0d1624
-
Upload from GitHub Actions: Add auto-translated datasets
Verified signaturec790fdb
-
Upload from GitHub Actions: updated and cleaned up scripts for new eval runs
Verified signature963cb78
-
Upload from GitHub Actions: Add Todos for using existing machine-translated datasets rather than our own ones
Verified signature56adaa2
-
Upload from GitHub Actions: updated translation functions
Verified signature8f5ce26 -
Upload from GitHub Actions: updated frontend and backend to fix bugs
Verified signature4e8cb1a
-
Upload from GitHub Actions: Merge pull request #9 from datenlabor-bmz/jn-dev
Verified signatureMajor Update - see description7c06aef
-
Upload from GitHub Actions: TruthfulQA translation WIP
Verified signaturefd102e9
-
Upload from GitHub Actions: Get more results, compute average based on all tasks
Verified signature98c6811
-
Upload from GitHub Actions: Translate MMLU and evaluate
Verified signature4c5c136 -
Upload from GitHub Actions: Correlation plot
Verified signatureb0aa389
-
Upload from GitHub Actions: Evaluate on autotranslated GSM dataset
Verified signaturef3a09a2
-
Upload from GitHub Actions: Add math benchmarks
Verified signature549360a -
Upload from GitHub Actions: Use FLORES+ via Huggingface
Verified signature913253a
-
Upload from GitHub Actions: Fix vibecoding
Verified signature75010c2
-
Pass through kwargs
David Pomerenke5fa433f -
Fix dataset loading
David Pomerenkec990cb9
-
Fix import paths
David Pomerenkec567aee
-
added download function and edited INFO
f529b7b
-
Only run tasks for which there is no result yet
David Pomerenke2f9dee1
-
Run on 40 languages, additional models
David Pomerenke260c1a3 -
Move functions for sharing them
David Pomerenke55406ba
-
Implement MMLU task
David Pomerenkea683732 -
MMLU data loader for 3 parallel datasets
David Pomerenke47170a5
-
Analyze MMLU datasets
David Pomerenke031925d
-
Refactor eval code into files
David Pomerenkeda6e1bc