Fetching metadata from the HF Docker repository...
-
Add 13 new benchmark datasets (batches 6-8)
Egor Bogomolov
Long Code Arena (6 tasks):
- Library-Based Code Gen, Project-Level Completion,
Bug Localization, Commit Message Gen,
CI Builds Repair, Module Summarization
Additional datasets:
- DPAIA EE-Dataset, Multi-SWE-bench, SWE-bench Multilingual,
CrossCodeEval, Defects4J, McEval, MultiPL-E
Key implementation details:
- CrossCodeEval: load per-language JSONL directly (inconsistent HF columns)
- Multi-SWE-bench: load individual JSONL files (40 files across languages)
- Defects4J: use train split, map bug_id/func_before/func_after fields
- RepoBench dropped (no data files, deprecated loading script)
- LCA CI logs: head+tail trimming (first/last 10k chars by line)
- LCA large fields: explicit trim markers showing original vs trimmed size
- GitHub repo/commit links for all LCA tasks where data is available
- Total datasets: 41 (up from 28) -
Add 28 benchmark datasets with rich visualization views
Egor Bogomolov
Datasets (28 total):
- Code Generation: REval, HumanEval+, MBPP+, MBPP, ClassEval, LiveCodeBench,
APPS, CodeContests, BigOBench, BigCodeBench, EffiBench, CodeSearchNet
- Code Reasoning: CRUXEval, HumanEvalPack (6 langs)
- Code Editing: SWE-bench Lite/Verified/Full, DebugBench, CanItEdit,
CodeEditorBench, CodeXGLUE Refinement, CommitBench
- Code Completion: SAFIM, HumanEval-X (5 langs)
- Vulnerability Detection: BigVul, DiverseVul, PrimeVul, Devign
View types:
- Simple view (code + inputs/outputs + tests)
- Before/After view with diff highlighting (DebugBench, CanItEdit, etc.)
- GitHub-style diff view with per-file sections and repo/issue/commit links (SWE-bench, CommitBench)
- Multi-language tabs (HumanEval-X, HumanEvalPack with canonical/buggy toggle)
- Fill-in-the-Middle view with inline hole markers (SAFIM)
- Vulnerability view with CWE badges (BigVul, DiverseVul, PrimeVul, Devign)
- Multi-solution view with complexity badges (BigOBench, CodeContests, APPS)
Architecture:
- Refac…
