arxiv:2611.08452

ReproBridge: Building Reproducibility into ML Research Artifacts

Published on Nov 5

· Submitted by

Yubo Wang on Nov 10

#3 Paper of the day

Google

Upvote

Authors:

Abstract

Reproducing machine learning experiments requires reconstructing procedures from information scattered across papers, released code, and supporting artifacts. For coding agents, reproduction success can therefore depend on their ability to fill in missing details, rather than on the released artifacts alone. We ask whether preserving recovered procedures and decisions improves subsequent independent reproduction. We define a standardized artifact design comprising a refined codebase, environment setup, experiment launchers, and a reproduction guide, and introduce ReproBridge, an agent workflow for constructing these refined reproducible artifacts. ReproBridge-Bench provides 1,029 tasks from 387 experiments across 36 papers, anchored by five human-verified reference scores per task. Agents attempt tasks in fresh sessions with raw code or refined artifacts; a separate Auditor checks execution evidence. Across four evaluated models, refined artifacts raise mean reproduction success from 76.4% to 92.6% and reduce downstream API spending by 19.7% on average. The cross-model success-rate spread narrows from 21.6 to 3.9 percentage points, with all four models achieving mean success rates above 90%. Removing any component reduces success in the ablation study, with the largest losses from code and environment removal. These results suggest that preserving execution knowledge helps even models with lower raw-code success reproduce experiments more reliably and at lower API cost.

View arXiv page View PDF Project page Add to collection

Community

ubowang

Paper submitter about 4 hours ago

ReproBridge starts from a simple question: when a coding agent reproduces a paper, how much of its success comes from the released artifacts, and how much from the agent filling in what the paper and code leave implicit?

What we built

Refined reproducible artifact (RRA): refined code, environment setup, experiment launchers, and a reproduction guide that records the rationale behind each setting and repair.
ReproBridge: an agent workflow that constructs RRAs from a paper and its official repository (Specify → Execute → Preserve, with an independent Reviewer).
ReproBridge-Bench: 1,029 tasks from 387 experiments across 36 ICML 2026 papers, each anchored by five human-verified reference scores.

Key results (four coding agents, fresh sessions, raw code vs. RRA)

Mean success 76.4% → 92.6%; downstream API spending −19.7%
Cross-model spread 21.6 → 3.9 pp; all four models above 90%
Every component matters in the ablation, code and environment the most
External check on CORE-Bench Hard: capsule success 62.2% → 73.3%, cost per successful capsule
4.34

How we evaluate research agents

Published scores are single observations → compatibility intervals from five independent human reproductions, applied in both directions.
Numbers can match for the wrong reasons → an Auditor traces each value to output files, execution records and scripts; against two human reviewers it reaches 88.6% precision / 79.5% recall (818 outcomes).
Hold the task fixed and vary only the handoff → this separates artifact quality from agent ability.

🌐 Project page with interactive results, a 36-paper explorer and case studies: https://wyyyb.github.io/reprobridge_website/ Code and data are under internal review and will be released once approved. Happy to answer questions!