Spaces:
Running
Running
| { | |
| "how_beam": [ | |
| "Triage filter, not a raw detector. A free AST symbol check proposes name and signature mismatches. Beam judged 56 candidates across 11 of 12 libraries (structlog failed; cattrs needed 2 calls). It rejected 0 of 32 true candidates and removed 15 of 24 false or unclear ones.", | |
| "Beam's triage kept all 24 true symbol-check drifts, kept 1 extra from the whole-repo pass's own candidates, and added 9 drifts the checker missed (7 new to the union): 24 + 1 + 9 = 34.", | |
| "Repro author. Beam wrote the doc-literal repro scripts (138,138 tokens). The starlette headline script was written by hand from Beam's verdict. 30 distinct bugs reproduce a real API error. 38 of 41 badges were real. 2 false + 1 unclear (werkzeug S38, S39 false; S41 unclear) received a strict badge." | |
| ], | |
| "limits": [ | |
| "Packs are capped at 400k so a whole repo plus 32k output fits with headroom; Beam accepted 947,446 in a probe. 5 of 12 first-sweep packs hit the remaining-budget cutoff (starlette, httpx, click, attrs, werkzeug). The packer puts tests, CI and examples last; exact dropped paths were not recorded in the traces.", | |
| "The 45-drift universe is the union of method output, so it favours those methods.", | |
| "Labels were checked against the source by an AI reviewer. (Field name in the dataset: `hand_verdict`.)", | |
| "Citation check means quote alignment of cited lines vs packed source, with a ±3-line shift allowed. It is not a verdict that the drift is real.", | |
| "A true drift can still fail the strict repro rule (marshmallow T11, structlog untriaged, werkzeug input_stream).", | |
| "2 false + 1 unclear (werkzeug S38, S39 false; S41 unclear) received a strict repro badge.", | |
| "Filed by this project. Open, not merged. Status checked 30 Oct 2026.", | |
| "httpx #3774 and arrow #1355 were already reported upstream by others.", | |
| "cattrs and click have drafts in the repo; they are not listed as filed.", | |
| "structlog: Beam's whole-repo reply had no parseable JSON (prompt 404,652 tokens); not retried.", | |
| "This Space is a static recording; it makes no Beam calls. Run DocDrift yourself with the CLI." | |
| ], | |
| "glossary": [ | |
| "Triage: Beam judges symbol-check and whole-repo candidates as true_drift, false, or unclear against one packed library prompt.", | |
| "Strict badge: TypeError, AttributeError, ImportError, or NameError raised on a DOC-LITERAL usage line.", | |
| "Line-shifted match: the cited quote matched packed source within ±3 lines of the model-cited line." | |
| ], | |
| "cli": "pip install git+https://github.com/OlsOls4/repoatlas-1m\nexport OLLAMA_API_KEY=YOUR_KEY\ndocdrift run https://github.com/OWNER/REPO --max-tokens 400000 --out findings", | |
| "cli_sweep": "", | |
| "ollama_key": "Create a free Ollama Cloud account at [ollama.com](https://ollama.com), open Settings, and create an API key. Export it as OLLAMA_API_KEY, or set OLLAMA_API_KEY_FILE to a file that contains the key. `docdrift run` packs the repository (docs and source first, up to the token cap), sends it to Beam for triage (32k output tokens, citation verify), then runs Beam-written doc-literal scripts in an isolated venv with no network. Skip the scripts with --no-repro. Cap repro tokens with --repro-budget. Stops on HTTP 402/quota (exit 4). No key is stored in this Space or in the archive.", | |
| "local": "", | |
| "links": { | |
| "code": "https://github.com/OlsOls4/repoatlas-1m", | |
| "dataset": "https://huggingface.co/datasets/ollymcbride4/docdrift-bugs", | |
| "thread": "https://huggingface.co/reflection/Beam-1.0-501B-A23B/discussions/5" | |
| }, | |
| "criteria": [ | |
| { | |
| "title": "Creativity", | |
| "body": "Treats docs as executable claims. Beam reads the whole library, then writes a script that **runs the docs**. A bug counts only if the doc's own line crashes." | |
| }, | |
| { | |
| "title": "Technical execution", | |
| "body": "- 12 pinned libraries\n- a strict repro gate: 30 bugs crash on the doc's own line\n- 38 of 41 badges real\n- ablation + stability rerun (26/27)\n- every number traces to the dataset" | |
| }, | |
| { | |
| "title": "Usefulness", | |
| "body": "34 real docs bugs in libraries people use (httpx, marshmallow, werkzeug, starlette, attrs, click…). 2 PRs filed upstream (starlette #3746, nox #1216), both open. A CLI for your own repo." | |
| }, | |
| { | |
| "title": "Beam", | |
| "body": "One whole-repo call per library (11 of 12; structlog failed, cattrs needed 2), with prompts up to 947,446 tokens accepted. With the whole repo, Beam cited the proof in another file 11/13 times, vs 0/13 with local files. Beam also wrote the repro scripts (138,138 tokens). At the same token budget, retrieval matched verdicts (46/56 vs 47/56) but grounded fewer cross-file cases (19/30 vs 25/30; p = 0.070, not shown at this n). A pre-registered extension at the same token budget: whole-repo grounded 52/78 cross-file items vs retrieval 38/78 (exact McNemar p = 0.0094). Outside-local evidence: 38/61 vs 25/61 (p = 0.0146)." | |
| } | |
| ], | |
| "why_caption": "Long context didn't change verdicts here. It changed whether Beam could point to the proof in another file. Cost: ~4.8× tokens (4,022,532 vs 830,527).", | |
| "why_matched_cost_note": "**Is it the whole repo, or just more tokens?** We gave a retrieval baseline the **same** Beam budget as whole-repo triage. That came to 3,957,434 against 4,022,532 tokens as spent (3,888,241 attributable; 96.7% of whole-repo) over the same 56 candidates. For each candidate, it got the local files plus the symbol definitions, imports and BM25 matches that fit in about 56k tokens.\n\nVerdict accuracy came out the same (46/56 against 47/56). Grounding still favoured the whole repo: the right verdict **and** the right cross-file line in 25/30 cases, against 19/30 for retrieval. Where the evidence lives outside the files the doc points at, the figures are 11/13 against 6/13. Retrieval pulled in importers or re-exports rather than the defining file in 5 of 13 outside-local items.\n\nThat gap is suggestive, not proven, at this sample size (exact McNemar p = 0.070). We pre-registered the test and report it as \"not shown at this n\".\n\n**Pre-registered extension.** We then added 63 items (57 planted renames, 6 real structlog candidates), frozen before any Beam call (design sha256 42f6b6fe…c996). Pooled over 78 cross-file items, the right verdict **and** the right line came out 52/78 for the whole repo against 38/78 for retrieval (exact McNemar p = 0.0094). Verdict accuracy still did not differ (108/119 against 105/118). The new items alone give p = 0.096 (\"not shown at this n\"), and the pooled result relies on one pre-registered re-send of a whole-repo call that failed to parse (p = 0.18 without it). The honest takeaway: whole-repo context doesn't make Beam **judge** better, but it helps Beam **point at the right line** more often, for no more spend than retrieval.", | |
| "why_subset_note": "On the 27-item rerun subset, local-only scored higher on verdict accuracy (0.926 vs 0.815). Whole-repo's advantage is citation: 4/4 vs 0/4 when the proof lives outside the candidate's files.", | |
| "video": "assets/docdrift-demo.mp4", | |
| "video_poster": "assets/docdrift-poster.png", | |
| "video_captions": "assets/docdrift-demo.vtt", | |
| "video_credit": "Voice: Kokoro-82M (Apache-2.0), generated locally", | |
| "sample_output": "[1/4] symbol check: tomlkit @ d7def7d\n 0 symbol flags\n[2/4] packing the whole repository (RepoAtlas packer)\n pack 141,326 tokens, 81 files, truncated=False; projected prompt 172,418 (guard 400,000)\n[3/4] Beam whole-repo triage: 1 call, 0 candidates + open search for cross-file drift\n[4/4] executable repro: doc-literal script per true_drift / extra finding, isolated venv, no network\n[venv] installing tomlkit@d7def7d\n[tomlkit:AD-1] attempt 1: could_not_run (lint) 'line 10: try/except catching Exception is not allowed'\n[tomlkit:AD-1] attempt 2: reproduced (expected_exception) 'AssertionError: docs claim 1.0.0 compliance but tomlkit accepts TOML 1.1.0-only inline-table syntax'\n[tomlkit:AD-2] attempt 1: reproduced (expected_exception) \"NameError: name 'datetime' is not defined\"\n\nDocDrift report: tomlkit @ d7def7dbcc59\n============================================================\ncandidates triaged: 0 \nextra findings from the whole-repo pass: 2\n AD-1 [other] The documentation still advertises TOML 1.0.0-compliance while the test suite an\n docs docs/index.rst:17 code tests/test_toml_tests.py:17 (verified)\n AD-2 [other] The quickstart writing snippet uses unqualified `datetime(...)` and `utc` names \n docs docs/quickstart.rst:121 code tomlkit/api.py:202 (UNVERIFIED)\nexecutable repro (isolated venv, no network):\n tomlkit:AD-1: reproduced AssertionError\n tomlkit:AD-2: reproduced NameError STRICT-CONFIRMED\n------------------------------------------------------------\nBeam tokens: triage 167,937 in + 4,975 out; repro 12,930 (3 calls); total 185,842\nwall time 96s (triage call 35s)\n", | |
| "sample_cast": "assets/docdrift_tomlkit.cast", | |
| "sample_note": "tomlkit @ d7def7d · 2 findings: a version-claim drift (citation verified; docs say TOML 1.0.0, tests accept 1.1.0) and a quickstart snippet missing its imports (NameError, strict-confirmed; arguable as drift) · 185,842 tokens · 96 s", | |
| "repro_caption": "Some bugs have 2 scripts (one per candidate); 30 distinct bugs reproduce.", | |
| "cross_file_note": "Cross-file means the documented claim and the cited implementation are in different files (including a docs/ page vs package source). It is true when recorded doc_path and code_path differ, or when the docs live under docs/ or a .md/.rst file and no code path was cited.", | |
| "bench_caption": "Recall denominator = 45 real drifts found by any method (benchmark pool) (a different 45 from the 45 true hybrid findings). 60 = what the hybrid flagged. 30 of 34 = bugs whose doc line crashes when run.", | |
| "bug_wall_note": "49 cards: 34 real bugs + 15 false/unclear findings (60 findings total; duplicates of the same bug are merged)", | |
| "cli_cost": "tomlkit: 185,842 tokens, 96 s", | |
| "cli_output_preview": "[1/4] symbol check: tomlkit @ d7def7d\n 0 symbol flags\n[2/4] packing the whole repository (RepoAtlas packer)", | |
| "built_at": "2026-10-30T14:45:16Z", | |
| "sources_public": "Every number is in the companion dataset (findings, triage verdicts, repros, ablation, token ledger).", | |
| "sources": [ | |
| "results_live/hybrid/hybrid.json", | |
| "results_live/hybrid/hybrid.csv", | |
| "results_live/hybrid/repros/repros.json", | |
| "results_live/baselines/baselines.json", | |
| "results_live/recall/", | |
| "results_live/index.json", | |
| "results_live/upstream_status.json", | |
| "results_live/traces/starlette-400000.json (summary and pack metadata only)", | |
| "results_live/ablation/results.json", | |
| "results_live/ablation/matched_cost/results.json", | |
| "results_live/ablation/matched_cost_ext/results.json", | |
| "results_live/beamv3/results.json", | |
| "results_live/fresh_run/findings.json", | |
| "results_live/fresh_run/stdout.log" | |
| ], | |
| "exclusions": [ | |
| "results_live/traces/*.json raw traces (only starlette-400000 summary and pack metadata are embedded in replay.json)", | |
| "results_live/hybrid/repros/*/attempts.json per-attempt logs", | |
| "results_live/hybrid/triage/*.json (read at build time, not shipped)", | |
| "results_live/ablation/raw/ API bodies", | |
| "results_live/ablation/matched_cost/raw/ API bodies", | |
| "results_live/beamv3/repros/*/ per-attempt logs", | |
| "dataset/ traces and other raw sweep artifacts", | |
| "raw pack traces (only starlette-400000 summary and pack metadata are embedded in replay.json)" | |
| ] | |
| } | |