Spaces:
Running
Running
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <title>DocDrift — docs that lie, caught by running them · Build with Beam</title> | |
| <meta name="description" content="DocDrift: Beam reads a whole Python library in one call, finds docs that disagree with the code, and proves it by running them. 34 real bugs in 11 of 12 libraries; 30 reproduce."> | |
| <meta property="og:title" content="DocDrift: docs that lie, caught by running them"> | |
| <meta property="og:description" content="DocDrift: Beam reads a whole Python library in one call, finds docs that disagree with the code, and proves it by running them. 34 real bugs in 11 of 12 libraries; 30 reproduce."> | |
| <meta property="og:image" content="https://huggingface.co/spaces/ollymcbride4/docdrift/resolve/main/assets/og.png"> | |
| <meta name="twitter:card" content="summary_large_image"> | |
| <meta name="twitter:image" content="https://huggingface.co/spaces/ollymcbride4/docdrift/resolve/main/assets/og.png"> | |
| <link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'%3E%3Crect width='32' height='32' rx='6' fill='%2317140f'/%3E%3Ctext x='16' y='23' font-size='20' text-anchor='middle' fill='%23e0795a'%3E≠%3C/text%3E%3C/svg%3E"> | |
| <link rel="stylesheet" href="assets/app.css"> | |
| <link rel="stylesheet" href="assets/asciinema-player.css"> | |
| </head> | |
| <body> | |
| <a class="skip" href="#main">Skip to content</a> | |
| <header class="hero" id="hero"> | |
| <div class="hero-copy"> | |
| <p class="eyebrow">Build with Beam · docs that lie, caught by running them</p> | |
| <p class="brand">DocDrift</p> | |
| <h1>Your docs say one thing. Your code does another.</h1> | |
| <div class="headline" id="headline"> | |
| <p class="lead-full">DocDrift gives Beam an entire Python library (docs + code, up to ~400k tokens) in one call. Beam checks every suspicious doc example against the whole codebase, then writes a script that runs it. Across 12 popular libraries (bugs in 11 of 12 libraries; pydantic-settings had none) it found 34 real docs bugs; 30 crash exactly as the docs are written.</p> | |
| <p class="lead-short">Beam read 12 whole Python libraries and ran their docs: 34 real bugs, 30 reproduce.</p> | |
| </div> | |
| <div class="tiles" id="tiles"><div class='tile'><strong>34</strong><span>real docs bugs · 11 of 12 libraries</span></div><div class='tile'><strong>30</strong><span>crash when you run the docs</span></div><div class='tile'><strong>2</strong><span>fixes filed upstream (starlette #3746, nox #1216)</span></div></div> | |
| <div class="note" id="hero-note">Labels: checked against the source by an AI reviewer, not a human. Recall is measured against the 45 drifts any method found, so it flatters the methods.</div> | |
| <p class="links" id="hero-links"><a href="https://github.com/OlsOls4/repoatlas-1m">Code</a> · <a href="https://huggingface.co/datasets/ollymcbride4/docdrift-bugs">Dataset</a> · <a href="https://huggingface.co/reflection/Beam-1.0-501B-A23B/discussions/5">Thread</a></p> | |
| </div> | |
| <aside class="drift" id="drift-card" aria-label="Starlette docs-drift example"> | |
| <p class="lbl">Docs say · docs/websockets.md:74 @8aa828c</p> | |
| <pre class="doc" id="drift-doc"></pre> | |
| <p class="lbl">Code says · starlette/websockets.py:136</p> | |
| <pre class="code" id="drift-code"></pre> | |
| <p class="lbl">Running the doc gives</p> | |
| <pre class="err" id="drift-err"></pre> | |
| <p class="drift-foot" id="drift-foot"></p> | |
| <p class="drift-honest" id="drift-honest"></p> | |
| </aside> | |
| <div id="demo-video" class="demo-video"> | |
| <video id="demo-video-el" src="assets/docdrift-demo.mp4" controls playsinline preload="metadata" poster="assets/docdrift-poster.png" width="1280" height="720" aria-label="DocDrift demo, 93 seconds, narrated (synthetic voice), captioned"> | |
| <track kind="captions" src="assets/docdrift-demo.vtt" srclang="en" label="English (same as on-screen)"> | |
| </video> | |
| <p class="video-credit">Voice: Kokoro-82M (Apache-2.0), generated locally</p><details class="transcript"><summary>Transcript</summary><p>DocDrift gives Beam a whole Python library in one call, to find docs that lie.</p><p>Starlette's docs say: call receive JSON with mode binary. Run it as written, and it crashes with a TypeError.</p><p>One Beam call, the whole library: 394,577 prompt tokens. 3 candidates; 2 kept as real drift, 1 rejected.</p><p>A fix is filed upstream: pull request 3746, open, not merged.</p><p>Every card shows what the docs say, what the code says, and what running it gives: 34 real docs bugs in 11 of 12 libraries.</p><p>Does the 1M-token context help? Accuracy is about the same. But grounding is better: the right cross-file line 25 of 30 times, versus 19 at matched cost. Off-file evidence: 11 of 13, versus 6. p = 0.070: suggestive, not proven.</p><p>On the benchmark: 34 of 45 known drifts found; 30 crash exactly as the docs are written. 32 of the 34 bugs ship a script.</p><p>Then a fresh run on tomlkit, a library DocDrift had never seen.</p><p>2 findings: 1 verified version drift, and 1 arguable missing-imports NameError, in 96 seconds.</p><p>Run it on your repo with docdrift run, using a free Ollama key.</p></details> | |
| </div> | |
| </header> | |
| <nav class="tabs" role="tablist" aria-label="Sections"> | |
| <button type="button" class="tab is-active" role="tab" id="tab-story" data-tab="story" aria-controls="story" aria-selected="true" tabindex="0">Story</button> | |
| <button type="button" class="tab" role="tab" id="tab-bugs" data-tab="bugs" aria-controls="bugs" aria-selected="false" tabindex="-1">Bugs</button> | |
| <button type="button" class="tab" role="tab" id="tab-benchmark" data-tab="benchmark" aria-controls="benchmark" aria-selected="false" tabindex="-1">Benchmark</button> | |
| <button type="button" class="tab" role="tab" id="tab-repros" data-tab="repros" aria-controls="repros" aria-selected="false" tabindex="-1">Repros</button> | |
| <button type="button" class="tab" role="tab" id="tab-run" data-tab="run" aria-controls="run" aria-selected="false" tabindex="-1">Run it</button> | |
| <button type="button" class="tab" role="tab" id="tab-limits" data-tab="limits" aria-controls="limits" aria-selected="false" tabindex="-1">Limits</button> | |
| </nav> | |
| <main id="main" tabindex="-1"> | |
| <section id="story" class="panel is-active" role="tabpanel" aria-labelledby="tab-story" tabindex="0"> | |
| <h2>How this uses Beam</h2> | |
| <div class="criteria" id="criteria-box"></div> | |
| <section id="recorded-run" class="player"> | |
| <details> | |
| <summary id="player-summary">Watch the recorded starlette run</summary> | |
| <p class="player-kicker">Recording — not a live call</p> | |
| <h3>Recorded starlette pack</h3> | |
| <aside class="banner" id="replay-banner"></aside> | |
| <div class="player-grid"> | |
| <div> | |
| <h4>Candidates</h4> | |
| <ol id="player-candidates"></ol> | |
| </div> | |
| <div> | |
| <h4>Beam verdicts</h4> | |
| <ol id="player-verdicts"></ol> | |
| </div> | |
| </div> | |
| <h4>Repro traceback</h4> | |
| <p id="player-repro-note" class="lede"></p> | |
| <div id="player-tracebacks"></div> | |
| <p id="player-pr" class="pr-line"></p> | |
| </details> | |
| </section> | |
| <h2>Why 1M context</h2> | |
| <p class="lede" id="why-caption"></p> | |
| <div id="why-chart" class="why-chart"></div> | |
| <p class="lede" id="why-subset"></p> | |
| <p class="note" id="why-placeholder"></p> | |
| <h2>What is docs drift?</h2> | |
| <p>Docs say one thing, the code does another. Worked example from starlette: docs call <code>receive_json(data, mode="binary")</code>; the implementation is <code>receive_json(self, mode: str = "text")</code>. Following the docs can raise <code>TypeError</code>.</p> | |
| <h2>What Beam did</h2> | |
| <ol class="method" id="how-beam"></ol> | |
| <svg class="method-diagram" viewBox="0 0 640 56" role="img" aria-label="AST check, then Beam whole-repo triage, then Beam-written repro, then a strict gate"> | |
| <text x="8" y="34" font-size="13">AST check (free)</text> | |
| <text x="168" y="34" font-size="16">→</text> | |
| <text x="196" y="34" font-size="13">Beam whole-repo triage</text> | |
| <text x="388" y="34" font-size="16">→</text> | |
| <text x="416" y="34" font-size="13">Beam-written repro</text> | |
| <text x="560" y="34" font-size="16">→</text> | |
| <text x="588" y="34" font-size="13">strict gate</text> | |
| </svg> | |
| <h2>Glossary</h2> | |
| <ul id="glossary-list"></ul> | |
| </section> | |
| <section id="bugs" class="panel" role="tabpanel" aria-labelledby="tab-bugs" tabindex="0" hidden> | |
| <h2>Bug wall</h2> | |
| <p class="lede" id="bug-wall-note"></p> | |
| <p class="lede">Docs say / code says / running it gives. Permalinks point at the pinned commit.</p> | |
| <div class="filters" id="bug-filters"> | |
| <div class="chips" id="library-chips"></div> | |
| <div class="chips" id="badge-chips"></div> | |
| </div> | |
| <p class="count" id="bug-count" aria-live="polite"></p> | |
| <p id="cross-file-def" class="note"></p> | |
| <div id="bug-grid" class="grid"></div> | |
| </section> | |
| <section id="benchmark" class="panel" role="tabpanel" aria-labelledby="tab-benchmark" tabindex="0" hidden> | |
| <h2>Benchmark</h2> | |
| <p class="lede" id="bench-caption"></p> | |
| <div class="headline-cards" id="bench-headlines"></div> | |
| <p class="lede" id="token-footnote"></p> | |
| <div class="table-wrap"> | |
| <table id="compare-table" class="cards"></table> | |
| </div> | |
| <h3>Hybrid funnel</h3> | |
| <ol id="funnel" class="funnel"></ol> | |
| <h3>Does long context help?</h3> | |
| <p id="ablation-claim" class="lede"></p> | |
| <div class="table-wrap"> | |
| <table id="ablation-table" class="cards"></table> | |
| </div> | |
| <p id="ablation-subset" class="lede"></p> | |
| <ul id="ablation-caveats"></ul> | |
| <h3 id="beamv3-title">Catching what one whole-repo call misses</h3> | |
| <p id="beamv3-claim" class="lede"></p> | |
| <p id="beamv3-recipe" class="lede"></p> | |
| <div class="table-wrap"> | |
| <table id="beamv3-table" class="cards beamv3"></table> | |
| </div> | |
| <details class="col-toggle" id="beamv3-cols"><summary>Show all columns</summary><p class="note">Extra columns are the false/unclear/unlabelled split, among-labelled precision, extra-of-45, new AI-verified, and marginal token cost.</p></details> | |
| <ul id="beamv3-caveats"></ul> | |
| <h3>Planted-drift distance probe</h3> | |
| <p id="recall-note"></p> | |
| <div id="recall-svg" class="recall-svg"></div> | |
| <details id="recall-all"><summary>Show all methods</summary><img id="recall-chart" alt="Planted-drift strict recall by distance, every recorded method" hidden loading="lazy"></details> | |
| <h3 id="pack-title">Pack coverage</h3> | |
| <p class="lede" id="pack-note"></p> | |
| <div class="table-wrap"> | |
| <table id="pack-table" class="cards"></table> | |
| </div> | |
| <div id="benchmark-notes" class="notes"></div> | |
| </section> | |
| <section id="repros" class="panel" role="tabpanel" aria-labelledby="tab-repros" tabindex="0" hidden> | |
| <h2>Repros</h2> | |
| <p class="lede">Each row is a script that ran the documented usage. Strict means TypeError, AttributeError, ImportError, or NameError on a documented-usage line. The starlette headline script was written by hand from Beam's verdict; the others were written by Beam.</p> | |
| <p class="lede" id="repro-caption"></p> | |
| <div class="table-wrap"> | |
| <table id="repro-table" class="cards"></table> | |
| </div> | |
| </section> | |
| <section id="run" class="panel" role="tabpanel" aria-labelledby="tab-run" tabindex="0" hidden> | |
| <h2>Run it on your repo</h2> | |
| <p id="ollama-key"></p> | |
| <details id="sample-output" hidden> | |
| <summary>See a real run on tomlkit, a library DocDrift had never seen (96 s)</summary> | |
| <p class="lede" id="sample-note"></p> | |
| <div id="sample-cast" hidden></div> | |
| <details> | |
| <summary>Plain text</summary> | |
| <pre id="sample-pre"></pre> | |
| </details> | |
| </details> | |
| <h3>CLI</h3> | |
| <p class="lede" id="cli-cost"></p> | |
| <pre><code id="cli-cmd"></code></pre> | |
| <pre><code id="cli-preview"></code></pre> | |
| <p class="links" id="run-links"></p> | |
| </section> | |
| <section id="limits" class="panel" role="tabpanel" aria-labelledby="tab-limits" tabindex="0" hidden> | |
| <h2>Limits and honesty</h2> | |
| <ul id="limits-list"></ul> | |
| <h3>Not shipped in this Space</h3> | |
| <ul id="exclusions-list"></ul> | |
| <p class="note" id="sources"></p> | |
| </section> | |
| </main> | |
| <footer> | |
| <p>Every number comes from the companion dataset. No API keys are embedded. This Space is a static recording; it makes no Beam calls.</p> | |
| <p class="links footer-links"><a href="https://github.com/OlsOls4/repoatlas-1m">Code</a> · <a href="https://huggingface.co/datasets/ollymcbride4/docdrift-bugs">Dataset</a> · <a href="https://huggingface.co/reflection/Beam-1.0-501B-A23B/discussions/5">Thread</a></p> | |
| </footer> | |
| <script src="assets/asciinema-player.min.js"></script> | |
| <script src="assets/app.js"></script> | |
| </body> | |
| </html> | |