Spaces:
Running
Running
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <title>RuleChef Live</title> | |
| <meta name="description" content="A request stream where a teacher model's answers become labelled examples and RuleChef turns them into regex rules."> | |
| <link rel="stylesheet" href="style.css"> | |
| </head> | |
| <body> | |
| <main class="wrap"> | |
| <header> | |
| <h1>RuleChef Live</h1> | |
| <p class="lede">Model calls turn into rules. A stream of <span data-n-stream>3,000</span> requests: sentences from European Court of | |
| Human Rights judgments (the public TAB anonymisation dataset), each asking for its dates and times, codes and reference numbers, | |
| and quantities. The current rule set tries each request first. If its gate says the rules cover the request, the rules answer | |
| and no model is called. Otherwise the teacher model, <a href="https://huggingface.co/vllm-sr/vela3-fast">vela3-fast</a>, answers, | |
| and its answer becomes a labelled example. After the first 50 examples and then every 100 new ones, | |
| <a href="https://github.com/KRLabsOrg/rulechef">RuleChef</a>'s search synthesis (no LLM) learns a new rule set from all of | |
| them on CPU.</p> | |
| <p class="note">The stream was run once offline with the code in this Space (<code>sim.py</code>); this page replays that run, | |
| so the rule sets and synthesis times are snapshots. The rules themselves run live in your browser, here and in the box at the | |
| bottom.</p> | |
| </header> | |
| <section class="controls" aria-label="Replay controls"> | |
| <button id="play" class="primary" type="button">Play</button> | |
| <label class="speed">Speed | |
| <select id="speed"><option value="10">slow</option><option value="30" selected>normal</option><option value="100">fast</option></select> | |
| </label> | |
| <input id="pos" type="range" min="0" max="3000" step="1" value="3000" aria-label="Requests processed"> | |
| <output id="posout" for="pos">3,000</output> | |
| </section> | |
| <section id="kpis" class="kpis" aria-live="polite"></section> | |
| <section class="charts"> | |
| <figure class="chart"><figcaption>Share of requests answered by rules <span class="sub">last 200 requests</span></figcaption><div id="c-cov"></div></figure> | |
| <figure class="chart"><figcaption>Model calls <span class="sub">cumulative</span></figcaption><div id="c-calls"></div></figure> | |
| <figure class="chart"><figcaption>Latency per request <span class="sub">mean of the last 200, ms</span></figcaption><div id="c-lat"></div></figure> | |
| </section> | |
| <section> | |
| <h2>Rule set versions, scored on the holdout when deployed</h2> | |
| <p class="help">Holdout: <span data-n-holdout>400</span> sentences from other documents. Matching is RuleChef's TAB scorer (same text, same type). | |
| "Cascade" means rules where the gate lets them answer, teacher elsewhere. The last column compares with TAB's human annotation; | |
| the rules learn from the teacher, mistakes included.</p> | |
| <div class="tablewrap"><table id="t-vers"></table></div> | |
| </section> | |
| <section> | |
| <h2>Rules of the current version</h2> | |
| <p class="help" id="rules-help"></p> | |
| <div class="tablewrap"><table id="t-rules" class="rules"></table></div> | |
| <details id="rules-rest" class="rest"></details> | |
| <div id="gate" class="gate"></div> | |
| </section> | |
| <section> | |
| <h2>Latest requests</h2> | |
| <ol id="recent" class="recent" reversed></ol> | |
| </section> | |
| <section class="try"> | |
| <h2>Try your own text</h2> | |
| <p class="help">The gate and the rules of the version at the slider position run in your browser. The teacher does not run on | |
| this static page: when the gate sends a request on, the page says so and shows what the rules found. For a holdout sentence | |
| it also shows the teacher's recorded answer.</p> | |
| <textarea id="inp" rows="3" aria-label="Text">On 14 March 1997 the applicant lodged application no. 36732/97 and claimed 250,000 French francs in damages.</textarea> | |
| <div class="row"> | |
| <button id="go" class="primary" type="button">Run rules</button> | |
| <button id="pick" type="button">Random holdout sentence</button> | |
| <button class="ex" type="button" data-ex="He was sentenced to two years and six months' imprisonment on 3 June 2004.">Example 2</button> | |
| <button class="ex" type="button" data-ex="The hearing took place the following day in Ankara.">Example 3</button> | |
| </div> | |
| <div id="status" class="status" aria-live="polite"></div> | |
| <div id="out" class="hl"></div> | |
| <div id="out-teacher" class="hl"></div> | |
| </section> | |
| <section class="about"> | |
| <h2>How the numbers are computed</h2> | |
| <p>All numbers come from this demo's own run over public data; nothing is copied from a paper. The page aggregates the run's | |
| per-request log (<code>data/replay.json</code>) as you move the slider.</p> | |
| <ul> | |
| <li><b>Stream and holdout.</b> TAB documents are split by document: <span data-n-stream>3,000</span> sentences for the stream, | |
| <span data-n-holdout>400</span> from other documents for the holdout. One request = one sentence of 40 to 400 characters.</li> | |
| <li><b>Teacher.</b> vela3-fast at revision <code>078c42f9</code>, one span question per request with three label names asked at | |
| request time: "date or time", "code or reference number", "quantity". Its answers were computed once offline on CPU (16 processes | |
| with 2 threads each); the per-request time of that run is the teacher latency shown.</li> | |
| <li><b>Rules.</b> RuleChef search synthesis (<code>synth="search"</code>, min support 2) on all labelled examples so far, with the | |
| teacher's answers as labels. A learned rule answers only if it matched at least <span data-trust-min>3</span> times on the | |
| labelled data with precision at least <span data-trust-prec>0.8</span> against the teacher. Rule latency is the measured time of the | |
| gate plus RuleChef's Python executor on one CPU core.</li> | |
| <li><b>Gate.</b> Rules answer a request only when every trigger token (anything with a digit, plus words the teacher often | |
| labels) is covered by an answering rule or sits in a learned skip context. A rule answer with no spans ("nothing to extract") | |
| means the gate saw no uncovered trigger.</li> | |
| <li><b>Spot checks and drift guard.</b> Every <span data-spot>10</span>th rule answer (by request-id hash) also goes to the teacher; | |
| it counts as a model call and becomes a labelled example. A rule whose spot-check precision drops below | |
| <span data-guard-prec>0.8</span> after <span data-guard-min>5</span> checked spans is switched off for the rest of the stream.</li> | |
| <li><b>Synthesis.</b> Times are wall-clock seconds of the offline run on one CPU core. Re-running <code>sim.py</code> gives the same | |
| rule sets in the same order (checked twice, in separate processes).</li> | |
| <li><b>Rules in your browser.</b> <code>engine.js</code> is a JavaScript port of RuleChef's executor and of the gate. On all | |
| 3,400 sentences of this demo and all rule set versions it returns the same spans and gate decisions as the Python code.</li> | |
| </ul> | |
| <h2>Credits and licences</h2> | |
| <ul> | |
| <li><b>RuleChef</b> (<a href="https://github.com/KRLabsOrg/rulechef">KRLabsOrg/rulechef</a>, Apache-2.0): rule learning from | |
| examples. The search synthesis used here was installed from commit <code>8173f96</code> of its <code>feat/search-synthesis</code> branch.</li> | |
| <li><b>vela3-fast</b> (<a href="https://huggingface.co/vllm-sr/vela3-fast">vllm-sr/vela3-fast</a>, Apache-2.0): the teacher.</li> | |
| <li><b>TAB</b>, the Text Anonymization Benchmark (Pilán et al., 2022, Norsk Regnesentral, | |
| <a href="https://github.com/NorskRegnesentral/text-anonymization-benchmark">repository</a>): MIT licence, copyright (C) 2021-2026 | |
| Norsk Regnesentral, full notice in <code>DATA_LICENSE_TAB.txt</code>. Used here: its public train split as published at | |
| <a href="https://huggingface.co/datasets/mattmdjaga/text-anonymization-benchmark-train">mattmdjaga/text-anonymization-benchmark-train</a> | |
| (revision <code>f0b7eeb</code>). The sentences, TAB's labels for the three types and the teacher's answers are in | |
| <code>data/requests.jsonl</code>.</li> | |
| <li>This Space's code: Apache-2.0.</li> | |
| </ul> | |
| </section> | |
| </main> | |
| <div id="tip" class="tip" role="tooltip" hidden></div> | |
| <script src="engine.js"></script> | |
| <script src="app.js"></script> | |
| </body> | |
| </html> | |