Spaces:
Running
Running
Download index.html from NagaYu/ClaimCheck: direct link, hf CLI and curl.
- Browser
- Download file 24.6 kB
-
https://huggingface.co/spaces/NagaYu/ClaimCheck/resolve/main/index.html
- Command line
-
hf download hf://spaces/NagaYu/ClaimCheck/index.html
-
curl -L -o index.html https://huggingface.co/spaces/NagaYu/ClaimCheck/resolve/main/index.html
24.6 kB
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8" /> | |
| <meta name="viewport" content="width=device-width, initial-scale=1" /> | |
| <title>ClaimCheck — verify LLM answers against their source</title> | |
| <meta name="description" content="Verify LLM answers against their source context before they reach users. Deterministic, client-side, no server." /> | |
| <style> | |
| :root{ | |
| --bg:#ffffff; --fg:#111827; --muted:#6b7280; --line:#e5e7eb; --panel:#f9fafb; | |
| --accent:#b45309; --accent-bg:#fffbeb; | |
| --supported:#16a34a; --supported-bg:#dcfce7; | |
| --derived:#2563eb; --derived-bg:#dbeafe; | |
| --approximate:#ca8a04; --approximate-bg:#fef9c3; | |
| --unsupported:#ea580c; --unsupported-bg:#ffedd5; | |
| --contradicted:#dc2626; --contradicted-bg:#fee2e2; | |
| } | |
| @media (prefers-color-scheme: dark){ | |
| :root{ --bg:#0b0f19; --fg:#e5e7eb; --muted:#9ca3af; --line:#1f2937; --panel:#111827; | |
| --accent:#fbbf24; --accent-bg:#1c1917; | |
| --supported-bg:#052e16; --derived-bg:#172554; --approximate-bg:#422006; | |
| --unsupported-bg:#431407; --contradicted-bg:#450a0a; } | |
| } | |
| *{box-sizing:border-box} | |
| body{margin:0;background:var(--bg);color:var(--fg); | |
| font:15px/1.6 ui-sans-serif,-apple-system,"Segoe UI",Roboto,"Helvetica Neue","Hiragino Sans","Noto Sans JP",sans-serif} | |
| .wrap{max-width:1100px;margin:0 auto;padding:28px 16px 64px} | |
| h1{font-size:28px;margin:0 0 6px;letter-spacing:-.02em} | |
| h2{font-size:18px;margin:28px 0 10px;letter-spacing:-.01em} | |
| .lede{color:var(--muted);margin:0 0 6px;font-size:15px} | |
| .badges{display:flex;gap:8px;flex-wrap:wrap;margin:14px 0 22px} | |
| .badge{font-size:12px;padding:3px 9px;border:1px solid var(--line);border-radius:999px;color:var(--muted);text-decoration:none} | |
| .badge:hover{border-color:var(--accent);color:var(--accent)} | |
| .note{background:var(--accent-bg);border:1px solid var(--line);border-left:3px solid var(--accent); | |
| padding:12px 14px;border-radius:6px;font-size:14px;margin:0 0 22px} | |
| .tabs{display:flex;gap:2px;border-bottom:1px solid var(--line);margin-bottom:20px;flex-wrap:wrap} | |
| .tab{padding:9px 14px;border:none;background:none;color:var(--muted);cursor:pointer;font-size:14px; | |
| border-bottom:2px solid transparent;font-family:inherit} | |
| .tab[aria-selected=true]{color:var(--fg);border-bottom-color:var(--accent);font-weight:600} | |
| .panel[hidden]{display:none} | |
| .grid{display:grid;grid-template-columns:1fr 1fr;gap:16px} | |
| @media(max-width:820px){.grid{grid-template-columns:1fr}} | |
| label{display:block;font-size:12px;font-weight:600;margin:0 0 4px;color:var(--muted)} | |
| textarea,input{width:100%;font:13px/1.55 ui-monospace,SFMono-Regular,Menlo,monospace;color:var(--fg); | |
| background:var(--panel);border:1px solid var(--line);border-radius:6px;padding:9px 11px;resize:vertical} | |
| textarea:focus,input:focus{outline:2px solid var(--accent);outline-offset:-1px} | |
| .field{margin-bottom:12px} | |
| button.go{background:var(--accent);color:#fff;border:none;border-radius:6px;padding:10px 22px; | |
| font-size:14px;font-weight:600;cursor:pointer;font-family:inherit;margin:6px 0 20px} | |
| button.go:hover{filter:brightness(1.08)} | |
| .verdict{font-size:22px;font-weight:700;margin:0 0 12px} | |
| table{width:100%;border-collapse:collapse;font-size:13px;margin:0 0 18px} | |
| th,td{text-align:left;padding:7px 9px;border-bottom:1px solid var(--line);vertical-align:top} | |
| th{color:var(--muted);font-weight:600;font-size:12px} | |
| td.num{font-weight:700;white-space:nowrap} | |
| .hl{background:var(--panel);border:1px solid var(--line);border-radius:6px;padding:14px; | |
| white-space:pre-wrap;word-break:break-word;line-height:2.1;margin:0 0 8px} | |
| mark{padding:2px 4px;border-radius:3px;font-weight:600;color:var(--fg)} | |
| mark.supported{background:var(--supported-bg);box-shadow:inset 0 -2px 0 var(--supported)} | |
| mark.derived{background:var(--derived-bg);box-shadow:inset 0 -2px 0 var(--derived)} | |
| mark.approximate{background:var(--approximate-bg);box-shadow:inset 0 -2px 0 var(--approximate)} | |
| mark.unsupported{background:var(--unsupported-bg);box-shadow:inset 0 -2px 0 var(--unsupported)} | |
| mark.contradicted{background:var(--contradicted-bg);box-shadow:inset 0 -2px 0 var(--contradicted)} | |
| .legend{display:flex;gap:14px;flex-wrap:wrap;font-size:12px;color:var(--muted);margin:0 0 20px} | |
| .key{display:inline-block;width:10px;height:10px;border-radius:2px;margin-right:5px;vertical-align:baseline} | |
| .st{font-weight:600} | |
| .st.supported{color:var(--supported)} .st.derived{color:var(--derived)} | |
| .st.approximate{color:var(--approximate)} .st.unsupported{color:var(--unsupported)} | |
| .st.contradicted{color:var(--contradicted)} | |
| details{background:var(--panel);border:1px solid var(--line);border-radius:6px;padding:10px 14px;margin:0 0 18px} | |
| summary{cursor:pointer;font-size:13px;font-weight:600;color:var(--muted)} | |
| pre{overflow:auto;font-size:12px;line-height:1.5;background:var(--panel);border:1px solid var(--line); | |
| border-radius:6px;padding:12px;margin:8px 0 0} | |
| code{font-family:ui-monospace,SFMono-Regular,Menlo,monospace;font-size:.92em} | |
| .muted{color:var(--muted);font-size:13px} | |
| .warn{color:var(--unsupported);font-weight:600} | |
| footer{margin-top:44px;padding-top:18px;border-top:1px solid var(--line);color:var(--muted);font-size:13px} | |
| a{color:var(--accent)} | |
| </style> | |
| </head> | |
| <body> | |
| <div class="wrap"> | |
| <h1>🔎 ClaimCheck</h1> | |
| <p class="lede"><strong>Verify LLM answers against their source context before they reach users.</strong></p> | |
| <p class="lede">We cannot catch every hallucination — but dangerous hallucinations are <em>specific</em>, and specific | |
| claims can be matched as strings. So ClaimCheck checks only what it can check deterministically, and always tells | |
| you <strong>how much it did not check</strong>.</p> | |
| <div class="badges"> | |
| <a class="badge" href="https://huggingface.co/NagaYu/claimcheck-rules" target="_blank" rel="noopener">📦 Rule pack</a> | |
| <a class="badge" href="https://huggingface.co/datasets/NagaYu/claimcheck-eval" target="_blank" rel="noopener">📊 158-case eval set</a> | |
| <span class="badge">⚡ ~0.2 ms typical</span> | |
| <span class="badge">🔒 100% client-side</span> | |
| </div> | |
| <div class="note"> | |
| <strong>Nothing leaves your browser.</strong> There is no server and no network call: the whole verifier is | |
| JavaScript running in this tab. Paste real model output if you want to — it stays on your machine. | |
| </div> | |
| <div class="tabs" role="tablist"> | |
| <button class="tab" role="tab" aria-selected="true" data-p="verify">Verify</button> | |
| <button class="tab" role="tab" aria-selected="false" data-p="retry">Retry</button> | |
| <button class="tab" role="tab" aria-selected="false" data-p="json">JSON repair</button> | |
| <button class="tab" role="tab" aria-selected="false" data-p="about">How it works</button> | |
| </div> | |
| <section class="panel" id="p-verify"> | |
| <div class="grid"> | |
| <div> | |
| <div class="field"><label for="answer">Answer — the LLM output to check</label> | |
| <textarea id="answer" rows="9"></textarea></div> | |
| <div class="field"><label for="context">Context — the source of truth given to the model</label> | |
| <textarea id="context" rows="9"></textarea></div> | |
| </div> | |
| <div> | |
| <div class="field"><label for="system">System prompt (optional — enables leak detection)</label> | |
| <textarea id="system" rows="4"></textarea></div> | |
| <div class="field"><label for="userin">User input (optional — enables injection-echo detection)</label> | |
| <textarea id="userin" rows="3"></textarea></div> | |
| <div class="field"><label for="schema">Schema JSON (optional — only if the answer must be JSON)</label> | |
| <textarea id="schema" rows="3" placeholder='{"title": "str", "score": "float"}'></textarea></div> | |
| <div class="field"><label for="policy">Policy JSON (optional — threshold overrides)</label> | |
| <textarea id="policy" rows="5"></textarea></div> | |
| </div> | |
| </div> | |
| <button class="go" id="run">Verify</button> | |
| <div id="out"></div> | |
| </section> | |
| <section class="panel" id="p-retry" hidden> | |
| <p class="muted">A blind retry re-sends the whole system prompt + context + question and regenerates everything. | |
| A targeted retry re-sends only the answer plus the specific failing claims. The difference is the estimate below. | |
| Run <strong>Verify</strong> first.</p> | |
| <button class="go" id="run-retry">Build retry instruction</button> | |
| <div id="retry-out"></div> | |
| </section> | |
| <section class="panel" id="p-json" hidden> | |
| <p class="muted">Structural repair only. <strong>The point is to never burn a retry on formatting.</strong> | |
| Fences, trailing commas, single quotes, raw newlines inside strings, fullwidth punctuation and Python literals | |
| are all fixed locally — retries are reserved for claims that are actually wrong.</p> | |
| <div class="grid"> | |
| <div class="field"><label for="raw">Raw model output</label><textarea id="raw" rows="12"></textarea></div> | |
| <div class="field"><label for="schema2">Schema JSON (optional)</label><textarea id="schema2" rows="12"></textarea></div> | |
| </div> | |
| <button class="go" id="run-json">Extract / repair / coerce</button> | |
| <div id="json-out"></div> | |
| </section> | |
| <section class="panel" id="p-about" hidden> | |
| <h2>Always read <code>coverage</code></h2> | |
| <p>Every result carries two independent numbers, and reporting one without the other is the most dangerous way to | |
| misuse this tool.</p> | |
| <table> | |
| <tr><th>field</th><th>question it answers</th></tr> | |
| <tr><td><code>grounding_score</code></td><td>Of the claims I checked, what share held up?</td></tr> | |
| <tr><td><code>coverage</code></td><td>What share of the answer did I check at all?</td></tr> | |
| </table> | |
| <p><code>grounding_score</code> alone is a trap. An answer of pure vague hedging produces <strong>zero</strong> | |
| claims and scores <strong>1.00</strong> on any naive groundedness metric — while being completely unverified. | |
| A high score at low coverage means <em>"I could not find anything to check"</em>, not <em>"everything is fine"</em>.</p> | |
| <h2>Claim statuses</h2> | |
| <table> | |
| <tr><th>status</th><th>meaning</th></tr> | |
| <tr><td><span class="st supported">supported</span></td><td>Appears literally in the context after normalisation</td></tr> | |
| <tr><td><span class="st derived">derived</span></td><td>Computable from context numbers by 2–3 term arithmetic; the formula is shown</td></tr> | |
| <tr><td><span class="st approximate">approximate</span></td><td>A near match exists — paraphrase, finer-grained date, related URL</td></tr> | |
| <tr><td><span class="st unsupported">unsupported</span></td><td>Nothing in the context backs it</td></tr> | |
| <tr><td><span class="st contradicted">contradicted</span></td><td>The context holds a <strong>near but different</strong> value</td></tr> | |
| </table> | |
| <p class="muted"><code>approximate</code> is deliberately <em>not</em> counted as grounded: a paraphrased quote is | |
| still an inaccurate quote.</p> | |
| <h2>Measured behaviour</h2> | |
| <p>On the <a href="https://huggingface.co/datasets/NagaYu/claimcheck-eval" target="_blank" rel="noopener">158-case | |
| bilingual evaluation set</a>:</p> | |
| <pre>answer-level recall 56/56 (100%) cases that should be flagged, and were | |
| clean answers passing 44/46 (96%) the rest were annotated, never blocked | |
| known false positives 22 (13.9%) | |
| known false negatives 3 (1.9%) | |
| JS/Python parity 158/158 (100%)</pre> | |
| <p>The false-positive rate is published deliberately. It is the number that decides whether anyone keeps a verifier | |
| switched on, and hiding it helps nobody.</p> | |
| <h2>The most serious failure: semantic reversal</h2> | |
| <p>All three false negatives are the same shape. A one-character change that reverses meaning keeps character | |
| similarity above 0.9, so a reversed quote reads as a faithful paraphrase:</p> | |
| <table> | |
| <tr><th>context</th><th>answer</th><th>similarity</th><th>status</th></tr> | |
| <tr><td>売上高は7%<strong>増加</strong>した</td><td>売上高は7%<strong>減少</strong>した</td><td>0.97</td><td><span class="st approximate">approximate</span> ❌</td></tr> | |
| <tr><td>提案は全会一致で<strong>承認</strong>された</td><td>提案は全会一致で<strong>否決</strong>された</td><td>0.92</td><td><span class="st approximate">approximate</span> ❌</td></tr> | |
| <tr><td>revenue <strong>grew</strong> 8%</td><td>revenue <strong>declined</strong> 8%</td><td>0.97</td><td><span class="st approximate">approximate</span> ❌</td></tr> | |
| </table> | |
| <p>Character-level matching cannot detect negation or antonym substitution. If your domain turns on | |
| approve/reject or increase/decrease, <strong>do not rely on quote status alone.</strong> In all three cases the | |
| answer-level verdict was still <code>retry</code>, because the NUMERIC and ENTITY claims inside the quote are the | |
| layer that catches what the quote check misses.</p> | |
| <h2>When it is wrong (false positives)</h2> | |
| <ul class="muted"> | |
| <li><strong>Faithful paraphrase</strong> — meaning preserved, wording replaced. The largest group.</li> | |
| <li><strong>Generic noun phrases</strong> — "Machine Learning" flagged as an unsupported entity.</li> | |
| <li><strong>Acronym expansion</strong> — context "World Health Organization", answer "WHO".</li> | |
| <li><strong>Rounding</strong> — context 12,345 vs answer "roughly 12,000": inside the near-miss window. | |
| Raise <code>numeric_tolerance</code> rather than lowering <code>contradiction_rel</code>.</li> | |
| <li><strong>Fiscal vs calendar year</strong> — FY2024 and 2024 treated as the same period.</li> | |
| <li><strong>Identifier variants</strong> — ISO 9001:2015 where the source says ISO 9001. The deliberate cost of | |
| catching one-digit-off fabrications.</li> | |
| <li><strong>URL query parameters</strong> — a tracking parameter breaks exact matching.</li> | |
| </ul> | |
| <p><strong>ENTITY is the noisiest check.</strong> Set <code>{"enable_entity": false}</code> in the policy box first | |
| if you see too much noise, and start from <code>annotate</code>, never <code>block</code>.</p> | |
| <h2>Not checked, on purpose</h2> | |
| <ul class="muted"> | |
| <li><strong>Relative dates</strong> ("last year", "昨年") — unresolvable without a generation date.</li> | |
| <li><strong>Bare small integers</strong> (0–10, no unit) are never marked contradicted: they sit near everything.</li> | |
| <li><strong>List markers</strong> ("1.", "2)") are not numeric claims.</li> | |
| <li><strong>Reasoning, tone, completeness, relevance.</strong> None are string-checkable.</li> | |
| </ul> | |
| <p class="warn">ClaimCheck is one layer of defence, not a correctness guarantee.</p> | |
| </section> | |
| <footer> | |
| Apache-2.0 · | |
| <a href="https://huggingface.co/NagaYu/claimcheck-rules" target="_blank" rel="noopener">rule pack</a> · | |
| <a href="https://huggingface.co/datasets/NagaYu/claimcheck-eval" target="_blank" rel="noopener">evaluation set</a> · | |
| verification runs entirely in your browser | |
| </footer> | |
| </div> | |
| <script type="module"> | |
| </script> | |
| </body> | |
| </html> | |