index.html

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>EchoCache — semantic cache for LLM APIs</title>
<meta name="description" content="A semantic cache with a MismatchGuard that blocks similar-looking prompts whose meaning differs. 0 wrong reuses out of 90 adversarial pairs at the default threshold.">
<style>
  :root {
    --bg:#ffffff; --fg:#1b1f23; --muted:#5b6672; --line:#e3e7eb; --card:#f7f9fb;
    --accent:#1f7a4d; --accent-soft:#e6f4ec; --warn:#8a5a00; --warn-bg:#fff6e0; --warn-line:#f0d089;
    --yes:#15803d; --yes-bg:#e8f6ee; --no:#b42318; --no-bg:#fdeceb; --code:#f2f4f7;
  }
  @media (prefers-color-scheme: dark) {
    :root:not([data-theme="light"]) {
      --bg:#0f1418; --fg:#e7edf2; --muted:#9aa7b2; --line:#232c34; --card:#161d23;
      --accent:#4ade80; --accent-soft:#12291c; --warn:#f2c14e; --warn-bg:#2a2313; --warn-line:#5a4a1c;
      --yes:#4ade80; --yes-bg:#12291c; --no:#f98b82; --no-bg:#2b1715; --code:#1a2228;
    }
  }
  * { box-sizing:border-box; }
  body {
    margin:0; background:var(--bg); color:var(--fg);
    font:16px/1.65 -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue","Hiragino Sans","Noto Sans JP",sans-serif;
    -webkit-font-smoothing:antialiased;
  }
  .wrap { max-width:920px; margin:0 auto; padding:0 16px 72px; }
  header { padding:56px 0 8px; }
  h1 { font-size:2.4rem; line-height:1.15; margin:0 0 .3em; letter-spacing:-.02em; }
  h1 .emoji { font-size:1em; }
  h2 { font-size:1.35rem; margin:2.6em 0 .6em; letter-spacing:-.01em; }
  h3 { font-size:1.05rem; margin:1.8em 0 .4em; }
  p { margin:.7em 0; }
  .lead { font-size:1.12rem; color:var(--fg); }
  .sub { color:var(--muted); }
  a { color:var(--accent); text-decoration:none; border-bottom:1px solid transparent; }
  a:hover { border-bottom-color:currentColor; }
  code { background:var(--code); padding:.12em .38em; border-radius:4px; font-size:.87em;
         font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace; word-break:break-word; }
  pre { background:var(--code); padding:14px 16px; border-radius:10px; overflow-x:auto; font-size:.84rem; line-height:1.55;
        border:1px solid var(--line); }
  pre code { background:none; padding:0; font-size:1em; }
  .notice { background:var(--warn-bg); border:1px solid var(--warn-line); color:var(--fg);
            border-radius:12px; padding:16px 18px; margin:26px 0; }
  .notice b { color:var(--warn); }
  .cards { display:grid; grid-template-columns:repeat(auto-fit,minmax(240px,1fr)); gap:12px; margin:22px 0; }
  .card { background:var(--card); border:1px solid var(--line); border-radius:12px; padding:16px 18px; }
  .card .k { font-size:.72rem; text-transform:uppercase; letter-spacing:.09em; color:var(--muted); }
  .card .v { font-size:1.02rem; font-weight:600; margin:.25em 0 .2em; }
  .card p { margin:.2em 0 0; font-size:.88rem; color:var(--muted); }
  .scroll { overflow-x:auto; margin:16px 0; border:1px solid var(--line); border-radius:12px; }
  table { border-collapse:collapse; width:100%; font-size:.9rem; }
  th, td { text-align:left; padding:10px 12px; border-bottom:1px solid var(--line); vertical-align:top; }
  th { background:var(--card); font-weight:600; font-size:.78rem; text-transform:uppercase; letter-spacing:.05em; color:var(--muted); white-space:nowrap; }
  tr:last-child td { border-bottom:none; }
  td.num { font-variant-numeric:tabular-nums; white-space:nowrap; font-family:ui-monospace,SFMono-Regular,Menlo,monospace; font-size:.88em; }
  td.p code { display:inline-block; margin:1px 0; }
  td.p .vs { color:var(--muted); font-size:.78rem; }
  td.reason { font-family:ui-monospace,SFMono-Regular,Menlo,monospace; font-size:.78rem; color:var(--muted); word-break:break-word; }
  .note { color:var(--muted); font-size:.78rem; margin-top:4px; }
  .yes { color:var(--yes); background:var(--yes-bg); padding:.12em .5em; border-radius:20px; font-size:.74rem; font-weight:700; letter-spacing:.04em; white-space:nowrap; }
  .no  { color:var(--no);  background:var(--no-bg);  padding:.12em .5em; border-radius:20px; font-size:.74rem; font-weight:700; letter-spacing:.04em; white-space:nowrap; }
  .hl { background:var(--accent-soft); }
  ul { padding-left:1.2em; } li { margin:.35em 0; }
  hr { border:none; border-top:1px solid var(--line); margin:3em 0 1.6em; }
  footer { color:var(--muted); font-size:.85rem; }
  .steps { counter-reset:s; padding-left:0; list-style:none; }
  .steps li { counter-increment:s; position:relative; padding-left:34px; margin:.9em 0; }
  .steps li::before { content:counter(s); position:absolute; left:0; top:1px; width:23px; height:23px;
    border-radius:50%; background:var(--accent-soft); color:var(--accent); font-size:.78rem; font-weight:700;
    display:flex; align-items:center; justify-content:center; }
</style>
</head>
<body>
<div class="wrap">

<header>
  <h1><span class="emoji">♻️</span> EchoCache</h1>
  <p class="lead">A semantic cache in front of your LLM API: it answers repeat questions from memory instead of paying for them again — and refuses to answer a <em>different</em> question from cache.</p>
  <p class="sub">Matching is 100% local (hashed character n-grams + SimHash, no model download, no network). Apache-2.0.</p>
</header>

<div class="notice">
  <p style="margin-top:0"><b>This page is documentation, not the running app.</b></p>
  <p>Hugging Face now requires a PRO subscription to host Gradio Spaces on free CPU hardware, so this Space is published as a <em>static</em> page. Everything below is real — the code, the measurements and the deploy steps — but there is no live demo to click here.</p>
  <p style="margin-bottom:0">To run it: <a href="https://huggingface.co/spaces/NagaYu/EchoCache/blob/main/app.py">take <code>app.py</code></a> and <code>requirements.txt</code> from this repository and either push them to a Gradio Space on paid hardware, or run <code>python3 app.py</code> locally. It is one file with no model download and it starts in seconds.</p>
</div>

<div class="cards">
  <div class="card">
    <div class="k">Space — this repo</div>
    <div class="v"><a href="https://huggingface.co/spaces/NagaYu/EchoCache">NagaYu/EchoCache</a></div>
    <p>The full application: cache, dashboard, threshold sweep, audit, JSON API. Static here; runnable anywhere.</p>
  </div>
  <div class="card">
    <div class="k">Model</div>
    <div class="v"><a href="https://huggingface.co/NagaYu/echocache-matcher">echocache-matcher</a></div>
    <p>The matching core as one numpy-only file. Deterministic and rule-based — <strong>no trained weights</strong>.</p>
  </div>
  <div class="card">
    <div class="k">Dataset</div>
    <div class="v"><a href="https://huggingface.co/datasets/NagaYu/echocache-guard-benchmark">echocache-guard-benchmark</a></div>
    <p>131 labelled pairs that measure wrong reuse. Scores any cache, not just this one.</p>
  </div>
</div>

<h2>The problem a similarity score cannot solve</h2>
<p>A cosine of 0.95 means <em>“these two strings look alike”</em>. It does not mean <em>“these two questions have the same answer”</em>. <code>Can I cancel my subscription?</code> and <code>Can I <b>not</b> cancel my subscription?</code> are near-identical to any similarity function and have opposite answers. Serving the cached answer there is not a cache miss — it is a wrong answer, and it costs far more than the tokens it saved.</p>
<p>EchoCache therefore splits the decision in two. Similarity picks <em>candidates</em>; a <strong>MismatchGuard</strong> decides whether a candidate may actually be served, vetoing any reuse where the meaning has shifted.</p>

<h2>Measured correctness</h2>
<p>Scored on the companion benchmark: 131 pairs — 41 a cache <em>may</em> reuse, 90 high-similarity pairs it <em>must not</em>. <strong>Wrong reuse</strong> counts cached answers served for a question whose meaning differs.</p>
<div class="scroll">
<table>
<thead><tr><th>Threshold</th><th>MismatchGuard</th><th>Accuracy</th><th>Reuse recall</th><th>Wrong reuse</th></tr></thead>
<tbody>
<tr class="hl"><td class="num">0.92 <span class="sub">(default)</span></td><td><b>on</b></td><td class="num">97.7%</td><td class="num">92.7%</td><td class="num"><b>0 / 90</b></td></tr>
<tr><td class="num">0.92</td><td>off</td><td class="num">93.1%</td><td class="num">92.7%</td><td class="num">6 / 90</td></tr>
<tr class="hl"><td class="num">0.80</td><td><b>on</b></td><td class="num">99.2%</td><td class="num">100%</td><td class="num"><b>1 / 90</b></td></tr>
<tr><td class="num">0.80</td><td>off</td><td class="num">77.9%</td><td class="num">100%</td><td class="num">29 / 90</td></tr>
<tr><td class="num">0.70</td><td>off</td><td class="num">69.5%</td><td class="num">100%</td><td class="num">40 / 90</td></tr>
</tbody>
</table>
</div>
<p>The guard is what makes an aggressive threshold survivable: at 0.80 it turns a 32% wrong-answer rate into 1%. Reproduce it with <code>python3 evaluate.py</code> in the dataset repo.</p>

<h2>Real decisions, recorded from the implementation</h2>
<p class="sub">Every similarity below was produced by running the published matcher — not estimated.</p>
<div class="scroll">
<table>
<thead><tr><th>Prompt pair</th><th>Similarity</th><th>Threshold</th><th>Verdict</th><th>Reason</th></tr></thead>
<tbody>
<tr>
<td class="p"><code>Hi team, how do I reset my password? Thanks in advance!</code><br><span class="vs">vs</span> <code>How do I reset my password?</code><div class="note">greeting and sign-off normalized away</div></td>
<td class="num">1.0000</td><td class="num">0.92</td><td><span class="yes">REUSE</span></td><td class="reason">exact key match</td>
</tr>
<tr>
<td class="p"><code>Could you please tell me how do I reset my password?</code><br><span class="vs">vs</span> <code>How do I reset my password?</code><div class="note">polite request frame is boilerplate</div></td>
<td class="num">1.0000</td><td class="num">0.92</td><td><span class="yes">REUSE</span></td><td class="reason">exact key match</td>
</tr>
<tr>
<td class="p"><code>お世話になっております。返品の方法を教えてください。よろしくお願いいたします。</code><br><span class="vs">vs</span> <code>返品の方法を教えてください</code><div class="note">Japanese honorific opening and closing removed</div></td>
<td class="num">1.0000</td><td class="num">0.92</td><td><span class="yes">REUSE</span></td><td class="reason">exact key match</td>
</tr>
<tr>
<td class="p"><code>How can I reset my password?</code><br><span class="vs">vs</span> <code>How do I reset my password?</code><div class="note">true paraphrase, reached at a lower threshold</div></td>
<td class="num">0.8476</td><td class="num">0.80</td><td><span class="yes">REUSE</span></td><td class="reason">—</td>
</tr>
<tr>
<td class="p"><code>Can I not cancel my subscription?</code><br><span class="vs">vs</span> <code>Can I cancel my subscription?</code><div class="note">negation flip</div></td>
<td class="num">0.9101</td><td class="num">0.70</td><td><span class="no">REFUSE</span></td><td class="reason">negation_mismatch:q=True,c=False</td>
</tr>
<tr>
<td class="p"><code>retry after 30 seconds</code><br><span class="vs">vs</span> <code>retry after 3 seconds</code><div class="note">quantity</div></td>
<td class="num">0.8840</td><td class="num">0.70</td><td><span class="no">REFUSE</span></td><td class="reason">numeric_mismatch:q={30},c={3}</td>
</tr>
<tr>
<td class="p"><code>Does the plan include 100MB of storage?</code><br><span class="vs">vs</span> <code>Does the plan include 100GB of storage?</code><div class="note">unit</div></td>
<td class="num">0.9231</td><td class="num">0.70</td><td><span class="no">REFUSE</span></td><td class="reason">unit_mismatch:q={100mb},c={100gb}</td>
</tr>
<tr>
<td class="p"><code>Does the X1 Yoga support 32GB?</code><br><span class="vs">vs</span> <code>Does the X1 Carbon support 32GB?</code><div class="note">product name</div></td>
<td class="num">0.7746</td><td class="num">0.70</td><td><span class="no">REFUSE</span></td><td class="reason">proper_noun_mismatch:q={x1,yoga},c={carbon,x1}</td>
</tr>
<tr>
<td class="p"><code>What is the 2025 tax rate?</code><br><span class="vs">vs</span> <code>What is the 2024 tax rate?</code><div class="note">time anchor</div></td>
<td class="num">0.8846</td><td class="num">0.70</td><td><span class="no">REFUSE</span></td><td class="reason">numeric_mismatch:q={2025},c={2024}</td>
</tr>
<tr>
<td class="p"><code>Why do I need to reset my password?</code><br><span class="vs">vs</span> <code>How do I reset my password?</code><div class="note">question type</div></td>
<td class="num">0.7327</td><td class="num">0.70</td><td><span class="no">REFUSE</span></td><td class="reason">question_type_mismatch:q={why},c={how}</td>
</tr>
<tr>
<td class="p"><code>Translate cheers into German</code><br><span class="vs">vs</span> <code>Translate cheers into French</code><div class="note">a sign-off word used as content</div></td>
<td class="num">0.7500</td><td class="num">0.70</td><td><span class="no">REFUSE</span></td><td class="reason">proper_noun_mismatch:q={german},c={french}</td>
</tr>
<tr>
<td class="p"><code>I forgot my login credentials, what now?</code><br><span class="vs">vs</span> <code>How do I reset my password?</code><div class="note">same intent, no shared surface - a known miss</div></td>
<td class="num">0.1174</td><td class="num">0.92</td><td><span class="no">REFUSE</span></td><td class="reason">temporal_mismatch:q={now},c={}</td>
</tr>
</tbody>
</table>
</div>

<h2>How a lookup works</h2>
<ol class="steps">
  <li><b>Exact.</b> sha256 of the normalized prompt. Normalization folds NFKC, case, whitespace and punctuation, and removes greetings, sign-offs, disclaimers and politeness wrappers in English and Japanese. Three very different-looking prompts collapse onto one key.</li>
  <li><b>SimHash band.</b> A 64-bit fingerprint of the same character n-grams narrows thousands of entries to a handful, with a vectorized popcount. No candidate, no cosine.</li>
  <li><b>Cosine + MismatchGuard.</b> Only survivors are scored against a hashed character-n-gram vector. Anything above the threshold must still pass five checks: negation flip, quantity/unit, proper noun or model number, time anchor, question type.</li>
</ol>
<p>Character n-grams mean <strong>no tokenizer and no word boundaries</strong>: Japanese, Chinese and Thai behave exactly like English. Measured on a laptop with 5,000 entries in one partition: ~0.2 ms per store, ~0.11 ms for an exact hit, ~0.3 ms for a full miss.</p>

<h2>What else is in the app</h2>
<ul>
  <li><b>Per-tenant isolation</b> — each tenant gets its own exact-key map, entry list, vector matrix, fingerprint array and LRU. No method takes two partitions, so there is no path that can surface one tenant's entry to another.</li>
  <li><b>A safety screen before storage</b> — Luhn-validated card numbers, credential prefixes, JWTs, PEM keys, high-entropy secrets and excess contact PII are refused with a machine-readable reason code.</li>
  <li><b>Threshold sweep</b> — upload a CSV of real traffic and see, per threshold, how much you would have reused and saved, with the guard on and off. That is the number you take to a review.</li>
  <li><b>Observability</b> — hit-rate over time, cumulative savings per tenant and route, latency split by hit/miss, most-reused entries, borderline decisions, index health, CSV export. Prompt text is not logged by default.</li>
  <li><b>Audit</b> — every candidate the guard rejected, with its reason, so a wrong hit can be traced and the entry invalidated.</li>
</ul>

<h2>The JSON API</h2>
<p>Nine endpoints, all returning an envelope. <code>ok: true</code> means the call completed — a miss is a successful call.</p>
<pre><code>POST /gradio_api/call/lookup   {"data": ["acme", "How do I reset my password?", 0.92, "support"]}

{"ok": true, "hit": true, "stage": "exact", "similarity": 1.0,
 "entry_id": "b67ac733...", "age_sec": 0.022, "response": "Open Settings &gt; Security &gt; Reset password.",
 "savings": {"saved_tokens": 18, "saved_cost_usd": 0.0},
 "summary": "HIT via exact (similarity 1.0000, age 0.022s, reused 1x)"}</code></pre>
<pre><code>{"ok": true, "hit": false, "stage": "miss", "reason": "guard_rejected",
 "best_similarity": 0.9101, "threshold": 0.7,
 "guard_reasons": ["negation_mismatch:q=True,c=False"],
 "summary": "MISS (candidate was above threshold but MismatchGuard rejected it)"}</code></pre>
<p><code>/store</code> · <code>/invalidate</code> · <code>/stats</code> · <code>/sweep</code> · <code>/audit</code> · <code>/export_index</code> · <code>/import_index</code> · <code>/health</code> complete the surface. The running app builds the exact paths for your deployment in its <em>API Docs</em> tab, because Gradio changes them between major versions.</p>

<h2>Run it yourself</h2>
<pre><code># locally - identical behaviour to a Space
git clone https://huggingface.co/spaces/NagaYu/EchoCache
cd EchoCache
pip install -r requirements.txt
python3 app.py            # http://localhost:7860</code></pre>
<p>To host it: create a <b>Gradio</b> Space (paid hardware or PRO), push <code>app.py</code>, <code>requirements.txt</code> and <code>README.md</code>, and it builds in a minute or two. No secrets are required — <code>HF_TOKEN</code> is optional and only enables an embedding re-rank that the cache never depends on. Full walkthrough: <a href="https://huggingface.co/spaces/NagaYu/EchoCache/blob/main/deploy.md">deploy.md</a> · <a href="https://huggingface.co/spaces/NagaYu/EchoCache/blob/main/deploy.ja.md">日本語版</a>.</p>

<h2>Use just the matcher</h2>
<pre><code>from huggingface_hub import hf_hub_download
import importlib.util, sys

path = hf_hub_download("NagaYu/echocache-matcher", "echocache_matcher.py")
spec = importlib.util.spec_from_file_location("echocache_matcher", path)
m = importlib.util.module_from_spec(spec); sys.modules["echocache_matcher"] = m
spec.loader.exec_module(m)

m.match("Can I cancel my subscription?", "Can I not cancel my subscription?", 0.5)
# {'reuse': False, 'similarity': 0.9101, 'guard_reasons': ['negation_mismatch:q=True,c=False'], ...}</code></pre>

<h2>Limitations</h2>
<ul>
  <li><b>Surface similarity only.</b> Paraphrases that share meaning but not characters are not matched, by design: <code>How do I reset my password?</code> vs <code>I forgot my login credentials, what now?</code> scores <b>0.12</b>. If your traffic is full of synonym-only rewordings, pair it with embeddings or accept the misses.</li>
  <li><b>The guard refuses valid reuse on purpose.</b> Any difference in the numeric-token set vetoes a hit, so <code>a 3 day trial</code> and <code>a three day trial</code> do not match. A miss costs one API call; a wrong hit costs trust.</li>
  <li><b>Free-tier reality.</b> Spaces have no persistent disk and sleep after 48h. The index lives in RAM: export it before you go idle and import it after wake-up. A cache is not a database.</li>
  <li><b>Savings are estimates.</b> Tokens are approximated locally and multiplied by the prices you configure. Useful for comparing thresholds, not for reconciling an invoice.</li>
  <li><b>The benchmark is not independent.</b> It was written alongside the matcher by the same author, and two matcher fixes came out of it. It is a regression baseline, not a third-party leaderboard.</li>
</ul>

<hr>
<footer>
  <p>Apache-2.0 · <a href="https://huggingface.co/spaces/NagaYu/EchoCache/tree/main">browse the source</a> · <a href="https://huggingface.co/NagaYu/echocache-matcher">model</a> · <a href="https://huggingface.co/datasets/NagaYu/echocache-guard-benchmark">dataset</a></p>
  <p>Semantic caching returns an answer written for a different prompt. Treat the threshold as a product decision, not a tuning knob.</p>
</footer>

</div>
</body>
</html>