| --- |
| title: EchoCache |
| emoji: ♻️ |
| colorFrom: green |
| colorTo: blue |
| sdk: static |
| app_file: index.html |
| pinned: false |
| license: apache-2.0 |
| short_description: Semantic cache and cost observability layer for LLM APIs |
| --- |
| |
| > ### ⚠️ This Space is a static documentation page, not the running app |
| > |
| > Hugging Face requires a PRO subscription to host **Gradio** Spaces on free CPU hardware, so this Space is |
| > published with `sdk: static`. Everything in this repository is the real, complete application — it simply |
| > is not being served here. |
| > |
| > **To run it:** clone this repo and `python3 app.py`, or push `app.py`, `requirements.txt` and |
| > **`README.gradio.md` (renamed to `README.md`)** to a Gradio Space on paid hardware. One file, no model |
| > download, starts in seconds. Full walkthrough in [`deploy.md`](deploy.md) · [日本語](deploy.ja.md). |
|
|
|
|
| # ♻️ EchoCache |
|
|
| **A semantic cache in front of your LLM API: it answers repeat questions from memory instead of paying for them again.** |
| **Matching is 100% local (character n-gram cosine + SimHash, no model download), so it runs on a free CPU Space.** |
| **A MismatchGuard blocks similar-looking prompts whose *meaning* differs, because a wrong cache hit costs far more than the tokens it saves.** |
|
|
| | | | |
| |---|---| |
| | 🚀 **Space** | [`NagaYu/EchoCache`](https://huggingface.co/spaces/NagaYu/EchoCache) — the running service (this repo) | |
| | 🧩 **Model** | [`NagaYu/echocache-matcher`](https://huggingface.co/NagaYu/echocache-matcher) — the matching core as a standalone module (no weights) | |
| | 📊 **Dataset** | [`NagaYu/echocache-guard-benchmark`](https://huggingface.co/datasets/NagaYu/echocache-guard-benchmark) — 131 labelled pairs that measure wrong reuse | |
|
|
| ## Measured correctness |
|
|
| Scored on the companion benchmark: 131 prompt pairs, 41 that a cache *may* reuse and 90 high-similarity pairs it *must not*. **Wrong reuse** counts cached answers served for a question whose meaning differs — each one is a wrong answer shipped to a user. |
|
|
| | threshold | MismatchGuard | accuracy | reuse recall | **wrong reuse** | |
| |---|---|---|---|---| |
| | 0.92 (default) | **on** | 97.7% | 92.7% | **0 / 90** | |
| | 0.92 | off | 93.1% | 92.7% | 6 / 90 | |
| | 0.80 | **on** | 99.2% | 100% | **1 / 90** | |
| | 0.80 | off | 77.9% | 100% | 29 / 90 | |
| | 0.70 | off | 69.5% | 100% | 40 / 90 | |
|
|
| Reproduce it yourself: `python3 evaluate.py` in the dataset repo. The guard is what makes an aggressive threshold survivable — without it, reuse at 0.80 comes with a 32% wrong-answer rate. |
|
|
| --- |
|
|
| ## What it does |
|
|
| | Tab | Purpose | |
| |---|---| |
| | **Lookup** | Ask the cache. Returns hit/miss, the matching stage, similarity, entry age and the cached response. | |
| | **Store** | Save a prompt/response pair — after a safety screen that refuses secrets and PII. | |
| | **Invalidate** | Drop entries of one tenant by entry-id prefix, by route, or all of them. | |
| | **Dashboard** | Hit rate over time, cumulative savings per tenant/route, latency distribution, most-reused entries, index health, CSV export. | |
| | **Sweep** | Upload a request log and see **per threshold** how much you would have reused and saved — with MismatchGuard on and off. | |
| | **Audit** | Borderline decisions and every candidate MismatchGuard rejected, with reasons. | |
| | **Backup** | Export/import the index (the free tier has no persistent disk). | |
| | **Health** | Index size, mode, uptime, configuration. | |
| | **API Docs** | The API paths **assembled at runtime** from the running server. | |
|
|
| ## How a lookup works — three stages, in order |
|
|
| 1. **Exact** — sha256 of the normalized prompt. Normalization does NFKC folding, lowercasing, whitespace collapsing, punctuation unification (including CJK punctuation) and removal of greetings, sign-offs and disclaimers. `"Hi team, how do I reset my password? Thanks in advance!"` and `"how do i reset my password?"` land on the same key. So do `"お世話になっております。返品の方法を教えてください。よろしくお願いいたします。"` and `"返品の方法を教えてください"`. |
| 2. **SimHash band** — a 64-bit fingerprint of the same character n-grams narrows thousands of entries to a handful (Hamming distance ≤ `SIMHASH_MAX_DIST`, vectorized popcount). Small partitions fall back to a full scan so recall does not suffer. |
| 3. **Cosine + MismatchGuard** — only the survivors get a cosine score against a hashed character-n-gram vector (`VECTOR_DIM`, L2-normalized, `np.add.at` — a HashingVectorizer written by hand). Anything at or above the threshold must still pass MismatchGuard. |
|
|
| Character n-grams mean **no tokenizer and no word boundaries**: Japanese, Chinese and Thai work exactly like English. |
|
|
| Measured on a developer laptop, CPU only, 5,000 entries in one partition: **~0.2 ms per store, ~0.11 ms for an exact hit, ~0.3 ms for a full miss.** A shared free-tier vCPU is slower, but this stays far below the latency of the API call you are avoiding. |
|
|
| ## MismatchGuard — why a similarity score is not enough |
|
|
| A cosine of 0.95 means *"these two strings look alike"*. It does **not** mean *"these two questions have the same answer"*. Serving a cached answer for a different question is not a cache miss, it is a **wrong answer**: a quality incident that no amount of saved tokens pays for. |
|
|
| Above the threshold, five checks can still veto a hit: |
|
|
| | Check | Rejected pair | Why it matters | |
| |---|---|---| |
| | **Negation flip** | `Can I cancel my subscription?` vs `Can I **not** cancel my subscription?` · `返品できますか` vs `返品できませんか` | Near-identical strings, opposite answers. | |
| | **Quantity / unit** | `retry after **3** seconds` vs `retry after **30** seconds` · `a 100**GB** plan` vs `a 100**MB** plan` | One character apart, order-of-magnitude apart. | |
| | **Proper noun / model number** | `Does the X1 **Carbon** support 32GB?` vs `Does the X1 **Yoga** support 32GB?` | Same shape, different product. | |
| | **Time anchor** | `What is the **2024** tax rate?` vs `What is the **2025** tax rate?` · `latest` vs `last year` | Time-anchored answers go stale when the anchor moves. | |
| | **Question type** | `**How** do I reset my password?` vs `**Why** do I need to reset my password?` | Procedure vs rationale. | |
|
|
| Every rejection is recorded with its reason and shown in the **Audit** tab, e.g. |
| `numeric_mismatch:q={2024},c={2025}`, `negation_mismatch:q=True,c=False`, `question_type_mismatch:q={why},c={how}`. |
|
|
| The guard is deliberately conservative: **when in doubt it returns a miss** (you pay for one API call) rather than a hit (you ship a wrong answer). Individual checks can be switched off with `GUARD_DISABLE`, and the **Sweep** tab quantifies exactly how much reuse the guard costs you. |
|
|
| > ⚠️ **Semantic caching returns an answer that was written for a different prompt.** Treat the threshold as a product decision, not a tuning knob. Start at the conservative default, look at the Sweep table, and only then move it. |
|
|
| ## Tenant isolation — and why it holds |
|
|
| Every tenant gets its **own partition**: its own exact-key map, its own entry list, its own vector matrix, its own fingerprint array, its own LRU. |
|
|
| ```python |
| {tenant_id: {"exact": dict, "entries": list, "matrix": np.ndarray, |
| "simhashes": np.ndarray, "lru": OrderedDict, "free": list, "stats": dict}} |
| ``` |
|
|
| The guarantee rests on three structural properties, not on a filter applied after the fact: |
|
|
| 1. **One access point.** Every read and write goes through `CacheIndex._partition(tenant_id)`, which returns exactly one dict. No method takes two partitions. |
| 2. **No cross-partition scan.** `lookup()` computes fingerprints and cosine against `part["matrix"]` and `part["simhashes"]` only. There is no code path that concatenates partitions, so there is nothing to leak through. |
| 3. **Scoped mutation.** `invalidate()` and `sweep_expired()` operate on the partition they were handed. Tenant ids are normalized (`[^A-Za-z0-9_.\-:@]` → `_`, max 64 chars), so `../`, unicode tricks and empty ids all collapse to a safe partition name. |
|
|
| Two operations deliberately cross partitions, and **both only ever delete** — neither reads data from one tenant nor moves it to another: |
|
|
| * **Global entry budget** (`MAX_TOTAL_ENTRIES`): the oldest entry of the *largest* partition is evicted. |
| * **Partition cap** (`MAX_TENANTS`): a first write by tenant N+1 evicts the least-recently-used tenant's partition *in full*. Size `MAX_TENANTS` above your real tenant count, or an idle tenant loses its whole cache (a cold cache, never a wrong answer). |
|
|
| ## API |
|
|
| Open the **API Docs** tab in the running Space: it prints the paths for *your* URL and *your* Gradio version. The **"Use via API"** link in the page footer is generated by Gradio itself and is always authoritative. |
|
|
| ### curl (POST returns an `event_id`, GET streams the result) |
| |
| ```bash |
| SPACE="https://<your-name>-echocache.hf.space" |
| |
| EVENT_ID=$(curl -s -X POST "$SPACE/gradio_api/call/lookup" \ |
| -H "Content-Type: application/json" \ |
| -d '{"data": ["acme", "How do I reset my password?", 0.92, "support"]}' \ |
| | python3 -c "import sys,json; print(json.load(sys.stdin)['event_id'])") |
|
|
| curl -s -N "$SPACE/gradio_api/call/lookup/$EVENT_ID" |
| ``` |
| |
| ```bash |
| curl -s -X POST "$SPACE/gradio_api/call/store" -H "Content-Type: application/json" \ |
| -d '{"data": ["acme", "How do I reset my password?", "Open Settings > Security > Reset password.", "support", 86400]}' |
| ``` |
| |
| > `/gradio_api` is the prefix used by Gradio 5 and 6; Gradio 4 has no prefix. Do not hardcode it — read it from the API Docs tab. |
|
|
| ### gradio_client |
| |
| ```python |
| from gradio_client import Client |
|
|
| client = Client("<your-name>/EchoCache") # or the full https URL |
|
|
| client.predict("acme", "How do I reset my password?", |
| "Open Settings > Security > Reset password.", "support", 86400, api_name="/store") |
| |
| hit = client.predict("acme", "Hi team, how do I reset my password? Thanks!", 0.92, "support", api_name="/lookup") |
| if hit["hit"]: |
| print(hit["response"]) # reuse it |
| else: |
| print(hit["reason"]) # call your LLM, then /store the answer |
| |
| client.predict("acme", "*", api_name="/invalidate") |
| client.predict("", "", api_name="/stats") |
| client.predict(api_name="/health") |
| ``` |
| |
| ### Typical integration |
| |
| ```python |
| res = client.predict(tenant, prompt, 0.92, route, api_name="/lookup") |
| if res.get("ok") and res.get("hit"): |
| answer = res["response"] |
| else: |
| answer = call_your_llm(prompt) |
| client.predict(tenant, prompt, answer, route, 86400, api_name="/store") |
| ``` |
| |
| ## JSON contracts |
| |
| Everything returns an envelope. `ok: true` means the call completed; a *miss* is a successful call. |
| Failures return `{"ok": false, "error": {"code": "...", "message": "..."}}` and never an HTTP 500. |
| |
| ### `/lookup` — hit |
| |
| ```json |
| { |
| "ok": true, |
| "hit": true, |
| "stage": "exact", |
| "similarity": 1.0, |
| "entry_id": "b67ac733828f356f2be7f21cd84ed8f8d4c1cdb6cd01ee57c403dbf360911dff", |
| "age_sec": 0.022, |
| "ttl_remaining_sec": 86399.978, |
| "response": "Open Settings > Security > Reset password.", |
| "route": "support", |
| "hits": 1, |
| "tokens_in": 7, |
| "tokens_out": 11, |
| "guard_reasons": [], |
| "cached_prompt_preview": "How do I reset my password?", |
| "candidates": 0, |
| "scan_mode": "exact", |
| "embed_used": false, |
| "borderline": false, |
| "guard_rejected": [], |
| "latency_ms": 0.188, |
| "savings": {"saved_tokens_in": 7, "saved_tokens_out": 11, "saved_tokens": 18, "saved_cost_usd": 0.0}, |
| "tenant_id": "acme", |
| "summary": "HIT via exact (similarity 1.0000, age 0.022s, reused 1x)" |
| } |
| ``` |
| |
| `stage` is `exact` (stage 1), `cosine` (stage 3, local only) or `embedding` (stage 3, blended with the optional embedding API). |
| |
| ### `/lookup` — miss |
| |
| ```json |
| { |
| "ok": true, |
| "hit": false, |
| "stage": "miss", |
| "similarity": 0.9101, |
| "entry_id": null, |
| "age_sec": null, |
| "reason": "guard_rejected", |
| "threshold": 0.7, |
| "candidates": 1, |
| "best_similarity": 0.9101, |
| "scan_mode": "simhash_band", |
| "embed_used": false, |
| "guard_reasons": ["negation_mismatch:q=True,c=False"], |
| "guard_rejected": [{"entry_id": "0f3c...", "similarity": 0.9101, "reasons": ["negation_mismatch:q=True,c=False"]}], |
| "borderline": false, |
| "latency_ms": 0.42, |
| "savings": {"saved_tokens_in": 0, "saved_tokens_out": 0, "saved_tokens": 0, "saved_cost_usd": 0.0}, |
| "tenant_id": "acme", |
| "summary": "MISS (candidate was above threshold but MismatchGuard rejected it)" |
| } |
| ``` |
| |
| `reason` is one of `empty_index`, `no_candidates`, `below_threshold`, `guard_rejected`, `empty_prompt`. |
| `age_sec` and `savings` are always present on a miss (null / all-zero) so a consumer can sum one field shape |
| across hits and misses. A stage-3 hit additionally carries `rank` (the candidate's position after re-ranking); |
| an `exact` hit does not. |
| |
| ### `/store` — stored, and refused |
| |
| ```json |
| {"ok": true, "stored": true, "updated": false, "tenant_id": "acme", |
| "entry_id": "b67ac733...", "route": "support", "reason": "ok", "ttl_sec": 86400, |
| "expires_at": 1789885968.18, "tokens_in": 7, "tokens_out": 11, |
| "truncated": false, "entries_in_tenant": 1, |
| "summary": "STORED entry b67ac733828f356f (ttl 86400s)"} |
| ``` |
| |
| ```json |
| {"ok": true, "stored": false, "tenant_id": "acme", "reason": "credit_card_luhn", |
| "reason_help": "A digit run passed the Luhn checksum (probable card number).", |
| "truncated": false, |
| "summary": "REFUSED: credit_card_luhn - A digit run passed the Luhn checksum (probable card number)."} |
| ``` |
| |
| Refusal codes: `credit_card_luhn` (Luhn-validated only — order numbers are not flagged), `credential_prefix` (`sk-`, `ghp_`, `AKIA`, `xoxb-`, `AIza`, `hf_`, `glpat-`, …), `jwt_structure`, `high_entropy_secret`, `contact_pii_excess`, `private_key_block`, `empty_prompt`, `empty_response`. |
| |
| The entropy screen needs two signals, not one. Measured over 340 random 24–48 character keys the *minimum* entropy was 3.97 bits/char, while ordinary identifiers sit in the same band (`com/blog/how-to-reset-your-password-2024` = 4.16, `UserAuthenticationServiceImpl2024Handler` = 4.36) — so no single threshold separates them. Tokens whose structure is word-shaped (separator-delimited words, CamelCase, snake_case, URL slugs) are therefore excluded before the entropy test. The result: **97.4% of random keys refused, ordinary URLs and filenames cached normally.** |
| |
| ## Configuration |
| |
| Everything is optional — the Space runs with no configuration at all. |
| |
| | Variable | Default | Meaning | |
| |---|---|---| |
| | `HF_TOKEN` (also read from `HUGGINGFACE_HUB_TOKEN`, `HUGGINGFACEHUB_API_TOKEN`) | *(unset)* | **Optional.** Enables the embedding re-rank. Without it the Space runs in **local-only** mode, which is the supported production mode. If one of the aliases is exported in your shell, you are in `local+embedding` mode — the header and `/health` always show which. | |
| | `EMBED_MODEL` | `sentence-transformers/all-MiniLM-L6-v2` | Feature-extraction model for the optional re-rank. | |
| | `EMBED_WEIGHT` | `0.35` | Weight of the embedding cosine in the blend (local similarity keeps the majority). | |
| | `MAX_ENTRIES` | `5000` | Max entries **per tenant** (LRU eviction beyond it). | |
| | `MAX_TOTAL_ENTRIES` | `20000` | Hard ceiling across all tenants — this is the real memory guard. | |
| | `MAX_TEXT_CHARS` | `8000` | Max stored characters per prompt and per response. | |
| | `DEFAULT_THRESHOLD` | `0.92` | Default similarity threshold. | |
| | `BORDERLINE_MARGIN` | `0.03` | A decision within ±this of the threshold is logged as borderline. | |
| | `DEFAULT_TTL_SEC` | `86400` | Default entry TTL. | |
| | `PRICE_IN_PER_1K` / `PRICE_OUT_PER_1K` | `0.0` | Your prices, used for the savings estimate. Leave at 0 and the USD columns stay 0. | |
| | `VECTOR_DIM` | `4096` | Hashed n-gram dimension. Memory ≈ entries × dim × 4 bytes. | |
| | `SIMHASH_MAX_DIST` | `12` | Stage-2 fingerprint band width. | |
| | `GUARD_DISABLE` | *(empty)* | Comma list of guard checks to switch off: `negation,numeric,proper_noun,temporal,question_type`. | |
| | `LOG_PROMPT_PREFIX` | `0` | Characters of prompt text to keep in the event log. **0 = never log prompt text** (the default). | |
| | `MAX_TENANTS` | `256` | LRU cap on partitions. Reaching it **drops the least-recently-used tenant's entire partition**. | |
| | `ENTROPY_THRESHOLD` | `4.0` | Entropy bar (bits/char) for the secret screen; identifier-shaped tokens are excluded first. | |
| | `SWEEP_MAX_ROWS` | `2000` | Row cap per sweep (truncation is reported, never silent). | |
| |
| ### Memory |
| |
| The vector matrix dominates: `entries × VECTOR_DIM × 4 bytes`. With the defaults, one full tenant of 5,000 entries is **≈ 80 MB**, and `MAX_TOTAL_ENTRIES=20000` caps the whole index at **≈ 330 MB** — comfortable inside the free tier's 16 GB alongside Python and Gradio. If you raise `MAX_ENTRIES`, raise `MAX_TOTAL_ENTRIES` deliberately, or lower `VECTOR_DIM` (2048 halves the memory and costs little accuracy). |
| |
| ## Choosing a threshold |
| |
| The default `0.92` is **conservative on purpose**: at that level you mostly reuse exact and near-duplicate prompts (different politeness, punctuation, casing, greetings). Genuine paraphrases with different wording typically score **0.80–0.90** with character n-grams. |
| |
| Do not guess. Export a day of real prompts to CSV (`tenant_id, prompt, response, route`), run the **Sweep** tab, and read the row you can defend: it shows reuse rate, estimated saving, borderline count and guard blocks for every threshold, with the guard on and off. The gap between the two curves is the reuse you are *deliberately* giving up to avoid wrong answers. |
| |
| ## Optional embedding enrichment |
| |
| With `HF_TOKEN` set, stage 3 blends the local cosine with an embedding cosine for the top few candidates: |
| `final = (1 - EMBED_WEIGHT) × local + EMBED_WEIGHT × embedding`. |
| |
| It is strictly optional and strictly non-critical: |
| |
| * every call has a timeout and is wrapped in a catch-all; |
| * `402` (out of credit), `429` (rate limited), timeouts and DNS failures all return `None`; |
| * after `EMBED_FAIL_THRESHOLD` consecutive failures a **circuit breaker** opens for `EMBED_COOLDOWN_SEC`, so a dead endpoint is not hammered; |
| * on any of these, the lookup completes using the local vector alone. Nothing degrades except re-ranking quality. |
| |
| A free Hugging Face account has roughly **$0.10/month** of inference credit — that is the whole reason the local path is the primary path. |
| |
| ## Free-tier constraints — read this before you rely on it |
| |
| * **The disk is not persistent.** Nothing survives a restart, a rebuild or a new commit. |
| * **The Space sleeps after 48h of inactivity**, and on the free tier that timer cannot be changed. |
| * **The index lives in RAM only.** A sleep or restart empties it. This is by design: a cache is not a database. |
| * **Carry it over with the Backup tab**: export before you go idle, import after wake-up. Exports always include prompts (they are needed to rebuild the vectors); response bodies are optional, and rows exported without a response are skipped on import. |
| * Events live in a bounded ring buffer (`EVENT_BUFFER`, default 5,000) — export the CSV if you need history. |
| * Cold start is a few seconds: no model is downloaded, nothing heavy runs at import time. |
| |
| ## Privacy |
| |
| * Prompt text is **not** written to the event log by default (`LOG_PROMPT_PREFIX=0`). |
| * The safety screen runs **before** storage, so secrets never enter the index. |
| * Nothing leaves the Space unless `HF_TOKEN` is set, and then only prompt text sent to the embedding endpoint. |
| |
| ## Disclaimer |
| |
| **Savings figures are estimates, not a bill.** Tokens are approximated locally (~4 ASCII characters or ~1 CJK character per token) and multiplied by the prices you configured. Real invoices depend on your provider's tokenizer, system prompts, retries and per-model pricing. Use these numbers to compare thresholds and to build a case — not to reconcile an invoice. |
| |
| ## The three repositories |
| |
| * **Space** — [`NagaYu/EchoCache`](https://huggingface.co/spaces/NagaYu/EchoCache): the running cache, dashboard, threshold sweep and JSON API. |
| * **Model** — [`NagaYu/echocache-matcher`](https://huggingface.co/NagaYu/echocache-matcher): `echocache_matcher.py`, the deterministic matching core (normalizer, hashed n-gram vectorizer, SimHash, secret screen, MismatchGuard) as a numpy-only module you can drop into your own cache. **It contains no trained weights** — it is published as a model because it fills the slot an embedding model would normally occupy. |
| * **Dataset** — [`NagaYu/echocache-guard-benchmark`](https://huggingface.co/datasets/NagaYu/echocache-guard-benchmark): the 131 labelled pairs behind the table above, plus the scoring script. Use it to check *any* semantic cache, not just this one. |
| |
| Licensed under Apache-2.0. |
| |
