README.md

metadata

title: EchoCache
emoji: ♻️
colorFrom: green
colorTo: blue
sdk: static
app_file: index.html
pinned: false
license: apache-2.0
short_description: Semantic cache and cost observability layer for LLM APIs

⚠️ This Space is a static documentation page, not the running app

Hugging Face requires a PRO subscription to host Gradio Spaces on free CPU hardware, so this Space is published with sdk: static. Everything in this repository is the real, complete application — it simply is not being served here.

To run it: clone this repo and python3 app.py, or push app.py, requirements.txt and README.gradio.md (renamed to README.md) to a Gradio Space on paid hardware. One file, no model download, starts in seconds. Full walkthrough in deploy.md · 日本語.

♻️ EchoCache

A semantic cache in front of your LLM API: it answers repeat questions from memory instead of paying for them again. Matching is 100% local (character n-gram cosine + SimHash, no model download), so it runs on a free CPU Space. A MismatchGuard blocks similar-looking prompts whose meaning differs, because a wrong cache hit costs far more than the tokens it saves.

🚀 Space	NagaYu/EchoCache — the running service (this repo)
🧩 Model	NagaYu/echocache-matcher — the matching core as a standalone module (no weights)
📊 Dataset	NagaYu/echocache-guard-benchmark — 131 labelled pairs that measure wrong reuse

Measured correctness

Scored on the companion benchmark: 131 prompt pairs, 41 that a cache may reuse and 90 high-similarity pairs it must not. Wrong reuse counts cached answers served for a question whose meaning differs — each one is a wrong answer shipped to a user.

threshold	MismatchGuard	accuracy	reuse recall	wrong reuse
0.92 (default)	on	97.7%	92.7%	0 / 90
0.92	off	93.1%	92.7%	6 / 90
0.80	on	99.2%	100%	1 / 90
0.80	off	77.9%	100%	29 / 90
0.70	off	69.5%	100%	40 / 90

Reproduce it yourself: python3 evaluate.py in the dataset repo. The guard is what makes an aggressive threshold survivable — without it, reuse at 0.80 comes with a 32% wrong-answer rate.

---

What it does

Tab	Purpose
Lookup	Ask the cache. Returns hit/miss, the matching stage, similarity, entry age and the cached response.
Store	Save a prompt/response pair — after a safety screen that refuses secrets and PII.
Invalidate	Drop entries of one tenant by entry-id prefix, by route, or all of them.
Dashboard	Hit rate over time, cumulative savings per tenant/route, latency distribution, most-reused entries, index health, CSV export.
Sweep	Upload a request log and see per threshold how much you would have reused and saved — with MismatchGuard on and off.
Audit	Borderline decisions and every candidate MismatchGuard rejected, with reasons.
Backup	Export/import the index (the free tier has no persistent disk).
Health	Index size, mode, uptime, configuration.
API Docs	The API paths assembled at runtime from the running server.

How a lookup works — three stages, in order

1. Exact — sha256 of the normalized prompt. Normalization does NFKC folding, lowercasing, whitespace collapsing, punctuation unification (including CJK punctuation) and removal of greetings, sign-offs and disclaimers. "Hi team, how do I reset my password? Thanks in advance!" and "how do i reset my password?" land on the same key. So do "お世話になっております。返品の方法を教えてください。よろしくお願いいたします。" and "返品の方法を教えてください".
2. SimHash band — a 64-bit fingerprint of the same character n-grams narrows thousands of entries to a handful (Hamming distance ≤ SIMHASH_MAX_DIST, vectorized popcount). Small partitions fall back to a full scan so recall does not suffer.
3. Cosine + MismatchGuard — only the survivors get a cosine score against a hashed character-n-gram vector (VECTOR_DIM, L2-normalized, np.add.at — a HashingVectorizer written by hand). Anything at or above the threshold must still pass MismatchGuard.

Character n-grams mean no tokenizer and no word boundaries: Japanese, Chinese and Thai work exactly like English.

Measured on a developer laptop, CPU only, 5,000 entries in one partition: ~0.2 ms per store, ~0.11 ms for an exact hit, ~0.3 ms for a full miss. A shared free-tier vCPU is slower, but this stays far below the latency of the API call you are avoiding.

MismatchGuard — why a similarity score is not enough

A cosine of 0.95 means "these two strings look alike". It does not mean "these two questions have the same answer". Serving a cached answer for a different question is not a cache miss, it is a wrong answer: a quality incident that no amount of saved tokens pays for.

Above the threshold, five checks can still veto a hit:

Check	Rejected pair	Why it matters
Negation flip	Can I cancel my subscription? vs Can I **not** cancel my subscription? · 返品できますか vs 返品できませんか	Near-identical strings, opposite answers.
Quantity / unit	retry after **3** seconds vs retry after **30** seconds · a 100**GB** plan vs a 100**MB** plan	One character apart, order-of-magnitude apart.
Proper noun / model number	Does the X1 **Carbon** support 32GB? vs Does the X1 **Yoga** support 32GB?	Same shape, different product.
Time anchor	What is the **2024** tax rate? vs What is the **2025** tax rate? · latest vs last year	Time-anchored answers go stale when the anchor moves.
Question type	**How** do I reset my password? vs **Why** do I need to reset my password?	Procedure vs rationale.

Every rejection is recorded with its reason and shown in the Audit tab, e.g. numeric_mismatch:q={2024},c={2025}, negation_mismatch:q=True,c=False, question_type_mismatch:q={why},c={how}.

The guard is deliberately conservative: when in doubt it returns a miss (you pay for one API call) rather than a hit (you ship a wrong answer). Individual checks can be switched off with GUARD_DISABLE, and the Sweep tab quantifies exactly how much reuse the guard costs you.

⚠️ Semantic caching returns an answer that was written for a different prompt. Treat the threshold as a product decision, not a tuning knob. Start at the conservative default, look at the Sweep table, and only then move it.

Tenant isolation — and why it holds

Every tenant gets its own partition: its own exact-key map, its own entry list, its own vector matrix, its own fingerprint array, its own LRU.

{tenant_id: {"exact": dict, "entries": list, "matrix": np.ndarray,
             "simhashes": np.ndarray, "lru": OrderedDict, "free": list, "stats": dict}}

The guarantee rests on three structural properties, not on a filter applied after the fact:

1. One access point. Every read and write goes through CacheIndex._partition(tenant_id), which returns exactly one dict. No method takes two partitions.
2. No cross-partition scan. lookup() computes fingerprints and cosine against part["matrix"] and part["simhashes"] only. There is no code path that concatenates partitions, so there is nothing to leak through.
3. Scoped mutation. invalidate() and sweep_expired() operate on the partition they were handed. Tenant ids are normalized ([^A-Za-z0-9_.\-:@] → _, max 64 chars), so ../, unicode tricks and empty ids all collapse to a safe partition name.

Two operations deliberately cross partitions, and both only ever delete — neither reads data from one tenant nor moves it to another:

• Global entry budget (MAX_TOTAL_ENTRIES): the oldest entry of the largest partition is evicted.
• Partition cap (MAX_TENANTS): a first write by tenant N+1 evicts the least-recently-used tenant's partition in full. Size MAX_TENANTS above your real tenant count, or an idle tenant loses its whole cache (a cold cache, never a wrong answer).

API

Open the API Docs tab in the running Space: it prints the paths for your URL and your Gradio version. The "Use via API" link in the page footer is generated by Gradio itself and is always authoritative.

curl (POST returns an event_id, GET streams the result)

SPACE="https://<your-name>-echocache.hf.space"

EVENT_ID=$(curl -s -X POST "$SPACE/gradio_api/call/lookup" \
  -H "Content-Type: application/json" \
  -d '{"data": ["acme", "How do I reset my password?", 0.92, "support"]}' \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['event_id'])")

curl -s -N "$SPACE/gradio_api/call/lookup/$EVENT_ID"

curl -s -X POST "$SPACE/gradio_api/call/store" -H "Content-Type: application/json" \
  -d '{"data": ["acme", "How do I reset my password?", "Open Settings > Security > Reset password.", "support", 86400]}'

/gradio_api is the prefix used by Gradio 5 and 6; Gradio 4 has no prefix. Do not hardcode it — read it from the API Docs tab.

gradio_client

from gradio_client import Client

client = Client("<your-name>/EchoCache")     # or the full https URL

client.predict("acme", "How do I reset my password?",
               "Open Settings > Security > Reset password.", "support", 86400, api_name="/store")

hit = client.predict("acme", "Hi team, how do I reset my password? Thanks!", 0.92, "support", api_name="/lookup")
if hit["hit"]:
    print(hit["response"])          # reuse it
else:
    print(hit["reason"])            # call your LLM, then /store the answer

client.predict("acme", "*", api_name="/invalidate")
client.predict("", "", api_name="/stats")
client.predict(api_name="/health")

Typical integration

res = client.predict(tenant, prompt, 0.92, route, api_name="/lookup")
if res.get("ok") and res.get("hit"):
    answer = res["response"]
else:
    answer = call_your_llm(prompt)
    client.predict(tenant, prompt, answer, route, 86400, api_name="/store")

JSON contracts

Everything returns an envelope. ok: true means the call completed; a miss is a successful call. Failures return {"ok": false, "error": {"code": "...", "message": "..."}} and never an HTTP 500.

/lookup — hit

{
  "ok": true,
  "hit": true,
  "stage": "exact",
  "similarity": 1.0,
  "entry_id": "b67ac733828f356f2be7f21cd84ed8f8d4c1cdb6cd01ee57c403dbf360911dff",
  "age_sec": 0.022,
  "ttl_remaining_sec": 86399.978,
  "response": "Open Settings > Security > Reset password.",
  "route": "support",
  "hits": 1,
  "tokens_in": 7,
  "tokens_out": 11,
  "guard_reasons": [],
  "cached_prompt_preview": "How do I reset my password?",
  "candidates": 0,
  "scan_mode": "exact",
  "embed_used": false,
  "borderline": false,
  "guard_rejected": [],
  "latency_ms": 0.188,
  "savings": {"saved_tokens_in": 7, "saved_tokens_out": 11, "saved_tokens": 18, "saved_cost_usd": 0.0},
  "tenant_id": "acme",
  "summary": "HIT via exact (similarity 1.0000, age 0.022s, reused 1x)"
}

stage is exact (stage 1), cosine (stage 3, local only) or embedding (stage 3, blended with the optional embedding API).

/lookup — miss

{
  "ok": true,
  "hit": false,
  "stage": "miss",
  "similarity": 0.9101,
  "entry_id": null,
  "age_sec": null,
  "reason": "guard_rejected",
  "threshold": 0.7,
  "candidates": 1,
  "best_similarity": 0.9101,
  "scan_mode": "simhash_band",
  "embed_used": false,
  "guard_reasons": ["negation_mismatch:q=True,c=False"],
  "guard_rejected": [{"entry_id": "0f3c...", "similarity": 0.9101, "reasons": ["negation_mismatch:q=True,c=False"]}],
  "borderline": false,
  "latency_ms": 0.42,
  "savings": {"saved_tokens_in": 0, "saved_tokens_out": 0, "saved_tokens": 0, "saved_cost_usd": 0.0},
  "tenant_id": "acme",
  "summary": "MISS (candidate was above threshold but MismatchGuard rejected it)"
}

reason is one of empty_index, no_candidates, below_threshold, guard_rejected, empty_prompt. age_sec and savings are always present on a miss (null / all-zero) so a consumer can sum one field shape across hits and misses. A stage-3 hit additionally carries rank (the candidate's position after re-ranking); an exact hit does not.

/store — stored, and refused

{"ok": true, "stored": true, "updated": false, "tenant_id": "acme",
 "entry_id": "b67ac733...", "route": "support", "reason": "ok", "ttl_sec": 86400,
 "expires_at": 1789885968.18, "tokens_in": 7, "tokens_out": 11,
 "truncated": false, "entries_in_tenant": 1,
 "summary": "STORED entry b67ac733828f356f (ttl 86400s)"}

{"ok": true, "stored": false, "tenant_id": "acme", "reason": "credit_card_luhn",
 "reason_help": "A digit run passed the Luhn checksum (probable card number).",
 "truncated": false,
 "summary": "REFUSED: credit_card_luhn - A digit run passed the Luhn checksum (probable card number)."}

Refusal codes: credit_card_luhn (Luhn-validated only — order numbers are not flagged), credential_prefix (sk-, ghp_, AKIA, xoxb-, AIza, hf_, glpat-, …), jwt_structure, high_entropy_secret, contact_pii_excess, private_key_block, empty_prompt, empty_response.

The entropy screen needs two signals, not one. Measured over 340 random 24–48 character keys the minimum entropy was 3.97 bits/char, while ordinary identifiers sit in the same band (com/blog/how-to-reset-your-password-2024 = 4.16, UserAuthenticationServiceImpl2024Handler = 4.36) — so no single threshold separates them. Tokens whose structure is word-shaped (separator-delimited words, CamelCase, snake_case, URL slugs) are therefore excluded before the entropy test. The result: 97.4% of random keys refused, ordinary URLs and filenames cached normally.

Configuration

Everything is optional — the Space runs with no configuration at all.

Variable	Default	Meaning
HF_TOKEN (also read from HUGGINGFACE_HUB_TOKEN, HUGGINGFACEHUB_API_TOKEN)	(unset)	Optional. Enables the embedding re-rank. Without it the Space runs in local-only mode, which is the supported production mode. If one of the aliases is exported in your shell, you are in local+embedding mode — the header and /health always show which.
EMBED_MODEL	sentence-transformers/all-MiniLM-L6-v2	Feature-extraction model for the optional re-rank.
EMBED_WEIGHT	0.35	Weight of the embedding cosine in the blend (local similarity keeps the majority).
MAX_ENTRIES	5000	Max entries per tenant (LRU eviction beyond it).
MAX_TOTAL_ENTRIES	20000	Hard ceiling across all tenants — this is the real memory guard.
MAX_TEXT_CHARS	8000	Max stored characters per prompt and per response.
DEFAULT_THRESHOLD	0.92	Default similarity threshold.
BORDERLINE_MARGIN	0.03	A decision within ±this of the threshold is logged as borderline.
DEFAULT_TTL_SEC	86400	Default entry TTL.
PRICE_IN_PER_1K / PRICE_OUT_PER_1K	0.0	Your prices, used for the savings estimate. Leave at 0 and the USD columns stay 0.
VECTOR_DIM	4096	Hashed n-gram dimension. Memory ≈ entries × dim × 4 bytes.
SIMHASH_MAX_DIST	12	Stage-2 fingerprint band width.
GUARD_DISABLE	(empty)	Comma list of guard checks to switch off: negation,numeric,proper_noun,temporal,question_type.
LOG_PROMPT_PREFIX	0	Characters of prompt text to keep in the event log. 0 = never log prompt text (the default).
MAX_TENANTS	256	LRU cap on partitions. Reaching it drops the least-recently-used tenant's entire partition.
ENTROPY_THRESHOLD	4.0	Entropy bar (bits/char) for the secret screen; identifier-shaped tokens are excluded first.
SWEEP_MAX_ROWS	2000	Row cap per sweep (truncation is reported, never silent).

Memory

The vector matrix dominates: entries × VECTOR_DIM × 4 bytes. With the defaults, one full tenant of 5,000 entries is ≈ 80 MB, and MAX_TOTAL_ENTRIES=20000 caps the whole index at ≈ 330 MB — comfortable inside the free tier's 16 GB alongside Python and Gradio. If you raise MAX_ENTRIES, raise MAX_TOTAL_ENTRIES deliberately, or lower VECTOR_DIM (2048 halves the memory and costs little accuracy).

Choosing a threshold

The default 0.92 is conservative on purpose: at that level you mostly reuse exact and near-duplicate prompts (different politeness, punctuation, casing, greetings). Genuine paraphrases with different wording typically score 0.80–0.90 with character n-grams.

Do not guess. Export a day of real prompts to CSV (tenant_id, prompt, response, route), run the Sweep tab, and read the row you can defend: it shows reuse rate, estimated saving, borderline count and guard blocks for every threshold, with the guard on and off. The gap between the two curves is the reuse you are deliberately giving up to avoid wrong answers.

Optional embedding enrichment

With HF_TOKEN set, stage 3 blends the local cosine with an embedding cosine for the top few candidates: final = (1 - EMBED_WEIGHT) × local + EMBED_WEIGHT × embedding.

It is strictly optional and strictly non-critical:

• every call has a timeout and is wrapped in a catch-all;
• 402 (out of credit), 429 (rate limited), timeouts and DNS failures all return None;
• after EMBED_FAIL_THRESHOLD consecutive failures a circuit breaker opens for EMBED_COOLDOWN_SEC, so a dead endpoint is not hammered;
• on any of these, the lookup completes using the local vector alone. Nothing degrades except re-ranking quality.

A free Hugging Face account has roughly $0.10/month of inference credit — that is the whole reason the local path is the primary path.

Free-tier constraints — read this before you rely on it

• The disk is not persistent. Nothing survives a restart, a rebuild or a new commit.
• The Space sleeps after 48h of inactivity, and on the free tier that timer cannot be changed.
• The index lives in RAM only. A sleep or restart empties it. This is by design: a cache is not a database.
• Carry it over with the Backup tab: export before you go idle, import after wake-up. Exports always include prompts (they are needed to rebuild the vectors); response bodies are optional, and rows exported without a response are skipped on import.
• Events live in a bounded ring buffer (EVENT_BUFFER, default 5,000) — export the CSV if you need history.
• Cold start is a few seconds: no model is downloaded, nothing heavy runs at import time.

Privacy

• Prompt text is not written to the event log by default (LOG_PROMPT_PREFIX=0).
• The safety screen runs before storage, so secrets never enter the index.
• Nothing leaves the Space unless HF_TOKEN is set, and then only prompt text sent to the embedding endpoint.

Disclaimer

Savings figures are estimates, not a bill. Tokens are approximated locally (~4 ASCII characters or ~1 CJK character per token) and multiplied by the prices you configured. Real invoices depend on your provider's tokenizer, system prompts, retries and per-model pricing. Use these numbers to compare thresholds and to build a case — not to reconcile an invoice.

The three repositories

• Space — NagaYu/EchoCache: the running cache, dashboard, threshold sweep and JSON API.
• Model — NagaYu/echocache-matcher: echocache_matcher.py, the deterministic matching core (normalizer, hashed n-gram vectorizer, SimHash, secret screen, MismatchGuard) as a numpy-only module you can drop into your own cache. It contains no trained weights — it is published as a model because it fills the slot an embedding model would normally occupy.
• Dataset — NagaYu/echocache-guard-benchmark: the 131 labelled pairs behind the table above, plus the scoring script. Use it to check any semantic cache, not just this one.

Licensed under Apache-2.0.