deploy.md

# Deploying EchoCache on a free Hugging Face Space

> 日本語版: [`deploy.ja.md`](deploy.ja.md)

About 10 minutes, copy-paste throughout. You need a browser, `git` and `python3`.
No paid plan, no GPU, no model download.

---

## 1. Create the Space

1. Open <https://huggingface.co/new-space>
2. Fill in:

   | Field | Value |
   |---|---|
   | **Owner** | your account |
   | **Space name** | `EchoCache` (referred to below as `<user>/EchoCache`) |
   | **License** | `apache-2.0` |
   | **SDK** | **Gradio** |
   | **Space hardware** | **CPU basic · 2 vCPU · 16 GB · FREE** |
   | **Visibility** | Public or Private — both work |

3. Press **Create Space**. If a Gradio template picker appears, choose "Blank"; every file gets replaced anyway.

---

## 2. Push the files

Log in once with a **write** token from <https://huggingface.co/settings/tokens>:

```bash
hf auth login          # older installs: huggingface-cli login
```

Then run this as-is, replacing `<user>` with your account name:

```bash
USER=<user>; SPACE=EchoCache; SRC="$(pwd)"
git clone https://huggingface.co/spaces/$USER/$SPACE ~/$SPACE-space && \
cp "$SRC/app.py" "$SRC/requirements.txt" "$SRC/deploy.md" "$SRC/deploy.ja.md" ~/$SPACE-space/ && \
cp "$SRC/README.gradio.md" ~/$SPACE-space/README.md && \
cd ~/$SPACE-space && git add app.py requirements.txt README.md deploy.md deploy.ja.md && \
git commit -m "Deploy EchoCache: semantic cache + cost observability" && git push
```

If `git clone` asks for credentials, use your account name as the username and your **access token** as the password.

The Space builds automatically (**Building** → **Running**, 1–2 minutes on a first push). Watch **Logs** on the Space page.

> Note the card swap: the published `README.md` carries `sdk: static`, so a Gradio deployment must use `README.gradio.md`.

> If the build complains about `sdk_version`, change that line in `README.md` to a version offered in your Space's Settings, then push again. Nothing else needs to change.

---

## 3. Set `HF_TOKEN` (optional — everything works without it)

Only needed if you want the optional embedding re-rank.

1. Space page → **Settings** → **Variables and secrets**
2. **New secret** → Name: `HF_TOKEN` / Value: your token → **Save**
   (`HUGGINGFACE_HUB_TOKEN` and `HUGGINGFACEHUB_API_TOKEN` are read as well. If one of them is exported in your shell during a local run, you will start in `local+embedding` mode and make outbound calls — the header and the `Health` tab always say which mode is active.)
3. **Restart this Space**

| | without `HF_TOKEN` | with `HF_TOKEN` |
|---|---|---|
| similarity | fully local (default, recommended) | local + embedding, weighted average |
| core features | **all working** | all working |
| on failure | — | 402/429/timeout falls back to local automatically |

A free Hugging Face account has roughly **$0.10/month** of inference credit. **Running without the token is the normal configuration.**

Useful optional variables, same screen:

```
PRICE_IN_PER_1K   = 0.003     # your input price; leave unset and savings stay $0
PRICE_OUT_PER_1K  = 0.015     # your output price
MAX_ENTRIES       = 5000      # per tenant
MAX_TOTAL_ENTRIES = 20000     # across all tenants — the real memory guard
MAX_TENANTS       = 256       # partition cap; exceeding it drops the LRU tenant's whole partition
DEFAULT_THRESHOLD = 0.92      # conservative by default; move it after running a Sweep
```

---

## 4. Verify it works (3 minutes in the UI)

1. **Open the Space** (`https://<user>-echocache.hf.space`). The header should read `Mode: local-only`.

2. **Store one entry** — `Store` tab, `tenant_id` = `acme`, `route` = `support`,
   prompt `How do I reset my password?`, response `Open Settings > Security > Reset password.` → **Store** returns `"stored": true` and an `entry_id`.
   (Or press *Seed 3 demo entries*.)

3. **Look it up**
   * `Lookup` tab, `tenant_id` = `acme`, prompt `Hi team, how do I reset my password? Thanks in advance!` → `"hit": true, "stage": "exact"`. The greeting and sign-off were normalized away onto the same key.
   * `How can I reset my password?` at threshold `0.80` → `"stage": "cosine"` (measured similarity ≈ **0.85**; at the default `0.92` it misses and returns `best_similarity` 0.85 — **read that number before choosing a threshold**).
   * **Wrong-hit protection:** `Why do I need to reset my password?` at threshold `0.70` → `"reason": "guard_rejected"` with `question_type_mismatch:q={why},c={how}`. High similarity is not enough.
   * **Tenant isolation:** switch `tenant_id` to `globex` and repeat → `"reason": "empty_index"`. Another tenant's entries are unreachable.

4. **Dashboard** → **Refresh**: one-line summary (requests / hits / misses / hit rate / saved tokens / estimated saving), hit-rate time series, cumulative savings per tenant, p90 latency, most-reused entries, index health. **Export events CSV** takes the log with you (prompt text is not recorded by default).

5. **Sweep** → **Download a sample log**, upload that same file, enter your prices, **Run sweep**. You get "at this threshold you reuse X% and save $Y", with MismatchGuard **on** and **off** side by side — the conservative row is the one you can defend in a review.

6. **API Docs** tab shows the real paths for your URL and your Gradio version. The **Use via API** link in the page footer is generated by Gradio itself and is always authoritative.

### From the command line

```bash
SPACE="https://<user>-echocache.hf.space"

EVENT_ID=$(curl -s -X POST "$SPACE/gradio_api/call/store" -H "Content-Type: application/json" \
  -d '{"data": ["acme","How do I reset my password?","Open Settings > Security > Reset password.","support",86400]}' \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['event_id'])")
curl -s -N "$SPACE/gradio_api/call/store/$EVENT_ID"

EVENT_ID=$(curl -s -X POST "$SPACE/gradio_api/call/lookup" -H "Content-Type: application/json" \
  -d '{"data": ["acme","Hi team, how do I reset my password? Thanks!",0.92,"support"]}' \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['event_id'])")
curl -s -N "$SPACE/gradio_api/call/lookup/$EVENT_ID"
```

```bash
pip install gradio_client
python3 - <<'PY'
from gradio_client import Client
c = Client("<user>/EchoCache")
print(c.predict("acme", "how do i reset my password", 0.92, "support", api_name="/lookup"))
print(c.predict(api_name="/health"))
PY
```

---

## 5. How to run it

### 5-1. Export before the Space sleeps, import after it wakes

The free tier has **no persistent disk** and **sleeps after 48h of inactivity** (the timer cannot be changed). The index lives in RAM only, so carrying it over is an explicit step:

1. `Backup` tab → tick *include response bodies* → **Export index**
2. Save `index.json` somewhere outside the Space
3. After a sleep or restart: `Backup` tab → upload `index.json` → **Import index**
4. Check `imported / skipped_no_response / expired / rejected_by_safety` in the result

* Vectors are **recomputed** on import, so a changed `VECTOR_DIM` is not a problem.
* The safety screen **runs again** on import (leave `skip_safety` off).
* An export without response bodies is for analysis only — those rows are skipped on import and counted under `skipped_no_response`.

Daily backup from your own machine:

```bash
python3 - <<'PY'
from gradio_client import Client
import datetime
c = Client("<user>/EchoCache")
path, receipt = c.predict(True, "", api_name="/export_index")
dst = "echocache-%s.json" % datetime.date.today()
open(dst, "w").write(open(path).read())
print(dst, receipt)
PY
```

### 5-2. Choose the threshold from a Sweep, not from intuition

* The default `0.92` is **conservative**: it mostly reuses exact matches and surface variations (greetings, honorifics, punctuation, casing).
* Genuine paraphrases with different wording land around **0.80–0.90** with character n-grams.
* Export a day of real traffic as CSV (`tenant_id, prompt, response, route`), run **Sweep**, and read reuse rate, estimated saving, borderline count and guard blocks before deciding.
* Before lowering the threshold, open **Audit** and read why the guard rejected candidates. Many rejections mean "these questions genuinely differ", not "the threshold is too high".
* On the published benchmark, threshold **0.80 with the guard on** scored 99.2% accuracy with 100% reuse recall and 1 wrong reuse in 90 — but that is one synthetic set, not your traffic. Measure yours.

### 5-3. What to watch

| Frequency | Where | What you are deciding |
|---|---|---|
| daily | Dashboard hit rate / saved cost | is reuse growing as expected |
| daily | Audit → guard rejections | rejection patterns; early warning of wrong hits |
| weekly | Dashboard index health (`evicted` / `expired`) | revisit TTL or the entry caps |
| weekly | Sweep on fresh logs | revisit the threshold |
| on change | Health → `mode` | did it silently switch away from `local-only` |

---

## 6. Troubleshooting

### Memory pressure, or the Space restarts

Index memory is roughly `entries × VECTOR_DIM × 4 bytes`, and **`MAX_ENTRIES` is per tenant**. With the defaults, a full tenant is ≈ 80 MB and the whole index is capped at ≈ 330 MB by `MAX_TOTAL_ENTRIES=20000`.

In order of effect (Settings → Variables and secrets):

```
MAX_TOTAL_ENTRIES = 8000      # the global ceiling — start here
MAX_ENTRIES       = 2000      # per tenant
MAX_TEXT_CHARS    = 4000      # per stored prompt / response
VECTOR_DIM        = 2048      # halves memory, costs little accuracy
EVENT_BUFFER      = 2000      # event ring buffer
```

`Health` shows the live `approx_memory_mb` and `vector_matrix_mb_per_full_tenant`.

### A tenant's cache went empty

Reaching `MAX_TENANTS` (default 256) **evicts the least-recently-used tenant's entire partition** on the next new tenant's first write. Deletion only — nothing is ever moved or read across tenants. Size `MAX_TENANTS` above your real tenant count. When the global entry budget is reached instead, entries are evicted one at a time from the largest partition.

### Nothing hits

Work through this in order before touching the threshold:

1. Read `reason` in the Lookup result:
   * `empty_index` — nothing stored for that tenant (a mistyped `tenant_id` is the usual cause)
   * `no_candidates` — no fingerprint-similar entry; the wording differs a lot
   * `below_threshold` — `best_similarity` is in the payload; decide from that number
   * `guard_rejected` — the meaning was judged different; go to step 2
2. Open **Audit** and read the reasons (`numeric_mismatch`, `negation_mismatch`, `proper_noun_mismatch`, `temporal_mismatch`, `question_type_mismatch`). If the reason is sound, that miss is **correct** and lowering the threshold will not fix it.
3. Compare the stored and queried prompts: `cached_prompt_preview` comes back in the response.
4. Still missing legitimate reuse? Step down through Sweep (0.86 → 0.82) and watch `borderline_hits` and `guard_blocks` grow.
5. Only if a specific check genuinely does not fit your domain, disable it individually: `GUARD_DISABLE=proper_noun`. Disabling all of them is not recommended.

### A wrong hit happened (deal with this first)

1. **Raise the threshold** (e.g. `0.92` → `0.95`). Immediate effect.
2. Find the `entry_id` in **Audit**, or in Dashboard → *most reused entries*.
3. Use **Invalidate entry** at the bottom of the Audit tab (`tenant_id` + the first 16 characters of the id). A route name invalidates that route; `*` clears the tenant.
4. If the same shape recurs, add those prompts to a CSV, re-run Sweep, and adopt a threshold that rejects them.
5. If the difference is not one the five checks can see, stop caching that category: give it its own `route` and invalidate that route on a schedule.

### The embedding API returns 402 or 429

* `402 Payment Required` — the free inference credit (~$0.10/month) is exhausted.
* `429 Too Many Requests` — rate limited.

**Neither stops the cache.** The exception is swallowed and matching continues on local vectors alone. After repeated failures a circuit breaker opens for `EMBED_COOLDOWN_SEC` (default 600s) so no further credit is spent. `Health` shows `embedding.last_error` and `cooldown_remaining_sec`. To stop permanently, delete `HF_TOKEN` from Secrets and restart — the header returns to `local-only`. If it does not, check for `HUGGINGFACE_HUB_TOKEN` / `HUGGINGFACEHUB_API_TOKEN`.

### The Space sleeps after 48h and wakes up empty

That is the free tier working as designed (no persistent disk, fixed sleep timer). Use the export/import routine in 5-1. If you need it always on, move to paid hardware or run `python app.py` on your own host — the same file works unchanged.

### `/store` returns `stored: false`

The safety screen refused the content. Read `reason`:

| reason | meaning |
|---|---|
| `credit_card_luhn` | a digit run passed the Luhn checksum; order numbers are not flagged |
| `credential_prefix` | `sk-`, `ghp_`, `AKIA`, `xoxb-`, `AIza`, `hf_`, … |
| `jwt_structure` | a three-part JWT |
| `high_entropy_secret` | a long high-entropy run that is not identifier-shaped |
| `contact_pii_excess` | too many e-mail addresses or phone numbers |
| `private_key_block` | a PEM private-key block |

**This is correct behaviour.** Before relaxing anything, ask whether that content belongs in a cache at all.

### The API path 404s

Gradio changes it between major versions (5 and 6 use `/gradio_api/call/<name>`, 4 uses `/call/<name>`). Do not hardcode it — read it from the **API Docs** tab or the **Use via API** footer link. `gradio_client` resolves it for you.

---

## Appendix: running locally

```bash
pip install -r requirements.txt
python3 app.py          # http://localhost:7860
```

Identical behaviour to the Space. Environment variables work the same way:

```bash
PRICE_IN_PER_1K=0.003 PRICE_OUT_PER_1K=0.015 MAX_ENTRIES=1000 python3 app.py
```