docs: log the 2026-09-30 multi-interest and starter-feed decisions

Doc 06 changelog entry for the four approved changes, with the replay
measurements behind them; CLAUDE.md sections 3.3, 3.5 and the quick
reference, and the CURRENT-STATE contract rows, updated to match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CLAUDE.md

@@ -106,7 +106,8 @@ If you find `alpha_long = 0.10` anywhere in code or config, it is a bug from doc
 - Algorithm: **Ward hierarchical agglomerative** via `scipy.cluster.hierarchy.ward`.
 - Code lives in: `app/recommend/clustering.py`.
 - **L2-normalize embeddings BEFORE Ward, then use Euclidean distance.** Cosine Ward via sklearn is mathematically not Ward (Murtagh and Legendre 2014). L2-norm + Euclidean is monotonically equivalent to cosine and gives the intended behavior. This normalization is already in the code.
-- **No fixed K.** Cut the dendrogram by adaptive gap-based threshold (see `_adaptive_threshold()`). Cap at `K_max = 7` (currently; doc 06 says `K_max = 20` for heavy users — raise this when users exist).
+- **No fixed K.** Cut the dendrogram by adaptive gap-based threshold (see `_adaptive_threshold()`). Cap at `K_max = 7` (currently; doc 06 says `K_max = 20` for heavy users — raise this when users exist) and at ceil(N/2) (`AVG_CLUSTER_SIZE_FLOOR = 2`). Take the finest cut within the cap that leaves no singleton; never fold a split pair into another cluster (doc 06, 2026-09-30).
+- **From 2 saves, not 5.** Below `MIN_PAPERS_FOR_CLUSTERING = 5` each save is its own interest (`per_save_clusters`) on the same quota/MMR path. Do not average unrelated early saves into one vector; Tier 2 EWMA is only the fallback.
 - **Medoid, not centroid.** Medoid = arg min over cluster members of sum of squared distances. Cache medoid paper IDs. This is implemented in `_find_medoid()`.
 - **Hungarian-match cluster IDs across reclusterings** — implemented via `stabilize_cluster_ids`.
 - Recompute on each feed request currently (not nightly batch — no batch job infrastructure yet).
@@ -144,7 +145,7 @@ If you find `alpha_long = 0.10` anywhere in code or config, it is a bug from doc
   took a correct 61/39 pool and selected 39/1, dumping 60 minority papers into the
   exploration pool where no later quota stage could recover them. See doc 06,
   2026-08-21.
-- Exploration injection: 2 serendipitous papers per feed. Code in `app/recommend/diversity.py` via `inject_exploration()`.
+- Exploration: 2 serendipitous papers per page, drawn in `_build_page()`. The pool is starter papers from ticked interests that no cluster medoid covers (`_uncovered_interest_papers`), otherwise the far half of the reader's own candidates. Draws stay uniform so propensities hold.
 - Quota (3.1) handles cross-cluster diversity. MMR handles within-quota redundancy.
   This is a statement about WHERE each runs, not just what it is for.
 - Do NOT use DPPs in v1.
@@ -479,7 +480,7 @@ If a topic is too large for a 06 changelog entry, create `docs/research/07-[topi
 | Reranker? | Personalized heuristic by default; optional LightGBM/auto. |
 | Latency budget? | <30ms end-to-end (compute only; metadata I/O excluded). |
 | Cold start? | Hybrid: categories + seed papers + popularity fallback (Phase 5 complete). Author import removed from current roadmap. |
-| When does behavioral take over? | ~10 saved papers. Currently activates at 5 (clustering) / 3 (EWMA) / 1 (BEST_SCORE). |
+| When does behavioral take over? | ~10 saved papers. Currently: 1 save = similar-to, 2–4 = one interest per save, 5+ = Ward clustering; EWMA single-vector is a fallback. |
 | When to add CF? | 500+ users (Phase 9). |
 | Current status? | Local implementation and validation are distinct from deployed status. See `docs/CURRENT-STATE.md`. |
 | ArXiv ID type? | String. Always. `dtype=str` in pandas. |

docs/CURRENT-STATE.md

@@ -1,6 +1,6 @@
 # Current technical contract
 
-Updated: 2026-09-28. Deployed to the Hugging Face Space at `67cba21` on
+Updated: 2026-09-30 (feed changes not yet deployed). Deployed to the Hugging Face Space at `67cba21` on
 2026-09-28; the live checks are in DISCOVERY-PLAN.md (deployment record).
 Re-verify with `/healthz/*` rather than trusting this date.
 
@@ -19,10 +19,11 @@ and deployed. Never copy old corpus counts or benchmark results into current cla
 | Search | Original + optional rewritten query; dense shard fanout and independent FTS5; Zilliz without FTS; RRF; optional top-10 MiniLM; title/citation boost | `app/hybrid_search_svc.py`, `app/config.py` |
 | Missing encodings | Available FTS query forms still run; actual contributing arms determine `retrieval_mode` | `app/hybrid_search_svc.py` |
 | Search total miss | arXiv keyword fallback; no invented hits for unfamiliar concepts | `app/routers/search.py` |
-| Detailed interests | Saved papers -> long/short/negative EWMA + Ward medoids; parameters unchanged | `app/recommend/profiles.py`, `clustering.py` |
+| Detailed interests | Saved papers -> long/short/negative EWMA. 2–4 saves: one interest per save; 5+: Ward medoids, K ≤ min(7, ceil(N/2)), finest singleton-free cut (2026-09-30) | `app/recommend/profiles.py`, `clustering.py` |
 | Multi-interest balance | Importance-weighted quota, within-cluster MMR, quota-bound served order | `fusion.py`, `app/routers/recommendations.py` |
 | Scoring | Personalized heuristic default; optional citation-trained LightGBM | `reranker.py`, `config.py` |
-| Cold start | Category popularity candidates; local per-category interleaving when sidecar exists; epsilon-greedy fresh block and deterministic recycled block | `app/discovery_svc.py`, recommendation router |
+| Cold start | Balanced per ticked interest; citations per month over 24 months, every third slot from the corpus's last 3 months (needs refreshed citations to rank those); epsilon-greedy fresh block and deterministic recycled block | `app/discovery_svc.py`, `local_meta.py`, recommendation router |
+| Exploration | 2 per page; from ticked interests no medoid covers, else the far half of own candidates | recommendation router |
 | Refresh | Personalized candidate retrieval excludes last seven days of served papers; one exhaustion retry allows labelled repeats | recommendation router, `app/db.py` |
 | Paging | Cached ordered pool per query ID, bound to user; per-user lock includes serving/impression write in one process | recommendation router |
 | Preferences | Editable `/interests`; zero to eight known categories; no deletion of existing saves | `app/routers/onboarding.py` |

docs/research/06-Deep-Research-Verdict.md

@@ -450,3 +450,33 @@ needed before HF candidates can enter the live feed. Bound retries, invalidate
 changed-text embeddings, and reject stale/future candidates. Report source failures.
 **Action items:** Implemented locally with pipeline regressions. Deploy a durable
 worker and establish real-model/relevance evidence before an opt-in serving pilot.
+
+### 2026-09-30 — Multi-interest from two saves, finer interest cap, grounded exploration
+**Decision:** The user approved four changes after a trace of their own interests
+(NLP, CV, ML, AI, Robotics, Software Engineering) with production vectors:
+1. The multi-interest path (quota + per-cluster MMR) starts at 2 saves. Below 5,
+   each save is its own interest (`per_save_clusters`); Ward still starts at 5.
+2. `AVG_CLUSTER_SIZE_FLOOR` 4 → 2, so K ≤ ceil(N/2) (still ≤ `K_max = 7`), and
+   `compute_clusters` takes the finest dendrogram cut at or below K that leaves
+   no singleton before the existing singleton merge.
+3. Exploration draws from ticked interests that no cluster medoid lists in its
+   categories. Otherwise it keeps the far-half-of-own-candidates pool.
+4. Tier 0 balances per ticked interest instead of per arXiv code. It ranks by
+   citations per month since publication and gives every third slot per
+   interest to a paper from the corpus's last three months.
+**Supersedes:** Tier 2 EWMA single-vector search as the 3–4-save tier (it remains
+the fallback); the soft cap of ceil(N/4); exploration drawn only from the reader's
+own retrieval neighbourhoods; Tier 0 per-code round-robin by raw citations.
+**Rationale:** Measured: averaging Llama 3 + π0 + OpenHands returned generic
+"LLM agents" surveys matching none of them. Seven saves across five interests
+allowed only 2 clusters, and 12 across six allowed only 3 (5 at the new floor),
+while single-interest readers are unchanged. Lowering the floor alone collapsed
+a noisy 3 + 2 reader into one cluster (the singleton merge folded the split
+pair into the majority), which the singleton-free cut fixes. Tier 0 held 0 of
+200 papers from 2026 and gave Computer Vision 22 of 200. Replay method and
+figures are in the 2026-09-30 feed trace. There is no judged relevance set yet,
+so these rest on the replay and on the invariants, not on an offline metric.
+**Action items:** Implemented with regression tests. Refresh citation counts
+(zero for every paper since 2025-06) and ingest papers after 2026-07-30, or
+Tier 0's fresh lane has no quality signal. Build a small judged set before
+tuning further; multi-medoid retrieval for loose clusters is deferred until then.