# Unihive recommendation engine: complete report

> **Correction (2026-10-07).** Version 2.0 removes the community name and flair as direct inputs and from the embedding text. It still receives each post's `domain` field, which for Reddit text posts is `self.<subreddit>`: 24 of the 30 domain tokens known to both models are subreddit names. Community information therefore remains indirectly available to the 2.0 models for text posts. A follow-up release removes `domain` entirely. Until then, treat v2.0's community independence as partial.

This report covers everything built, tested and measured for the Unihive post recommender: from the raw Reddit telemetry to the shipped **hybrid engine 2.0** and its backend adapter. It also explains every file of engine code in this branch.

**Contents**
1. Summary
2. Data
3. Enriching the posts: image captions and text embeddings
4. Features and label
5. Phase 1: five RecBole models, ONNX export
6. Phase 2: benchmark on standard metrics
7. LLM tagging with a closed 114-tag taxonomy
8. The first two-stage engine (tag match, then model) and why it failed
9. Every alternative pipeline we tried
10. The hybrid engine: design
11. Building the engine and the training/serving bug we found
12. Version 1.1: gentler age decay and topic-first slots
13. Version 2.0: no community, no flair
14. Backend compatibility
15. Code walkthrough (every file of the engine)
16. How to reproduce
17. Limits and next steps
18. Glossary of metrics

---

## 1. Summary

| Item | Result |
|---|---|
| Data | 4 real users (P02, P06, P09, P25), 5 Reddit sessions, 1,794 post exposures, 1,677 distinct posts, plus 2 later P09 sessions (89 and 142 posts) for out-of-time checks |
| Label | engaged = click / vote / save / join / reply / opened detail, **or** focal dwell ≥ 3 s |
| Shipped engine | **Hybrid engine 2.0** (`models/hybrid_engine/`): v2b for new users, blended toward a warm DeepFM as engaged history grows (w = engaged / (engaged + 10), capped at 0.7), gentle age decay, cold-start push, topic-first ordering (7 of every 10 slots go to topic matches) |
| What 2.0 changed | Community (subreddit) and flair removed as direct inputs and from the text that the content vectors are built from (your instruction). The per-user community history rates stay. **Correction:** the `domain` input still carries subreddit names for text posts (see the note at the top). |
| Verified | ONNX == PyTorch (≤ 2.4e-7); the engine rebuilds the training inputs from raw data (≤ 0.0021); engine scores == offline predictions (≤ 2.5e-4); 9 automated tests, including the backend's real `reco.store` on fakeredis |
| Quality (NDCG@10) | Clearly above random in every setting. Example, validation period whole feeds: 0.65 vs 0.35 random. Below Reddit's own feed order, which knows far more (feed AUC 0.77–0.92 vs 0.60). |
| Backend | `backend_adapter.py` maps Redis post meta, user state, burst-buffer events and `interaction_events` rows to engine inputs. `build_pool_hybrid` is a drop-in for `reco.service.build_pool`; with the same data it returns the same candidates as the original. |
| Biggest open item | The backend stores CLIP-512 vectors only. The engine needs a Qwen3-Embedding-0.6B vector per post (one extra call in the tagging worker). Without it the engine still runs, at about 0.02 lower NDCG@10 on average. |

---

## 2. Data

**Source.** Trumio Trace browser telemetry of real Reddit scrolling: `post_impression`, `attention_update` (visible / focal / idle ms), hovers, clicks, votes, saves, joins, replies and `post_detail_summary` (time on the opened post and its comments).

**Interaction table** (`out/interactions_v2.csv`, built by `build_interactions.py`). One row per (user, session, post exposure), 64 columns:
- **Context:** rank in feed, page kind, view context, recommendation source, time in session.
- **Post:** subreddit, post type, flair, title, body, image caption + OCR, author hash, score, comments, upvote ratio, age, is_ad, NSFW, spoiler, membership, link domain, language, awards.
- **Attention:** visible / focal / active / idle ms, re-exposures, hover ms and count, clicked, opened detail, detail time on body and comments.
- **Actions:** up/down vote, saved, joined, reposted, reply opened, media watch.

| | rows | positives | positive rate |
|---|---|---|---|
| train (first 70% of each user's timeline) | 1,255 | 308 | 24.5% |
| validation (next 15%) | 269 | 65 | 24.2% |
| test (last 15%) | 270 | 44 | 16.3% |

The split is **chronological per user**, so the model never trains on a user's future.

**Out-of-time sessions** (P09, recorded about two days after all training data):
- **Session 1** (`P09_2026-10-07_04-51-26`): 89 posts, 11 engaged.
- **Session 2** (`P09_2026-10-07_05-10-53`): 142 posts, 10 engaged; 59% of posts came from subreddits never seen in training.

Raw session events are **not** pushed (they stay local).

---

## 3. Enriching the posts: image captions and text embeddings

**Image captions.**
- Model: Florence-2 (detailed caption + OCR), run as parallel instances on the GPU (`caption_images.py`).
- Missing images were recovered through pullpush.io (`pullpush_recover.py`) and captioned in later rounds.
- `attach_captions.py` joins the captions to the events. In the end, **391 posts have a caption**.

**Text embeddings.**
1. **MiniLM-L12** (384-d) first, then **Qwen3-Embedding-0.6B** (1024-d, GPU, float16) for all 1,677 posts (`embed_posts_minilm.py`, `embed_posts_qwen3.py`).
2. The first pass missed 137 posts that appear only in the event log. They were added and the embeddings rebuilt.
3. Three vectors per post: text, caption, and **fused** (text + " Image: " + caption). The models use the fused vector.
4. Qwen3 beat MiniLM by about 0.02 AUC, and captions added about 0.013 (both within noise; section 6).

**Versions of the embedding text.**
- **Engines 1.x:** `"title. subreddit. flair. body"` (+ caption).
- **Engine 2.0:** `"title. body"` (+ caption), from `embed_posts_qwen3_nocomm.py` → `out/post_emb_qwen3_0p6b_nocomm.npz`.

**Content components.** PCA reduces the 1024-d vector to **16 components**, each scaled to unit standard deviation (`pca0`–`pca15`). The fitted mean, components and scale are stored in `meta.json`, so the engine computes them for new posts: `pca16(v) = ((v − mean) @ components.T) / component_std`.

---

## 4. Features and label

`prep_recbole.py` turns the interaction table into RecBole files. **Rule: an input may only use information available before the user reacts.** The row's own dwell, hover and clicks define the label and are never inputs.

**Post inputs (known at exposure)**
- `log_score`, `log_comments`, `log_age_h` (missing age → median, 121 h), `title_len`, `body_len` (log of character counts).
- `has_caption`, `is_ad`, `upvote_ratio` (missing → 1.0, plus an `upvote_missing` flag), `log_awards`, `is_spoiler`.
- `pca0`–`pca15`.
- Categories: post type, link domain (top 30, else "other"), language. In 1.x also subreddit and flair (removed in 2.0, section 13).

**Causal history inputs.** These summarise the user's earlier rows whose viewing had already ended (`ts_j + visible_ms_j ≤ ts_i`). R is the last 20 such rows. The constant 3 is a prior that keeps short histories from producing extreme rates.

| Feature | Formula / meaning |
|---|---|
| `h_n` | log(1 + number of past interactions) |
| `h_pos_rate` | (engaged in R + 3·global rate) / (\|R\| + 3) |
| `h_mean_focal` | Σ log(1 + focal s) over R / (\|R\| + 3) |
| `h_hover` | Σ log(1 + hover ms) over R / (\|R\| + 3) |
| `h_click` | (clicks in R + 3·0.08) / (\|R\| + 3) |
| `sub_seen`, `sub_rate` | how often this user saw / engaged with posts from the **same community** before (smoothed toward the user's own rate) |
| `auth_seen`, `auth_rate` | the same for the same author |
| `aff_recent` | cosine between this post's vector and the mean vector of recently engaged posts |
| `aff_neg` | cosine with the mean vector of recently skipped posts |
| `uemb0`–`uemb15` | mean content components of recently engaged posts (the user's recent taste) |

Context features also existed: feed rank, attention focus, session position, session time and feed speed (`h_expo_rate`). The warm model of the engine **does not** use them, because a recommender cannot know them before it ranks (section 9).

**Switches in `prep_recbole.py`** (environment variables). With none set, the output is byte-identical to the original; this was checked with `cmp`.

| Switch | Effect |
|---|---|
| `SUFFIX` | write a variant dataset `reco_ctr<SUFFIX>` instead of the main one |
| `LABEL_MODE` | dwell2000 / dwell3000 (default) / dwell5000 / engaged |
| `EMB_NPZ`, `EMB_KEY` | which embedding file / vector to use |
| `SHUFFLE=1` | shuffle train/valid labels (sanity check: test AUC must fall to ~0.5) |
| `LOUO=Pxx` | leave one user out entirely (new-user test) |
| `MASK_IDS=1` | replace user and post IDs with a constant |
| `HIST_CAP=n` | simulate a user with only the last n interactions |
| `STRICT_PAST=1` | the serving rule: only interactions strictly **before** the request count (section 11) |
| `META_DIR` | where standardisation and PCA are written and read (engine 2.0 uses `package_meta_nc`, so the original `package_meta` is never touched) |

---

## 5. Phase 1: five RecBole models, ONNX export

**Setup.** RecBole 1.2.0 on PyTorch 2.11 with CUDA. It needed compatibility fixes:
- `.agg(len)` → `.map(len)` (pandas), via `apply_recbole_patch.py`;
- NumPy alias shims;
- `torch.load(..., weights_only=False)`;
- missing `pydantic` and `kmeans_pytorch` installed;
- float fields fed as (value, 1) pairs;
- `alias_of_item_id` for sequential models.

DIN and SASRecF crashed inside RecBole 1.2 (embedding index out of range) and were dropped.

**Models and tuned settings** (search on validation, then 5 seeds each):

| Model | Tuned settings | Test AUC (mean of 5 seeds) | Test logloss |
|---|---|---|---|
| LR | weight_decay 0.1 | 0.694 | 0.502 |
| DeepFM | emb 16, dropout 0.3, wd 1e-3 | **0.739** | 1.418 |
| Wide&Deep | emb 16, dropout 0.3, wd 1e-3 | 0.732 | 1.014 |
| DCN V2 | emb 4, dropout 0.3, wd 1e-2 | 0.711 | 0.660 |
| SASRec (next post) | hidden 32, dropout 0.6, wd 1e-2 | Hit@10 0.073 vs random 0.028 | — |

**Export.**
- Every model was exported to ONNX and checked against PyTorch.
- Each export comes with metadata (vocabularies, feature order, standardisation, PCA), an explainer Word document and a runnable example.
- Pushed to the GitHub `models` branch, including the full PyTorch checkpoints and a retraining kit: datasets, tuned settings, `continue_training.py` and the data pipeline.

---

## 6. Phase 2: benchmark on standard metrics

All numbers are on the 270-row test split (44 positives), as the mean over 5 seeds. **The AUC standard error is about 0.046**, so gaps below about 0.13 AUC cannot be detected reliably.

**Headline findings**
1. The models beat chance and popularity: test AUC 0.69–0.74 vs 0.50 random and 0.56 popularity. All four are significantly better than popularity.
2. **Reddit's own feed order is the strongest ranker per user**: NDCG@10 0.66 vs 0.49 for the best model, GAUC 0.82. Engagement is dominated by feed position: rank 0–2 → 44% engaged, rank 21+ → 21%.
3. The deep models were badly calibrated: raw logloss 1.39 (DeepFM) and 1.07 (Wide&Deep), worse than a constant (0.465). **Platt scaling** fitted on validation fixes it: 0.454–0.484 for all four.
4. **History features drive everything.** Removing them takes the average AUC from 0.714 to 0.619. Feed context and embedding features each add about 0.02. **Categories and IDs add about nothing**: permutation importance of the category group is 0.001, and masking IDs slightly raises AUC.

**Per-user ranking (test, macro over 4 users)**

| Method | NDCG@10 | Hit@10 | MRR |
|---|---|---|---|
| DeepFM, 5-seed ensemble | 0.491 | 1.00 | 0.69 |
| DCN V2 | 0.465 | 1.00 | 0.68 |
| LR | 0.451 | 0.75 | 0.64 |
| Reddit feed order | **0.659** | 1.00 | 1.00 |
| Popularity | 0.328 | 1.00 | 0.47 |
| Random | 0.199 | 0.81 | 0.33 |

**Other checks**
- **Leave-one-user-out** (new-user test): LR keeps 0.69 mean AUC; the deep models drop to about 0.65.
- **Shuffled labels** give AUC 0.44–0.53: no leakage.
- **Adversarial validation** separates train from test perfectly, through session-position features that grow along the timeline. Removing them costs only 0.013.
- **Slices:** weakest for user P09 (AUC about 0.58) and r/AskPhysics (about 0.39). Weaker in the second half of the test period (interest drift).
- **Cost:** every model scores a post in about 0.2 ms on CPU.

Details: `benchmark_report/REPORT.md` and its CSV tables.

---

## 7. LLM tagging with a closed 114-tag taxonomy

**Tagger.** Groq `openai/gpt-oss-120b`, temperature 0, JSON output, choosing only from the app's taxonomy: 13 broad interests, 114 child tags (`backend/crud/taxonomy.py`). Two API keys were rotated; they live only in a git-ignored `.env` and were never committed.

**Results**
- 1,677 posts tagged, 1,997 tags, 1.6% invalid tags rejected, mean 1.19 tags per post, mean confidence 0.80.
- **24.4% of posts got no tag** (careers, college and off-topic content the taxonomy does not cover). 75 of 114 tags were ever used.

**Reliability**
- Agreement with gpt-oss-20b: Jaccard 0.51. With qwen3-27b: 0.53. A re-run of the same model at temperature 0 reproduces the exact tag set only 47% of the time (same top tag 71%).
- Broad domains are stable (kappa 0.76). 91% of posts carry their subreddit's majority domain.
- Manual spot check: 36 of 40 acceptable.

---

## 8. The first two-stage engine (tag match, then model) and why it failed

**Design.**
- **Stage 1** scores how close a post's tags are to the user's picked tags, using 8 tag-match features:
  - exact overlap;
  - parent-domain overlap;
  - rarity-weighted overlap;
  - soft similarity through a 114×114 tag-similarity matrix;
  - whether the post's top tag was picked;
  - number of matches;
  - number of tags;
  - mean confidence.

  A logistic regression combines them into **tag closeness**.
- **Stage 2** is a Wide&Deep model on post features + tag-match features: no user ID, no history, no feed position. It works for a brand-new user (test AUC 0.692).
- **Final step:** order by tag closeness, then drop posts with model probability < 0.3 (or blend the two).

**Result: below the model alone.** Test NDCG@10: engine 0.29 and blend 0.37 vs the Stage-2 model alone 0.40. The reason: for these real users, posts matching their inferred interests were engaged with at the **same rate** as non-matching posts (20.2% vs 20.2%). For P09, engagement moved to posts the taxonomy cannot tag (37% engaged among untagged vs 3% among tagged).

**Random-interest simulation.** 2,000 synthetic users per interest type, test-period posts only:
- Coherent interests got lists of median 5 posts; 5.7% got nothing.
- Only 36% of the pool was ever recommended, because untagged posts can never appear.

---

## 9. Every alternative pipeline we tried

All were evaluated with the same protocol: per-user test pools, settings tuned on validation, test scored once, 25-post-page bootstrap. Simulated users drew random interests from the test-period pool.

| Pipeline | Idea | Test NDCG@10 | Verdict |
|---|---|---|---|
| Reddit feed order | reference | 0.659 | needs Reddit's own ranking; not available to us |
| DeepFM warm (history + feed rank) | warm CTR | 0.491 | uses feed rank, which a recommender cannot know before ranking |
| **v2b: model + 0.1 × tag closeness** | model first, small tag bonus | **0.467** | best cold-start option → became the engine's new-user score |
| v5: 10% exploration slots | exploration quota | 0.447 | small cost for variety |
| v5: 20% exploration + light diversity | exploration quota | 0.429 | more variety, ILD 0.85 |
| v2a: Stage-2 model ranks all | model only | 0.402 | |
| Two-tower retrieval + model ranking | Google two-tower | 0.390 | retrieval drops good posts |
| v1 blend (25% tag, 75% model) | first engine | 0.373 | |
| Phoenix ranker, dwell head | X Thunder + Phoenix | 0.365 | |
| Two-tower, topics + history | Google two-tower | 0.359 | |
| v4: interest lane only | multi-lane | 0.356 | |
| Popularity | baseline | 0.328 | |
| Two-tower, topics only | Google two-tower | 0.309 | |
| v1 engine (tag order + filter) | first engine | 0.291 | |
| v2c: topic gate, then model | gate first | 0.282 | |
| v4: four-lane mixer + MMR | multi-lane + diversity | 0.274 | most diverse (ILD 0.91), lowest accuracy |
| Phoenix full | X Thunder + Phoenix | 0.258 | |
| v1 tag closeness only | Stage 1 only | 0.248 | |
| Random | baseline | 0.192 | |
| v3: learned intent embedding | predict the user's next-intent vector | 0.180 | at chance with 4 users of data |

**Hybrid experiments that shaped the final design**
- **v6.** Blending v2b with a warm DeepFM by history depth helped on validation (0.633 vs 0.547 for v2b) but not on test.
- **Fault probe.**
  - Every real user in validation and test has ≥ 61 past interactions, so "new user" behaviour could only be studied by truncating histories (`HIST_CAP`).
  - Per-seed AUC of the warm model varied by up to 0.06.
  - A position-debiased warm model and a "clean" warm model (no IDs, no feed context, no session signals) were both better than v6's.
- **v7.** Tuned warm variants on validation. The clean warm model became the shipped warm model.

---

## 10. The hybrid engine: design

Your directive: v2b for new users; gradually shift toward a warm DeepFM as meaningful history builds up, while keeping some v2b for freshness and exploration; weight decay as posts age; a cold-start push for new posts.

```
v2b    = Platt(cold model probability) + tag_bonus · rank01(tag closeness)            tag_bonus = 0.1
w      = 0                                   for a user with no engaged posts
       = min(w_max, engaged / (engaged + n0)) otherwise                               n0 = 10, w_max = 0.7
blend  = (1 − w) · rank01(v2b) + w · rank01(warm DeepFM probability)
fresh  = floor + (1 − floor) · 0.5 ^ (age_hours / half_life)                          1.1+: half_life 720 h, floor 0.85
push   = beta · [age_hours ≤ 24] / sqrt(1 + impressions)                              beta = 0.1
score  = blend · fresh + push
order  = in every block of 10 positions: the 7 best topic-matching posts, then the 3 best of any kind
```

**Why each piece is there**
- **rank01** (rank inside the request, scaled to (0, 1]) makes the two models' scores comparable, whatever their calibration.
- **w grows with engaged posts**, not all impressions, because the warm model only learns from engagement. It never reaches 1, so v2b always keeps at least 30%.
- **Freshness is multiplicative and gentle.** A week-old post keeps 98%, a month-old post 93%, and no post drops below 85%. The models already see post age, so a strong decay double-counts it (section 12).
- **The push is additive and fades with impressions**, so a new post gets a few showings and then must earn its place.
- **Topic-first** guarantees that the top of the feed reflects the picked interests. A **topic match** means the post shares a picked tag or its broad domain.
- **Exploration is not the engine's job.** The backend's `compose.pick_explore` keeps reserving 15–20% out-of-interest slots.

---

## 11. Building the engine and the training/serving bug we found

**Cold model** (v2b's probability): the Stage-2 Wide&Deep, 5 seeds averaged inside one ONNX graph, with Platt calibration fitted on validation.

**Warm model:** DeepFM without user/post IDs, feed context (rank, recommendation source, page kind, view context, attention focus) or session signals (session position, session time, feed speed). It uses post features + 27 history features, 5 seeds in one graph.

**The bug: a training/serving skew.** The first parity check compared engine-built inputs with the training files. The warm history features differed by up to **6.46 standard deviations**, and the scores by up to 0.9.
- **Cause:** the training data counted impressions logged at the **same timestamp** with zero visibility as "history". A live request can never see those.
- **Fix:** a new dataset switch `STRICT_PAST=1` (only interactions strictly before the request), the same rule in the engine (`ts < now` and viewing ended), and a retrained warm model.
- **Effect:** warm test AUC dropped from 0.727 to 0.671. The lower number is the honest one.
- **After the fix:** features match within 0.003 and scores within 3e-4.

**Engine 1.0 replay (NDCG@10)**

| Setting | Validation, whole pool | Test, whole pool |
|---|---|---|
| Shipped defaults | 0.62 | 0.39 |
| v2b only | 0.51 | 0.48 |
| `w_max` 0.3 | 0.55 | 0.48 |
| No decay / push | 0.66 | 0.40 |

The blend wins on validation and loses on test. With only 4 long-history users, these defaults are a starting point to re-tune on real logs.

---

## 12. Version 1.1: gentler age decay and topic-first slots

Your request: soften the age penalty, and make topic matches find the top posts.
- **Decay:** half-life 168 h → 720 h, floor 0.7 → 0.85.
- **Topic-first:** 7 of every 10 positions go to topic matches.

**Results (NDCG@10, topic share of the top 10)**

| Feed | 1.0 | 1.1 |
|---|---|---|
| New session 1, one feed at session start | 0.25 (10%) | **0.41 (70%)**, engaged posts in the top 10: 3 → 5 |
| New session 1, 25-post pages | 0.52 | **0.56** |
| Validation, whole feeds / pages | 0.65 / 0.60 | **0.69 / 0.64** |
| Test, whole feeds / pages | **0.45 / 0.55** | 0.32 / 0.46 |
| New session 2: AUC / pages | 0.58 / 0.34 | **0.60 / 0.38** (Reddit order AUC 0.77, random 0.50) |

The topic rule helps when engagement follows the picked topics (new session 1, validation). It costs in the test period, where P09 had drifted to untagged posts. You chose to keep 1.1 as it was ("no improvement" round).

---

## 13. Version 2.0: no community, no flair

> **Correction (2026-10-07).** Version 2.0 removes the community name and flair as direct inputs and from the embedding text. It still receives each post's `domain` field, which for Reddit text posts is `self.<subreddit>`: 24 of the 30 domain tokens known to both models are subreddit names. Community information therefore remains indirectly available to the 2.0 models for text posts. A follow-up release removes `domain` entirely. Until then, treat v2.0's community independence as partial.

**What was wrong.** Since the first RecBole dataset, every post metadata column had been an input, including **subreddit** and **flair**. The subreddit name was also inside the embedding text. You had never asked for this, and it cannot transfer: Unihive communities are not Reddit subreddits, so every backend community would be "unknown" to the 1.x models.

**What changed**
1. **Content vectors** re-embedded from `"title. body"` (+ image caption) for all 1,677 posts (`embed_posts_qwen3_nocomm.py`).
2. **New datasets**, built by `prep_recbole.py` with `EMB_NPZ=post_emb_qwen3_0p6b_nocomm.npz META_DIR=package_meta_nc`:
   - `reco_ctr_nc`;
   - `reco_ctr_nc_noid_strict` (also `MASK_IDS=1 STRICT_PAST=1`);
   - `reco_engine_nc`, the cold dataset: tag-match features + post features, tokens post type / domain / language only.

   All originals are untouched (checksums verified).
3. **Both models retrained** (`engine_v2/build_engine_v2.py`), with the same tuned settings, 5 seeds, export and checks. Test AUC:
   - **Cold:** valid 0.777, test **0.602** (1.1: 0.692). Inputs: post type, domain, language + 35 floats.
   - **Warm:** valid 0.713, test **0.565** (1.1: 0.671). Per-seed test AUC 0.55–0.60.
4. **Kept:** `sub_seen` / `sub_rate`, the user's own engagement rate with a community. They compare community ids inside one user's history and need no learned names. On the backend, `community_id` provides them.
5. **Guard test:** `test_no_community_or_flair_in_models` fails if subreddit or flair is a model input, if the embedding text includes them, or if renaming them changes any score.

**Where the AUC drop comes from** (`engine_v2/diag_engine_v2.py`: cold model, same recipe, 5 seeds)

| Variant | Test AUC |
|---|---|
| A: 1.1 setup (community + flair as inputs and in the text) | 0.692 (reproduces 1.1 exactly) |
| B: inputs removed, old text | 0.624 (−0.07) |
| C: inputs kept, new text | 0.653 (−0.04) |
| D: both removed = 2.0 | 0.602 (−0.09) |

On this Reddit trace the community name genuinely predicted engagement. For example, P09 engaged with r/AskPhysics in a way the models picked up.

**Verification of 2.0**
- ONNX == PyTorch: 2.4e-7.
- Feature parity on 539 rows: 0.0021, 0 token mismatches.
- Score parity: cold 1.1e-4, warm 2.5e-4.
- 0.7 ms per candidate.

**Ranking quality under backend conditions** (NDCG@10; `artifacts/backend_modes_report.json`)
- *Full* = every Reddit field, as in training (offline upper bound).
- *Unknown community names* = every community gets a distinct name the models never saw, as Unihive communities will.
- *Backend data* = also no flair, tags without confidences, one dwell value (hover 0, focal = visible), no upvote ratio or awards.
- *No Qwen3* = also no content vectors (today's backend).

| Feed | 1.1 full | 1.1 unknown comm. | 1.1 backend | 1.1 backend, no Qwen3 | **2.0 full** | **2.0 backend** | **2.0 backend, no Qwen3** | random |
|---|---|---|---|---|---|---|---|---|
| Validation, whole pool | 0.685 | 0.673 | 0.574 | 0.625 | 0.650 | 0.579 | 0.604 | 0.350 |
| Validation, 25-post pages | 0.642 | 0.652 | 0.582 | 0.534 | 0.559 | 0.595 | 0.582 | 0.350 |
| Test, whole pool | 0.316 | 0.333 | 0.291 | 0.267 | 0.340 | 0.329 | 0.268 | 0.201 |
| Test, 25-post pages | 0.458 | 0.459 | 0.483 | 0.465 | 0.415 | 0.397 | 0.441 | 0.286 |
| New session 1, start | 0.413 | 0.354 | 0.290 | 0.225 | 0.255 | 0.159 | 0.155 | 0.119 |
| New session 1, pages | 0.559 | 0.487 | 0.396 | 0.359 | 0.468 | 0.486 | 0.362 | 0.282 |
| New session 2, start | 0.110 | 0.095 | 0.139 | 0.139 | 0.110 | 0.110 | 0.110 | 0.079 |
| New session 2, pages | 0.381 | 0.394 | 0.522 | 0.338 | 0.403 | 0.382 | 0.372 | 0.245 |
| **Mean** | 0.446 | 0.431 | **0.410** | **0.369** | 0.400 | **0.380** | **0.362** | 0.239 |

**Reading the table**
- 2.0 gives identical scores with real or unknown values of the `community` field (checked: the "unknown community names" column equals "full" for 2.0, so it is not repeated). That test changed the `community` field only; it did not change the `domain` token, which still carries subreddit names for text posts (see the correction at the top).
- Under backend conditions, 2.0 is 0.03 (with Qwen3) and 0.01 (without) below 1.1. The 1.1 figures are optimistic, because its content vectors still contain the Reddit subreddit name, which Unihive posts will not have.
- Both are well above random everywhere.
- Single feeds (one request) are noisy: a 10-position list with 10–11 engaged posts out of 89–142.

**Re-check on your two later sessions** (`engine_v2/validate_two_sessions.py` → `artifacts/validation_two_sessions.json`; each engine with its own content vectors)

| | Session 1 (89 posts, 11 engaged) | | | Session 2 (142 posts, 10 engaged) | | |
|---|---|---|---|---|---|---|
| | NDCG@10 | engaged in top 10 | AUC | NDCG@10 | engaged in top 10 | AUC |
| Engine 2.0 | 0.255 | 3 | 0.688 | 0.110 | 1 | 0.498 |
| 2.0 without content vectors | 0.249 | 3 | 0.760 | 0.110 | 1 | 0.532 |
| Engine 1.1 | 0.413 | 5 | 0.780 | 0.110 | 1 | 0.602 |
| Reddit feed order | 0.641 | 6 | 0.917 | 0.312 | 3 | 0.773 |
| Random | 0.124 | 1.1 | 0.50 | 0.070 | 0.6 | 0.50 |

25-post pages: 2.0 scored 0.468 and 0.403 (1.1: 0.559 and 0.381). 2.0 lost two engaged posts from the session-1 top 10. Its top two posts there were finance posts promoted by the topic-first rule, because "investing-and-markets" is one of P09's inferred interests.

**New users** (`engine_v2/validate_new_users.py` → `artifacts/new_user_report.json`)

The same real feeds, ranked as if each user had just signed up: picked interests only, no history (the code checks that w = 0). NDCG@10:

| Feed | New user (interests only) | New user, no interests | Same user, full history | Random |
|---|---|---|---|---|
| Validation, whole feeds | 0.525 | 0.564 | 0.650 | 0.350 |
| Validation, 25-post pages | 0.525 | 0.568 | 0.559 | 0.350 |
| Test, whole feeds | 0.410 | 0.303 | 0.340 | 0.201 |
| Test, 25-post pages | 0.426 | 0.460 | 0.415 | 0.286 |
| Session 1, one feed | 0.221 | 0.445 | 0.255 | 0.119 |
| Session 1, pages | 0.341 | 0.527 | 0.468 | 0.282 |
| Session 2, one feed | 0.139 | 0.139 | 0.110 | 0.079 |
| Session 2, pages | 0.359 | 0.377 | 0.403 | 0.245 |
| **Mean** | **0.368** | **0.423** | **0.400** | **0.239** |

**Reading the results**
- A brand-new user gets a useful feed from the first request: about 54% above random.
- History adds on top (0.368 → 0.400).
- A user with no picked interests is still served, ranked on post quality and content.

These users never picked interests themselves; theirs were inferred from behaviour, and some guesses were wrong. That is why "no interests" scored highest. Real signup picks should match better. If they don't, lower `topic_first_slots` for new users (not changed in 2.0).

---

## 14. Backend compatibility

**Checked against the real backend code** (`backend/reco/*`, `backend/api/schemas.py`, migrations 004/005, `db/trumio_schema_dump.sql`):
- **Taxonomy:** the backend's `reco/vocab.py` holds exactly the engine's 13 broad and 114 child slugs (compared slug by slug).
- **Post meta** (`store.posts_meta`): tags (frozenset of slugs, no confidences), cat, created, likes, comments, views, author.
- **Rows:** `posts` (community_id, post_type ∈ TEXT/LINK/MEDIA/NEWS/REPOST/QUESTION/POLL, title, body, url, counters) and `post_reco` (tag_slugs, CLIP-512 `emb_fused`).
- **User state** (`store.get_state`): base_tags (the interests picked at onboarding, broad or child slugs), tag_weights, a 512-d embedding.
- **Events:** the client's IMPRESSION / VIEW / LONG_VIEW / SKIP / CLICK / SHARE / SAVE / HIDE / NOT_INTERESTED. Likes and comments are written to `interaction_events` by `crud/posts.py`. The Redis burst buffer holds `{type, post_id, dwell_ms, t, surface, context}`.

**Mapping (`backend_adapter.py`)**

| Engine input | From the backend |
|---|---|
| `interests` | `state["base_tags"]`; a broad pick ("ai-ml") expands to all of its child tags |
| `score`, `comments`, `impressions` | likes, comments, views |
| `age_hours` | (now − created) / 3600 |
| `tags` | post_reco slugs, confidence 1.0 |
| `post_type` | TEXT/QUESTION/POLL → text, LINK/NEWS → link, REPOST → crosspost, MEDIA → image (or video, from the media kind) |
| `domain` | parsed from the post url |
| `embedding` | Qwen3 vector under `reco:post:{id}:q3` (new). Missing → neutral content |
| `community` | `posts.community_id`, only for the user's per-community history rates |
| history | burst buffer (read without draining) + `interaction_events` (with the post's community and author). Events on one post within 30 min = one interaction. engaged = LONG_VIEW / CLICK / SAVE / SHARE / LIKE / VOTE / COMMENT / REPLY / JOIN / POLL_VOTE or dwell ≥ 3 s, unless SKIP / HIDE / NOT_INTERESTED. `ts_ms` = received time − dwell |

**`build_pool_hybrid`** keeps the backend's retrieval exactly: `graph.expand_weights`, the tag indexes, `seen_among`, the own-post filter and `compose.split_counts`. Only the scoring is replaced. It writes the pool with `store.set_pool` and `store.set_level`, and returns `[(post_id, score, "cat|tags")]`, the same shape as `build_pool`.

**Tests** (`tests/test_backend_adapter.py`, 7 tests)
- Interests, post mapping and event→history mapping, including the 30-min visit merge, NOT_INTERESTED overriding dwell, the shown-time correction and future events being dropped.
- `rank_pool` with backend shapes:
  - a new user with no Qwen3 vectors gets pure v2b and topic-first;
  - CLIP vectors are refused;
  - with Qwen3 + history the warm weight is exactly 0.6 at 15 engaged posts.
- Qwen3 codec and embedding text.
- Postgres + buffer merge, with a fake connection that returns `HISTORY_SQL`-shaped rows.
- **The backend's real `reco.store` on fakeredis** (`index_post`, `save_state`, `buf_push`, `posts_meta`, `set_pool`, `pool_all`). It also checks that `build_pool_hybrid` returns the same candidates as the backend's own `service.build_pool`.

All pass. **Not tested:** `HISTORY_SQL` against a live Postgres (Docker was not running). It was checked against the schema dump.

**Integration steps:** see `models/hybrid_engine/INTEGRATION.md`.
1. Store a Qwen3 vector in the tagging worker.
2. Delegate `build_pool` to `build_pool_hybrid`.
3. Add `onnxruntime` to the requirements.
4. Shadow mode → replay → A/B test.

---

## 15. Code walkthrough (every file of the engine)

### `models/hybrid_engine/` (the shipped package)

**`__init__.py`**: exports `HybridRecommender`, `FeatureBuilder`, `rank01`.

**`features.py`**: rebuilds the exact training inputs from raw data.
- `clean(s)` collapses whitespace.
- `token(v)` maps a category value to its training token: 'none' for missing or empty, whitespace → '_'.
- `unit(v)` scales a vector to unit length.
- `EMB_DIM = 1024`; `content_vector(emb)` returns a unit vector, or **None** when the embedding is missing, the wrong size (e.g. CLIP-512), all zeros or not finite.
- `rank01(x)` gives ranks inside one request: lowest → 1/n, highest → 1.
- `FeatureBuilder(meta)` loads the constants, PCA, taxonomy, tag similarity, Stage-1 weights and IDF from `meta.json`.
  - `pca16(emb)`: the 16 content components.
  - `z(name, raw)`: standardise with the training mean/std.
  - `post_floats(post)`: the 11 post statistics + 16 components (all zeros when the content is unknown). Missing age → median; missing upvote ratio → 1.0 + flag.
  - `post_tokens(post)`: category tokens. User/post IDs are always 'unk'; the domain falls back to 'other' outside the top 30; membership → yes/no/unk. The models read only the tokens listed in their `meta.json` (2.0: post_type, domain, post_language, + member for warm).
  - `tag_vector(tags)`: `{slug: confidence}` or a list → a 114-d confidence vector (unknown slugs ignored).
  - `interest_profile(picks)`: q (picked tags), r (their broad domains), ms (best similarity of each tag to any pick).
  - `tag_match(P, q, r, ms)`: the 8 tag-match features (section 8).
  - `tag_closeness(tm)`: Stage-1 logistic score.
  - `history_floats(history, post, now_ms, post_X)`: the causal history features of section 4. J = interactions with `ts < now` whose viewing ended by now; R = the last 20. Content features use only past posts that have a vector. Returns the features and the number of engaged posts in J (which drives w).

**`engine.py`**: `HybridRecommender`.
- `__init__(artifacts_dir, **overrides)` loads `meta.json` and two single-threaded CPU ONNX sessions. Overrides replace the defaults in `meta["defaults"]`.
- `_tokens` / `_floats` order a feature dict into the model's input vector. Unknown tokens → index 0.
- `_prep_history` sorts the history by time, checks each vector with `content_vector`, and precomputes `pca16`.
- `warm_weight(engaged)` returns 0 for no engagement, else `min(w_max, e / (e + n0))`.
- `score(user, candidates, now_ms)`: one pass over the candidates.
  1. Content vector, post floats, tokens.
  2. Tag match → tag closeness and the topic-match flag (shares a picked tag or its broad domain).
  3. Cold inputs = standardised post floats + standardised tag-match features.
  4. History features + engaged count → warm inputs.
  5. Run the cold ONNX → Platt → `v2b = p_cal + tag_bonus · rank01(closeness)`.
  6. Run the warm ONNX only if some w > 0.
  7. Blend the ranks, multiply by freshness, add the push.

  `now_ms` may be a list (one time per candidate) for replays. Returns one dict per candidate with every component.
- `order(scored)`: topic-first. While posts remain: take the best `topic_first_slots` topic matches, fill the block of `topic_block` with the best remaining posts, repeat. Too few matches → the slots fall back to any post. Sets `position`.
- `rank` = `order(score(...))`; `recommend` = the top k of `rank`.

**`backend_adapter.py`**: the bridge to the Unihive backend (section 14).
- Event vocabularies: `ENGAGED_EVENTS`, `NEGATIVE_EVENTS`, `CLICK_EVENTS`. Constants: 3 s engagement threshold, 30-min visit gap, 2-min dwell cap.
- `POST_TYPE_MAP` and `MEDIA_KIND_MAP` translate backend post types to the trained ones.
- `q3_key`, `encode_q3`, `decode_q3` store a unit Qwen3 vector as float16 in Redis. A wrong size is refused.
- `qwen3_text(title, body, caption)`: the exact embedding text of 2.0.
- `expand_interests(slugs, taxonomy)`: broad picks → children; unknown slugs dropped; order kept.
- `post_from_meta(pid, meta, now_s, emb_q3, extra)` and `post_from_row(row, now_s, emb_q3)`: backend post → engine candidate.
- `_norm_event` accepts buffer dicts and `interaction_events` rows.
- `history_from_events(events, posts, now_s)`: merges visits, decides engaged/clicked, corrects the shown time, fills community/author/vector.
- `user_from_backend(state, history, taxonomy)`.
- `HISTORY_SQL` reads `interaction_events` joined with `posts` (newest first, limited).
- `fetch_q3(r, pids)` (one MGET), `peek_buffer(r, uid)` (LRANGE, no drain).
- `load_history(conn, r, uid, buffered, limit, now_s)`: Postgres rows + buffer → history with vectors.
- `build_pool_hybrid(r, uid, state, level, *, engine, conn, ...)`: the drop-in `build_pool`.
- `rank_pool(engine, state, meta, history, now_s, q3, extra, size)`: returns the pool scores (strictly decreasing, engine order) and the explanation dicts.

**`example.py`**: a demo: the same candidates for a brand-new user and after 30 engaged posts.

**`tests/test_engine.py`**
- `test_smoke`:
  - a new user gets pure v2b;
  - the weight grows (0.091 / 0.333 / 0.667 / 0.7 at 1 / 5 / 20 / 100 engaged posts) and is capped;
  - future events are ignored;
  - freshness is monotone and the push applies only to fresh posts;
  - topic-first fills the first 7 slots, `slots=0` gives plain score order, and few matches fall back gracefully.
- `test_no_community_or_flair_in_models`: the 2.0 guard (section 13).

**`tests/test_backend_adapter.py`**: section 14.

**`requirements.txt`**: numpy, onnxruntime.

**`artifacts/`**: section 13 and the package README.

### `models/training/engine_v2/` (how the engine was built and checked)
- **`evalkit.py`**: shared protocol.
  - Loads the frozen v1 rows (user, split, label, inferred interest picks Q, post tag vectors P) and the offline predictions.
  - Platt on validation only.
  - `user_metrics` (NDCG@10, Hit@10, MAP@10, AUC, topic match per user), `pooled_auc`, `calib_report`.
- **`engine_lib.py`**: taxonomy, tag embeddings and similarity, interest profiles (picked = 1.0, implied parent = 0.6), the 8 tag-match features, the hand-set Stage-1 rule.
- **`build_hybrid_engine.py`**: engine 1.0 build.
  - `Ens` averages the seeds' sigmoid outputs inside one module.
  - `export` writes ONNX (opset 17, dynamic batch) with the input order and vocabularies.
  - `check` compares ONNX with PyTorch on the validation and test files.
  - Writes `meta.json` and `build_report.json`.
- **`retrain_warm_strict.py`**: retrains the 1.x warm model on `reco_ctr_noid_strict` (the serving rule) and updates the artifacts.
- **`build_engine_v2.py`**: engine 2.0 build (section 13). It reuses the 1.0 helpers and asserts that subreddit/flair are not inputs.
- **`diag_engine_v2.py`**: the four-variant diagnosis of the 2.0 AUC drop.
- **`verify_hybrid_engine.py`**: runs the engine's real code path on raw data:
  1. feature parity on every valid/test row;
  2. score parity against the offline ensembles;
  3. a replay (whole pools and 25-post pages);
  4. a cold-start vs long-history example.

  It detects the engine version from `meta.json` and uses the matching embeddings, datasets and offline scores.
- **`sweep_hybrid_engine.py`**: knob sensitivity (`w_max`, decay/push) on validation and test.
- **`validate_v11.py`**: 1.0 vs 1.1 variants on new session 1, test and validation.
- **`validate_new_session.py`** and **`validate_session.py`**: rebuild per-post outcomes from raw Trace events with the training rules, tag new posts with Groq (cached), embed them with the version-matching text (cached), and rank as live requests with P09's earlier history.
- **`validate_backend_modes.py`**: section 13's backend-conditions table, run once per engine version.

### Data pipeline files (`models/training/`)
- **`prep_recbole.py`**: section 4 (features, split, switches).
- **`embed_posts_qwen3.py`** / **`embed_posts_qwen3_nocomm.py`**: the 1.x / 2.0 content vectors.
- **`run_models.py`**, **`tune_and_final.py`**, **`continue_training.py`**, **`export_onnx.py`**, **`apply_recbole_patch.py`**: phase 1 training and export.
- **`data_pipeline/`**: captions, pullpush recovery, interaction table.

---

## 16. How to reproduce

```bash
# engine tests (any machine): numpy + onnxruntime (+ redis, fakeredis for the backend store test)
cd models && python -m hybrid_engine.tests.test_engine && python -m hybrid_engine.tests.test_backend_adapter
UNIHIVE_BACKEND=/path/to/backend python -m hybrid_engine.tests.test_backend_adapter     # include the backend's real reco.store

# rebuild engine 2.0 from the data (GPU recommended; RecBole env from training/RETRAIN.md)
python embed_posts_qwen3_nocomm.py
EMB_NPZ=post_emb_qwen3_0p6b_nocomm.npz SUFFIX=_nc META_DIR=package_meta_nc python prep_recbole.py
EMB_NPZ=post_emb_qwen3_0p6b_nocomm.npz SUFFIX=_nc_noid_strict META_DIR=package_meta_nc MASK_IDS=1 STRICT_PAST=1 python prep_recbole.py
python engine_v2/build_engine_v2.py          # writes hybrid_engine/artifacts_v2 -> install as hybrid_engine/artifacts
python engine_v2/verify_hybrid_engine.py     # parity must stay near zero
python engine_v2/validate_backend_modes.py   # ranking quality under backend conditions
```
`build_engine_v2.py` needs the frozen v1 outputs (`engine_v1_frozen/out_engine`) and the `reco_engine` dataset from the earlier build. Both are in the local `reco_data` workspace; the main datasets are in `models/training/`.

---

## 17. Limits and next steps

**Limits**
- **Tiny data.** 4 users, 44 positive test examples: an AUC standard error of about 0.05. Most differences between variants are within noise. Treat all defaults as starting points.
- **Reddit is not Unihive.** Post types, popularity scales and the engagement mix will differ. Retraining on Unihive logs is the most important next step, and the pipeline supports it.
- **Feed position dominated engagement in the trace.** Reddit's own order (which knows position and much more) stays far ahead.
- **Not yet available in the backend:** Qwen3 vectors, tag confidences, and hover / focal dwell. The cost of each is measured in section 13.

**Next steps**
1. Add the Qwen3 vector to the tagging worker (INTEGRATION step 1).
2. Shadow-mode logging of engine vs current scores.
3. Collect 2–4 weeks of logs, then re-tune `w_max`, `n0` and decay/push, and retrain both models on Unihive data.
4. Keep exploration in `compose` at 15–20%.
5. Add tag confidences to `post_reco` (a small gain).

---

## 18. Glossary of metrics
- **AUC:** probability that a random engaged post is scored above a random non-engaged one. 0.5 = chance.
- **GAUC:** AUC per user, averaged.
- **NDCG@10:** how close the top 10 is to the best possible ordering of the engaged posts. 1 = all engaged posts first, with position-discounted credit.
- **Hit@10:** at least one engaged post in the top 10.
- **MRR:** 1 / (position of the first engaged post).
- **Logloss / Brier / ECE:** how good the probabilities themselves are (lower is better). ECE = calibration error.
- **Platt scaling:** a one-feature logistic regression on the model's logit, fitted on validation, that repairs over-confidence.
- **Parity:** the largest difference between two computations that should be identical (engine vs training file, ONNX vs PyTorch).
- **ILD:** intra-list diversity, the mean pairwise distance of the posts in a list.