REPORT.md

Unihive recommendation engine: complete report

Correction (2026-10-07). Version 2.0 removes the community name and flair as direct inputs and from the embedding text. It still receives each post's domain field, which for Reddit text posts is self.<subreddit>: 24 of the 30 domain tokens known to both models are subreddit names. Community information therefore remains indirectly available to the 2.0 models for text posts. A follow-up release removes domain entirely. Until then, treat v2.0's community independence as partial.

This report covers everything built, tested and measured for the Unihive post recommender: from the raw Reddit telemetry to the shipped hybrid engine 2.0 and its backend adapter. It also explains every file of engine code in this branch.

Contents

1. Summary
2. Data
3. Enriching the posts: image captions and text embeddings
4. Features and label
5. Phase 1: five RecBole models, ONNX export
6. Phase 2: benchmark on standard metrics
7. LLM tagging with a closed 114-tag taxonomy
8. The first two-stage engine (tag match, then model) and why it failed
9. Every alternative pipeline we tried
10. The hybrid engine: design
11. Building the engine and the training/serving bug we found
12. Version 1.1: gentler age decay and topic-first slots
13. Version 2.0: no community, no flair
14. Backend compatibility
15. Code walkthrough (every file of the engine)
16. How to reproduce
17. Limits and next steps
18. Glossary of metrics

---

1. Summary

Item	Result
Data	4 real users (P02, P06, P09, P25), 5 Reddit sessions, 1,794 post exposures, 1,677 distinct posts, plus 2 later P09 sessions (89 and 142 posts) for out-of-time checks
Label	engaged = click / vote / save / join / reply / opened detail, or focal dwell ≥ 3 s
Shipped engine	Hybrid engine 2.0 (models/hybrid_engine/): v2b for new users, blended toward a warm DeepFM as engaged history grows (w = engaged / (engaged + 10), capped at 0.7), gentle age decay, cold-start push, topic-first ordering (7 of every 10 slots go to topic matches)
What 2.0 changed	Community (subreddit) and flair removed as direct inputs and from the text that the content vectors are built from (your instruction). The per-user community history rates stay. Correction: the domain input still carries subreddit names for text posts (see the note at the top).
Verified	ONNX == PyTorch (≤ 2.4e-7); the engine rebuilds the training inputs from raw data (≤ 0.0021); engine scores == offline predictions (≤ 2.5e-4); 9 automated tests, including the backend's real reco.store on fakeredis
Quality (NDCG@10)	Clearly above random in every setting. Example, validation period whole feeds: 0.65 vs 0.35 random. Below Reddit's own feed order, which knows far more (feed AUC 0.77–0.92 vs 0.60).
Backend	backend_adapter.py maps Redis post meta, user state, burst-buffer events and interaction_events rows to engine inputs. build_pool_hybrid is a drop-in for reco.service.build_pool; with the same data it returns the same candidates as the original.
Biggest open item	The backend stores CLIP-512 vectors only. The engine needs a Qwen3-Embedding-0.6B vector per post (one extra call in the tagging worker). Without it the engine still runs, at about 0.02 lower NDCG@10 on average.

---

2. Data

Source. Trumio Trace browser telemetry of real Reddit scrolling: post_impression, attention_update (visible / focal / idle ms), hovers, clicks, votes, saves, joins, replies and post_detail_summary (time on the opened post and its comments).

Interaction table (out/interactions_v2.csv, built by build_interactions.py). One row per (user, session, post exposure), 64 columns:

• Context: rank in feed, page kind, view context, recommendation source, time in session.
• Post: subreddit, post type, flair, title, body, image caption + OCR, author hash, score, comments, upvote ratio, age, is_ad, NSFW, spoiler, membership, link domain, language, awards.
• Attention: visible / focal / active / idle ms, re-exposures, hover ms and count, clicked, opened detail, detail time on body and comments.
• Actions: up/down vote, saved, joined, reposted, reply opened, media watch.

	rows	positives	positive rate
train (first 70% of each user's timeline)	1,255	308	24.5%
validation (next 15%)	269	65	24.2%
test (last 15%)	270	44	16.3%

The split is chronological per user, so the model never trains on a user's future.

Out-of-time sessions (P09, recorded about two days after all training data):

• Session 1 (P09_2026-10-07_04-51-26): 89 posts, 11 engaged.
• Session 2 (P09_2026-10-07_05-10-53): 142 posts, 10 engaged; 59% of posts came from subreddits never seen in training.

Raw session events are not pushed (they stay local).

---

3. Enriching the posts: image captions and text embeddings

Image captions.

• Model: Florence-2 (detailed caption + OCR), run as parallel instances on the GPU (caption_images.py).
• Missing images were recovered through pullpush.io (pullpush_recover.py) and captioned in later rounds.
• attach_captions.py joins the captions to the events. In the end, 391 posts have a caption.

Text embeddings.

1. MiniLM-L12 (384-d) first, then Qwen3-Embedding-0.6B (1024-d, GPU, float16) for all 1,677 posts (embed_posts_minilm.py, embed_posts_qwen3.py).
2. The first pass missed 137 posts that appear only in the event log. They were added and the embeddings rebuilt.
3. Three vectors per post: text, caption, and fused (text + " Image: " + caption). The models use the fused vector.
4. Qwen3 beat MiniLM by about 0.02 AUC, and captions added about 0.013 (both within noise; section 6).

Versions of the embedding text.

• Engines 1.x: "title. subreddit. flair. body" (+ caption).
• Engine 2.0: "title. body" (+ caption), from embed_posts_qwen3_nocomm.py → out/post_emb_qwen3_0p6b_nocomm.npz.

Content components. PCA reduces the 1024-d vector to 16 components, each scaled to unit standard deviation (pca0–pca15). The fitted mean, components and scale are stored in meta.json, so the engine computes them for new posts: pca16(v) = ((v − mean) @ components.T) / component_std.

---

4. Features and label

prep_recbole.py turns the interaction table into RecBole files. Rule: an input may only use information available before the user reacts. The row's own dwell, hover and clicks define the label and are never inputs.

Post inputs (known at exposure)

• log_score, log_comments, log_age_h (missing age → median, 121 h), title_len, body_len (log of character counts).
• has_caption, is_ad, upvote_ratio (missing → 1.0, plus an upvote_missing flag), log_awards, is_spoiler.
• pca0–pca15.
• Categories: post type, link domain (top 30, else "other"), language. In 1.x also subreddit and flair (removed in 2.0, section 13).

Causal history inputs. These summarise the user's earlier rows whose viewing had already ended (ts_j + visible_ms_j ≤ ts_i). R is the last 20 such rows. The constant 3 is a prior that keeps short histories from producing extreme rates.

Feature	Formula / meaning
h_n	log(1 + number of past interactions)
h_pos_rate	(engaged in R + 3·global rate) / (|R| + 3)
h_mean_focal	Σ log(1 + focal s) over R / (|R| + 3)
h_hover	Σ log(1 + hover ms) over R / (|R| + 3)
h_click	(clicks in R + 3·0.08) / (|R| + 3)
sub_seen, sub_rate	how often this user saw / engaged with posts from the same community before (smoothed toward the user's own rate)
auth_seen, auth_rate	the same for the same author
aff_recent	cosine between this post's vector and the mean vector of recently engaged posts
aff_neg	cosine with the mean vector of recently skipped posts
uemb0–uemb15	mean content components of recently engaged posts (the user's recent taste)

Context features also existed: feed rank, attention focus, session position, session time and feed speed (h_expo_rate). The warm model of the engine does not use them, because a recommender cannot know them before it ranks (section 9).

Switches in prep_recbole.py (environment variables). With none set, the output is byte-identical to the original; this was checked with cmp.

Switch	Effect
SUFFIX	write a variant dataset reco_ctr<SUFFIX> instead of the main one
LABEL_MODE	dwell2000 / dwell3000 (default) / dwell5000 / engaged
EMB_NPZ, EMB_KEY	which embedding file / vector to use
SHUFFLE=1	shuffle train/valid labels (sanity check: test AUC must fall to ~0.5)
LOUO=Pxx	leave one user out entirely (new-user test)
MASK_IDS=1	replace user and post IDs with a constant
HIST_CAP=n	simulate a user with only the last n interactions
STRICT_PAST=1	the serving rule: only interactions strictly before the request count (section 11)
META_DIR	where standardisation and PCA are written and read (engine 2.0 uses package_meta_nc, so the original package_meta is never touched)

---

5. Phase 1: five RecBole models, ONNX export

Setup. RecBole 1.2.0 on PyTorch 2.11 with CUDA. It needed compatibility fixes:

• .agg(len) → .map(len) (pandas), via apply_recbole_patch.py;
• NumPy alias shims;
• torch.load(..., weights_only=False);
• missing pydantic and kmeans_pytorch installed;
• float fields fed as (value, 1) pairs;
• alias_of_item_id for sequential models.

DIN and SASRecF crashed inside RecBole 1.2 (embedding index out of range) and were dropped.

Models and tuned settings (search on validation, then 5 seeds each):

Model	Tuned settings	Test AUC (mean of 5 seeds)	Test logloss
LR	weight_decay 0.1	0.694	0.502
DeepFM	emb 16, dropout 0.3, wd 1e-3	0.739	1.418
Wide&Deep	emb 16, dropout 0.3, wd 1e-3	0.732	1.014
DCN V2	emb 4, dropout 0.3, wd 1e-2	0.711	0.660
SASRec (next post)	hidden 32, dropout 0.6, wd 1e-2	Hit@10 0.073 vs random 0.028	—

Export.

• Every model was exported to ONNX and checked against PyTorch.
• Each export comes with metadata (vocabularies, feature order, standardisation, PCA), an explainer Word document and a runnable example.
• Pushed to the GitHub models branch, including the full PyTorch checkpoints and a retraining kit: datasets, tuned settings, continue_training.py and the data pipeline.

---

6. Phase 2: benchmark on standard metrics

All numbers are on the 270-row test split (44 positives), as the mean over 5 seeds. The AUC standard error is about 0.046, so gaps below about 0.13 AUC cannot be detected reliably.

Headline findings

1. The models beat chance and popularity: test AUC 0.69–0.74 vs 0.50 random and 0.56 popularity. All four are significantly better than popularity.
2. Reddit's own feed order is the strongest ranker per user: NDCG@10 0.66 vs 0.49 for the best model, GAUC 0.82. Engagement is dominated by feed position: rank 0–2 → 44% engaged, rank 21+ → 21%.
3. The deep models were badly calibrated: raw logloss 1.39 (DeepFM) and 1.07 (Wide&Deep), worse than a constant (0.465). Platt scaling fitted on validation fixes it: 0.454–0.484 for all four.
4. History features drive everything. Removing them takes the average AUC from 0.714 to 0.619. Feed context and embedding features each add about 0.02. Categories and IDs add about nothing: permutation importance of the category group is 0.001, and masking IDs slightly raises AUC.

Per-user ranking (test, macro over 4 users)

Method	NDCG@10	Hit@10	MRR
DeepFM, 5-seed ensemble	0.491	1.00	0.69
DCN V2	0.465	1.00	0.68
LR	0.451	0.75	0.64
Reddit feed order	0.659	1.00	1.00
Popularity	0.328	1.00	0.47
Random	0.199	0.81	0.33

Other checks

• Leave-one-user-out (new-user test): LR keeps 0.69 mean AUC; the deep models drop to about 0.65.
• Shuffled labels give AUC 0.44–0.53: no leakage.
• Adversarial validation separates train from test perfectly, through session-position features that grow along the timeline. Removing them costs only 0.013.
• Slices: weakest for user P09 (AUC about 0.58) and r/AskPhysics (about 0.39). Weaker in the second half of the test period (interest drift).
• Cost: every model scores a post in about 0.2 ms on CPU.

Details: benchmark_report/REPORT.md and its CSV tables.

---

7. LLM tagging with a closed 114-tag taxonomy

Tagger. Groq openai/gpt-oss-120b, temperature 0, JSON output, choosing only from the app's taxonomy: 13 broad interests, 114 child tags (backend/crud/taxonomy.py). Two API keys were rotated; they live only in a git-ignored .env and were never committed.

Results

• 1,677 posts tagged, 1,997 tags, 1.6% invalid tags rejected, mean 1.19 tags per post, mean confidence 0.80.
• 24.4% of posts got no tag (careers, college and off-topic content the taxonomy does not cover). 75 of 114 tags were ever used.

Reliability

• Agreement with gpt-oss-20b: Jaccard 0.51. With qwen3-27b: 0.53. A re-run of the same model at temperature 0 reproduces the exact tag set only 47% of the time (same top tag 71%).
• Broad domains are stable (kappa 0.76). 91% of posts carry their subreddit's majority domain.
• Manual spot check: 36 of 40 acceptable.

---

8. The first two-stage engine (tag match, then model) and why it failed

Design.

• Stage 1 scores how close a post's tags are to the user's picked tags, using 8 tag-match features:

  • exact overlap;
  • parent-domain overlap;
  • rarity-weighted overlap;
  • soft similarity through a 114×114 tag-similarity matrix;
  • whether the post's top tag was picked;
  • number of matches;
  • number of tags;
  • mean confidence.

  A logistic regression combines them into tag closeness.

• Stage 2 is a Wide&Deep model on post features + tag-match features: no user ID, no history, no feed position. It works for a brand-new user (test AUC 0.692).

• Final step: order by tag closeness, then drop posts with model probability < 0.3 (or blend the two).

Result: below the model alone. Test NDCG@10: engine 0.29 and blend 0.37 vs the Stage-2 model alone 0.40. The reason: for these real users, posts matching their inferred interests were engaged with at the same rate as non-matching posts (20.2% vs 20.2%). For P09, engagement moved to posts the taxonomy cannot tag (37% engaged among untagged vs 3% among tagged).

Random-interest simulation. 2,000 synthetic users per interest type, test-period posts only:

• Coherent interests got lists of median 5 posts; 5.7% got nothing.
• Only 36% of the pool was ever recommended, because untagged posts can never appear.

---

9. Every alternative pipeline we tried

All were evaluated with the same protocol: per-user test pools, settings tuned on validation, test scored once, 25-post-page bootstrap. Simulated users drew random interests from the test-period pool.

Pipeline	Idea	Test NDCG@10	Verdict
Reddit feed order	reference	0.659	needs Reddit's own ranking; not available to us
DeepFM warm (history + feed rank)	warm CTR	0.491	uses feed rank, which a recommender cannot know before ranking
v2b: model + 0.1 × tag closeness	model first, small tag bonus	0.467	best cold-start option → became the engine's new-user score
v5: 10% exploration slots	exploration quota	0.447	small cost for variety
v5: 20% exploration + light diversity	exploration quota	0.429	more variety, ILD 0.85
v2a: Stage-2 model ranks all	model only	0.402
Two-tower retrieval + model ranking	Google two-tower	0.390	retrieval drops good posts
v1 blend (25% tag, 75% model)	first engine	0.373
Phoenix ranker, dwell head	X Thunder + Phoenix	0.365
Two-tower, topics + history	Google two-tower	0.359
v4: interest lane only	multi-lane	0.356
Popularity	baseline	0.328
Two-tower, topics only	Google two-tower	0.309
v1 engine (tag order + filter)	first engine	0.291
v2c: topic gate, then model	gate first	0.282
v4: four-lane mixer + MMR	multi-lane + diversity	0.274	most diverse (ILD 0.91), lowest accuracy
Phoenix full	X Thunder + Phoenix	0.258
v1 tag closeness only	Stage 1 only	0.248
Random	baseline	0.192
v3: learned intent embedding	predict the user's next-intent vector	0.180	at chance with 4 users of data

Hybrid experiments that shaped the final design

• v6. Blending v2b with a warm DeepFM by history depth helped on validation (0.633 vs 0.547 for v2b) but not on test.
• Fault probe.
  • Every real user in validation and test has ≥ 61 past interactions, so "new user" behaviour could only be studied by truncating histories (HIST_CAP).
  • Per-seed AUC of the warm model varied by up to 0.06.
  • A position-debiased warm model and a "clean" warm model (no IDs, no feed context, no session signals) were both better than v6's.
• v7. Tuned warm variants on validation. The clean warm model became the shipped warm model.

---

10. The hybrid engine: design

Your directive: v2b for new users; gradually shift toward a warm DeepFM as meaningful history builds up, while keeping some v2b for freshness and exploration; weight decay as posts age; a cold-start push for new posts.

v2b    = Platt(cold model probability) + tag_bonus · rank01(tag closeness)            tag_bonus = 0.1
w      = 0                                   for a user with no engaged posts
       = min(w_max, engaged / (engaged + n0)) otherwise                               n0 = 10, w_max = 0.7
blend  = (1 − w) · rank01(v2b) + w · rank01(warm DeepFM probability)
fresh  = floor + (1 − floor) · 0.5 ^ (age_hours / half_life)                          1.1+: half_life 720 h, floor 0.85
push   = beta · [age_hours ≤ 24] / sqrt(1 + impressions)                              beta = 0.1
score  = blend · fresh + push
order  = in every block of 10 positions: the 7 best topic-matching posts, then the 3 best of any kind

Why each piece is there

• rank01 (rank inside the request, scaled to (0, 1]) makes the two models' scores comparable, whatever their calibration.
• w grows with engaged posts, not all impressions, because the warm model only learns from engagement. It never reaches 1, so v2b always keeps at least 30%.
• Freshness is multiplicative and gentle. A week-old post keeps 98%, a month-old post 93%, and no post drops below 85%. The models already see post age, so a strong decay double-counts it (section 12).
• The push is additive and fades with impressions, so a new post gets a few showings and then must earn its place.
• Topic-first guarantees that the top of the feed reflects the picked interests. A topic match means the post shares a picked tag or its broad domain.
• Exploration is not the engine's job. The backend's compose.pick_explore keeps reserving 15–20% out-of-interest slots.

---

11. Building the engine and the training/serving bug we found

Cold model (v2b's probability): the Stage-2 Wide&Deep, 5 seeds averaged inside one ONNX graph, with Platt calibration fitted on validation.

Warm model: DeepFM without user/post IDs, feed context (rank, recommendation source, page kind, view context, attention focus) or session signals (session position, session time, feed speed). It uses post features + 27 history features, 5 seeds in one graph.

The bug: a training/serving skew. The first parity check compared engine-built inputs with the training files. The warm history features differed by up to 6.46 standard deviations, and the scores by up to 0.9.

• Cause: the training data counted impressions logged at the same timestamp with zero visibility as "history". A live request can never see those.
• Fix: a new dataset switch STRICT_PAST=1 (only interactions strictly before the request), the same rule in the engine (ts < now and viewing ended), and a retrained warm model.
• Effect: warm test AUC dropped from 0.727 to 0.671. The lower number is the honest one.
• After the fix: features match within 0.003 and scores within 3e-4.

Engine 1.0 replay (NDCG@10)

Setting	Validation, whole pool	Test, whole pool
Shipped defaults	0.62	0.39
v2b only	0.51	0.48
w_max 0.3	0.55	0.48
No decay / push	0.66	0.40

The blend wins on validation and loses on test. With only 4 long-history users, these defaults are a starting point to re-tune on real logs.

---

12. Version 1.1: gentler age decay and topic-first slots

Your request: soften the age penalty, and make topic matches find the top posts.

• Decay: half-life 168 h → 720 h, floor 0.7 → 0.85.
• Topic-first: 7 of every 10 positions go to topic matches.

Results (NDCG@10, topic share of the top 10)

Feed	1.0	1.1
New session 1, one feed at session start	0.25 (10%)	0.41 (70%), engaged posts in the top 10: 3 → 5
New session 1, 25-post pages	0.52	0.56
Validation, whole feeds / pages	0.65 / 0.60	0.69 / 0.64
Test, whole feeds / pages	0.45 / 0.55	0.32 / 0.46
New session 2: AUC / pages	0.58 / 0.34	0.60 / 0.38 (Reddit order AUC 0.77, random 0.50)

The topic rule helps when engagement follows the picked topics (new session 1, validation). It costs in the test period, where P09 had drifted to untagged posts. You chose to keep 1.1 as it was ("no improvement" round).

---

13. Version 2.0: no community, no flair

Correction (2026-10-07). Version 2.0 removes the community name and flair as direct inputs and from the embedding text. It still receives each post's domain field, which for Reddit text posts is self.<subreddit>: 24 of the 30 domain tokens known to both models are subreddit names. Community information therefore remains indirectly available to the 2.0 models for text posts. A follow-up release removes domain entirely. Until then, treat v2.0's community independence as partial.

What was wrong. Since the first RecBole dataset, every post metadata column had been an input, including subreddit and flair. The subreddit name was also inside the embedding text. You had never asked for this, and it cannot transfer: Unihive communities are not Reddit subreddits, so every backend community would be "unknown" to the 1.x models.

What changed

1. Content vectors re-embedded from "title. body" (+ image caption) for all 1,677 posts (embed_posts_qwen3_nocomm.py).

2. New datasets, built by prep_recbole.py with EMB_NPZ=post_emb_qwen3_0p6b_nocomm.npz META_DIR=package_meta_nc:

   • reco_ctr_nc;
   • reco_ctr_nc_noid_strict (also MASK_IDS=1 STRICT_PAST=1);
   • reco_engine_nc, the cold dataset: tag-match features + post features, tokens post type / domain / language only.

   All originals are untouched (checksums verified).

3. Both models retrained (engine_v2/build_engine_v2.py), with the same tuned settings, 5 seeds, export and checks. Test AUC:

   • Cold: valid 0.777, test 0.602 (1.1: 0.692). Inputs: post type, domain, language + 35 floats.
   • Warm: valid 0.713, test 0.565 (1.1: 0.671). Per-seed test AUC 0.55–0.60.

4. Kept: sub_seen / sub_rate, the user's own engagement rate with a community. They compare community ids inside one user's history and need no learned names. On the backend, community_id provides them.

5. Guard test: test_no_community_or_flair_in_models fails if subreddit or flair is a model input, if the embedding text includes them, or if renaming them changes any score.

Where the AUC drop comes from (engine_v2/diag_engine_v2.py: cold model, same recipe, 5 seeds)

Variant	Test AUC
A: 1.1 setup (community + flair as inputs and in the text)	0.692 (reproduces 1.1 exactly)
B: inputs removed, old text	0.624 (−0.07)
C: inputs kept, new text	0.653 (−0.04)
D: both removed = 2.0	0.602 (−0.09)

On this Reddit trace the community name genuinely predicted engagement. For example, P09 engaged with r/AskPhysics in a way the models picked up.

Verification of 2.0

• ONNX == PyTorch: 2.4e-7.
• Feature parity on 539 rows: 0.0021, 0 token mismatches.
• Score parity: cold 1.1e-4, warm 2.5e-4.
• 0.7 ms per candidate.

Ranking quality under backend conditions (NDCG@10; artifacts/backend_modes_report.json)

• Full = every Reddit field, as in training (offline upper bound).
• Unknown community names = every community gets a distinct name the models never saw, as Unihive communities will.
• Backend data = also no flair, tags without confidences, one dwell value (hover 0, focal = visible), no upvote ratio or awards.
• No Qwen3 = also no content vectors (today's backend).

Feed	1.1 full	1.1 unknown comm.	1.1 backend	1.1 backend, no Qwen3	2.0 full	2.0 backend	2.0 backend, no Qwen3	random
Validation, whole pool	0.685	0.673	0.574	0.625	0.650	0.579	0.604	0.350
Validation, 25-post pages	0.642	0.652	0.582	0.534	0.559	0.595	0.582	0.350
Test, whole pool	0.316	0.333	0.291	0.267	0.340	0.329	0.268	0.201
Test, 25-post pages	0.458	0.459	0.483	0.465	0.415	0.397	0.441	0.286
New session 1, start	0.413	0.354	0.290	0.225	0.255	0.159	0.155	0.119
New session 1, pages	0.559	0.487	0.396	0.359	0.468	0.486	0.362	0.282
New session 2, start	0.110	0.095	0.139	0.139	0.110	0.110	0.110	0.079
New session 2, pages	0.381	0.394	0.522	0.338	0.403	0.382	0.372	0.245
Mean	0.446	0.431	0.410	0.369	0.400	0.380	0.362	0.239

Reading the table

• 2.0 gives identical scores with real or unknown values of the community field (checked: the "unknown community names" column equals "full" for 2.0, so it is not repeated). That test changed the community field only; it did not change the domain token, which still carries subreddit names for text posts (see the correction at the top).
• Under backend conditions, 2.0 is 0.03 (with Qwen3) and 0.01 (without) below 1.1. The 1.1 figures are optimistic, because its content vectors still contain the Reddit subreddit name, which Unihive posts will not have.
• Both are well above random everywhere.
• Single feeds (one request) are noisy: a 10-position list with 10–11 engaged posts out of 89–142.

Re-check on your two later sessions (engine_v2/validate_two_sessions.py → artifacts/validation_two_sessions.json; each engine with its own content vectors)

	Session 1 (89 posts, 11 engaged)			Session 2 (142 posts, 10 engaged)
	NDCG@10	engaged in top 10	AUC	NDCG@10	engaged in top 10	AUC
Engine 2.0	0.255	3	0.688	0.110	1	0.498
2.0 without content vectors	0.249	3	0.760	0.110	1	0.532
Engine 1.1	0.413	5	0.780	0.110	1	0.602
Reddit feed order	0.641	6	0.917	0.312	3	0.773
Random	0.124	1.1	0.50	0.070	0.6	0.50

25-post pages: 2.0 scored 0.468 and 0.403 (1.1: 0.559 and 0.381). 2.0 lost two engaged posts from the session-1 top 10. Its top two posts there were finance posts promoted by the topic-first rule, because "investing-and-markets" is one of P09's inferred interests.

New users (engine_v2/validate_new_users.py → artifacts/new_user_report.json)

The same real feeds, ranked as if each user had just signed up: picked interests only, no history (the code checks that w = 0). NDCG@10:

Feed	New user (interests only)	New user, no interests	Same user, full history	Random
Validation, whole feeds	0.525	0.564	0.650	0.350
Validation, 25-post pages	0.525	0.568	0.559	0.350
Test, whole feeds	0.410	0.303	0.340	0.201
Test, 25-post pages	0.426	0.460	0.415	0.286
Session 1, one feed	0.221	0.445	0.255	0.119
Session 1, pages	0.341	0.527	0.468	0.282
Session 2, one feed	0.139	0.139	0.110	0.079
Session 2, pages	0.359	0.377	0.403	0.245
Mean	0.368	0.423	0.400	0.239

Reading the results

• A brand-new user gets a useful feed from the first request: about 54% above random.
• History adds on top (0.368 → 0.400).
• A user with no picked interests is still served, ranked on post quality and content.

These users never picked interests themselves; theirs were inferred from behaviour, and some guesses were wrong. That is why "no interests" scored highest. Real signup picks should match better. If they don't, lower topic_first_slots for new users (not changed in 2.0).

---

14. Backend compatibility

Checked against the real backend code (backend/reco/*, backend/api/schemas.py, migrations 004/005, db/trumio_schema_dump.sql):

• Taxonomy: the backend's reco/vocab.py holds exactly the engine's 13 broad and 114 child slugs (compared slug by slug).
• Post meta (store.posts_meta): tags (frozenset of slugs, no confidences), cat, created, likes, comments, views, author.
• Rows: posts (community_id, post_type ∈ TEXT/LINK/MEDIA/NEWS/REPOST/QUESTION/POLL, title, body, url, counters) and post_reco (tag_slugs, CLIP-512 emb_fused).
• User state (store.get_state): base_tags (the interests picked at onboarding, broad or child slugs), tag_weights, a 512-d embedding.
• Events: the client's IMPRESSION / VIEW / LONG_VIEW / SKIP / CLICK / SHARE / SAVE / HIDE / NOT_INTERESTED. Likes and comments are written to interaction_events by crud/posts.py. The Redis burst buffer holds {type, post_id, dwell_ms, t, surface, context}.

Mapping (backend_adapter.py)

Engine input	From the backend
interests	state["base_tags"]; a broad pick ("ai-ml") expands to all of its child tags
score, comments, impressions	likes, comments, views
age_hours	(now − created) / 3600
tags	post_reco slugs, confidence 1.0
post_type	TEXT/QUESTION/POLL → text, LINK/NEWS → link, REPOST → crosspost, MEDIA → image (or video, from the media kind)
domain	parsed from the post url
embedding	Qwen3 vector under reco:post:{id}:q3 (new). Missing → neutral content
community	posts.community_id, only for the user's per-community history rates
history	burst buffer (read without draining) + interaction_events (with the post's community and author). Events on one post within 30 min = one interaction. engaged = LONG_VIEW / CLICK / SAVE / SHARE / LIKE / VOTE / COMMENT / REPLY / JOIN / POLL_VOTE or dwell ≥ 3 s, unless SKIP / HIDE / NOT_INTERESTED. ts_ms = received time − dwell

build_pool_hybrid keeps the backend's retrieval exactly: graph.expand_weights, the tag indexes, seen_among, the own-post filter and compose.split_counts. Only the scoring is replaced. It writes the pool with store.set_pool and store.set_level, and returns [(post_id, score, "cat|tags")], the same shape as build_pool.

Tests (tests/test_backend_adapter.py, 7 tests)

• Interests, post mapping and event→history mapping, including the 30-min visit merge, NOT_INTERESTED overriding dwell, the shown-time correction and future events being dropped.
• rank_pool with backend shapes:
  • a new user with no Qwen3 vectors gets pure v2b and topic-first;
  • CLIP vectors are refused;
  • with Qwen3 + history the warm weight is exactly 0.6 at 15 engaged posts.
• Qwen3 codec and embedding text.
• Postgres + buffer merge, with a fake connection that returns HISTORY_SQL-shaped rows.
• The backend's real reco.store on fakeredis (index_post, save_state, buf_push, posts_meta, set_pool, pool_all). It also checks that build_pool_hybrid returns the same candidates as the backend's own service.build_pool.

All pass. Not tested: HISTORY_SQL against a live Postgres (Docker was not running). It was checked against the schema dump.

Integration steps: see models/hybrid_engine/INTEGRATION.md.

1. Store a Qwen3 vector in the tagging worker.
2. Delegate build_pool to build_pool_hybrid.
3. Add onnxruntime to the requirements.
4. Shadow mode → replay → A/B test.

---

15. Code walkthrough (every file of the engine)

models/hybrid_engine/ (the shipped package)

__init__.py: exports HybridRecommender, FeatureBuilder, rank01.

features.py: rebuilds the exact training inputs from raw data.

• clean(s) collapses whitespace.
• token(v) maps a category value to its training token: 'none' for missing or empty, whitespace → '_'.
• unit(v) scales a vector to unit length.
• EMB_DIM = 1024; content_vector(emb) returns a unit vector, or None when the embedding is missing, the wrong size (e.g. CLIP-512), all zeros or not finite.
• rank01(x) gives ranks inside one request: lowest → 1/n, highest → 1.
• FeatureBuilder(meta) loads the constants, PCA, taxonomy, tag similarity, Stage-1 weights and IDF from meta.json.
  • pca16(emb): the 16 content components.
  • z(name, raw): standardise with the training mean/std.
  • post_floats(post): the 11 post statistics + 16 components (all zeros when the content is unknown). Missing age → median; missing upvote ratio → 1.0 + flag.
  • post_tokens(post): category tokens. User/post IDs are always 'unk'; the domain falls back to 'other' outside the top 30; membership → yes/no/unk. The models read only the tokens listed in their meta.json (2.0: post_type, domain, post_language, + member for warm).
  • tag_vector(tags): {slug: confidence} or a list → a 114-d confidence vector (unknown slugs ignored).
  • interest_profile(picks): q (picked tags), r (their broad domains), ms (best similarity of each tag to any pick).
  • tag_match(P, q, r, ms): the 8 tag-match features (section 8).
  • tag_closeness(tm): Stage-1 logistic score.
  • history_floats(history, post, now_ms, post_X): the causal history features of section 4. J = interactions with ts < now whose viewing ended by now; R = the last 20. Content features use only past posts that have a vector. Returns the features and the number of engaged posts in J (which drives w).

engine.py: HybridRecommender.

• __init__(artifacts_dir, **overrides) loads meta.json and two single-threaded CPU ONNX sessions. Overrides replace the defaults in meta["defaults"].

• _tokens / _floats order a feature dict into the model's input vector. Unknown tokens → index 0.

• _prep_history sorts the history by time, checks each vector with content_vector, and precomputes pca16.

• warm_weight(engaged) returns 0 for no engagement, else min(w_max, e / (e + n0)).

• score(user, candidates, now_ms): one pass over the candidates.

  1. Content vector, post floats, tokens.
  2. Tag match → tag closeness and the topic-match flag (shares a picked tag or its broad domain).
  3. Cold inputs = standardised post floats + standardised tag-match features.
  4. History features + engaged count → warm inputs.
  5. Run the cold ONNX → Platt → v2b = p_cal + tag_bonus · rank01(closeness).
  6. Run the warm ONNX only if some w > 0.
  7. Blend the ranks, multiply by freshness, add the push.

  now_ms may be a list (one time per candidate) for replays. Returns one dict per candidate with every component.

• order(scored): topic-first. While posts remain: take the best topic_first_slots topic matches, fill the block of topic_block with the best remaining posts, repeat. Too few matches → the slots fall back to any post. Sets position.

• rank = order(score(...)); recommend = the top k of rank.

backend_adapter.py: the bridge to the Unihive backend (section 14).

• Event vocabularies: ENGAGED_EVENTS, NEGATIVE_EVENTS, CLICK_EVENTS. Constants: 3 s engagement threshold, 30-min visit gap, 2-min dwell cap.
• POST_TYPE_MAP and MEDIA_KIND_MAP translate backend post types to the trained ones.
• q3_key, encode_q3, decode_q3 store a unit Qwen3 vector as float16 in Redis. A wrong size is refused.
• qwen3_text(title, body, caption): the exact embedding text of 2.0.
• expand_interests(slugs, taxonomy): broad picks → children; unknown slugs dropped; order kept.
• post_from_meta(pid, meta, now_s, emb_q3, extra) and post_from_row(row, now_s, emb_q3): backend post → engine candidate.
• _norm_event accepts buffer dicts and interaction_events rows.
• history_from_events(events, posts, now_s): merges visits, decides engaged/clicked, corrects the shown time, fills community/author/vector.
• user_from_backend(state, history, taxonomy).
• HISTORY_SQL reads interaction_events joined with posts (newest first, limited).
• fetch_q3(r, pids) (one MGET), peek_buffer(r, uid) (LRANGE, no drain).
• load_history(conn, r, uid, buffered, limit, now_s): Postgres rows + buffer → history with vectors.
• build_pool_hybrid(r, uid, state, level, *, engine, conn, ...): the drop-in build_pool.
• rank_pool(engine, state, meta, history, now_s, q3, extra, size): returns the pool scores (strictly decreasing, engine order) and the explanation dicts.

example.py: a demo: the same candidates for a brand-new user and after 30 engaged posts.

tests/test_engine.py

• test_smoke:
  • a new user gets pure v2b;
  • the weight grows (0.091 / 0.333 / 0.667 / 0.7 at 1 / 5 / 20 / 100 engaged posts) and is capped;
  • future events are ignored;
  • freshness is monotone and the push applies only to fresh posts;
  • topic-first fills the first 7 slots, slots=0 gives plain score order, and few matches fall back gracefully.
• test_no_community_or_flair_in_models: the 2.0 guard (section 13).

tests/test_backend_adapter.py: section 14.

requirements.txt: numpy, onnxruntime.

artifacts/: section 13 and the package README.

models/training/engine_v2/ (how the engine was built and checked)

• evalkit.py: shared protocol.

  • Loads the frozen v1 rows (user, split, label, inferred interest picks Q, post tag vectors P) and the offline predictions.
  • Platt on validation only.
  • user_metrics (NDCG@10, Hit@10, MAP@10, AUC, topic match per user), pooled_auc, calib_report.

• engine_lib.py: taxonomy, tag embeddings and similarity, interest profiles (picked = 1.0, implied parent = 0.6), the 8 tag-match features, the hand-set Stage-1 rule.

• build_hybrid_engine.py: engine 1.0 build.

  • Ens averages the seeds' sigmoid outputs inside one module.
  • export writes ONNX (opset 17, dynamic batch) with the input order and vocabularies.
  • check compares ONNX with PyTorch on the validation and test files.
  • Writes meta.json and build_report.json.

• retrain_warm_strict.py: retrains the 1.x warm model on reco_ctr_noid_strict (the serving rule) and updates the artifacts.

• build_engine_v2.py: engine 2.0 build (section 13). It reuses the 1.0 helpers and asserts that subreddit/flair are not inputs.

• diag_engine_v2.py: the four-variant diagnosis of the 2.0 AUC drop.

• verify_hybrid_engine.py: runs the engine's real code path on raw data:

  1. feature parity on every valid/test row;
  2. score parity against the offline ensembles;
  3. a replay (whole pools and 25-post pages);
  4. a cold-start vs long-history example.

  It detects the engine version from meta.json and uses the matching embeddings, datasets and offline scores.

• sweep_hybrid_engine.py: knob sensitivity (w_max, decay/push) on validation and test.

• validate_v11.py: 1.0 vs 1.1 variants on new session 1, test and validation.

• validate_new_session.py and validate_session.py: rebuild per-post outcomes from raw Trace events with the training rules, tag new posts with Groq (cached), embed them with the version-matching text (cached), and rank as live requests with P09's earlier history.

• validate_backend_modes.py: section 13's backend-conditions table, run once per engine version.

Data pipeline files (models/training/)

• prep_recbole.py: section 4 (features, split, switches).
• embed_posts_qwen3.py / embed_posts_qwen3_nocomm.py: the 1.x / 2.0 content vectors.
• run_models.py, tune_and_final.py, continue_training.py, export_onnx.py, apply_recbole_patch.py: phase 1 training and export.
• data_pipeline/: captions, pullpush recovery, interaction table.

---

16. How to reproduce

# engine tests (any machine): numpy + onnxruntime (+ redis, fakeredis for the backend store test)
cd models && python -m hybrid_engine.tests.test_engine && python -m hybrid_engine.tests.test_backend_adapter
UNIHIVE_BACKEND=/path/to/backend python -m hybrid_engine.tests.test_backend_adapter     # include the backend's real reco.store

# rebuild engine 2.0 from the data (GPU recommended; RecBole env from training/RETRAIN.md)
python embed_posts_qwen3_nocomm.py
EMB_NPZ=post_emb_qwen3_0p6b_nocomm.npz SUFFIX=_nc META_DIR=package_meta_nc python prep_recbole.py
EMB_NPZ=post_emb_qwen3_0p6b_nocomm.npz SUFFIX=_nc_noid_strict META_DIR=package_meta_nc MASK_IDS=1 STRICT_PAST=1 python prep_recbole.py
python engine_v2/build_engine_v2.py          # writes hybrid_engine/artifacts_v2 -> install as hybrid_engine/artifacts
python engine_v2/verify_hybrid_engine.py     # parity must stay near zero
python engine_v2/validate_backend_modes.py   # ranking quality under backend conditions

build_engine_v2.py needs the frozen v1 outputs (engine_v1_frozen/out_engine) and the reco_engine dataset from the earlier build. Both are in the local reco_data workspace; the main datasets are in models/training/.

---

17. Limits and next steps

Limits

• Tiny data. 4 users, 44 positive test examples: an AUC standard error of about 0.05. Most differences between variants are within noise. Treat all defaults as starting points.
• Reddit is not Unihive. Post types, popularity scales and the engagement mix will differ. Retraining on Unihive logs is the most important next step, and the pipeline supports it.
• Feed position dominated engagement in the trace. Reddit's own order (which knows position and much more) stays far ahead.
• Not yet available in the backend: Qwen3 vectors, tag confidences, and hover / focal dwell. The cost of each is measured in section 13.

Next steps

1. Add the Qwen3 vector to the tagging worker (INTEGRATION step 1).
2. Shadow-mode logging of engine vs current scores.
3. Collect 2–4 weeks of logs, then re-tune w_max, n0 and decay/push, and retrain both models on Unihive data.
4. Keep exploration in compose at 15–20%.
5. Add tag confidences to post_reco (a small gain).

---

18. Glossary of metrics

• AUC: probability that a random engaged post is scored above a random non-engaged one. 0.5 = chance.
• GAUC: AUC per user, averaged.
• NDCG@10: how close the top 10 is to the best possible ordering of the engaged posts. 1 = all engaged posts first, with position-discounted credit.
• Hit@10: at least one engaged post in the top 10.
• MRR: 1 / (position of the first engaged post).
• Logloss / Brier / ECE: how good the probabilities themselves are (lower is better). ECE = calibration error.
• Platt scaling: a one-feature logistic regression on the model's logit, fitted on validation, that repairs over-confidence.
• Parity: the largest difference between two computations that should be identical (engine vs training file, ONNX vs PyTorch).
• ILD: intra-list diversity, the mean pairwise distance of the posts in a list.