README: engine v2.0 is the model to use; older models moved to archive/

README.md

@@ -1,344 +1,328 @@
----
-license: apache-2.0
-library_name: onnx
-language:
-- en
-pipeline_tag: tabular-classification
-tags:
-- recommender-system
-- feed-ranking
-- ctr-prediction
-- cold-start
-- deepfm
-- wide-and-deep
-- onnx
-- recbole
----
-
-# UniHive RecSys: hybrid feed-ranking engine (v2.0)
-
-A small, production-shaped **post recommendation engine** for the UniHive student social app. It ranks candidate posts for a user and handles the full lifecycle of a user and a post:
-
-- **Brand-new users** are ranked by **v2b**: an engagement model (Wide&Deep) plus a small bonus for closeness between the post's tags and the user's picked interests.
-- **As a user builds up engaged history**, the engine **gradually blends in a warm DeepFM model** that reads their behaviour. It always keeps at least 30% v2b for freshness.
-- **New posts** get a **cold-start push** that fades as they collect impressions. **Older posts** sink through a **gentle freshness decay**.
-- **Topic-first ordering:** in every block of 10 positions, 7 go to posts matching the user's interests.
-
-Everything runs on CPU through ONNX Runtime: about **0.7 ms per candidate**, about 2 MB of models.
-
-> **Honest scope.** The models were trained on a small real-world trace: 4 users, 1,794 Reddit post exposures, 44 positive test examples. The engine consistently beats random and popularity, but trails Reddit's own feed order. Treat the defaults as a well-tested starting point to re-tune on your own logs. All numbers below come from held-out or later-recorded data.
-
----
-
-## Contents
-- [How it works](#how-it-works)
-- [Quick start](#quick-start)
-- [Inputs and outputs](#inputs-and-outputs)
-- [Using it inside a backend (UniHive adapter)](#using-it-inside-a-backend-unihive-adapter)
-- [All scores](#all-scores)
-- [Where the engine's parts come from (industry lineage)](#where-the-engines-parts-come-from-industry-lineage)
-- [Model sizes (why the files are small)](#model-sizes-why-the-files-are-small)
-- [Files in this repo](#files-in-this-repo)
-- [Training data and procedure](#training-data-and-procedure)
-- [Limitations and responsible use](#limitations-and-responsible-use)
-- [Citation](#citation)
-
----
-
-## How it works
-
-```
-                 candidate posts (already retrieved by your backend)
-                                     │
-        ┌────────────────────────────┼─────────────────────────────┐
-        ▼                            ▼                             ▼
-  tag closeness              COLD model (Wide&Deep)          WARM model (DeepFM)
-  8 tag-match features       post stats + content            post inputs + 27 user-history
-  → logistic score           + tag-match features            features (no IDs, no feed position)
-        │                    → Platt-calibrated P(engage)    → P(engage)
-        └──────────► v2b = P_cold + 0.1 · rank(closeness)          │
-                                     │                             │
-                 blend = (1 − w) · rank(v2b) + w · rank(warm)      w = min(0.7, engaged / (engaged + 10))
-                                     │
-                 score = blend · freshness(age) + push(age, impressions)
-                                     │
-                 topic-first order: each block of 10 = 7 best topic matches + 3 best of any kind
-```
-
-| Piece | Formula | Default |
-|---|---|---|
-| Warm weight | `w = 0` with no engaged posts, else `min(w_max, e / (e + n0))` | n0 = 10, w_max = 0.7 → 0.09 at 1 engaged post, 0.33 at 5, 0.50 at 10, 0.67 at 20 |
-| Freshness | `floor + (1 − floor) · 0.5^(age_h / half_life)` | half-life 720 h, floor 0.85 (a week-old post keeps 98%, a month-old 93%) |
-| Cold-start push | `beta · [age_h ≤ 24] / sqrt(1 + impressions)` | beta 0.1 |
-| Topic-first | 7 of every 10 positions go to posts sharing a picked tag or its broad domain | `topic_first_slots = 7` |
-
-**What the models do NOT use.** User or post IDs, feed position, scroll speed, community (subreddit) name and flair are all excluded. The content vectors are built from title + body + image caption only. A built-in test fails if community or flair ever becomes an input again.
-
----
-
-## Quick start
-
-```bash
-pip install huggingface_hub numpy onnxruntime
-hf download Tirthanker/UniHive_RecSys --local-dir UniHive_RecSys
-cd UniHive_RecSys
-python -m hybrid_engine.example                 # demo: a brand-new user vs the same user after 30 engaged posts
-python -m hybrid_engine.tests.test_engine       # smoke tests + the no-community guard
-```
-
-```python
-from hybrid_engine import HybridRecommender
-
-engine = HybridRecommender()                        # loads hybrid_engine/artifacts once (CPU, ONNX)
-
-user = {
-    "interests": ["machine-learning", "computer-vision"],   # tag slugs the user picked (114-tag taxonomy, see meta.json)
-    "history": [                                             # past interactions; anything not before now_ms is ignored
-        {"post_id": "p1", "ts_ms": 1791141039157, "visible_ms": 5200, "focal_ms": 4800, "hover_ms": 300,
-         "clicked": True, "engaged": True, "community": "c-ml", "author": "a-17", "embedding": None},
-    ],
-}
-posts = [
-    {"post_id": "p9", "post_type": "text", "score": 42, "comments": 7, "age_hours": 3.5,
-     "title": "Fine-tuning ViTs on a laptop GPU", "body": "...", "tags": {"computer-vision": 0.9, "deep-learning": 0.7},
-     "embedding": None,           # optional Qwen3-Embedding-0.6B vector (1024 floats); None = content unknown
-     "author": "a-3", "impressions": 0, "community": "c-cv"},
-    # ... more candidates
-]
-
-feed = engine.recommend(user, posts, now_ms=1791145000000, k=10)
-for d in feed:
-    print(d["position"], d["post_id"], round(d["score"], 3), "warm weight", d["warm_weight"])
-```
-
-**Content vectors (recommended).** Embed `"title. body"` (+ `" Image: <caption>"` when there is one) with `Qwen/Qwen3-Embedding-0.6B`, `normalize_embeddings=True`. `hybrid_engine.backend_adapter.qwen3_text()` builds the exact training text. Without vectors the engine still works; it simply loses the content signal (see the scores).
-
-**Knobs.** `HybridRecommender(w_max=0.5, n0_engaged_posts=5, topic_first_slots=0, push_beta=0, decay_floor=1.0, ...)`. The defaults are in `artifacts/meta.json`.
-
----
-
-## Inputs and outputs
-
-**Candidate post fields:**
-- `post_id`, `post_type` (text / image / video / link / ...), `score` (likes), `comments`, `age_hours`, `title`, `body`, `image_caption`;
-- `tags` (`{slug: confidence}` or a list), `embedding` (1024 floats or None), `author`, `impressions` (how often your platform has shown it);
-- optional: `domain`, `language`, `is_ad`, `is_spoiler`, `upvote_ratio`, `awards`, `is_member`;
-- `community` is used only for the user's *own* per-community engagement history, never as a model input.
-
-**History item fields:** `post_id`, `ts_ms` (when shown), `visible_ms`, `focal_ms`, `hover_ms`, `clicked`, `engaged` (dwell ≥ 3 s or click / like / save / join / reply), `community`, `author`, `embedding`.
-
-**Output per candidate:**
-- `score`, `position`, `topic_match`;
-- `v2b`, `cold_probability`, `tag_closeness`, `warm_probability`, `warm_weight`, `engaged_history`;
-- `freshness_multiplier`, `push`, `has_content_embedding`.
-
-Log these to explain every feed.
-
----
-
-## Using it inside a backend (UniHive adapter)
-
-`hybrid_engine/backend_adapter.py` connects the engine to a FastAPI + Redis + Postgres feed service:
-- **`post_from_meta` / `post_from_row`:** backend post rows → candidates. Post types are mapped, the link domain is parsed and age is computed.
-- **`history_from_events`:** raw client events (VIEW, LONG_VIEW, CLICK, LIKE, SAVE, SKIP, HIDE, ...) → interactions. Events on one post within 30 min count as one visit, an explicit "not interested" overrides dwell, and the shown time is corrected for when the event arrived.
-- **`expand_interests`:** broad interest picks → child tags.
-- **`build_pool_hybrid`:** a drop-in replacement for a `build_pool(r, uid, state, level)` scoring step. The backend's own retrieval is kept; only the ranking changes.
-- **`encode_q3` / `fetch_q3` / `q3_key`:** store and read float16 Qwen3 vectors in Redis.
-
-Step-by-step integration, latency and rollout advice are in [`hybrid_engine/INTEGRATION.md`](hybrid_engine/INTEGRATION.md). The adapter tests can also run against the real backend store on fakeredis: `UNIHIVE_BACKEND=/path/to/backend python -m hybrid_engine.tests.test_backend_adapter`.
-
----
-
-## All scores
-
-Metrics:
-- **AUC:** chance that an engaged post is scored above a non-engaged one (0.5 = random).
-- **NDCG@10:** quality of the top 10, where 1.0 means every engaged post comes first.
-- **25-post pages:** a feed ranked one page at a time, with history updated as the user scrolls.
-
-### 1. Engine models (test split: last 15% of each user's timeline, 270 exposures, 44 engaged)
-| Model | Validation AUC | Test AUC | Notes |
-|---|---|---|---|
-| Cold model (Wide&Deep, 5-seed ensemble) | 0.777 | **0.602** | post + tag-match features only; works for a brand-new user |
-| Warm model (DeepFM, 5-seed ensemble) | 0.713 | **0.565** | per-seed test AUC 0.55–0.60 |
-| v1.1 cold model (used community + flair) | 0.799 | 0.692 | legacy, see below |
-| v1.1 warm model (used community + flair) | — | 0.671 | legacy |
-
-The AUC standard error with 44 positives is about 0.046.
-
-### 2. Engineering checks
-| Check | Result |
-|---|---|
-| ONNX vs PyTorch, max difference | 2.4e-7 |
-| Engine-built features vs training files (539 rows) | max 0.0021 standardized units; 0 categorical mismatches |
-| Engine scores vs offline predictions | cold 1.1e-4, warm 2.5e-4 |
-| Latency | 0.7 ms per candidate on 1 CPU thread, including history features |
-| Automated tests | 9 pass (smoke, no-community guard, 7 backend-adapter tests including the real backend store on fakeredis) |
-
-### 3. Ranking quality, offline periods and later sessions (NDCG@10)
-Each feed is ranked as one live request. "Backend data" adds the limits of a typical app backend: no flair, tags without confidences, a single dwell value, no Reddit vote ratio.
-
-| Feed | v2.0 | v2.0, backend data | v2.0, backend data, no content vectors | Random |
-|---|---|---|---|---|
-| Validation period, whole feeds | **0.650** | 0.579 | 0.604 | 0.350 |
-| Validation period, 25-post pages | 0.559 | 0.595 | 0.582 | 0.350 |
-| Test period, whole feeds | **0.340** | 0.329 | 0.268 | 0.201 |
-| Test period, 25-post pages | 0.415 | 0.397 | 0.441 | 0.286 |
-| Later session 1, one feed | 0.255 | 0.159 | 0.155 | 0.119 |
-| Later session 1, 25-post pages | 0.468 | 0.486 | 0.362 | 0.282 |
-| Later session 2, one feed | 0.110 | 0.110 | 0.110 | 0.079 |
-| Later session 2, 25-post pages | 0.403 | 0.382 | 0.372 | 0.245 |
-| **Mean** | **0.400** | **0.380** | **0.362** | **0.239** |
-
-### 4. The two later real sessions in detail (recorded two days after all training data)
-| | Session 1: 89 posts, 11 engaged | | | Session 2: 142 posts, 10 engaged | | |
-|---|---|---|---|---|---|---|
-| | NDCG@10 | engaged in top 10 | AUC | NDCG@10 | engaged in top 10 | AUC |
-| **Engine v2.0** | 0.255 | 3 | 0.688 | 0.110 | 1 | 0.498 |
-| v2.0 without content vectors | 0.249 | 3 | 0.760 | 0.110 | 1 | 0.532 |
-| Engine v1.1 (with community) | 0.413 | 5 | 0.780 | 0.110 | 1 | 0.602 |
-| Reddit's own feed order | 0.641 | 6 | 0.917 | 0.312 | 3 | 0.773 |
-| Random | 0.124 | 1.1 | 0.50 | 0.070 | 0.6 | 0.50 |
-
-On 25-post pages, v2.0 scored 0.468 and 0.403 (v1.1: 0.559 and 0.381).
-
-### 5. Why v2.0 removed community and flair, and what it cost
-Community names and flairs are specific to the source platform: a new platform's communities are unknown to a model trained on Reddit. Removing them costs offline accuracy on Reddit data. Cold-model diagnosis, 5 seeds:
-
-| Variant | Test AUC |
-|---|---|
-| With community + flair inputs and in the embedding text (v1.1) | 0.692 |
-| Inputs removed only | 0.624 |
-| Embedding text changed only | 0.653 |
-| Both removed (v2.0) | 0.602 |
-
-Under conditions a new platform would actually have (unknown community names, backend data), the mean NDCG@10 gap shrinks: **0.380 for v2.0 vs 0.410 for v1.1**, and 0.362 vs 0.369 without content vectors. The v1.1 figures are optimistic, because its content vectors still contain Reddit community names.
-
-### 6. Earlier research results that shaped the design (test period, NDCG@10)
-| Approach | NDCG@10 |
-|---|---|
-| Reddit feed order (reference, uses information a recommender cannot have) | 0.659 |
-| DeepFM with history + feed position | 0.491 |
-| **v2b (model + 0.1 × tag closeness): the engine's new-user score** | **0.467** |
-| 10% exploration slots | 0.447 |
-| Stage-2 model alone | 0.402 |
-| Two-tower retrieval + model ranking | 0.390 |
-| First two-stage engine (tag order + probability filter) | 0.291 |
-| X-style Thunder + Phoenix pipeline, full | 0.258 |
-| Popularity | 0.328 |
-| Learned next-intent embedding | 0.180 |
-| Random | 0.192 |
-
-Full benchmark tables (calibration, slices, ablations, significance tests, the random-interest simulation) are in [`REPORT.md`](REPORT.md).
-
----
-
-## Where the engine's parts come from (industry lineage)
-
-The engine is assembled from techniques that run in widely used production recommenders. Each part below is an established method we implemented or evaluated. This repo is an independent implementation, not affiliated with these companies.
-
-| Part of this engine | Technique | Where it is used in industry |
-|---|---|---|
-| **Cold model** | **Wide & Deep** (Cheng et al., 2016) | Introduced by Google for **Google Play** app recommendations, where it was deployed in production |
-| **Warm model** | **DeepFM** (Guo et al., 2017) | Developed by Huawei Noah's Ark Lab and evaluated on the **Huawei App Market**; a standard CTR model across ads and app stores |
-| Phase-1 benchmark model | **DCN V2** (Wang et al., 2021) | Google's deep & cross network, reported in production at **Google** for ranking |
-| Phase-1 next-post model | **SASRec** (Kang & McAuley, 2018) | Self-attentive sequential recommendation; the basis of many "next item" systems |
-| **Hybrid cold → warm blend** | content/popularity model for new users, behavioural model as history grows | The standard cold-start pattern in e-commerce and streaming feeds |
-| **Probability calibration** | **Platt scaling** | Common for CTR models in ads systems, where calibrated probabilities feed auctions |
-| **Freshness decay** | exponential time decay of a post's score | The same family as the time-decayed ranking of **Hacker News** and **Reddit "hot"** |
-| **Cold-start push** | boost for new items that fades with exposure | Used by feeds such as **TikTok** to give every new video an initial audience, and in bandit-style new-item exploration |
-| **Topic-first slots + exploration share** | slot quotas, with 15–20% exploration kept in the serving layer | Quota-based feed blending and exploration as used in large feeds (**YouTube**, **Instagram**) |
-| **Content vectors** | **Qwen3-Embedding-0.6B** (Alibaba Qwen team) | A widely used open embedding model for search and retrieval |
-| **Interest tags** | LLM tagging with a closed taxonomy (gpt-oss-120b on Groq) | LLM-generated item labels are increasingly used to cold-start catalogs |
-| Evaluated, not shipped | **Two-tower retrieval** (Yi et al., 2019) | **YouTube**'s candidate retrieval and Google's recommendation stacks |
-| Evaluated, not shipped | **Thunder + Phoenix** pipeline | Modelled on **X (Twitter)**'s open-sourced recommendation algorithm: in-network candidate source + transformer ranker |
-| Evaluated, not shipped | **MMR diversification** (Carbonell & Goldstein, 1998) | Classic re-ranking for result diversity in search and feeds |
-| Training framework | **RecBole** | Open-source library from Renmin University of China, used to train all five phase-1 models |
-
----
-
-## Model sizes (why the files are small)
-
-The trained models are deliberately small: they learn from about 1,800 rows, and a larger network would only memorise them. Parameter counts below are read from the checkpoints.
-
-| Model | Parameters | One copy (float32) | Shipped as |
-|---|---|---|---|
-| **Engine v2.0 cold model** (Wide&Deep: 4-wide embeddings → MLP 64 → 32) | 13,048 | 51 KB | `cold_stage2.onnx`, 5 seeds in one graph, 319 KB |
-| **Engine v2.0 warm model** (DeepFM: 16-wide embeddings → MLP 64 → 32) | 66,457 | 260 KB | `warm_deepfm.onnx`, 5 seeds in one graph, 1.4 MB |
-| Engine v1.1 cold / warm (with community + flair) | 15,170 / 73,979 | 59 / 289 KB | `legacy_v1_1/` |
-| Phase-1 LR / Wide&Deep / DeepFM / DCN V2 / SASRec | 37,622 / 74,167 / 111,479 / 189,189 / 30,848 | 0.1–0.7 MB | `phase1_models/` |
-| v1 two-stage engine, Stage-2 LR / Wide&Deep / DeepFM | 7,617 / 15,170 / 52,802 | 0.03–0.2 MB | `v1_two_stage_engine/` |
-
-The `.pth` checkpoints are larger than these numbers because they also store the RecBole configuration and optimizer state.
-
-**The one large model** in the pipeline is the pre-trained text encoder **[`Qwen/Qwen3-Embedding-0.6B`](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B)** (about 0.6 B parameters, 1.2 GB). It turns post text into the 1024-d content vectors. It is used as is, without fine-tuning, so it is not copied here; download it from the Qwen repository. Image captions in the training data came from Florence-2, also used as is.
-
----
-
-## Files in this repo
-
-```
-README.md                         this model card
-REPORT.md                         full project report: data, every experiment, all results, code walkthrough
-hybrid_engine/
-  engine.py                       HybridRecommender: score / rank / recommend, blend, decay, push, topic-first
-  features.py                     rebuilds the exact training features from raw post + user data
-  backend_adapter.py              connects a FastAPI + Redis + Postgres feed backend (drop-in build_pool)
-  example.py                      runnable demo
-  tests/                          smoke tests, no-community guard, backend adapter tests
-  README.md, INTEGRATION.md       usage and step-by-step backend integration
-  requirements.txt                numpy, onnxruntime
-  artifacts/
-    cold_stage2.onnx              cold model (Wide&Deep, 5 seeds averaged in one graph)
-    warm_deepfm.onnx              warm model (DeepFM, 5 seeds averaged in one graph)
-    meta.json                     vocabularies, feature order, standardisation, PCA, taxonomy, calibration, defaults
-    build_report.json             ONNX == PyTorch checks, test AUC
-    verification_report.json      feature / score parity, replay
-    backend_modes_report.json     ranking quality under backend data conditions
-    validation_two_sessions.json  the two later real sessions
-pytorch_checkpoints/engine_v2.0/  the original RecBole checkpoints of the shipped engine (5 seeds per model) for further training
-legacy_v1_1/                      the previous engine (with community + flair): ONNX, meta.json, reports, pytorch_checkpoints/ (5 seeds per model)
-phase1_models/                    the five phase-1 models (LR, DeepFM, Wide&Deep, DCN V2, SASRec): ONNX, PyTorch checkpoints,
-                                  metadata (vocabularies, standardisation, PCA, export report), Model_Explainer.docx, example_inference.py
-v1_two_stage_engine/              the first two-stage engine's Stage-2 models (LR, Wide&Deep, DeepFM): ONNX, checkpoints,
-                                  metadata (Stage-1 tag-matcher weights, feature lists, back-test, simulation and tag-quality results)
-```
-
-**Loading a PyTorch checkpoint** (RecBole 1.2.0): `torch.load(path, weights_only=False)` returns `config` and `state_dict`. Rebuilding the model needs the RecBole datasets from the training kit (GitHub repository, branch `recommender`, `models/training/`).
-
----
-
-## Training data and procedure
-
-- **Data:** Trumio Trace browser telemetry of real Reddit scrolling (impressions, visible/focal attention, hovers, clicks, votes, saves, joins, replies, opened posts).
-  - 4 consenting participants, 5 sessions, 1,794 exposures, 1,677 posts.
-  - Usernames hashed, post bodies truncated.
-  - The training data itself is not published here.
-- **Label:** engaged = click / vote / save / join / reply / opened detail, or focal dwell ≥ 3 s.
-- **Split:** chronological per user, 70% / 15% / 15% (train 1,255 rows, 308 engaged; validation 269, 65; test 270, 44).
-- **History features:** causal only. A past interaction counts only if its viewing ended before the request, and only if it happened strictly before it (the serving rule; a training/serving mismatch found here was fixed).
-- **Content:** Qwen3-Embedding-0.6B of title + body (+ Florence-2 image caption and OCR), reduced to 16 PCA components.
-- **Training:** RecBole 1.2.0, hyperparameters tuned on validation, 5 seeds per model, averaged inside one ONNX graph. The cold model gets Platt calibration fitted on validation.
-
----
-
-## Limitations and responsible use
-
-- **Small data.** 4 users and 44 positive test examples: most differences below about 0.1 AUC are within noise. Re-tune `w_max`, `n0`, decay and push on your own logs, and retrain once you have data from your platform.
-- **Trained on Reddit.** Popularity scales, post types and engagement patterns of other platforms differ.
-- **Topic-first can hide drifting interests.** If a user's engagement moves to topics outside their picked tags, the 7-of-10 rule pushes those posts down. Keep an exploration share (15–20%) in your serving layer, and monitor off-topic engagement.
-- **About 24% of posts got no tag** in the trace. Untagged posts get no closeness bonus but are still ranked by the models.
-- **It ranks; it does not retrieve or moderate.** Candidate retrieval, safety filtering and policy enforcement must happen before the engine.
-
----
-
-## Citation
-```bibtex
-@misc{unihive_recsys_2026,
-  title  = {UniHive RecSys: a hybrid cold-to-warm feed-ranking engine},
-  author = {Singh, Tirthanker},
-  year   = {2026},
-  url    = {https://huggingface.co/Tirthanker/UniHive_RecSys}
-}
-```
-Underlying methods: Cheng et al. 2016 (Wide & Deep); Guo et al. 2017 (DeepFM); Wang et al. 2021 (DCN V2); Kang & McAuley 2018 (SASRec); Yi et al. 2019 (two-tower sampling-bias correction); Platt 1999 (calibration); Carbonell & Goldstein 1998 (MMR); Zhao et al. 2021 (RecBole).
+---
+license: apache-2.0
+library_name: onnx
+language:
+- en
+pipeline_tag: tabular-classification
+tags:
+- recommender-system
+- feed-ranking
+- ctr-prediction
+- cold-start
+- deepfm
+- wide-and-deep
+- onnx
+---
+
+# UniHive RecSys: hybrid feed-ranking engine (v2.0)
+
+A small, fast **post recommendation engine** for a student social app. Give it a user (interests + past interactions) and a list of candidate posts; it returns the posts in feed order, with a score and an explanation for each.
+
+> ### ✅ This repository is engine v2.0: our final, intended design. Use `hybrid_engine/`.
+> - **Run it:** `hybrid_engine/` (code) + `hybrid_engine/artifacts/` (the weights it loads). Nothing else is needed.
+> - **Train it further:** `pytorch_checkpoints/engine_v2.0/` (the same v2.0 models as PyTorch files).
+> - **Ignore `archive/`** unless you are studying the project's history. Those are superseded experiments, built on different inputs, and they will not work with `hybrid_engine/`.
+
+---
+
+## Contents
+1. [Which files you need](#1-which-files-you-need)
+2. [Install and run in 2 minutes](#2-install-and-run-in-2-minutes)
+3. [Use it in your code](#3-use-it-in-your-code)
+4. [Content vectors (recommended)](#4-content-vectors-recommended)
+5. [Inputs and outputs](#5-inputs-and-outputs)
+6. [Settings](#6-settings)
+7. [Plug it into a backend](#7-plug-it-into-a-backend)
+8. [Continue training](#8-continue-training)
+9. [How it works](#9-how-it-works)
+10. [Scores](#10-scores)
+11. [Model size](#11-model-size)
+12. [Where the engine's parts come from](#12-where-the-engines-parts-come-from)
+13. [Training data and limitations](#13-training-data-and-limitations)
+14. [Archive (older versions)](#14-archive-older-versions)
+
+---
+
+## 1. Which files you need
+
+| You want to... | Files |
+|---|---|
+| **Rank posts** (most users) | `hybrid_engine/` (code + `artifacts/cold_stage2.onnx`, `artifacts/warm_deepfm.onnx`, `artifacts/meta.json`) |
+| Turn post text into content vectors | the public encoder [`Qwen/Qwen3-Embedding-0.6B`](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) (section 4) |
+| Continue training the models | `pytorch_checkpoints/engine_v2.0/` (section 8) |
+| Read the full project report | `REPORT.md` |
+
+The engine reads exactly the files in `hybrid_engine/artifacts/`. Don't replace them with files from `archive/`: the feature lists, vocabularies and standardisation differ.
+
+---
+
+## 2. Install and run in 2 minutes
+
+```bash
+pip install huggingface_hub numpy onnxruntime
+hf download Tirthanker/UniHive_RecSys --include "hybrid_engine/*" --local-dir UniHive_RecSys
+cd UniHive_RecSys
+python -m hybrid_engine.example                 # demo: a brand-new user vs the same user after 30 engaged posts
+python -m hybrid_engine.tests.test_engine       # smoke tests; should end with "community / flair guard passed"
+```
+Python 3.10+. Runs on CPU; no GPU needed.
+
+---
+
+## 3. Use it in your code
+
+```python
+import time
+from hybrid_engine import HybridRecommender
+
+engine = HybridRecommender()          # load once at startup; reuse for every request
+
+user = {
+    "interests": ["machine-learning", "computer-vision"],    # tag slugs the user picked (list: meta.json -> taxonomy.child)
+    "history": [                                              # what the user saw before; may be empty for a new user
+        {"post_id": "p1", "ts_ms": 1791141039157, "visible_ms": 5200, "focal_ms": 4800, "hover_ms": 300,
+         "clicked": True, "engaged": True, "community": "c-ml", "author": "a-17", "embedding": None},
+    ],
+}
+candidates = [
+    {"post_id": "p9", "post_type": "text", "score": 42, "comments": 7, "age_hours": 3.5,
+     "title": "Fine-tuning ViTs on a laptop GPU", "body": "Notes from a weekend project ...",
+     "tags": {"computer-vision": 0.9, "deep-learning": 0.7}, "embedding": None,
+     "author": "a-3", "impressions": 0, "community": "c-cv"},
+    {"post_id": "p10", "post_type": "image", "score": 300, "comments": 40, "age_hours": 50,
+     "title": "Campus fest photos", "body": "", "tags": {"photography": 0.8}, "embedding": None,
+     "author": "a-8", "impressions": 120, "community": "c-fest"},
+]
+
+feed = engine.recommend(user, candidates, now_ms=time.time() * 1000, k=10)
+for d in feed:
+    print(d["position"], d["post_id"], round(d["score"], 3), "topic" if d["topic_match"] else "", "warm weight", d["warm_weight"])
+```
+- `recommend(user, candidates, now_ms, k)`: the top k in final feed order.
+- `rank(...)`: every candidate in feed order.
+- `score(...)`: raw scores without the topic-first reordering.
+
+`now_ms` is the request time in milliseconds. History items at or after it are ignored automatically.
+
+---
+
+## 4. Content vectors (recommended)
+
+The models read a 1024-d content vector per post. Build it **exactly** like this (the same recipe the models were trained on):
+
+```python
+from sentence_transformers import SentenceTransformer        # pip install sentence-transformers
+from hybrid_engine.backend_adapter import qwen3_text
+
+encoder = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B"); encoder.max_seq_length = 512   # load once
+text = qwen3_text(title, body, image_caption)                 # "title. body" (+ " Image: caption"); no community, no flair
+post["embedding"] = encoder.encode(text, normalize_embeddings=True)
+```
+- Compute it once per post when the post is created, and store it.
+- Without vectors (`"embedding": None`) the engine still ranks, but loses the content signal: about 0.02 lower NDCG@10 on average (section 10).
+- A vector of the wrong size (e.g. a 512-d CLIP vector) is ignored on purpose: it lives in a different space.
+
+---
+
+## 5. Inputs and outputs
+
+**Candidate post**
+
+| Field | Type | Meaning |
+|---|---|---|
+| `post_id` | str | your id |
+| `post_type` | str | `text`, `image`, `video`, `link`, `gallery`, ... |
+| `score`, `comments` | int | likes / upvotes and comment count |
+| `age_hours` | float | hours since the post was created |
+| `title`, `body`, `image_caption` | str | used for length features (and for the content vector, section 4) |
+| `tags` | dict or list | `{slug: confidence}` from the 114-tag taxonomy, or a list of slugs (confidence 1.0) |
+| `embedding` | 1024 floats or None | section 4 |
+| `author`, `community` | str | used only for the user's *own* history with that author / community, never as model inputs |
+| `impressions` | int | how often your platform has shown the post (drives the new-post push) |
+| optional | | `domain`, `language`, `is_ad`, `is_spoiler`, `upvote_ratio`, `awards`, `is_member` |
+
+**History item:** `post_id`, `ts_ms` (when shown), `visible_ms`, `focal_ms` (attention), `hover_ms`, `clicked`, `engaged`, `community`, `author`, `embedding`. Set `engaged` to True for dwell ≥ 3 s or a click / like / save / join / reply. If you log only one dwell value, use it for both `visible_ms` and `focal_ms`, and set `hover_ms` to 0.
+
+**Output per candidate**
+- `score`, `position`, `topic_match`;
+- `v2b`, `cold_probability`, `tag_closeness`;
+- `warm_probability`, `warm_weight`, `engaged_history`;
+- `freshness_multiplier`, `push`, `has_content_embedding`.
+
+Log them to explain any feed.
+
+---
+
+## 6. Settings
+
+`HybridRecommender(**overrides)`; the defaults are stored in `hybrid_engine/artifacts/meta.json`.
+
+| Setting | Default | Effect |
+|---|---|---|
+| `n0_engaged_posts` | 10 | engaged posts at which the warm model reaches half of its maximum share |
+| `w_max` | 0.7 | maximum warm-model share (the rest stays with the new-user model) |
+| `tag_bonus` | 0.1 | weight of interest-tag closeness in the new-user score |
+| `decay_half_life_hours`, `decay_floor` | 720, 0.85 | freshness: a week-old post keeps 98%, a month-old 93%, never below 85% |
+| `push_beta`, `push_max_age_hours` | 0.1, 24 | lift for posts under 24 h old, fading with impressions |
+| `topic_first_slots`, `topic_block` | 7, 10 | 7 of every 10 positions go to posts matching the user's interests (0 = off) |
+
+Example: `HybridRecommender(w_max=0.5, topic_first_slots=5)`.
+
+---
+
+## 7. Plug it into a backend
+
+`hybrid_engine/backend_adapter.py` maps the data of a FastAPI + Redis + Postgres feed service onto the engine:
+- **Posts:** `post_from_meta`, `post_from_row` (backend posts → candidates).
+- **History:** `history_from_events` (raw VIEW / LONG_VIEW / CLICK / LIKE / SAVE / SKIP / HIDE events → history, merging visits and correcting timestamps).
+- **Interests:** `expand_interests` (broad interests → tags).
+- **Ranking:** `build_pool_hybrid`, a drop-in replacement for a `build_pool(r, uid, state, level)` scoring step.
+- **Vectors in Redis:** `encode_q3`, `fetch_q3`.
+
+Step-by-step guide: [`hybrid_engine/INTEGRATION.md`](hybrid_engine/INTEGRATION.md). Speed: about 0.7 ms per candidate on one CPU thread, so a 200-post pool takes about 140 ms.
+
+---
+
+## 8. Continue training
+
+`pytorch_checkpoints/engine_v2.0/` holds the RecBole 1.2.0 originals of both models, 5 seeds each (`cold_widedeep_seed0-4.pth`, `warm_deepfm_seed0-4.pth`). The shipped ONNX files average these 5 seeds.
+
+```bash
+pip install torch recbole==1.2.0      # the checkpoints store a RecBole config, so RecBole must be importable to load them
+```
+```python
+import torch
+ckpt = torch.load("pytorch_checkpoints/engine_v2.0/warm_deepfm_seed0.pth", map_location="cpu", weights_only=False)
+print(ckpt["config"]["model"], ckpt["config"]["dataset"])          # DeepFM reco_ctr_nc_noid_strict
+state = ckpt["state_dict"]                                         # weights
+```
+Rebuilding and retraining needs the RecBole datasets and scripts of the training kit: GitHub repository, branch `recommender`, folder `models/training/`. Use `engine_v2/build_engine_v2.py`, then `engine_v2/verify_hybrid_engine.py`. After retraining, re-export **both** ONNX files and `meta.json` together.
+
+---
+
+## 9. How it works
+
+```
+candidate posts ─┬─► tag closeness (8 tag-match features → logistic score)
+                 ├─► COLD model, Wide&Deep: post stats + content + tag match → calibrated P(engage)
+                 └─► WARM model, DeepFM: post inputs + 27 user-history features → P(engage)
+
+v2b   = P_cold + 0.1 · rank(tag closeness)                              (the new-user score)
+w     = 0 for a user with no engaged posts, else min(0.7, engaged / (engaged + 10))
+blend = (1 − w) · rank(v2b) + w · rank(P_warm)
+score = blend · (0.85 + 0.15 · 0.5^(age_h / 720))  +  0.1 · [age_h ≤ 24] / sqrt(1 + impressions)
+order = in each block of 10 positions: 7 best interest matches, then 3 best of any kind
+```
+- **New users:** w = 0, so pure v2b.
+- **The warm model takes over gradually:** w = 0.09 after 1 engaged post, 0.33 after 5, 0.50 after 10, and 0.70 at most. The new-user model always keeps at least 30%.
+- **Excluded from the models:** user and post IDs, feed position, scroll speed, community name and flair.
+
+---
+
+## 10. Scores
+
+All numbers are for v2.0, on data the models never trained on.
+- **AUC:** chance an engaged post is scored above a non-engaged one (0.5 = random).
+- **NDCG@10:** quality of the top 10 (1.0 = all engaged posts first).
+
+**Models** (test split: last 15% of each user's timeline, 270 exposures, 44 engaged)
+
+| Model | Validation AUC | Test AUC |
+|---|---|---|
+| Cold model (Wide&Deep, 5-seed ensemble) | 0.777 | **0.602** |
+| Warm model (DeepFM, 5-seed ensemble) | 0.713 | **0.565** |
+
+**Ranking quality** (NDCG@10; each feed ranked as one live request)
+
+| Feed | v2.0 | v2.0 with typical backend data | same, no content vectors | Random |
+|---|---|---|---|---|
+| Validation period, whole feeds | **0.650** | 0.579 | 0.604 | 0.350 |
+| Validation period, 25-post pages | 0.559 | 0.595 | 0.582 | 0.350 |
+| Test period, whole feeds | **0.340** | 0.329 | 0.268 | 0.201 |
+| Test period, 25-post pages | 0.415 | 0.397 | 0.441 | 0.286 |
+| Later real session 1, one feed | 0.255 | 0.159 | 0.155 | 0.119 |
+| Later real session 1, 25-post pages | 0.468 | 0.486 | 0.362 | 0.282 |
+| Later real session 2, one feed | 0.110 | 0.110 | 0.110 | 0.079 |
+| Later real session 2, 25-post pages | 0.403 | 0.382 | 0.372 | 0.245 |
+| **Mean** | **0.400** | **0.380** | **0.362** | **0.239** |
+
+"Typical backend data" means no flair, tags without confidences, a single dwell value and no vote ratio.
+
+**The two later real sessions** (recorded two days after all training data)
+
+| | Session 1 (89 posts, 11 engaged) | Session 2 (142 posts, 10 engaged) |
+|---|---|---|
+| v2.0: NDCG@10 / engaged in top 10 / AUC | 0.255 / 3 / 0.688 | 0.110 / 1 / 0.498 |
+| Reddit's own feed order | 0.641 / 6 / 0.917 | 0.312 / 3 / 0.773 |
+| Random | 0.124 / 1.1 / 0.50 | 0.070 / 0.6 / 0.50 |
+
+**Engineering checks**
+- ONNX equals PyTorch within 2.4e-7.
+- The engine rebuilds the training features from raw data within 0.0021, with 0 category mismatches.
+- Engine scores match the offline predictions within 2.5e-4.
+- 9 automated tests pass.
+
+**Compared with the previous version:** v1.1 used the community name and scored higher on its Reddit training data (test AUC 0.692 / 0.671, mean NDCG@10 0.446). Community names don't carry over to a new platform. Under realistic conditions the gap is small: 0.380 for v2.0 vs 0.410 for v1.1, and 0.362 vs 0.369 without content vectors. Details are in `REPORT.md`, section 13.
+
+---
+
+## 11. Model size
+
+| Model | Parameters | Shipped file |
+|---|---|---|
+| Cold model (Wide&Deep: 4-wide embeddings → MLP 64 → 32) | 13,048 per seed | `cold_stage2.onnx`, 5 seeds, 319 KB |
+| Warm model (DeepFM: 16-wide embeddings → MLP 64 → 32) | 66,457 per seed | `warm_deepfm.onnx`, 5 seeds, 1.4 MB |
+
+The models are small on purpose: they learn from about 1,800 interactions, and a larger network would only memorise them. The `.pth` files are bigger than the parameter count because they also store the training configuration and optimizer state. The only large model in the pipeline is the public **Qwen3-Embedding-0.6B** text encoder (about 1.2 GB). It is used as is and downloaded from its own repository, so it is not copied here.
+
+---
+
+## 12. Where the engine's parts come from
+
+This is an independent implementation built from methods used in well-known production recommenders. It is not affiliated with any of these companies.
+
+| Part | Method | Known use |
+|---|---|---|
+| Cold model | **Wide & Deep** (Cheng et al., 2016) | Google Play app recommendations (Google) |
+| Warm model | **DeepFM** (Guo et al., 2017) | Huawei App Market CTR prediction (Huawei Noah's Ark Lab) |
+| Cold → warm blend | new-user model handing over to a behavioural model as history grows | the standard cold-start pattern of e-commerce and streaming feeds |
+| Calibration | **Platt scaling** | CTR models in online advertising |
+| Freshness decay | exponential time decay | the family of Hacker News and Reddit "hot" ranking |
+| New-post push | a boost for new items that fades with exposure | initial-audience boosts for new content in short-video feeds such as TikTok |
+| Topic-first slots + exploration share | slot quotas, with 15–20% exploration kept in the serving layer | quota-based feed blending in large feeds (YouTube, Instagram) |
+| Content vectors | **Qwen3-Embedding-0.6B** | Alibaba Qwen's open embedding model |
+| Interest tags | LLM tagging into a closed taxonomy | cold-starting item catalogs with LLM labels |
+| Evaluated during research, not in v2.0 | two-tower retrieval (YouTube), X's Thunder + Phoenix pipeline, MMR diversification, DCN V2 (Google), SASRec | see `REPORT.md` |
+
+---
+
+## 13. Training data and limitations
+
+**Data**
+- Browser telemetry of real Reddit scrolling: 4 consenting participants, 1,794 post exposures, 1,677 posts. Usernames were hashed. The data is not published.
+- **Label:** engaged = click / vote / save / join / reply / opened post, or attention ≥ 3 s.
+- **Split:** chronological per user (70 / 15 / 15).
+- **No look-ahead:** history features only count interactions that ended before the ranked post was shown.
+
+**Limitations**
+- **Small data.** Differences below about 0.1 AUC are within noise. Treat the settings as a starting point and re-tune `w_max`, `n0_engaged_posts` and decay/push on your own logs.
+- **Trained on Reddit.** Other platforms differ; retrain on your own data when you have it.
+- **The topic-first rule can hide a user's new interests.** Keep a 15–20% exploration share in your serving layer.
+- **It ranks; it doesn't retrieve or moderate.** Filter candidates for safety before ranking.
+
+---
+
+## 14. Archive (older versions)
+
+Not for use. `archive/` holds superseded research models, kept only so the history in `REPORT.md` can be checked. Everything in this README, and everything `hybrid_engine/` loads, is v2.0. See [`archive/README.md`](archive/README.md).
+
+---
+
+```bibtex
+@misc{unihive_recsys_2026,
+  title  = {UniHive RecSys: a hybrid cold-to-warm feed-ranking engine},
+  author = {Singh, Tirthanker},
+  year   = {2026},
+  url    = {https://huggingface.co/Tirthanker/UniHive_RecSys}
+}
+```

archive/README.md

@@ -0,0 +1,11 @@
+# Archive: earlier models (reference only)
+
+**Do not use these files with `hybrid_engine/`.** The current model is v2.0 in `hybrid_engine/artifacts/` (see the main README). The models here were trained on different inputs (they use community name and flair, user/post IDs or feed position) and have their own vocabularies and standardisation. They are kept so the results in `REPORT.md` can be checked and compared.
+
+| Folder | What it is | Status |
+|---|---|---|
+| `engine_v1_1/` | the previous hybrid engine (same design, but its models used community name + flair). ONNX, `meta.json`, reports, `pytorch_checkpoints/` (5 seeds per model) | replaced by v2.0 |
+| `phase1_models/` | the first five models (LR, DeepFM, Wide&Deep, DCN V2, SASRec), trained with user/post IDs and feed position. ONNX, checkpoints, metadata, `Model_Explainer.docx`, `example_inference.py` | research baseline |
+| `v1_two_stage_engine/` | Stage-2 models of the first two-stage engine (tag match, then model). ONNX, checkpoints, Stage-1 weights and results | research baseline, outperformed |
+
+The engine v1.1 files can be loaded on their own with `HybridRecommender(artifacts_dir="archive/engine_v1_1")`, but that version needs content vectors built with its own text recipe ("title. community. flair. body"), not the v2.0 one.