UniHive RecSys hybrid engine v2.0: ONNX models, code, checkpoints, model card, full report

1f9a1bf verified 27 days ago

7.63 kB

# Integrating the hybrid engine into the Unihive backend

The engine is a plain Python package (numpy + onnxruntime). `backend_adapter.py` connects it to the backend's own data. It plugs into the feed at one place: **where `backend/reco/service.py::build_pool` scores candidate posts**. Everything else stays as it is: candidate retrieval, the explore lane, interleaving and the queue.
```
GET /reco/feed ──► service.next_page ──► refill_queue ──► build_pool(r, uid, state, level)
                                                              │  candidate ids (tag indexes in Redis, widened by graph.expand_weights)
                                                              │  minus seen posts, minus own posts
                                                              ▼
                                         today: scoring.score_post(...)  per post
                                         new:   backend_adapter.build_pool_hybrid(...)  one engine call per pool
                                                              ▼
                                         compose.block_sizes / pick_explore / interleave   (unchanged)
```
## What was checked against the backend code
| Backend piece (file) | Shape | Adapter function |
|---|---|---|
| `reco/store.py::posts_meta` | `{pid: {tags: frozenset, cat, created, likes, comments, views, author}}` | `post_from_meta`: likes→score, views→impressions, created→age_hours, tags→list (confidence 1.0) |
| `posts` + `post_reco` rows (`db/trumio_schema_dump.sql`, migration 005) | community_id, post_type (TEXT/LINK/MEDIA/NEWS/REPOST/QUESTION/POLL), title, body, url, counters, created_at, tag_slugs | `post_from_row`: post types mapped to the trained ones; domain parsed from url |
| `reco/store.py::get_state` | `{embedding, tag_weights, base_tags, version, synced_at}` | `user_from_backend`: `base_tags`; broad picks ("ai-ml") expand to their child tags |
| Redis burst buffer (`ingest_events` → `buf_push`) | `{type, post_id, dwell_ms, t, surface, context}` | `peek_buffer` (reads without draining), `history_from_events` |
| `interaction_events` (schema dump) | user_id, event_type, entity_type, entity_id, dwell_ms, created_at | `HISTORY_SQL` + `load_history` (adds the post's community_id and author_id) |
| Tag vocabulary (`reco/vocab.py`, `crud/taxonomy.py`) | 13 broad + 114 child tags | identical to the engine's taxonomy (checked slug by slug) |
**Event mapping.** Events on the same post within 30 minutes form one interaction. **Engaged** means any of LONG_VIEW, CLICK, SAVE, SHARE, LIKE, VOTE, COMMENT, REPLY, JOIN or POLL_VOTE, or dwell ≥ 3 s. An explicit SKIP, HIDE or NOT_INTERESTED in the same visit overrides this. The backend stamps an event when it is *received*, so the adapter sets `ts_ms = received - dwell`, the time the post was shown. The engine's "viewing had ended before the request" rule then holds exactly.
## Step 1. Store a Qwen3 vector for every post (tagging worker)
`reco/tagger.py::process_post` already tags posts and computes CLIP vectors. Add one call:
```python
from hybrid_engine.backend_adapter import qwen3_text, encode_q3, q3_key
Q3 = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B"); Q3.max_seq_length = 512           # load once in the worker
vec = Q3.encode(qwen3_text(post["title"], post["body"], image_caption), normalize_embeddings=True)
await r.set(q3_key(post_id), encode_q3(vec))                                               # reco:post:{id}:q3, float16, 2 KB
```
- The text is **title + body (+ image caption)**, with no community and no flair. Build it with `qwen3_text` so it matches training exactly.
- The backend's CLIP-512 vectors are a different space. The engine refuses them and treats such posts as "content unknown".
- **Until this step is done the engine still works.** Posts without the vector get neutral content features. The cost is measured in REPORT.md: about 0.02 NDCG@10 on average, more on some feeds.

## Step 2. Call the engine in `build_pool`
Load it once at startup. Then let `build_pool` delegate (same arguments, same return value, same Redis writes):
```python
# backend/reco/service.py
from hybrid_engine import HybridRecommender
from hybrid_engine import backend_adapter
ENGINE = HybridRecommender()          # ~2 MB of ONNX + metadata, loaded once per process
async def build_pool(r, uid, state, level):
    async with use_read_conn() as conn:                    # persisted events; pass conn=None to use only the Redis buffer
        return await backend_adapter.build_pool_hybrid(r, uid, state, level, engine=ENGINE, conn=conn,
                                                       candidates_per_tag=CANDIDATES_PER_TAG, max_query_tags=MAX_QUERY_TAGS)
```
- Retrieval is the backend's own code path: `graph.expand_weights`, the tag indexes, `seen_among`, the own-post filter and `compose.split_counts`. Only the scoring changes. The test runs both functions on the same fakeredis data and checks that they return the same candidates.
- The pool score is the engine's position (strictly decreasing), so the `reco:u:{uid}:pool` ZSET keeps the engine's order, including the topic-first blocks of 10.
- For explanations, call `backend_adapter.rank_pool(...)` directly. It also returns the engine's per-post dicts (`topic_match`, `warm_weight`, `cold_probability`, `warm_probability`, `freshness_multiplier`, `push`, `has_content_embedding`) for `log_served`.
- **Exploration stays where it is.** `compose.pick_explore` still reserves out-of-interest slots (keep it at 15–20% of each block). The engine ranks the interest pool.
- Add `onnxruntime>=1.18` to `backend/requirements.txt`. The Docker image is `python:3.12-slim`, which onnxruntime supports. The worker also needs `sentence-transformers` for step 1.

## Step 3. Latency and caching
About 0.7 ms per candidate on one CPU thread, including the history features. A 200-post pool takes about 140 ms. Run it in `asyncio.to_thread` if the event loop must stay free. Cache per user: the loaded history. Cache per post: the 16 content components (`engine.fb.pca16(vec)`).

## Step 4. Roll out safely
1. **Shadow mode.** Compute engine scores next to the current `score_post` and log both, without changing the feed.
2. **Offline replay** on your own logs. Re-tune `w_max`, `n0_engaged_posts`, `decay_*` and `push_*` on earlier days, then check on later days.
3. **A/B test** (or interleaving) on a share of users. Primary metric: engaged posts per session. Guardrails: diversity, freshness, share of posts never shown, and engagement on off-topic posts (the topic-first rule can hide them if a user's interests drift).
4. **Retrain on Unihive data** once there are enough logged sessions. Use the training kit in `models/training/` and `engine_v2/build_engine_v2.py`. Build the datasets with `prep_recbole.py` (`EMB_NPZ=… META_DIR=…`, plus `MASK_IDS=1 STRICT_PAST=1` for the warm model). Then re-run `engine_v2/verify_hybrid_engine.py`: parity must stay near zero.

## Known limits
- Trained on a 4-user Reddit trace. Re-tune on your own data before trusting the defaults.
- The DeepFM share only matters after a user has engaged posts; brand-new users are served by v2b alone.
- The backend logs one dwell value per event (no hover or separate focal time). The adapter uses it for both; REPORT.md measures the effect.
- About 24% of posts got no tag in the trace. Untagged posts get no tag-closeness bonus but are still ranked by the models.
- `HISTORY_SQL` was checked against the schema dump and tested with a fake connection, not against a live Postgres (Docker was not running).

	# Integrating the hybrid engine into the Unihive backend

	The engine is a plain Python package (numpy + onnxruntime). `backend_adapter.py` connects it to the backend's own data. It plugs into the feed at one place: where `backend/reco/service.py::build_pool` scores candidate posts. Everything else stays as it is: candidate retrieval, the explore lane, interleaving and the queue.

	```
	GET /reco/feed ──► service.next_page ──► refill_queue ──► build_pool(r, uid, state, level)
	│ candidate ids (tag indexes in Redis, widened by graph.expand_weights)
	│ minus seen posts, minus own posts
	▼
	today: scoring.score_post(...) per post
	new: backend_adapter.build_pool_hybrid(...) one engine call per pool
	▼
	compose.block_sizes / pick_explore / interleave (unchanged)
	```

	## What was checked against the backend code
	\| Backend piece (file) \| Shape \| Adapter function \|
	\|---\|---\|---\|
	\| `reco/store.py::posts_meta` \| `{pid: {tags: frozenset, cat, created, likes, comments, views, author}}` \| `post_from_meta`: likes→score, views→impressions, created→age_hours, tags→list (confidence 1.0) \|
	\| `posts` + `post_reco` rows (`db/trumio_schema_dump.sql`, migration 005) \| community_id, post_type (TEXT/LINK/MEDIA/NEWS/REPOST/QUESTION/POLL), title, body, url, counters, created_at, tag_slugs \| `post_from_row`: post types mapped to the trained ones; domain parsed from url \|
	\| `reco/store.py::get_state` \| `{embedding, tag_weights, base_tags, version, synced_at}` \| `user_from_backend`: `base_tags`; broad picks ("ai-ml") expand to their child tags \|
	\| Redis burst buffer (`ingest_events` → `buf_push`) \| `{type, post_id, dwell_ms, t, surface, context}` \| `peek_buffer` (reads without draining), `history_from_events` \|
	\| `interaction_events` (schema dump) \| user_id, event_type, entity_type, entity_id, dwell_ms, created_at \| `HISTORY_SQL` + `load_history` (adds the post's community_id and author_id) \|
	\| Tag vocabulary (`reco/vocab.py`, `crud/taxonomy.py`) \| 13 broad + 114 child tags \| identical to the engine's taxonomy (checked slug by slug) \|

	Event mapping. Events on the same post within 30 minutes form one interaction. Engaged means any of LONG_VIEW, CLICK, SAVE, SHARE, LIKE, VOTE, COMMENT, REPLY, JOIN or POLL_VOTE, or dwell ≥ 3 s. An explicit SKIP, HIDE or NOT_INTERESTED in the same visit overrides this. The backend stamps an event when it is received, so the adapter sets `ts_ms = received - dwell`, the time the post was shown. The engine's "viewing had ended before the request" rule then holds exactly.

	## Step 1. Store a Qwen3 vector for every post (tagging worker)
	`reco/tagger.py::process_post` already tags posts and computes CLIP vectors. Add one call:
	```python
	from hybrid_engine.backend_adapter import qwen3_text, encode_q3, q3_key
	Q3 = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B"); Q3.max_seq_length = 512 # load once in the worker
	vec = Q3.encode(qwen3_text(post["title"], post["body"], image_caption), normalize_embeddings=True)
	await r.set(q3_key(post_id), encode_q3(vec)) # reco:post:{id}:q3, float16, 2 KB
	```
	- The text is title + body (+ image caption), with no community and no flair. Build it with `qwen3_text` so it matches training exactly.
	- The backend's CLIP-512 vectors are a different space. The engine refuses them and treats such posts as "content unknown".
	- Until this step is done the engine still works. Posts without the vector get neutral content features. The cost is measured in REPORT.md: about 0.02 NDCG@10 on average, more on some feeds.

	## Step 2. Call the engine in `build_pool`
	Load it once at startup. Then let `build_pool` delegate (same arguments, same return value, same Redis writes):
	```python
	# backend/reco/service.py
	from hybrid_engine import HybridRecommender
	from hybrid_engine import backend_adapter
	ENGINE = HybridRecommender() # ~2 MB of ONNX + metadata, loaded once per process

	async def build_pool(r, uid, state, level):
	async with use_read_conn() as conn: # persisted events; pass conn=None to use only the Redis buffer
	return await backend_adapter.build_pool_hybrid(r, uid, state, level, engine=ENGINE, conn=conn,
	candidates_per_tag=CANDIDATES_PER_TAG, max_query_tags=MAX_QUERY_TAGS)
	```
	- Retrieval is the backend's own code path: `graph.expand_weights`, the tag indexes, `seen_among`, the own-post filter and `compose.split_counts`. Only the scoring changes. The test runs both functions on the same fakeredis data and checks that they return the same candidates.
	- The pool score is the engine's position (strictly decreasing), so the `reco:u:{uid}:pool` ZSET keeps the engine's order, including the topic-first blocks of 10.
	- For explanations, call `backend_adapter.rank_pool(...)` directly. It also returns the engine's per-post dicts (`topic_match`, `warm_weight`, `cold_probability`, `warm_probability`, `freshness_multiplier`, `push`, `has_content_embedding`) for `log_served`.
	- Exploration stays where it is. `compose.pick_explore` still reserves out-of-interest slots (keep it at 15–20% of each block). The engine ranks the interest pool.
	- Add `onnxruntime>=1.18` to `backend/requirements.txt`. The Docker image is `python:3.12-slim`, which onnxruntime supports. The worker also needs `sentence-transformers` for step 1.

	## Step 3. Latency and caching
	About 0.7 ms per candidate on one CPU thread, including the history features. A 200-post pool takes about 140 ms. Run it in `asyncio.to_thread` if the event loop must stay free. Cache per user: the loaded history. Cache per post: the 16 content components (`engine.fb.pca16(vec)`).

	## Step 4. Roll out safely
	1. Shadow mode. Compute engine scores next to the current `score_post` and log both, without changing the feed.
	2. Offline replay on your own logs. Re-tune `w_max`, `n0_engaged_posts`, `decay_` and `push_` on earlier days, then check on later days.
	3. A/B test (or interleaving) on a share of users. Primary metric: engaged posts per session. Guardrails: diversity, freshness, share of posts never shown, and engagement on off-topic posts (the topic-first rule can hide them if a user's interests drift).
	4. Retrain on Unihive data once there are enough logged sessions. Use the training kit in `models/training/` and `engine_v2/build_engine_v2.py`. Build the datasets with `prep_recbole.py` (`EMB_NPZ=… META_DIR=…`, plus `MASK_IDS=1 STRICT_PAST=1` for the warm model). Then re-run `engine_v2/verify_hybrid_engine.py`: parity must stay near zero.

	## Known limits
	- Trained on a 4-user Reddit trace. Re-tune on your own data before trusting the defaults.
	- The DeepFM share only matters after a user has engaged posts; brand-new users are served by v2b alone.
	- The backend logs one dwell value per event (no hover or separate focal time). The adapter uses it for both; REPORT.md measures the effect.
	- About 24% of posts got no tag in the trace. Untagged posts get no tag-closeness bonus but are still ranked by the models.
	- `HISTORY_SQL` was checked against the schema dump and tested with a fake connection, not against a live Postgres (Docker was not running).