hybrid_engine/INTEGRATION.md

Integrating the hybrid engine into the Unihive backend

The engine is a plain Python package (numpy + onnxruntime). backend_adapter.py connects it to the backend's own data. It plugs into the feed at one place: where backend/reco/service.py::build_pool scores candidate posts. Everything else stays as it is: candidate retrieval, the explore lane, interleaving and the queue.

GET /reco/feed ──► service.next_page ──► refill_queue ──► build_pool(r, uid, state, level)
                                                              │  candidate ids (tag indexes in Redis, widened by graph.expand_weights)
                                                              │  minus seen posts, minus own posts
                                                              ▼
                                         today: scoring.score_post(...)  per post
                                         new:   backend_adapter.build_pool_hybrid(...)  one engine call per pool
                                                              ▼
                                         compose.block_sizes / pick_explore / interleave   (unchanged)

What was checked against the backend code

Backend piece (file)	Shape	Adapter function
reco/store.py::posts_meta	{pid: {tags: frozenset, cat, created, likes, comments, views, author}}	post_from_meta: likes→score, views→impressions, created→age_hours, tags→list (confidence 1.0)
posts + post_reco rows (db/trumio_schema_dump.sql, migration 005)	community_id, post_type (TEXT/LINK/MEDIA/NEWS/REPOST/QUESTION/POLL), title, body, url, counters, created_at, tag_slugs	post_from_row: post types mapped to the trained ones; domain parsed from url
reco/store.py::get_state	{embedding, tag_weights, base_tags, version, synced_at}	user_from_backend: base_tags; broad picks ("ai-ml") expand to their child tags
Redis burst buffer (ingest_events → buf_push)	{type, post_id, dwell_ms, t, surface, context}	peek_buffer (reads without draining), history_from_events
interaction_events (schema dump)	user_id, event_type, entity_type, entity_id, dwell_ms, created_at	HISTORY_SQL + load_history (adds the post's community_id and author_id)
Tag vocabulary (reco/vocab.py, crud/taxonomy.py)	13 broad + 114 child tags	identical to the engine's taxonomy (checked slug by slug)

Event mapping. Events on the same post within 30 minutes form one interaction. Engaged means any of LONG_VIEW, CLICK, SAVE, SHARE, LIKE, VOTE, COMMENT, REPLY, JOIN or POLL_VOTE, or dwell ≥ 3 s. An explicit SKIP, HIDE or NOT_INTERESTED in the same visit overrides this. The backend stamps an event when it is received, so the adapter sets ts_ms = received - dwell, the time the post was shown. The engine's "viewing had ended before the request" rule then holds exactly.

Step 1. Store a Qwen3 vector for every post (tagging worker)

reco/tagger.py::process_post already tags posts and computes CLIP vectors. Add one call:

from hybrid_engine.backend_adapter import qwen3_text, encode_q3, q3_key
Q3 = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B"); Q3.max_seq_length = 512           # load once in the worker
vec = Q3.encode(qwen3_text(post["title"], post["body"], image_caption), normalize_embeddings=True)
await r.set(q3_key(post_id), encode_q3(vec))                                               # reco:post:{id}:q3, float16, 2 KB

• The text is title + body (+ image caption), with no community and no flair. Build it with qwen3_text so it matches training exactly.
• The backend's CLIP-512 vectors are a different space. The engine refuses them and treats such posts as "content unknown".
• Until this step is done the engine still works. Posts without the vector get neutral content features. The cost is measured in REPORT.md: about 0.02 NDCG@10 on average, more on some feeds.

Step 2. Call the engine in build_pool

Load it once at startup. Then let build_pool delegate (same arguments, same return value, same Redis writes):

# backend/reco/service.py
from hybrid_engine import HybridRecommender
from hybrid_engine import backend_adapter
ENGINE = HybridRecommender()          # ~2 MB of ONNX + metadata, loaded once per process

async def build_pool(r, uid, state, level):
    async with use_read_conn() as conn:                    # persisted events; pass conn=None to use only the Redis buffer
        return await backend_adapter.build_pool_hybrid(r, uid, state, level, engine=ENGINE, conn=conn,
                                                       candidates_per_tag=CANDIDATES_PER_TAG, max_query_tags=MAX_QUERY_TAGS)

• Retrieval is the backend's own code path: graph.expand_weights, the tag indexes, seen_among, the own-post filter and compose.split_counts. Only the scoring changes. The test runs both functions on the same fakeredis data and checks that they return the same candidates.
• The pool score is the engine's position (strictly decreasing), so the reco:u:{uid}:pool ZSET keeps the engine's order, including the topic-first blocks of 10.
• For explanations, call backend_adapter.rank_pool(...) directly. It also returns the engine's per-post dicts (topic_match, warm_weight, cold_probability, warm_probability, freshness_multiplier, push, has_content_embedding) for log_served.
• Exploration stays where it is. compose.pick_explore still reserves out-of-interest slots (keep it at 15–20% of each block). The engine ranks the interest pool.
• Add onnxruntime>=1.18 to backend/requirements.txt. The Docker image is python:3.12-slim, which onnxruntime supports. The worker also needs sentence-transformers for step 1.

Step 3. Latency and caching

About 0.7 ms per candidate on one CPU thread, including the history features. A 200-post pool takes about 140 ms. Run it in asyncio.to_thread if the event loop must stay free. Cache per user: the loaded history. Cache per post: the 16 content components (engine.fb.pca16(vec)).

Step 4. Roll out safely

1. Shadow mode. Compute engine scores next to the current score_post and log both, without changing the feed.
2. Offline replay on your own logs. Re-tune w_max, n0_engaged_posts, decay_* and push_* on earlier days, then check on later days.
3. A/B test (or interleaving) on a share of users. Primary metric: engaged posts per session. Guardrails: diversity, freshness, share of posts never shown, and engagement on off-topic posts (the topic-first rule can hide them if a user's interests drift).
4. Retrain on Unihive data once there are enough logged sessions. Use the training kit in models/training/ and engine_v2/build_engine_v2.py. Build the datasets with prep_recbole.py (EMB_NPZ=… META_DIR=…, plus MASK_IDS=1 STRICT_PAST=1 for the warm model). Then re-run engine_v2/verify_hybrid_engine.py: parity must stay near zero.

Known limits

• Trained on a 4-user Reddit trace. Re-tune on your own data before trusting the defaults.
• The DeepFM share only matters after a user has engaged posts; brand-new users are served by v2b alone.
• The backend logs one dwell value per event (no hover or separate focal time). The adapter uses it for both; REPORT.md measures the effect.
• About 24% of posts got no tag in the trace. Untagged posts get no tag-closeness bonus but are still ranked by the models.
• HISTORY_SQL was checked against the schema dump and tested with a fake connection, not against a live Postgres (Docker was not running).