snl-intern / docs /ARCHITECTURE.md
xPushkal's picture
SHL Assessment Recommender β€” deploy snapshot
ff17b25
|
Raw History Blame Contribute Delete
4.45 kB

Architecture

Request lifecycle

POST /chat  (stateless: full history every call)
   β”‚
   β–Ό
FastAPI layer (app/main.py)
   β€’ lenient request parsing (unknown fields ignored β†’ never a 422)
   β€’ run pipeline in a worker thread under a whole-turn timeout
   β€’ RequestValidationError + Exception handlers β†’ schema-valid 200
   β–Ό
Turn orchestrator (app/agent/turn.py) β€” defense-in-depth try/except
   β–Ό
LangGraph: understand β†’ act β†’ respond  (app/agent/graph.py)
   β”‚
   β”œβ”€ understand (app/agent/understand.py)
   β”‚     deterministic keyword router by default (vagueness-aware) β†’
   β”‚     {in_scope, intent, search_query, hard, soft, compare_names,
   β”‚      remove_names, add_query, clarifying_question, user_done}
   β”‚     optional (ENABLE_LLM): one Groq JSON call over the WHOLE history,
   β”‚     with the deterministic router as the fallback on any failure
   β”‚
   β”œβ”€ act (app/agent/dispatch.py) β€” deterministic
   β”‚     refuse | clarify(≀1) | recommend | refine | compare
   β”‚     reconstructs prior shortlist from the last assistant reply
   β”‚
   └─ respond (app/agent/render.py + app/responder.py)
         templated reply + anti-hallucination/schema gate
   β–Ό
Retrieval (app/retrieval/*) β€” rank the FULL catalog
   HARD filters (never relaxed) β†’ RRF(BM25, dense) β†’ SOFT boosts β†’ top-10
   [MMR / LLM-rerank optional, OFF by default]

Why these choices

Stateless state lives in the reply. The wire schema is fixed (reply, recommendations, end_of_conversation), and the grader replays the full history each call. So the numbered list embedded in the reply text is the only cross-turn state carrier. Every recommendation line prints its catalog URL; render.parse_prior_shortlist reads those URLs back out of the most recent assistant message to reconstruct the prior shortlist (URLs are unambiguous keys, unlike names with parentheses). This is what makes "remove the second one" work without a database.

≀2 LLM calls per turn. One understand call routes + extracts; replies are templated; only compare (and the optional reranker) make a second call. Groq is sub-second, so a turn finishes well inside the 30s cap with timeout headroom.

Rank the whole catalog. With ~370 recommendable items, there is no need for an ANN index or a candidate cap β€” we score every document with a brute-force matrix multiply (dense, embeddings @ query) plus BM25, fuse with Reciprocal Rank Fusion, then add soft boosts. Exact, instant, and one fewer dependency (no FAISS).

Hard vs soft. understand tags each constraint. Hard constraints (max_duration, required languages, required test_types) are applied as filters before scoring and are never relaxed β€” if fewer than 10 survive, we return fewer than 10. Soft preferences (job_levels, preferred types) only add a small ranking boost and can never exclude an item, which protects recall.

Reliability / fallback matrix

Failure Handling Result
Malformed request body lenient schema + RequestValidationError handler 200, safe reply
understand LLM error/timeout 1 stricter retry β†’ deterministic router valid route
Deterministic router on vague input vagueness check β†’ clarify (not blind recommend) empty recs, clarify
Rerank disabled/fails deterministic fusion order top-10
Compare LLM fails deterministic field-by-field template grounded compare
Off-catalog / duplicate item responder canonicalization gate dropped
Whole turn exceeds budget asyncio.wait_for in main.py 200, safe reply

test_type derivation

The catalog has no test_type; its keys[] are SHL category names (trace-confirmed: Personality & Behavior β†’ P). app/data/test_type_map.py maps each category to its letter and exposes all codes plus a single primary. 338/377 records have one category; for the 39 multi-category records the catalog lists categories alphabetically (so order is not meaningful) and we apply a documented priority K>A>P>B>C>S>E>D.

Scope

Verified empirically: all 10 sample-trace gold shortlists are 100% present in the provided catalog, so it is the recommendable universe (no website scraping). The 7 pre-packaged Job Solutions (all literally "… Solution") are tagged is_individual=False and excluded from recommendations, but remain in the catalog for validation.