Spaces:
Sleeping
Sleeping
Download docs/APPROACH.html from xPushkal/snl-intern: direct link, hf CLI and curl.
- Browser
- Download file 10.5 kB
-
https://huggingface.co/spaces/xPushkal/snl-intern/resolve/main/docs/APPROACH.html
- Command line
-
hf download hf://spaces/xPushkal/snl-intern/docs/APPROACH.html
-
curl -L -o APPROACH.html https://huggingface.co/spaces/xPushkal/snl-intern/resolve/main/docs/APPROACH.html
10.5 kB
| <html xmlns="http://www.w3.org/1999/xhtml"> | |
| <head> | |
| <meta charset="utf-8" /> | |
| <meta name="generator" content="pandoc" /> | |
| <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" /> | |
| <title>SHL Assessment Recommender — Approach</title> | |
| <style> | |
| code{white-space: pre-wrap;} | |
| span.smallcaps{font-variant: small-caps;} | |
| div.columns{display: flex; gap: min(4vw, 1.5em);} | |
| div.column{flex: auto; overflow-x: auto;} | |
| div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;} | |
| ul.task-list[class]{list-style: none;} | |
| ul.task-list li input[type="checkbox"] { | |
| font-size: inherit; | |
| width: 0.8em; | |
| margin: 0 0.8em 0.2em -1.6em; | |
| vertical-align: middle; | |
| } | |
| .display.math{display: block; text-align: center; margin: 0.5rem auto;} | |
| </style> | |
| <style type="text/css">@page { size: A4; margin: 14mm 16mm; } | |
| html { font-size: 10pt; } | |
| body { | |
| font-family: -apple-system, "Segoe UI", Roboto, Helvetica, Arial, sans-serif; | |
| line-height: 1.34; color: #1a1a1a; max-width: 190mm; margin: 0 auto; | |
| } | |
| h1 { font-size: 16pt; margin: 0 0 2pt; } | |
| h2 { font-size: 11.5pt; margin: 9pt 0 3pt; border-bottom: 1px solid #ccc; padding-bottom: 2pt; } | |
| h1 + p, h2 + p { margin-top: 2pt; } | |
| p, li { margin: 3pt 0; } | |
| ul { margin: 3pt 0; padding-left: 16pt; } | |
| code { background: #f2f2f2; padding: 0 3px; border-radius: 3px; font-size: 9pt; } | |
| table { border-collapse: collapse; font-size: 9pt; margin: 4pt 0; } | |
| th, td { border: 1px solid #bbb; padding: 2pt 6pt; text-align: left; } | |
| th { background: #f0f0f0; } | |
| strong { color: #000; } | |
| a { color: #1a1a1a; text-decoration: none; } | |
| </style> | |
| </head> | |
| <body> | |
| <header id="title-block-header"> | |
| <h1 class="title">SHL Assessment Recommender — Approach</h1> | |
| </header> | |
| <h1 id="approach-conversational-shl-assessment-recommender">Approach — | |
| Conversational SHL Assessment Recommender</h1> | |
| <h2 id="problem-design-stance">Problem & design stance</h2> | |
| <p>Build a stateless <code>/chat</code> agent that moves a hiring | |
| manager from a vague intent to a grounded shortlist of SHL assessments — | |
| clarifying, recommending, refining, comparing, and refusing out-of-scope | |
| asks — scored on schema hard-evals, Recall@10, and behavior probes. My | |
| stance: <strong>reliability and grounding first, recall second, | |
| cleverness last.</strong> A turn that 500s or returns a hallucinated | |
| item scores zero regardless of how good the ranking is, so the | |
| architecture makes invalid output structurally impossible and treats the | |
| LLM as an <em>enhancement</em> over a fully-working deterministic | |
| core.</p> | |
| <h2 id="data-retrieval">Data & retrieval</h2> | |
| <p>The provided catalog (377 records) has no <code>test_type</code>; its | |
| <code>keys[]</code> are SHL category names, which the sample traces | |
| confirm map to the letter codes | |
| (<code>Personality & Behavior</code>→<code>P</code>). I derive | |
| <code>test_type</code> from that mapping (priority | |
| <code>K>A>P>B>C>S>E>D</code> for the 39 | |
| multi-category records). I verified that <strong>all 10 trace gold | |
| shortlists are 100% in the file</strong>, so it is the recommendable | |
| universe — no website scraping. The 7 pre-packaged “… Solution” bundles | |
| are scoped out via a narrow, auditable rule.</p> | |
| <p>Retrieval <strong>ranks the entire catalog</strong> (it is tiny): | |
| hard filters first (duration cap / required language / required type — | |
| never relaxed), then <strong>Reciprocal Rank Fusion of BM25 and dense | |
| <code>bge-small</code> cosine</strong> (<code>RRF_K=40</code>, | |
| grid-searched — a stable 30–45 plateau), then small soft-preference | |
| boosts, then top-10. Soft preferences only re-rank; if fewer than 10 | |
| pass the hard filters we return fewer. No FAISS — a brute-force matrix | |
| multiply over ~370 vectors is exact and instant. One domain prior sits | |
| on top: expert gold shortlists pair role-specific skill tests with a | |
| broad <strong>personality</strong> instrument (OPQ32r appears in 7/10 | |
| gold sets) that keyword/semantic retrieval never surfaces from a role | |
| query, so a <em>full</em> shortlist reserves its last slot for OPQ32r | |
| unless a hard constraint excludes it. This is the single biggest recall | |
| lever (below).</p> | |
| <h2 id="agent">Agent</h2> | |
| <p>A 3-node LangGraph (<code>understand → act → respond</code>). | |
| <code>understand</code> routes intent and extracts cumulative | |
| constraints from the whole history — deterministically by default, or | |
| via one Groq JSON call when <code>ENABLE_LLM=true</code>; | |
| <code>act</code> is deterministic; replies are templated. Because the | |
| wire schema is fixed and the API is stateless, the <strong>numbered list | |
| in the reply is the only cross-turn state</strong> — each line prints | |
| its catalog URL, which I parse back to reconstruct the prior shortlist | |
| for refinements like “remove the second one”. Clarification is capped at | |
| one (with a turn-budget backstop so we never starve the 8-turn cap into | |
| an empty final shortlist). <code>end_of_conversation</code> is true only | |
| on explicit user completion — matching the traces.</p> | |
| <h2 id="prompt-design">Prompt design</h2> | |
| <p>The routing prompt is deliberately small and contract-shaped: it | |
| returns a fixed JSON object | |
| (<code>in_scope, intent, search_query, hard/soft constraints, compare/remove names, clarifying_question, user_done</code>) | |
| and never writes prose or invents names. Comparison uses a separate | |
| prompt fed <strong>only catalog facts</strong> (no priors), with a | |
| deterministic field-by-field template as fallback.</p> | |
| <h2 id="guardrails-anti-hallucination">Guardrails | |
| (anti-hallucination)</h2> | |
| <p>Every response passes one gate: each recommendation is canonicalized | |
| against the catalog (URL returned verbatim, type checked), off-catalog | |
| and duplicate items are dropped, the list is capped at 10. Combined with | |
| lenient input parsing + global exception/timeout handlers, <strong>every | |
| call returns a schema-valid 200</strong> — request errors, LLM failures, | |
| and timeouts included.</p> | |
| <h2 id="evaluation">Evaluation</h2> | |
| <p>Three harnesses (<code>pytest</code> — 70 tests, | |
| <code>eval.replay</code>, <code>eval.ablation</code>). Replay mirrors | |
| the grader: feed each trace’s user turns one at a time, carry full | |
| history, score Recall@10 on the final shortlist, plus latency and 11 | |
| behavior probes (both scripted-final and grader-like | |
| stop-on-first-shortlist modes).</p> | |
| <p><strong>Retrieval ablation (fixed query, retrieval isolated from | |
| agent behavior):</strong></p> | |
| <table> | |
| <thead> | |
| <tr> | |
| <th>Config</th> | |
| <th>Mean Recall@10</th> | |
| </tr> | |
| </thead> | |
| <tbody> | |
| <tr> | |
| <td>BM25 only</td> | |
| <td>0.387</td> | |
| </tr> | |
| <tr> | |
| <td>Dense only</td> | |
| <td>0.468</td> | |
| </tr> | |
| <tr> | |
| <td><strong>Hybrid (RRF)</strong></td> | |
| <td><strong>0.551</strong></td> | |
| </tr> | |
| <tr> | |
| <td>Hybrid + MMR</td> | |
| <td>0.551</td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| <p><strong>Multi-turn replay (default deterministic config):</strong> | |
| mean <strong>Recall@10 = 0.733</strong> (scripted-final) / | |
| <strong>0.700</strong> (stop-on-first-shortlist). <strong>Behavior | |
| probes:</strong> 11/11 — refuses off-topic (incl. novel/paraphrased), | |
| resists injection (incl. paraphrased), no turn-1 recommend on vague, | |
| honors edits, multi-edit refine, three-way compare, duration | |
| hard-constraint, completion sets end, no hallucinated items. | |
| <strong>Latency:</strong> deterministic p50 ≈ 0.05s, p95 ≈ 0.12s; with | |
| the 8B router p50 ≈ 0.5s (≈9s in a network-degraded sandbox) — inside | |
| the 30s cap either way.</p> | |
| <h2 id="what-didnt-work-how-i-measured">What didn’t work / how I | |
| measured</h2> | |
| <p>Every flag is off until a measured Recall@10 gain justifies it, and | |
| the reverse — every adopted change moved the replay number: - | |
| <strong>Battery-composition prior (adopted):</strong> reserving a full | |
| shortlist’s tail slot for OPQ32r lifted scripted-final <strong>0.625 → | |
| 0.699</strong>; with the <code>RRF_K=40</code> retune and two | |
| state-preservation fixes (below) the config reaches | |
| <strong>0.733</strong>. - <strong>State-preservation fixes | |
| (adopted):</strong> a shortlist <em>question</em> (“do we really need | |
| Verify G+?”) was rebuilding the list from scratch and discarding | |
| accumulated edits — now it re-renders unchanged; and “keep <named item>” | |
| now adds that exact instrument when absent. Both recovered C9 gold (0.57 | |
| → 0.71). - <strong>Plural folding / light stemming (rejected):</strong> | |
| <em>lowered</em> Recall@10 (0.713 → 0.693) — it collided distinct | |
| product tokens → left out. - <strong>MMR (rejected):</strong> | |
| <strong>no</strong> ablation gain (0.551 = 0.551) → left | |
| <strong>off</strong> (flagged). - <strong>Family/variant boost | |
| (rejected):</strong> <em>reduced</em> Recall@10 — a strong family | |
| displaces other-family gold → left <strong>off</strong>. - <strong>70B | |
| routing</strong> exhausted the Groq free-tier daily token budget (100k | |
| TPD) in a single replay and pushed p95 latency to ~10s → switched | |
| routing to <strong><code>llama-3.1-8b-instant</code></strong> (separate, | |
| larger budget; faster) and slashed prompt tokens. - <strong>LLM routing | |
| under-performed and is OFF by default.</strong> The deterministic | |
| broad-query core out-scores the 8B route (LLM query | |
| <em>distillation</em> hurt multi-faceted/semantic needs like “senior | |
| Rust”, where gold items are inferred), at ~0.05s/turn vs ~9s and with no | |
| token/network risk. <code>ENABLE_LLM</code> defaults to false (same | |
| discipline as MMR/rerank); the LLM is one env var away for scope/routing | |
| robustness, and any LLM failure falls back to this same core.</p> | |
| <h2 id="reliability-robustness-hardening">Reliability & robustness | |
| hardening</h2> | |
| <p>Beyond the never-4xx/5xx contract: request bodies are size-capped and | |
| history/message length is clamped (bounded CPU under adversarial input, | |
| still degrading to a valid 200); the BM25 index is refit from the | |
| catalog at load rather than unpickled (no | |
| deserialization-of-untrusted-data surface); dependencies are pinned to | |
| CVE-clear versions; and the container runs as a non-root user.</p> | |
| <h2 id="ai-tool-usage">AI-tool usage</h2> | |
| <p>Built with AI-assisted coding (Claude Code) for scaffolding, | |
| retrieval/agent plumbing, and the eval harness; all design decisions | |
| (hybrid-rank-all, hard/soft split, state-in-reply, | |
| flags-off-until-measured, model choice) are documented in | |
| <code>docs/DECISIONS.md</code> and were driven by the measurements | |
| above.</p> | |
| <h2 id="stack">Stack</h2> | |
| <p><code>bge-small-en-v1.5</code> (local, baked) + BM25 retrieval · | |
| FastAPI + Pydantic v2 · LangGraph · rapidfuzz · NumPy · Hugging Face | |
| Spaces (Docker) · Groq <code>llama-3.1-8b-instant</code> (opt-in).</p> | |
| </body> | |
| </html> | |