Spaces:
Sleeping
Sleeping
File size: 10,492 Bytes
ff17b25 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 | <!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta charset="utf-8" />
<meta name="generator" content="pandoc" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
<title>SHL Assessment Recommender — Approach</title>
<style>
code{white-space: pre-wrap;}
span.smallcaps{font-variant: small-caps;}
div.columns{display: flex; gap: min(4vw, 1.5em);}
div.column{flex: auto; overflow-x: auto;}
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
ul.task-list[class]{list-style: none;}
ul.task-list li input[type="checkbox"] {
font-size: inherit;
width: 0.8em;
margin: 0 0.8em 0.2em -1.6em;
vertical-align: middle;
}
.display.math{display: block; text-align: center; margin: 0.5rem auto;}
</style>
<style type="text/css">@page { size: A4; margin: 14mm 16mm; }
html { font-size: 10pt; }
body {
font-family: -apple-system, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
line-height: 1.34; color: #1a1a1a; max-width: 190mm; margin: 0 auto;
}
h1 { font-size: 16pt; margin: 0 0 2pt; }
h2 { font-size: 11.5pt; margin: 9pt 0 3pt; border-bottom: 1px solid #ccc; padding-bottom: 2pt; }
h1 + p, h2 + p { margin-top: 2pt; }
p, li { margin: 3pt 0; }
ul { margin: 3pt 0; padding-left: 16pt; }
code { background: #f2f2f2; padding: 0 3px; border-radius: 3px; font-size: 9pt; }
table { border-collapse: collapse; font-size: 9pt; margin: 4pt 0; }
th, td { border: 1px solid #bbb; padding: 2pt 6pt; text-align: left; }
th { background: #f0f0f0; }
strong { color: #000; }
a { color: #1a1a1a; text-decoration: none; }
</style>
</head>
<body>
<header id="title-block-header">
<h1 class="title">SHL Assessment Recommender — Approach</h1>
</header>
<h1 id="approach-conversational-shl-assessment-recommender">Approach —
Conversational SHL Assessment Recommender</h1>
<h2 id="problem-design-stance">Problem & design stance</h2>
<p>Build a stateless <code>/chat</code> agent that moves a hiring
manager from a vague intent to a grounded shortlist of SHL assessments —
clarifying, recommending, refining, comparing, and refusing out-of-scope
asks — scored on schema hard-evals, Recall@10, and behavior probes. My
stance: <strong>reliability and grounding first, recall second,
cleverness last.</strong> A turn that 500s or returns a hallucinated
item scores zero regardless of how good the ranking is, so the
architecture makes invalid output structurally impossible and treats the
LLM as an <em>enhancement</em> over a fully-working deterministic
core.</p>
<h2 id="data-retrieval">Data & retrieval</h2>
<p>The provided catalog (377 records) has no <code>test_type</code>; its
<code>keys[]</code> are SHL category names, which the sample traces
confirm map to the letter codes
(<code>Personality & Behavior</code>→<code>P</code>). I derive
<code>test_type</code> from that mapping (priority
<code>K>A>P>B>C>S>E>D</code> for the 39
multi-category records). I verified that <strong>all 10 trace gold
shortlists are 100% in the file</strong>, so it is the recommendable
universe — no website scraping. The 7 pre-packaged “… Solution” bundles
are scoped out via a narrow, auditable rule.</p>
<p>Retrieval <strong>ranks the entire catalog</strong> (it is tiny):
hard filters first (duration cap / required language / required type —
never relaxed), then <strong>Reciprocal Rank Fusion of BM25 and dense
<code>bge-small</code> cosine</strong> (<code>RRF_K=40</code>,
grid-searched — a stable 30–45 plateau), then small soft-preference
boosts, then top-10. Soft preferences only re-rank; if fewer than 10
pass the hard filters we return fewer. No FAISS — a brute-force matrix
multiply over ~370 vectors is exact and instant. One domain prior sits
on top: expert gold shortlists pair role-specific skill tests with a
broad <strong>personality</strong> instrument (OPQ32r appears in 7/10
gold sets) that keyword/semantic retrieval never surfaces from a role
query, so a <em>full</em> shortlist reserves its last slot for OPQ32r
unless a hard constraint excludes it. This is the single biggest recall
lever (below).</p>
<h2 id="agent">Agent</h2>
<p>A 3-node LangGraph (<code>understand → act → respond</code>).
<code>understand</code> routes intent and extracts cumulative
constraints from the whole history — deterministically by default, or
via one Groq JSON call when <code>ENABLE_LLM=true</code>;
<code>act</code> is deterministic; replies are templated. Because the
wire schema is fixed and the API is stateless, the <strong>numbered list
in the reply is the only cross-turn state</strong> — each line prints
its catalog URL, which I parse back to reconstruct the prior shortlist
for refinements like “remove the second one”. Clarification is capped at
one (with a turn-budget backstop so we never starve the 8-turn cap into
an empty final shortlist). <code>end_of_conversation</code> is true only
on explicit user completion — matching the traces.</p>
<h2 id="prompt-design">Prompt design</h2>
<p>The routing prompt is deliberately small and contract-shaped: it
returns a fixed JSON object
(<code>in_scope, intent, search_query, hard/soft constraints, compare/remove names, clarifying_question, user_done</code>)
and never writes prose or invents names. Comparison uses a separate
prompt fed <strong>only catalog facts</strong> (no priors), with a
deterministic field-by-field template as fallback.</p>
<h2 id="guardrails-anti-hallucination">Guardrails
(anti-hallucination)</h2>
<p>Every response passes one gate: each recommendation is canonicalized
against the catalog (URL returned verbatim, type checked), off-catalog
and duplicate items are dropped, the list is capped at 10. Combined with
lenient input parsing + global exception/timeout handlers, <strong>every
call returns a schema-valid 200</strong> — request errors, LLM failures,
and timeouts included.</p>
<h2 id="evaluation">Evaluation</h2>
<p>Three harnesses (<code>pytest</code> — 70 tests,
<code>eval.replay</code>, <code>eval.ablation</code>). Replay mirrors
the grader: feed each trace’s user turns one at a time, carry full
history, score Recall@10 on the final shortlist, plus latency and 11
behavior probes (both scripted-final and grader-like
stop-on-first-shortlist modes).</p>
<p><strong>Retrieval ablation (fixed query, retrieval isolated from
agent behavior):</strong></p>
<table>
<thead>
<tr>
<th>Config</th>
<th>Mean Recall@10</th>
</tr>
</thead>
<tbody>
<tr>
<td>BM25 only</td>
<td>0.387</td>
</tr>
<tr>
<td>Dense only</td>
<td>0.468</td>
</tr>
<tr>
<td><strong>Hybrid (RRF)</strong></td>
<td><strong>0.551</strong></td>
</tr>
<tr>
<td>Hybrid + MMR</td>
<td>0.551</td>
</tr>
</tbody>
</table>
<p><strong>Multi-turn replay (default deterministic config):</strong>
mean <strong>Recall@10 = 0.733</strong> (scripted-final) /
<strong>0.700</strong> (stop-on-first-shortlist). <strong>Behavior
probes:</strong> 11/11 — refuses off-topic (incl. novel/paraphrased),
resists injection (incl. paraphrased), no turn-1 recommend on vague,
honors edits, multi-edit refine, three-way compare, duration
hard-constraint, completion sets end, no hallucinated items.
<strong>Latency:</strong> deterministic p50 ≈ 0.05s, p95 ≈ 0.12s; with
the 8B router p50 ≈ 0.5s (≈9s in a network-degraded sandbox) — inside
the 30s cap either way.</p>
<h2 id="what-didnt-work-how-i-measured">What didn’t work / how I
measured</h2>
<p>Every flag is off until a measured Recall@10 gain justifies it, and
the reverse — every adopted change moved the replay number: -
<strong>Battery-composition prior (adopted):</strong> reserving a full
shortlist’s tail slot for OPQ32r lifted scripted-final <strong>0.625 →
0.699</strong>; with the <code>RRF_K=40</code> retune and two
state-preservation fixes (below) the config reaches
<strong>0.733</strong>. - <strong>State-preservation fixes
(adopted):</strong> a shortlist <em>question</em> (“do we really need
Verify G+?”) was rebuilding the list from scratch and discarding
accumulated edits — now it re-renders unchanged; and “keep <named item>”
now adds that exact instrument when absent. Both recovered C9 gold (0.57
→ 0.71). - <strong>Plural folding / light stemming (rejected):</strong>
<em>lowered</em> Recall@10 (0.713 → 0.693) — it collided distinct
product tokens → left out. - <strong>MMR (rejected):</strong>
<strong>no</strong> ablation gain (0.551 = 0.551) → left
<strong>off</strong> (flagged). - <strong>Family/variant boost
(rejected):</strong> <em>reduced</em> Recall@10 — a strong family
displaces other-family gold → left <strong>off</strong>. - <strong>70B
routing</strong> exhausted the Groq free-tier daily token budget (100k
TPD) in a single replay and pushed p95 latency to ~10s → switched
routing to <strong><code>llama-3.1-8b-instant</code></strong> (separate,
larger budget; faster) and slashed prompt tokens. - <strong>LLM routing
under-performed and is OFF by default.</strong> The deterministic
broad-query core out-scores the 8B route (LLM query
<em>distillation</em> hurt multi-faceted/semantic needs like “senior
Rust”, where gold items are inferred), at ~0.05s/turn vs ~9s and with no
token/network risk. <code>ENABLE_LLM</code> defaults to false (same
discipline as MMR/rerank); the LLM is one env var away for scope/routing
robustness, and any LLM failure falls back to this same core.</p>
<h2 id="reliability-robustness-hardening">Reliability & robustness
hardening</h2>
<p>Beyond the never-4xx/5xx contract: request bodies are size-capped and
history/message length is clamped (bounded CPU under adversarial input,
still degrading to a valid 200); the BM25 index is refit from the
catalog at load rather than unpickled (no
deserialization-of-untrusted-data surface); dependencies are pinned to
CVE-clear versions; and the container runs as a non-root user.</p>
<h2 id="ai-tool-usage">AI-tool usage</h2>
<p>Built with AI-assisted coding (Claude Code) for scaffolding,
retrieval/agent plumbing, and the eval harness; all design decisions
(hybrid-rank-all, hard/soft split, state-in-reply,
flags-off-until-measured, model choice) are documented in
<code>docs/DECISIONS.md</code> and were driven by the measurements
above.</p>
<h2 id="stack">Stack</h2>
<p><code>bge-small-en-v1.5</code> (local, baked) + BM25 retrieval ·
FastAPI + Pydantic v2 · LangGraph · rapidfuzz · NumPy · Hugging Face
Spaces (Docker) · Groq <code>llama-3.1-8b-instant</code> (opt-in).</p>
</body>
</html>
|