Spaces:
Running
Running
File size: 7,418 Bytes
3e012b6 1bbf278 4be82f6 1bbf278 3e012b6 1bbf278 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | # HireTrace: Solution Video Walkthrough Script (β€5 Minutes)
**micro1 Agentic Workflows Hackathon**
**Tagline:** *"Every recommendation traces back to evidence."*
---
## Video Outline & Timing
| Segment | Time | Screen Visual | Narration & Key Points |
|---|---|---|---|
| **1. Problem Framing** | 0:00 - 0:40 | β’ Architecture diagram & fragmented candidate docs (JD, CV, interview notes, assessment report, RFC) | β’ "Hiring teams evaluate candidates across disconnected silos: CVs, interviews, coding assessments, and project docs."<br>β’ "Reviewing each in isolation causes teams to miss cross-source contradictions or overweight single signals."<br>β’ "HireTrace introduces evidence-first assessment: answering not *'Should we hire?'*, but *'How well does the evidence support the role, and where does it disagree with itself?'*" |
| **2. Zero Cost & Local Architecture** | 0:40 - 1:05 | β’ Terminal showing Ollama running `qwen2.5:3b` at `localhost:11434`<br>β’ Network audit showing $0.00 external API calls<br>β’ Architecture diagram showing Pydantic repair loop & 3x self-consistency voting | β’ "Hard architectural constraint: **$0.00 cost**."<br>β’ "Zero paid API calls, zero cloud subscriptions. All inference runs locally via Ollama (`qwen2.5:3b`), and all retrieval uses local `faiss-cpu`."<br>β’ "To ensure local 3B open-weights reliability, HireTrace enforces JSON-schema constrained decoding with Pydantic self-repair loops and 3x self-consistency majority voting with tie-break fallbacks." |
| **3. Baseline A & B Comparison** | 1:05 - 1:45 | β’ Code walkthrough of `/baseline/rubric_scorer.py` and `/baseline/naive_llm.py` | β’ "We benchmark against two realistic baselines:"<br>β’ "β’ **Baseline A (Deterministic Resume Rubric)**: A deterministic rubric scoring Open Source (0β35), Self Projects (0β30), Production (0β25), Tech Skills (0β10), and Bonus Points (0β20). Fast, but blind to cross-source contradictions."<br>β’ "β’ **Baseline B (Naive Single-Prompt LLM)**: Concatenates all documents into local Ollama in one prompt. While it can produce apparently grounded individual claims, it lacks the cross-source verification machinery needed to reliably detect contradictions." |
| **4. The Centerpiece Case Walkthrough** | 1:45 - 3:00 | β’ **HireTrace Interactive Dashboard (`http://127.0.0.1:8080`)**<br>β’ Hovering over candidate dots on the **Interactive 2D Quadrant Canvas** (showing real-time tooltips & crosshairs at 70/70 thresholds)<br>β’ Selecting **Alexander Sterling** (`case_15_deceptive_centerpiece`)<br>β’ Side-by-side discrepancy diffs with **verbatim quote highlighting** and **'π View in Dossier'** jump navigation<br>β’ Priority Questions panel with **'π Copy All Questions'** one-click export | β’ "Here is our centerpiece demo: Alexander Sterling."<br>β’ "On paper, his CV looks stellar: 8+ years experience, 250 GitHub stars, claiming: *'Led migration of legacy monolithic core to Apache Kafka for a 7-person team.'*"<br>β’ "Baseline B assigns Alexander 85.0 and misses the deception entirely. But HireTrace's decomposed multi-agent pipeline catches it:"<br>β’ "1. **Requirement Mapping Agent** extracts 5 core requirements from the JD."<br>β’ "2. **FAISS Retrieval Layer** indexes all evidence spans across CV, interview, and project docs."<br>β’ "3. **Cross-Source Verification Agent** isolates 5 critical discrepancies across tenure and project authority:"<br> β’ *Discrepancy 1 (Tenure)*: CV claims 3.5 years at FinFlow; interview notes prove he joined ~18 months ago.<br> β’ *Discrepancies 2β5 (Leadership & Authority)*: CV claims he *led* the 7-person Kafka migration and authored the master architectural blueprint; Project RFC documents prove Dr. Robert Vance was Principal Architect/Lead and Alexander was a junior contributing member writing Python serialization adapters.<br>β’ "Notice our new **'View in Dossier'** button: clicking it jumps directly to the exact source tab and highlights the raw text."<br>β’ "4. **Claim Critic Agent** verifies 100% exact substring containment in raw sources, distinguishing asserted facts from synthesized inferences."<br>β’ "5. **Recommendation Writer Agent** plots him on our **2D Quadrant**: Role Fit (82.7/100) vs. Evidence Consistency (0.0/100) -> **[REVIEW REQUIRED]**."<br>β’ "Recruiters click **'Copy All Questions'** to immediately get an interview debrief template for panel review." |
| **5. Audit & Governance Tab: Fairness, Calibration & Defense** | 3:00 - 4:00 | β’ Switching to **'π‘ Audit & Governance'** tab<br>β’ Highlighting the 4 audit cards: EEOC Fairness, Claim Grounding, Uncertainty Calibration, Adversarial Defense | β’ "In enterprise hiring, compliance and governance are non-negotiable. Our new **Audit & Governance** tab exposes live telemetry:"<br>β’ "β’ **Algorithmic Fairness & Parity:** Audited under EEOC Uniform Guidelines (4 CFR Part 60) across 44 counterfactual candidate dossiers and 11 demographic groups. Yields **0.00 pts score drift** and **1.0000 Disparate Impact Ratio (EEOC compliant)**."<br>β’ "β’ **Evidence Grounding & Critic:** 100% quote containment, 100% citation span validity, 66.7% claim grounding on 15 synthetic cases."<br>β’ "β’ **Uncertainty Calibration:** Brier score 0.2281 and Expected Calibration Error of 23.9% across confidence bins."<br>β’ "β’ **Adversarial Defense:** 100% containment across 8 evaluated attack vectors on synthetic benchmarks (prompt injection, delimiter escapes, conversational jailbreaks, and credential/metric fabrications)." |
| **6. Evaluation Benchmark & Closing Hot Take** | 4:00 - 5:00 | β’ Benchmark matrix tab<br>β’ 50-case benchmark results<br>β’ Final hot take slide | β’ "Across our expanded 50-case benchmark suite spanning 5 engineering domains:"<br>β’ "β’ **Spearman Rank Correlation (Ο):** Baseline A (0.579) -> **HireTrace Agent (0.816)** (95% CI: [0.446, 0.983])."<br>β’ "β’ **Contradiction Detection:** **100.0% precision, 100.0% recall, 1.000 F1** (zero false alarms on clean controls)."<br>β’ "β’ **Reviewer Time Saved:** Under our standardized cognitive reviewer-effort model, estimated review time falls from 18.0 min to 3.5 min (**80.6% estimated time saved**)."<br>β’ "**Closing Hot Take:** *'Verification establishes consistency, not physical truth.'* If a CV and interview agree, that proves internal consistency across documents β not physical truth. That is why automated agents must never make autonomous hiring decisions, and why HireTrace traces every claim to evidence for human reviewers." |
---
## Production Tips for Recording
1. **Resolution:** 1920x1080 (1080p).
2. **Terminal:** Run `python -m ui.server 8080` and show the clean web UI at `http://127.0.0.1:8080`.
3. **Tabs to Demonstrate:**
- Tab 1: **Candidate Assessment** (Scatter quadrant canvas with crosshairs and hover tooltip, Alexander Sterling centerpiece, Priority Questions + Copy button).
- Tab 2: **Discrepancy Diffs** (Side-by-side quote highlight and 'View in Dossier' button).
- Tab 3: **Dossier Viewer** (Multi-doc tabs).
- Tab 4: **Benchmark Matrix** (Spearman rho, Brier score, ECE).
- Tab 5: **Audit & Governance** (EEOC fairness, grounding fidelity, adversarial defense).
4. **Ollama:** Confirm Ollama daemon is running with `qwen2.5:3b` or running in offline mock mode ($0.00 cost).
|