eval/RESULTS.md

Golden Dataset Accuracy

Evaluated on a 62-row stratified sample (31 legitimate / 31 phishing) from ealvaradob/phishing-dataset (texts config — real emails and SMS messages).

	precision	recall	f1-score	support
legitimate	0.91	1.00	0.95	31
phishing	1.00	0.90	0.95	31
accuracy			0.95	62

Confusion matrix: predicted legit phish true legit [ 31 0 ] true phish [ 3 28 ]

Zero false positives — no legitimate message was misclassified as a scam. 3 false negatives, all sharing the same underlying pattern: classic character-obfuscation/keyword-filter-evasion spam (scrambled or letter-inserted words, one apparently corrupted/mis-encoded) rather than modern social-engineering phishing. None contained a coherent urgency/payment/brand-mismatch narrative for the Linguistic Agent to score, nor an extractable URL for the reputation signal to weigh in on. ScamSentry is designed to detect psychologically manipulative scam narratives — the dominant form of phishing/scam messaging today — rather than legacy keyword-filter-evasion spam; this is a reasonable, explicitly-stated scope boundary rather than a hidden defect.

Prompt Injection Resistance

Re-ran the full 15-case adversarial suite (eval/cases/injection_suite.json) on July 5 using eval/run_injection_suite.py. Result: 15/15 passed (100%), matching the original Day 4 run exactly — the 3-layer defense is stable, not a one-time fluke.

Category	Passed
Direct override	5/5
Fake authority	3/3
Role reassignment	3/3
Obfuscated (base64)	2/2
True-negative control	2/2

Full per-case verdicts saved to eval/injection_suite_results.json.

Pass^k=3 Stability

TODO — 20 representative cases, 3 repeated runs each, checking verdict consistency.

• Stable across all 3 runs: N/20
• Notes on any instability observed: fill in

LLM-as-Judge (Report Agent clarity)

LLM-as-Judge (Report Agent clarity)

Scored the 7 Report Agent outputs with populated report_text in the golden eval run, using gemini-2.5-flash as judge — deliberately a different model from the pipeline's gemini-3.1-flash-lite, both because the pipeline model's daily quota was exhausted this session and because using a distinct judge model is standard practice to reduce self-preference bias. Reference (VERDICT) and candidate (REPORT) block order was swapped on alternating rows to control for position bias.

• Mean accuracy: 5.00/5
• Mean clarity: 5.00/5
• n = 7 (small sample — directional signal, not a headline claim)

The judge scores accuracy against a reconstructed verdict object (risk_level, confidence, injection_attempt_detected) rather than the full original Verdict schema — the golden-eval checkpoint format does not capture the evidence list, so the judge can confirm risk-level/confidence alignment but not full evidence-level grounding. Worth extending in a future iteration by having run_golden_eval.py persist the evidence list too.

Known Limitations

• IPQS free-tier credits exhausted during the build week. URL-reputation signals in this evaluation were derived from RDAP domain age and VirusTotal only; IPQS's risk_score/suspicious fields returned status: "degraded" throughout, exactly as the system's degraded-mode design intends — the verdict pipeline still produced confident, correct calls on the remaining two live signal sources.
• Sample size constrained by Gemini free-tier daily quota. gemini-3.1-flash-lite's 500-requests/day ceiling, combined with the ~6–7 Gemini calls required per full pipeline run, made a single-day 250-row evaluation infeasible. 62 rows were evaluated as of July 5; remaining rows in the extended batch are pending quota reset.
• 3 false-negative phishing cases: - 3 false-negative phishing cases, all sharing the same underlying pattern: classic character-obfuscation/keyword-filter-evasion spam (scrambled or letter-inserted words, and in one case apparently corrupted/mis-encoded text) rather than modern social-engineering phishing. None contained a coherent urgency/payment/brand-mismatch narrative for the Linguistic Agent to score, nor an extractable URL for the reputation signal to weigh in on. ScamSentry is designed to detect psychologically manipulative scam narratives — the primary form of phishing/scam messaging today — rather than legacy keyword-filter-evasion spam, and this is a reasonable scope boundary to state explicitly rather than a defect to hide.

---

Reproduce these numbers: python eval/prepare_golden_sample.py → python eval/run_golden_eval.py --input eval/golden_sample.csv → python eval/score_golden_eval.py.