## Golden Dataset Accuracy

Evaluated on a 62-row stratified sample (31 legitimate / 31 phishing) from
`ealvaradob/phishing-dataset` (`texts` config — real emails and SMS messages).

| | precision | recall | f1-score | support |
|---|---|---|---|---|
| legitimate | 0.91 | 1.00 | 0.95 | 31 |
| phishing | 1.00 | 0.90 | 0.95 | 31 |
| **accuracy** | | | **0.95** | 62 |

Confusion matrix:
          predicted
          legit  phish
true legit  [   31      0  ]
true phish  [    3     28  ]

**Zero false positives** — no legitimate message was misclassified as a scam.
**3 false negatives**, all sharing the same underlying pattern: classic
character-obfuscation/keyword-filter-evasion spam (scrambled or
letter-inserted words, one apparently corrupted/mis-encoded) rather than
modern social-engineering phishing. None contained a coherent
urgency/payment/brand-mismatch narrative for the Linguistic Agent to score,
nor an extractable URL for the reputation signal to weigh in on. ScamSentry
is designed to detect psychologically manipulative scam narratives — the
dominant form of phishing/scam messaging today — rather than legacy
keyword-filter-evasion spam; this is a reasonable, explicitly-stated scope
boundary rather than a hidden defect.

## Prompt Injection Resistance

Re-ran the full 15-case adversarial suite (`eval/cases/injection_suite.json`) on July 5
using `eval/run_injection_suite.py`. Result: **15/15 passed (100%)**, matching the
original Day 4 run exactly — the 3-layer defense is stable, not a one-time fluke.

| Category | Passed |
|---|---|
| Direct override | 5/5 |
| Fake authority | 3/3 |
| Role reassignment | 3/3 |
| Obfuscated (base64) | 2/2 |
| True-negative control | 2/2 |

Full per-case verdicts saved to `eval/injection_suite_results.json`.

## Pass^k=3 Stability

_TODO — 20 representative cases, 3 repeated runs each, checking verdict consistency._

- Stable across all 3 runs: _N/20_
- Notes on any instability observed: _fill in_

## LLM-as-Judge (Report Agent clarity)

## LLM-as-Judge (Report Agent clarity)

Scored the 7 Report Agent outputs with populated `report_text` in the golden
eval run, using `gemini-2.5-flash` as judge — deliberately a different model
from the pipeline's `gemini-3.1-flash-lite`, both because the pipeline
model's daily quota was exhausted this session and because using a distinct
judge model is standard practice to reduce self-preference bias. Reference
(VERDICT) and candidate (REPORT) block order was swapped on alternating
rows to control for position bias.

- **Mean accuracy: 5.00/5**
- **Mean clarity: 5.00/5**
- n = 7 (small sample — directional signal, not a headline claim)

The judge scores accuracy against a reconstructed verdict object
(`risk_level`, `confidence`, `injection_attempt_detected`) rather than the
full original `Verdict` schema — the golden-eval checkpoint format does not
capture the `evidence` list, so the judge can confirm risk-level/confidence
alignment but not full evidence-level grounding. Worth extending in a future
iteration by having `run_golden_eval.py` persist the evidence list too.

## Known Limitations

- **IPQS free-tier credits exhausted** during the build week. URL-reputation
  signals in this evaluation were derived from RDAP domain age and VirusTotal
  only; IPQS's `risk_score`/`suspicious` fields returned `status: "degraded"`
  throughout, exactly as the system's degraded-mode design intends — the
  verdict pipeline still produced confident, correct calls on the remaining
  two live signal sources.
- **Sample size constrained by Gemini free-tier daily quota.** `gemini-3.1-flash-lite`'s
  500-requests/day ceiling, combined with the ~6–7 Gemini calls required per
  full pipeline run, made a single-day 250-row evaluation infeasible. 62 rows
  were evaluated as of July 5; remaining rows in the extended batch are
  pending quota reset.
- **3 false-negative phishing cases**: - **3 false-negative phishing cases**, all sharing the same underlying pattern:
  classic character-obfuscation/keyword-filter-evasion spam (scrambled or
  letter-inserted words, and in one case apparently corrupted/mis-encoded
  text) rather than modern social-engineering phishing. None contained a
  coherent urgency/payment/brand-mismatch narrative for the Linguistic Agent
  to score, nor an extractable URL for the reputation signal to weigh in on.
  ScamSentry is designed to detect psychologically manipulative scam
  narratives — the primary form of phishing/scam messaging today — rather
  than legacy keyword-filter-evasion spam, and this is a reasonable scope
  boundary to state explicitly rather than a defect to hide.

---
_Reproduce these numbers: `python eval/prepare_golden_sample.py` →
`python eval/run_golden_eval.py --input eval/golden_sample.csv` →
`python eval/score_golden_eval.py`._