v10 model card: context-gated fusion, version history v2-v10, plain-language architecture guide

README.md

@@ -26,9 +26,9 @@ tags:
   - interpretable-ml
   - tfidf
   - residual-network
-  - cross-signal-fusion
-  - lexical-analysis
+  - context-gating
   - calibrated-probabilities
+  - dmarc-alignment
 datasets:
   - ceas-2008
   - enron-email
@@ -48,20 +48,20 @@ model-index:
           type: text-classification
           name: Phishing Email Detection
         dataset:
-          name: 7-corpus benchmark (CEAS, Enron, SpamAssassin, Ling-Spam, Nazario, Nigerian, farshad72)
+          name: 7-source benchmark (CEAS, Enron, SpamAssassin, Ling-Spam, Nazario, Nigerian, farshad72, puyang2025)
           type: ceas-2008
         metrics:
           - type: f1
             value: 0.9445
             name: F1 Score
           - type: accuracy
-            value: 0.9470
+            value: 0.9477
             name: Accuracy
           - type: precision
-            value: 0.9369
+            value: 0.9402
             name: Precision
           - type: recall
-            value: 0.9523
+            value: 0.9489
             name: Recall
 widget:
   - text: "From: PayPal Security <security@paypa1-alert.tk>\nReply-To: attacker@evil-domain.ru\nSubject: URGENT: Your account will be suspended\n\nDear Customer, your PayPal account has been suspended. Verify now at http://paypal-login.tk/verify"
@@ -70,22 +70,17 @@ widget:
     example_title: "Legitimate email example"
 ---
 
-# Phish_Byte v8
+# Phish_Byte v10
 
 A from-scratch PyTorch model for **email phishing detection** — no pretrained weights, no transformers, no fine-tuning.
 
-**F1 0.944** on 5,000 held-out samples from a 7-corpus, 166K-email benchmark.
-**716K parameters** (~90× smaller than DistilBERT).
-**815 emails/sec** on a laptop GPU.
-**104 engineered features** across 8 analysis modules.
-**Temperature-calibrated confidence** — probability outputs are empirically calibrated, not just monotonic scores.
-Every verdict explains itself with full per-feature attribution.
+**743K parameters.** Every signal is a feature computed directly from the email — domain consistency, live SPF/DKIM/DMARC validation, link forensics, character-level lexical analysis — fed through a **context-gating architecture** that learns how one signal should change the interpretation of another, rather than a fixed hand-written rule.
 
 > The only non-transformer phishing detection model on HuggingFace.
 
 ---
 
-## ⚠️ Install — no PyPI package yet
+## Install — no PyPI package yet
 
 `pip install phishbyte` does **not** work. Clone the source repository:
 
@@ -94,10 +89,14 @@ git clone https://github.com/AnonymousSingh-007/Phish_Byte.git
 cd Phish_Byte
 python -m venv venv && source venv/bin/activate   # Windows: .\venv\Scripts\Activate.ps1
 pip install -r requirements.txt
-python verify_install.py    # confirms everything before you start
+python verify_install.py
 ```
 
-Then from inside the cloned folder:
+`verify_install.py` checks every dependency and source file, then does a live test download from this Hub repo. Run it before anything else — it identifies exactly what's wrong instead of a confusing traceback.
+
+---
+
+## Usage
 
 ```python
 from phishbyte import PhishByteEngine
@@ -106,63 +105,45 @@ engine  = PhishByteEngine.from_pretrained("SamSec007/phishbyte")
 verdict = engine.analyze(raw_email_string)
 
 print(verdict.label)             # "phishing" or "legitimate"
-print(verdict.probability)       # calibrated P(phish) in [0.0, 1.0]
+print(verdict.probability)       # calibrated P(phish), 0.0–1.0
 print(verdict.confidence)        # "high" / "medium" / "low"
-print(verdict.layer_used)        # 1 = rules decided, 2 = MLP decided
-print(verdict.feature_weights)   # all 104 signal values
+print(verdict.layer_used)        # 1 = fast rule veto, 2 = full network
+print(verdict.feature_weights)   # every signal computed for this email
 ```
 
-`from_pretrained()` downloads ~3 MB (weights + thresholds + TF-IDF vocab) and caches locally. Every call after the first is instant.
-
----
+`from_pretrained()` downloads the weights, calibration thresholds, and TF-IDF vocabulary from this repo on first call and caches them locally — every call after that is instant.
 
-## Analyse a real email from Gmail
+### Analyze a real email from Gmail
 
 1. Open the email → **⋮** → **Show original**
 2. Select all (Ctrl+A), copy (Ctrl+C)
 3. Run `python cli.py`, paste when prompted, press Enter then Ctrl+Z (Windows) or Ctrl+D (Mac/Linux)
 
-Or save as `.eml`:
+Or save as `.eml` and run:
 
 ```bash
 python cli.py --file suspicious.eml
 ```
 
----
-
-## What changed in v8 vs v7
-
-| | v7 | v8 |
-|---|:---:|:---:|
-| Parameters | 254K | **716K** |
-| Features | 85 | **104** |
-| Architecture | 85→360→180×2→90→48→1 | **104→620→310×2→155→76→1** |
-| Training corpus | 83K emails, 6 sources | **166K emails, 7 sources** |
-| F1 (held-out) | 0.950 | **0.944** |
-| Cross-signal fusion | None | **Yes — 5 inter-module features** |
-| Lexical domain analysis | None | **Yes — character-level on sender + link domain** |
-| BDI features | 3 | **5 (+ IP-target forms, open redirects)** |
-| Confidence calibration | Post-hoc threshold | **Learned temperature parameter (Platt scaling)** |
-| Training metric | Naive 0.5 cutoff | **Youden-optimal threshold** |
+### CLI quick demo
 
-The F1 drop from 0.950 to 0.944 reflects a harder benchmark — the corpus doubled in size and the evaluation pool now includes modern notification-style legitimate emails (CNN news digests, mailing lists, marketing email) that the 83K model never saw.
+```bash
+python cli.py --demo phish     # a known phishing example, no files needed
+python cli.py --demo legit     # a known legitimate example
+```
 
 ---
 
-## How it works — plain language
+## How it works, in plain language
 
-Eight independent analysis modules run on every email:
+Every email is analyzed by several small, independent checks — the sender's domain, the links in the body, whether the mail server proved its identity via SPF/DKIM/DMARC, the subject line, the wording of the message. Each check produces a handful of numbers between 0 and 1.
 
-1. **Domain analysis** — checks whether From, Reply-To, and Return-Path addresses are internally consistent; detects display-name spoofing (e.g. "PayPal Security" sending from an unrelated domain); flags suspicious domain patterns
-2. **URL and body analysis** — HTTPS/HTTP ratio, anchor text vs href mismatch, urgency language normalized per 100 words, link density by unique destinations
-3. **SPF validation** — live DNS lookup to verify the sending server is authorized
-4. **Subject line analysis** — urgency, currency, brand names, ALL-CAPS, fake RE: prefixes
-5. **Body Domain Identification (BDI)** — finds the most common link destination domain; flags form actions pointing at raw IP addresses; detects open-redirect URL patterns
-6. **Lexical domain analysis** — character-level forensics on the sender domain AND the most-linked domain: digit runs, hyphen stacking, Shannon entropy, typosquat distance to known brands (leet-normalized: `paypa1` → `paypal`)
-7. **Cross-signal fusion** — computes interaction features across all six modules: trust consistency (SPF pass + domain agreement + BDI match), multi-module agreement score, domain/BDI compounding, lexical brand confusion
-8. **TF-IDF vocabulary** — 50 discriminative unigrams learned from the training corpus (no pretrained LM)
+The key design choice is what happens next. Rather than mixing every number together and hoping a network untangles the relationships on its own, Phish_Byte splits its evidence into two groups before the network ever sees them:
 
-All 104 outputs concatenate into one vector and feed a residual MLP with a learned temperature scalar for calibrated confidence.
+- **Raw evidence** — facts about this specific email, like "the Reply-To address differs from the sender" or "the most common link goes somewhere else."
+- **Context evidence** — facts that change how the raw evidence should be read, like "the server cryptographically proved its identity via DMARC and DKIM," or "this urgency is coercive rather than just professionally formal."
+
+A dedicated layer in the network learns a *gate* — for every email, how much should the context evidence turn up or down the weight given to each piece of raw evidence. This isn't a hardcoded rule like "if DMARC passes, ignore link mismatches." The network learns the actual relationship from real data, so it can distinguish a Gmail Reply-To on an authenticated academic mailing list from a Gmail Reply-To on an unauthenticated bank impersonation attempt — two situations a fixed rule would treat identically.
 
 ---
 
@@ -170,105 +151,97 @@ All 104 outputs concatenate into one vector and feed a residual MLP with a learn
 
 ```
 raw email
-  → 8 analysis modules → 104-dim feature vector
-  → Layer 1 gate: composite score ≥ 0.85 → fast PHISHING verdict
-  → Layer 2: residual MLP
-      104 → 620 → 310 (×2 ResBlock) → 155 → 76 → 1
-      + input-to-output skip connection
-      + learned temperature scalar (Platt scaling)
-  → calibrated P(phish) + PhishVerdict with 104-feature attribution
+  → domain / URL / auth (SPF+DKIM+DMARC) / subject / link-forensics / lexical analysis
+  → cross-signal agreement check
+  → context feature computation (coercive vs professional urgency,
+     auth-verified sender context, freemail Reply-To context)
+  → RAW evidence (38 numbers) and CONTEXT evidence (13 numbers), kept separate
+  → Context Fusion Layer: learns a gate deciding how context reweights raw evidence
+  → fused representation + 50 TF-IDF word signals
+  → residual MLP: 620 → 310 (×2 residual blocks) → 155 → 76 → 1
+  → temperature-calibrated confidence score
+  → PhishVerdict with every raw and context signal that fired
 ```
 
 ---
 
-## Benchmarks
+## Version history
 
-Evaluated on 5,000 held-out samples, self-reported.
+**v2 — 12K parameters, 29 features, single dataset (CEAS-2008, ~39K emails).** The original prototype: a small MLP over hand-picked domain, URL, and subject features. Established the cascading Layer 1 (cheap rules) → Layer 2 (neural network) design that every later version kept.
 
-| Metric | Phish_Byte v8 | DistilBERT fine-tuned\* |
-|--------|:------------:|:---------------------:|
-| F1 score | 0.944 | ~0.967 |
-| Accuracy | 94.70% | ~97% |
-| Parameters | **716K** | 66,000,000 |
-| Model size | **~3 MB** | ~263 MB |
-| Throughput (GPU) | **815/sec** | ~50/sec |
-| GPU required | **No** | Practically yes |
-| Header + SPF analysis | **Yes** | No |
-| Per-feature attribution | **104 features** | Token-level SHAP |
-| Confidence calibrated | **Yes (temperature)** | No |
+**v7 — 254K parameters, 85 features, 83K emails across 6 datasets.** Added a TF-IDF vocabulary learned directly from the training corpus and Body Domain Identification (most-common-link-domain analysis). First version with real generalization testing beyond a single dataset.
 
-\* Self-reported by a different author on a different split. Not apples-to-apples — treat both F1 numbers as directional.
+**v8 — 716K parameters, 104 features, 166K emails across 7 datasets.** Added character-level lexical domain analysis (digit runs, entropy, typosquat distance) and a cross-signal layer that checks whether independent modules agree with each other, rather than scoring each in isolation.
 
----
+**v9 — 718K parameters, 107 features.** Replaced a naive SPF-only authentication check — which misfired constantly on legitimate marketing platforms like Marketo and Mailgun that relay mail on a brand's behalf — with a proper DMARC/DKIM alignment check that reads what the receiving mail server already validated.
 
-## Feature groups (104 total)
-
-| Group | Count | What it captures |
-|-------|:-----:|-----------------|
-| Domain | 7 | header consistency, brand impersonation, display-name spoof, suspicious pattern |
-| URL + Body | 10 | link security, anchor mismatch, urgency, caps ratio, digit ratio |
-| SPF | 3 | live DNS sender authorization |
-| Subject | 7 | urgency, security theme, brand, currency, caps, fake RE, fake txn ID |
-| BDI | 5 | MCLD mismatch, form action mismatch, external link ratio, IP-target form, open redirect |
-| Lexical — sender domain | 6 | digit runs, hyphen runs, entropy, vowel anomaly, typosquat distance, length |
-| Lexical — most-linked domain | 6 | same six on the dominant link destination |
-| Cross-signal fusion | 5 | trust consistency, SPF-pass URL discount, multi-module agreement, domain/BDI compounding, lexical brand confusion |
-| TF-IDF | 50 | top-50 discriminative unigrams from training corpus |
-| Composite | 5 | per-module summary scores |
+**v10 (current) — 743K parameters, context-gated fusion architecture.** Split urgency detection into two independent signals (coercive pressure vs. professional formality, previously conflated into one number). Restructured the network so raw evidence and context evidence enter as separate inputs to a learned fusion layer, instead of being concatenated and left for the network to disentangle unaided.
 
 ---
 
-## Training corpus (166K emails, 7 sources)
-
-| Source | Emails |
-|--------|-------:|
-| CEAS-2008, Enron, SpamAssassin, Ling-Spam, Nazario, Nigerian Fraud | ~83K |
-| farshad72/spam_email (HuggingFace) | ~83K |
-| **Total after dedup** | **~166K** |
-
-Balance: ~56% phishing / 44% legitimate.
+## Feature groups
+
+| Group | Count | What it measures |
+|-------|:-----:|-------------------|
+| Domain (raw) | 7 | header consistency, brand impersonation, display-name spoofing |
+| URL + body (raw) | 4 | link security, anchor/href mismatch, link density |
+| SPF (raw) | 3 | basic sender authorization signal |
+| Subject (raw) | 7 | brand mentions, currency, formatting, fake reply prefixes |
+| Char-level (raw) | 5 | capitalization, digit density, HTML/text ratio |
+| BDI (raw) | 5 | most-common-link-domain mismatch, IP-target forms, open redirects |
+| Lexical — sender domain (raw) | 6 | digit runs, hyphen runs, entropy, typosquat distance |
+| Coercive urgency (raw) | 1 | pressure/threat language |
+| Auth-verified ESP context | 1 | how strongly DMARC/DKIM validate the sender |
+| Professional formality (context) | 1 | independent of coercive urgency |
+| Freemail Reply-To context | 2 | raw fact + how much authentication offsets it |
+| Tracking pattern (observational) | 1 | structural resemblance to ESP tracking infrastructure — never used to exempt anything |
+| Cross-signal fusion (context) | 5 | agreement between independent modules |
+| Auth alignment (context) | 3 | DMARC pass, DKIM alignment, composite score |
+| TF-IDF | 50 | words learned directly from the training corpus |
 
 ---
 
-## Troubleshooting
+## Training data
 
-Run `python verify_install.py` first — it identifies the exact problem rather than a confusing traceback.
-
-| Error | Fix |
-|-------|-----|
-| `ModuleNotFoundError: No module named 'phishbyte'` | Not in cloned folder or venv not activated |
-| `ImportError: cannot import name 'X'` | `git pull origin main` |
-| `pip install phishbyte` fails | No PyPI package yet — clone the repo |
-| Model download hangs | Check internet — Hub: huggingface.co/SamSec007/phishbyte |
-| Windows symlink warning | Harmless — ignore or enable Developer Mode |
+| Dataset | Source |
+|---------|--------|
+| CEAS-2008, Enron, SpamAssassin, Ling-Spam, Nazario, Nigerian Fraud | Kaggle |
+| farshad72/spam_email | HuggingFace, 83K rows |
+| puyang2025/seven-phishing-email-datasets | HuggingFace, 203K rows, 7-source unified |
+| **Combined, after deduplication** | **~166,000 emails, ~56% phishing / 44% legitimate** |
 
 ---
 
-## Limitations — read before deploying
+## Benchmarks
 
-- **Most training data predates 2010.** Modern phishing (OAuth abuse, QR lures, redirect chains through legitimate cloud services) is underrepresented. Recall on 2020s attacks is not independently verified.
-- **No DMARC feature yet.** Emails sent via legitimate ESPs (Marketo, Mailgun, SendGrid) will trigger `spf_fail` even when `dmarc=pass`. This causes false positives on marketing email from large organizations. DMARC extraction is the next planned feature.
-- **No adversarial robustness testing.** Use as one signal in defence-in-depth, not a standalone gate.
-- **F1 0.944 is self-reported** on a held-out split of the training corpus.
-- **English-language only.**
+Evaluated on 12,000 held-out samples, self-reported.
+
+| Metric | Value |
+|--------|:-----:|
+| F1 score | 0.9445 |
+| Accuracy | 94.77% |
+| Precision | 0.9402 |
+| Recall | 0.9489 |
+| Parameters | 743K |
+| Model size | ~3 MB |
+| GPU required | No |
 
 ---
 
-## Roadmap
+## Limitations
 
-- [ ] DMARC feature — fixes false positives on ESP-delivered legitimate email
-- [ ] Retrain on 2020–2024 phishing data (PhishTank, OpenPhish, APWG eCrime)
-- [ ] HuggingFace Space demo (try in-browser, zero install)
-- [ ] PyPI package (`pip install phishbyte`)
-- [ ] arXiv preprint
-- [ ] Adversarial robustness test suite
+- **Most training data predates 2010.** Modern phishing (OAuth abuse, QR lures, redirect chains through legitimate cloud services) is underrepresented.
+- **False positives on legitimate marketing/notification email are reduced but not eliminated.** The context-gating architecture measurably helps but this remains an active area of work.
+- **No adversarial robustness testing has been performed.** Use as one signal in defence-in-depth, not a standalone gate.
+- **Benchmark numbers are self-reported**, not independently verified.
+- **English-language only.**
 
 ## Citation
 
 ```bibtex
 @software{phishbyte2026,
   author = {Singh, Samratth},
-  title  = {Phish_Byte: Cascading from-scratch PyTorch phishing detection},
+  title  = {Phish_Byte: Context-gated fusion architecture for from-scratch email phishing detection},
   year   = {2026},
   url    = {https://github.com/AnonymousSingh-007/Phish_Byte}
 }