v8 model card: 716K params, 104 features, 166K corpus, cross-signal fusion, temperature calibration, DMARC limitation noted

README.md

@@ -1,19 +1,41 @@
 ---
-language: en
+language:
+  - en
 license: mit
 library_name: phishbyte
 pipeline_tag: text-classification
 tags:
   - phishing-detection
   - email-security
+  - cybersecurity
+  - security
   - pytorch
   - from-scratch
   - no-pretrained-weights
   - cascading-inference
   - lightweight
   - explainable-ai
+  - nlp
+  - phishing
+  - spam-detection
+  - malware-detection
+  - threat-detection
+  - email-classification
+  - text-classification
+  - feature-engineering
+  - interpretable-ml
+  - tfidf
+  - residual-network
+  - cross-signal-fusion
+  - lexical-analysis
+  - calibrated-probabilities
 datasets:
-  - CEAS-2008
+  - ceas-2008
+  - enron-email
+  - spamassassin
+  - ling-spam
+  - nazario-phishing
+  - nigerian-fraud
 metrics:
   - f1
   - precision
@@ -26,102 +48,229 @@ model-index:
           type: text-classification
           name: Phishing Email Detection
         dataset:
-          name: CEAS-2008
+          name: 7-corpus benchmark (CEAS, Enron, SpamAssassin, Ling-Spam, Nazario, Nigerian, farshad72)
           type: ceas-2008
         metrics:
           - type: f1
-            value: 0.948
+            value: 0.9445
+            name: F1 Score
           - type: accuracy
-            value: 0.944
+            value: 0.9470
+            name: Accuracy
           - type: precision
-            value: 0.954
+            value: 0.9369
+            name: Precision
           - type: recall
-            value: 0.943
+            value: 0.9523
+            name: Recall
+widget:
+  - text: "From: PayPal Security <security@paypa1-alert.tk>\nReply-To: attacker@evil-domain.ru\nSubject: URGENT: Your account will be suspended\n\nDear Customer, your PayPal account has been suspended. Verify now at http://paypal-login.tk/verify"
+    example_title: "Phishing email example"
+  - text: "From: alice@company.com\nReply-To: alice@company.com\nSubject: Team lunch tomorrow\n\nHi everyone, lunch is at noon in the usual spot. See you there!"
+    example_title: "Legitimate email example"
 ---
 
-# Phish_Byte
+# Phish_Byte v8
 
-A from-scratch PyTorch model for **email phishing detection**.
-**F1 0.948** on CEAS-2008. **12,545 parameters** (≈9,000× smaller than DistilBERT).
-**1,500+ emails/sec** on a laptop GPU. Every verdict explains itself.
+A from-scratch PyTorch model for **email phishing detection** — no pretrained weights, no transformers, no fine-tuning.
 
-## Why this exists
+**F1 0.944** on 5,000 held-out samples from a 7-corpus, 166K-email benchmark.
+**716K parameters** (~90× smaller than DistilBERT).
+**815 emails/sec** on a laptop GPU.
+**104 engineered features** across 8 analysis modules.
+**Temperature-calibrated confidence** — probability outputs are empirically calibrated, not just monotonic scores.
+Every verdict explains itself with full per-feature attribution.
 
-Every phishing detection model on HuggingFace is currently a fine-tuned
-transformer (DistilBERT, BERT, RoBERTa) — 65 to 110 million parameters,
-~250 MB on disk, ~50 ms per email on GPU. Phish_Byte takes a different
-bet: a small custom MLP trained from scratch, fed by 29 carefully chosen
-features, routed through a cascading inference pipeline. The model is
-**9,000× smaller** than DistilBERT, performs competitively, deploys
-without a GPU, and explains every decision.
+> The only non-transformer phishing detection model on HuggingFace.
 
-## Usage
+---
+
+## ⚠️ Install — no PyPI package yet
+
+`pip install phishbyte` does **not** work. Clone the source repository:
+
+```bash
+git clone https://github.com/AnonymousSingh-007/Phish_Byte.git
+cd Phish_Byte
+python -m venv venv && source venv/bin/activate   # Windows: .\venv\Scripts\Activate.ps1
+pip install -r requirements.txt
+python verify_install.py    # confirms everything before you start
+```
+
+Then from inside the cloned folder:
 
 ```python
 from phishbyte import PhishByteEngine
 
-engine  = PhishByteEngine.from_pretrained("AnonymousSingh-007/phishbyte")
+engine  = PhishByteEngine.from_pretrained("SamSec007/phishbyte")
 verdict = engine.analyze(raw_email_string)
 
-print(verdict.label)             # 'phishing'
-print(verdict.probability)       # 0.9735
-print(verdict.confidence)        # 'high'
-print(verdict.layer_used)        # 2 — MLP made this call
-print(verdict.feature_weights)   # full per-feature attribution
+print(verdict.label)             # "phishing" or "legitimate"
+print(verdict.probability)       # calibrated P(phish) in [0.0, 1.0]
+print(verdict.confidence)        # "high" / "medium" / "low"
+print(verdict.layer_used)        # 1 = rules decided, 2 = MLP decided
+print(verdict.feature_weights)   # all 104 signal values
 ```
 
+`from_pretrained()` downloads ~3 MB (weights + thresholds + TF-IDF vocab) and caches locally. Every call after the first is instant.
+
+---
+
+## Analyse a real email from Gmail
+
+1. Open the email → **⋮** → **Show original**
+2. Select all (Ctrl+A), copy (Ctrl+C)
+3. Run `python cli.py`, paste when prompted, press Enter then Ctrl+Z (Windows) or Ctrl+D (Mac/Linux)
+
+Or save as `.eml`:
+
+```bash
+python cli.py --file suspicious.eml
+```
+
+---
+
+## What changed in v8 vs v7
+
+| | v7 | v8 |
+|---|:---:|:---:|
+| Parameters | 254K | **716K** |
+| Features | 85 | **104** |
+| Architecture | 85→360→180×2→90→48→1 | **104→620→310×2→155→76→1** |
+| Training corpus | 83K emails, 6 sources | **166K emails, 7 sources** |
+| F1 (held-out) | 0.950 | **0.944** |
+| Cross-signal fusion | None | **Yes — 5 inter-module features** |
+| Lexical domain analysis | None | **Yes — character-level on sender + link domain** |
+| BDI features | 3 | **5 (+ IP-target forms, open redirects)** |
+| Confidence calibration | Post-hoc threshold | **Learned temperature parameter (Platt scaling)** |
+| Training metric | Naive 0.5 cutoff | **Youden-optimal threshold** |
+
+The F1 drop from 0.950 to 0.944 reflects a harder benchmark — the corpus doubled in size and the evaluation pool now includes modern notification-style legitimate emails (CNN news digests, mailing lists, marketing email) that the 83K model never saw.
+
+---
+
+## How it works — plain language
+
+Eight independent analysis modules run on every email:
+
+1. **Domain analysis** — checks whether From, Reply-To, and Return-Path addresses are internally consistent; detects display-name spoofing (e.g. "PayPal Security" sending from an unrelated domain); flags suspicious domain patterns
+2. **URL and body analysis** — HTTPS/HTTP ratio, anchor text vs href mismatch, urgency language normalized per 100 words, link density by unique destinations
+3. **SPF validation** — live DNS lookup to verify the sending server is authorized
+4. **Subject line analysis** — urgency, currency, brand names, ALL-CAPS, fake RE: prefixes
+5. **Body Domain Identification (BDI)** — finds the most common link destination domain; flags form actions pointing at raw IP addresses; detects open-redirect URL patterns
+6. **Lexical domain analysis** — character-level forensics on the sender domain AND the most-linked domain: digit runs, hyphen stacking, Shannon entropy, typosquat distance to known brands (leet-normalized: `paypa1` → `paypal`)
+7. **Cross-signal fusion** — computes interaction features across all six modules: trust consistency (SPF pass + domain agreement + BDI match), multi-module agreement score, domain/BDI compounding, lexical brand confusion
+8. **TF-IDF vocabulary** — 50 discriminative unigrams learned from the training corpus (no pretrained LM)
+
+All 104 outputs concatenate into one vector and feed a residual MLP with a learned temperature scalar for calibrated confidence.
+
+---
+
 ## Architecture
 
 ```
-Layer 1 — rule scorers (~1 ms): domain + URL + SPF + subject
-            │
-            ├──► obvious phishing? short-circuit verdict
-            │
-            └──► otherwise route to MLP
-                       │
-Layer 2 — MLP (~3 ms): 29 → 96 → 48 → 1 (sigmoid)
-            │
-            ▼
-        PhishVerdict {label, probability, confidence, layer_used, feature_weights}
+raw email
+  → 8 analysis modules → 104-dim feature vector
+  → Layer 1 gate: composite score ≥ 0.85 → fast PHISHING verdict
+  → Layer 2: residual MLP
+      104 → 620 → 310 (×2 ResBlock) → 155 → 76 → 1
+      + input-to-output skip connection
+      + learned temperature scalar (Platt scaling)
+  → calibrated P(phish) + PhishVerdict with 104-feature attribution
 ```
 
-## Performance (CEAS-2008, n=2000 held-out)
+---
+
+## Benchmarks
+
+Evaluated on 5,000 held-out samples, self-reported.
+
+| Metric | Phish_Byte v8 | DistilBERT fine-tuned\* |
+|--------|:------------:|:---------------------:|
+| F1 score | 0.944 | ~0.967 |
+| Accuracy | 94.70% | ~97% |
+| Parameters | **716K** | 66,000,000 |
+| Model size | **~3 MB** | ~263 MB |
+| Throughput (GPU) | **815/sec** | ~50/sec |
+| GPU required | **No** | Practically yes |
+| Header + SPF analysis | **Yes** | No |
+| Per-feature attribution | **104 features** | Token-level SHAP |
+| Confidence calibrated | **Yes (temperature)** | No |
+
+\* Self-reported by a different author on a different split. Not apples-to-apples — treat both F1 numbers as directional.
+
+---
+
+## Feature groups (104 total)
+
+| Group | Count | What it captures |
+|-------|:-----:|-----------------|
+| Domain | 7 | header consistency, brand impersonation, display-name spoof, suspicious pattern |
+| URL + Body | 10 | link security, anchor mismatch, urgency, caps ratio, digit ratio |
+| SPF | 3 | live DNS sender authorization |
+| Subject | 7 | urgency, security theme, brand, currency, caps, fake RE, fake txn ID |
+| BDI | 5 | MCLD mismatch, form action mismatch, external link ratio, IP-target form, open redirect |
+| Lexical — sender domain | 6 | digit runs, hyphen runs, entropy, vowel anomaly, typosquat distance, length |
+| Lexical — most-linked domain | 6 | same six on the dominant link destination |
+| Cross-signal fusion | 5 | trust consistency, SPF-pass URL discount, multi-module agreement, domain/BDI compounding, lexical brand confusion |
+| TF-IDF | 50 | top-50 discriminative unigrams from training corpus |
+| Composite | 5 | per-module summary scores |
+
+---
+
+## Training corpus (166K emails, 7 sources)
+
+| Source | Emails |
+|--------|-------:|
+| CEAS-2008, Enron, SpamAssassin, Ling-Spam, Nazario, Nigerian Fraud | ~83K |
+| farshad72/spam_email (HuggingFace) | ~83K |
+| **Total after dedup** | **~166K** |
+
+Balance: ~56% phishing / 44% legitimate.
+
+---
 
-| Metric           | Value     |
-|------------------|----------:|
-| F1 score         | **0.948** |
-| Accuracy         | 94.40%    |
-| Precision        | 0.9537    |
-| Recall           | 0.9432    |
-| Parameters       | 12,545    |
-| Model size       | ~50 KB    |
-| Throughput (GPU) | 1,527 /s  |
-| Throughput (CPU) | ~800 /s   |
+## Troubleshooting
 
-## Features (29 inputs)
+Run `python verify_install.py` first — it identifies the exact problem rather than a confusing traceback.
 
-- **Domain (5)**: From/Reply-To/Return-Path mismatch, freemail flag, brand impersonation
-- **URL (5)**: HTTPS ratio, anchor mismatch, suspicious TLD, urgency, link density
-- **SPF (3)**: SPF fail, no record, no sending IP
-- **Subject (7)**: urgency, security theme, brand name, currency, all caps, fake RE, fake transaction ID
-- **Character-level (5)**: caps ratio, digit ratio, special chars, avg word length, HTML/text ratio
-- **Composite (4)**: per-layer normalized scores
+| Error | Fix |
+|-------|-----|
+| `ModuleNotFoundError: No module named 'phishbyte'` | Not in cloned folder or venv not activated |
+| `ImportError: cannot import name 'X'` | `git pull origin main` |
+| `pip install phishbyte` fails | No PyPI package yet — clone the repo |
+| Model download hangs | Check internet — Hub: huggingface.co/SamSec007/phishbyte |
+| Windows symlink warning | Harmless — ignore or enable Developer Mode |
+
+---
+
+## Limitations — read before deploying
+
+- **Most training data predates 2010.** Modern phishing (OAuth abuse, QR lures, redirect chains through legitimate cloud services) is underrepresented. Recall on 2020s attacks is not independently verified.
+- **No DMARC feature yet.** Emails sent via legitimate ESPs (Marketo, Mailgun, SendGrid) will trigger `spf_fail` even when `dmarc=pass`. This causes false positives on marketing email from large organizations. DMARC extraction is the next planned feature.
+- **No adversarial robustness testing.** Use as one signal in defence-in-depth, not a standalone gate.
+- **F1 0.944 is self-reported** on a held-out split of the training corpus.
+- **English-language only.**
+
+---
 
-## Limitations
+## Roadmap
 
-- ~5% of decisions are wrong (F1 0.948, not 1.0). Use as one signal in defence-in-depth, not the only gate.
-- Trained on CEAS-2008 — English-language phishing from 2008. Modern attack patterns and non-English emails will degrade performance.
-- SPF validation is bypassed for training (historical domains don't resolve) but runs live at inference time.
-- Adversarial emails crafted specifically to game these features will get through.
+- [ ] DMARC feature — fixes false positives on ESP-delivered legitimate email
+- [ ] Retrain on 2020–2024 phishing data (PhishTank, OpenPhish, APWG eCrime)
+- [ ] HuggingFace Space demo (try in-browser, zero install)
+- [ ] PyPI package (`pip install phishbyte`)
+- [ ] arXiv preprint
+- [ ] Adversarial robustness test suite
 
 ## Citation
 
 ```bibtex
 @software{phishbyte2026,
-  author  = {Singh, Samratth},
-  title   = {Phish_Byte: A cascading from-scratch PyTorch model for email phishing detection},
-  year    = {2026},
-  url     = {https://github.com/AnonymousSingh-007/Phish_Byte}
+  author = {Singh, Samratth},
+  title  = {Phish_Byte: Cascading from-scratch PyTorch phishing detection},
+  year   = {2026},
+  url    = {https://github.com/AnonymousSingh-007/Phish_Byte}
 }
 ```