Akash-Sakala/bert-phishing-classifier_teacher

BERT Phishing URL Classifier (Teacher)

A fine-tuned bert-base-uncased that classifies a URL as phishing or benign from its text alone. It is the teacher in a three-stage compression pipeline:

BERT-base teacher (110M, 12 layers)  ──KD──▶  DistilBERT student (~52M, 4 layers)  ──NF4──▶  4-bit student
        this repo                              _student                                  _student_4bit
      acc 89.7%                               acc 96.0%                                 see card

In this pipeline the distilled student ends up better than the teacher. The temperature-softened teacher outputs most likely acted as a strong regulariser. See the student card for details.

Results

Test split (33,000 URLs, balanced):

Model	Accuracy	Precision	Recall	F1
BERT teacher (this)	0.8971	0.9136	0.8763	0.8945
DistilBERT student	0.9601	0.9710	0.9483	0.9595

Validation split, per epoch:

Epoch	Train loss	Val loss	Accuracy	Precision	Recall	F1	ROC-AUC
1	0.3139	0.2644	0.892	0.918	0.860	0.888	0.961
2	0.3096	0.2569	0.895	0.912	0.873	0.892	0.963
3	0.3057	0.2569	0.895	0.918	0.867	0.892	0.963

Training

Data	Akash-Sakala/phishing-site-classification: 154k train / 33k val / 33k test
Labels	0 = benign, 1 = phishing
Learning rate	2e-5, linear schedule, 10% warm-up
Batch size	16
Epochs	3 (28,875 steps)
Optimizer	AdamW (fused), β = (0.9, 0.999), ε = 1e-8
Precision	Native AMP (fp16)
Seed	42
Frameworks	Transformers 5.0.0 · PyTorch 2.10.0 · Datasets 4.0.0 · Tokenizers 0.22.2

Usage

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

repo = "Akash-Sakala/bert-phishing-classifier_teacher"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()

urls = ["paypal.com.secure-login.verify-account.ru/webscr", "wikipedia.org/wiki/Phishing"]
inputs = tokenizer(urls, return_tensors="pt", truncation=True, padding=True, max_length=128)
with torch.no_grad():
    probs = model(**inputs).logits.softmax(-1)[:, 1]
for u, p in zip(urls, probs):
    print(f"{p:.2f}  {'PHISHING' if p > 0.5 else 'benign'}  {u}")

For deployment, prefer the student. It is smaller, faster and more accurate.

Limitations

• The model sees only the URL string. It doesn't check page content, certificates, domain age or reputation feeds, so treat it as one signal in a broader phishing-defence stack.
• Most URLs in the dataset have no http(s):// scheme. Strip the scheme before inference to match the training distribution.
• Phishing patterns drift over time, so accuracy on new campaigns will be lower than on this static test set.

Related

• Student (KD): bert-phishing-classifier_student
• 4-bit student: bert-phishing-classifier_student_4bit
• Dataset: phishing-site-classification