Scam Shield ยท DistilBERT

Reads one text message and says scam or safe.

A fine-tuned DistilBERT (67M parameters) trained on real scam reports from around the world, real everyday SMS, and written examples of both scams and safe messages. It runs on a normal laptop CPU.

How to use

Messages must be masked the same way as in training: links โ†’ <URL>, phone numbers โ†’ <PHONE_NUMBER>, emails โ†’ <EMAIL_ADDRESS>, long numbers โ†’ <NUMBER>. The masking code ships with this model as text_utils.py.

import importlib.util
from huggingface_hub import hf_hub_download
from transformers import pipeline

repo = "rasheq/scam-shield-distilbert"

# Load the masking code that ships with the model
path = hf_hub_download(repo, "text_utils.py", revision="v1.1.0")
spec = importlib.util.spec_from_file_location("text_utils", path)
text_utils = importlib.util.module_from_spec(spec)
spec.loader.exec_module(text_utils)

classify = pipeline("text-classification", model=repo, revision="v1.1.0")

message = "Hi Mum, I dropped my phone, this is my new number. Can you send me $300 today?"
print(classify(text_utils.prepare(message)))
# [{'label': 'scam', 'score': 0.99...}]

Labels: safe (0) and scam (1). A scam score above 0.5 means scam. The model was trained on messages with at least 3 words; shorter messages can't be judged reliably.

Results

Test B: 400 new messages (the honest score)

Written by a different AI from a generic prompt and never used for training or tuning: 199 scams and 201 safe messages from 18 countries, both easy and hard.

Model Accuracy Scams caught False alarms
v1.0.0 75.2% 67% (134 of 199) 17% (34 of 201)
v1.1.0 91.5% 89% (178 of 199) 6% (13 of 201)

On hard messages v1.1.0 scores 81.4% (79% of scams caught, 16% false alarms); on easy ones 95.7%.

Held-out test set: 1,510 real messages

Metric Score
Accuracy 98.5%
Precision 99.0%
Recall (scams caught) 98.8% (1,032 of 1,045)
F1 98.9%
False alarm rate 2.2% (10 of 465 safe messages)

Read this before trusting the numbers. The safe messages in the held-out test set are personal chats only, so its scores are higher than real life. Expect about 90%, as on Test B.

Training

Setting Value
Base model distilbert/distilbert-base-uncased
Extra tokens <URL>, <PHONE_NUMBER>, <EMAIL_ADDRESS>, <NUMBER>
Training data 21,054 messages (10,411 scam, 10,643 safe)
Max length 128 tokens
Batch size 16
Learning rate 2e-5, 10% warm-up, linear decay, AdamW (weight decay 0.01)
Epochs up to 4, early stop on validation loss; best epoch 2
Hardware Apple M3 Pro (MPS), about 3.5 minutes per epoch

Known limitations

  • English only, mostly SMS-style messages. Emails and long chats are outside what it learned.
  • It can't see link addresses. Every link becomes <URL>, so scams whose only clue is a fake-looking domain can slip through.
  • Long, multi-line real notices (bank statements, sign-in alerts) can still be flagged as scams.
  • Real messages with links or money (receipts, promos, fines, charity appeals) are the most common false alarms.
  • Some scams slip through: slow "wrong number" chats that turn into investment pitches, and some bank-alert scams that ask you to call a number.
  • Sensitive to wording: small rewording can flip the answer.
  • Label noise: some public scam reports used for training are real messages that users were unsure about.
  • A helper, not a guarantee. Never treat a "safe" result as proof that a message is safe.

Changes in 1.1.0

  • 4,000 new written training messages (hard cases, scams and safe).
  • 163 mislabelled public scam reports removed after a hand review.
  • Test B accuracy 75.2% โ†’ 91.5%, false alarms 17% โ†’ 6%.

Training data and credits

  • IMC 2025 smishing reports (CC BY 4.0): Agarwal, Papasavva, Suarez-Tangil, Vasek. Fishing for Smishing: Understanding SMS Phishing Infrastructure and Strategies by Mining Public User Reports. IMC '25. https://doi.org/10.1145/3730567.3764431
  • SMS Phishing Dataset for Machine Learning and Pattern Recognition (CC BY 4.0): Mishra, Soni. Mendeley Data, 2022. https://doi.org/10.17632/f45bkkt8pr.1
  • Synthetic safe service messages and hard cases (CC BY 4.0): written for this project with an AI model, then checked.
  • Base model: DistilBERT (Sanh et al., 2019), Apache-2.0.
Downloads last month
43
Safetensors
Model size
67M params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for rasheq/scam-shield-distilbert

Finetuned
(12,888)
this model

Dataset used to train rasheq/scam-shield-distilbert

Evaluation results