Instructions to use rasheq/scam-shield-distilbert with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use rasheq/scam-shield-distilbert with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="rasheq/scam-shield-distilbert")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("rasheq/scam-shield-distilbert") model = AutoModelForSequenceClassification.from_pretrained("rasheq/scam-shield-distilbert", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Scam Shield ยท DistilBERT
Reads one text message and says scam or safe.
A fine-tuned DistilBERT (67M parameters) trained on real scam reports from around the world, real everyday SMS, and written examples of both scams and safe messages. It runs on a normal laptop CPU.
- Code, training and data pipeline: https://github.com/Mohamedrasheq/scam-shield
- Dataset: https://huggingface.co/datasets/rasheq/scam-shield-sms
- Version: 1.1.0 ยท English only
How to use
Messages must be masked the same way as in training: links โ <URL>, phone numbers โ <PHONE_NUMBER>, emails โ <EMAIL_ADDRESS>, long numbers โ <NUMBER>. The masking code ships with this model as text_utils.py.
import importlib.util
from huggingface_hub import hf_hub_download
from transformers import pipeline
repo = "rasheq/scam-shield-distilbert"
# Load the masking code that ships with the model
path = hf_hub_download(repo, "text_utils.py", revision="v1.1.0")
spec = importlib.util.spec_from_file_location("text_utils", path)
text_utils = importlib.util.module_from_spec(spec)
spec.loader.exec_module(text_utils)
classify = pipeline("text-classification", model=repo, revision="v1.1.0")
message = "Hi Mum, I dropped my phone, this is my new number. Can you send me $300 today?"
print(classify(text_utils.prepare(message)))
# [{'label': 'scam', 'score': 0.99...}]
Labels: safe (0) and scam (1). A scam score above 0.5 means scam. The model was trained on messages with at least 3 words; shorter messages can't be judged reliably.
Results
Test B: 400 new messages (the honest score)
Written by a different AI from a generic prompt and never used for training or tuning: 199 scams and 201 safe messages from 18 countries, both easy and hard.
| Model | Accuracy | Scams caught | False alarms |
|---|---|---|---|
| v1.0.0 | 75.2% | 67% (134 of 199) | 17% (34 of 201) |
| v1.1.0 | 91.5% | 89% (178 of 199) | 6% (13 of 201) |
On hard messages v1.1.0 scores 81.4% (79% of scams caught, 16% false alarms); on easy ones 95.7%.
Held-out test set: 1,510 real messages
| Metric | Score |
|---|---|
| Accuracy | 98.5% |
| Precision | 99.0% |
| Recall (scams caught) | 98.8% (1,032 of 1,045) |
| F1 | 98.9% |
| False alarm rate | 2.2% (10 of 465 safe messages) |
Read this before trusting the numbers. The safe messages in the held-out test set are personal chats only, so its scores are higher than real life. Expect about 90%, as on Test B.
Training
| Setting | Value |
|---|---|
| Base model | distilbert/distilbert-base-uncased |
| Extra tokens | <URL>, <PHONE_NUMBER>, <EMAIL_ADDRESS>, <NUMBER> |
| Training data | 21,054 messages (10,411 scam, 10,643 safe) |
| Max length | 128 tokens |
| Batch size | 16 |
| Learning rate | 2e-5, 10% warm-up, linear decay, AdamW (weight decay 0.01) |
| Epochs | up to 4, early stop on validation loss; best epoch 2 |
| Hardware | Apple M3 Pro (MPS), about 3.5 minutes per epoch |
Known limitations
- English only, mostly SMS-style messages. Emails and long chats are outside what it learned.
- It can't see link addresses. Every link becomes
<URL>, so scams whose only clue is a fake-looking domain can slip through. - Long, multi-line real notices (bank statements, sign-in alerts) can still be flagged as scams.
- Real messages with links or money (receipts, promos, fines, charity appeals) are the most common false alarms.
- Some scams slip through: slow "wrong number" chats that turn into investment pitches, and some bank-alert scams that ask you to call a number.
- Sensitive to wording: small rewording can flip the answer.
- Label noise: some public scam reports used for training are real messages that users were unsure about.
- A helper, not a guarantee. Never treat a "safe" result as proof that a message is safe.
Changes in 1.1.0
- 4,000 new written training messages (hard cases, scams and safe).
- 163 mislabelled public scam reports removed after a hand review.
- Test B accuracy 75.2% โ 91.5%, false alarms 17% โ 6%.
Training data and credits
- IMC 2025 smishing reports (CC BY 4.0): Agarwal, Papasavva, Suarez-Tangil, Vasek. Fishing for Smishing: Understanding SMS Phishing Infrastructure and Strategies by Mining Public User Reports. IMC '25. https://doi.org/10.1145/3730567.3764431
- SMS Phishing Dataset for Machine Learning and Pattern Recognition (CC BY 4.0): Mishra, Soni. Mendeley Data, 2022. https://doi.org/10.17632/f45bkkt8pr.1
- Synthetic safe service messages and hard cases (CC BY 4.0): written for this project with an AI model, then checked.
- Base model: DistilBERT (Sanh et al., 2019), Apache-2.0.
- Downloads last month
- 43
Model tree for rasheq/scam-shield-distilbert
Base model
distilbert/distilbert-base-uncasedDataset used to train rasheq/scam-shield-distilbert
Evaluation results
- accuracy on Scam Shield SMS (test split, real messages only)test set self-reported0.985
- precision on Scam Shield SMS (test split, real messages only)test set self-reported0.990
- recall on Scam Shield SMS (test split, real messages only)test set self-reported0.988
- f1 on Scam Shield SMS (test split, real messages only)test set self-reported0.989