Instructions to use danjai/scamlens-qs-distilbert-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use danjai/scamlens-qs-distilbert-v2 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="danjai/scamlens-qs-distilbert-v2")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("danjai/scamlens-qs-distilbert-v2") model = AutoModelForSequenceClassification.from_pretrained("danjai/scamlens-qs-distilbert-v2", device_map="auto") - Notebooks
- Google Colab
- Kaggle
ScamLens quick_score: distilbert-v2
A classifier that reads the wording of an SMS or chat message and gives the probability that it is a scam.
It is the quick_score tool of ScamLens, a scam-investigation agent built for a university course project (IS469).
This is one of the candidate models that was trained and compared. It is not the one ScamLens runs with.
Training
| Setting | Value |
|---|---|
| base_model | distilbert/distilbert-base-uncased |
| epochs | 2 |
| lr | 2e-05 |
| batch_size | 8 |
| max_length | 128 |
| warmup_ratio | 0.1 |
| weight_decay | 0.01 |
| scheduler | linear |
| class_weights | balanced |
| val_ratio | 0.1 |
| seed | 42 |
Data
- Training examples: 7267 (6541 to train, 726 to validate)
- Files: sg_quick_score_train.csv, uci_sms_spam
- Training data SHA-256:
c7ee14ad903570865923493246450518c7027781be93ed94211d714b80b35491 - Test data SHA-256:
7ffae3e4dad2531b825b5e9ab44278eb3a9e5d09a3e6b4c140ea32bf21e8be23
The training data is Singapore-style scam and genuine messages: messages taken from or rebuilt from public scam advisories, synthetic messages, generated bank alerts in genuine and scam pairs, and everyday SMS from public corpora.
Results
- Validation F1: 0.989
- Test set: 178 hand-written Singapore messages that are never in the training data. A message counts as a scam at a probability of 0.50 or more. Ranges are 95% confidence intervals.
- Accuracy: 0.820 (0.757 to 0.870)
- Scam recall: 0.894 (0.820 to 0.940)
- Scam precision: 0.816 (0.735 to 0.876)
- F1: 0.853
- Genuine messages called a scam: 21 of 74
- Scams missed: 11 of 104
How to use
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
repo = "danjai/scamlens-qs-distilbert-v2"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()
text = "DBS: Your account is suspended. Verify at dbs-sg-unlock.com"
inputs = tokenizer(text, return_tensors="pt", truncation=True)
with torch.no_grad():
scam_prob = torch.softmax(model(**inputs).logits, dim=-1)[0, 1].item()
print(scam_prob)
Labels: 0 = genuine, 1 = scam.
In ScamLens
Add this line to .env. The model downloads the first time quick_score runs.
SCAMLENS_QUICK_SCORE_MODEL=danjai/scamlens-qs-distilbert-v2
ScamLens reads the probability as: high from 0.70, medium from 0.50, low below that.
Limitations
- It judges wording only. It does not check links, phone numbers or senders. In ScamLens those checks are separate tools, and this score is only a first opinion.
- It was trained on English messages in a Singapore setting (local banks, agencies and payment services). Expect it to be less reliable elsewhere.
- The test set is small (178 messages), so the confidence intervals are wide.
- Don't use it alone to block or accuse anyone.
- Downloads last month
- 53
Model tree for danjai/scamlens-qs-distilbert-v2
Base model
distilbert/distilbert-base-uncased