Sift — SMS Scam Classifier

Model overview

Sift is an AI-assisted SMS scam screening project designed to help international students review unfamiliar English messages. This repository contains its primary classification model, a trained scikit-learn pipeline that predicts Normal (0) or Scam (1) and outputs a score for the Scam class.

Developers: Can Jin and Gwen Guan (CMU course project, 2026).
Model type: TF-IDF word and character features + binary Logistic Regression, trained for this project (not a fine-tuned transformer).
Saved decision threshold: 0.38, chosen on validation data.
Secondary explanation model: Qwen2.5-1.5B-Instruct, used off the shelf in the application to explain the input message. Qwen does not change the classifier's prediction.
Augmentation model: google/flan-t5-small, used during data preparation for contextual paraphrasing, with a rule-based fallback. FLAN-T5 is not part of the deployed classifier.

Training data and provenance

The original labeled dataset has 500 examples: 250 Scam and 250 Normal. It includes SMS-style messages, adapted messages, and some source material from other communication channels; not every example was an independently observed real-world SMS.

Scam: 150 records from Mishra and Soni's SMS Phishing Dataset, 92 from the NCSU WSPR SMS Phishing Dataset, and eight examples drawn from university safety resources (Iowa, UC Davis, RPI, UC Berkeley).
Normal: 238 synthetic or adapted examples based on official information from organizations, including ANZ, ITV, U.S. Bank, O2, and Unity Bank, plus 12 ham-labeled records from the published Financial Scams Detection Dataset. Additional sources and references are documented in the Sift Final Report.
Source details and original record IDs are maintained in the Scam records and Normal records.

The original examples were divided before augmentation using seed 24679:

Split	Normal	Scam	Total
Original training	175	175	350
Validation	38	37	75
Test	37	38	75

The training set was expanded with 500 additional examples: 200 contextual paraphrases, 120 entity substitutions, 100 controlled typing variations, and 80 SMS-style variations. Only original training examples were used as augmentation parents. The final training set therefore has 850 examples; validation and test sets each contain 75 original examples. The augmentation record documents parent IDs and methods.

Licensing note: The project combines sources with different licenses or terms. For example, the Mishra–Soni dataset is attributed as CC BY 4.0 and the NCSU WSPR repository as MIT in the project documentation. These terms do not establish one uniform license for all original, adapted, or augmented messages. Review the underlying sources before redistributing data.

Model development and selection

Five configurations were compared: word unigram TF-IDF; word unigram/bigram TF-IDF; and three combined word-and-character TF-IDF models with Logistic Regression C values of 0.5, 1.0, and 2.0. Validation average precision was the main model-selection measure, followed by validation F1, recall, and training time.

The selected configuration was word_char_c2:

Word TF-IDF: unigrams and bigrams, maximum 20,000 features.
Character TF-IDF: char_wb character 3–5 grams, maximum 30,000 features.
Logistic Regression: C=2.0, solver='liblinear', L2 regularization, max_iter=3000.
Decision threshold: 0.38, selected from validation-set thresholds 0.10 through 0.90 in steps of 0.01.

The test set was evaluated after the configuration and threshold were fixed.

Evaluation

Metric (Scam is positive)	Validation (n=75)	Test (n=75)
Accuracy	97.33%	98.67%
Precision	97.30%	97.44%
Recall	97.30%	100.00%
F1	97.30%	98.70%

Test confusion matrix: 36 true negatives, 1 false positive, 0 false negatives, and 38 true positives. The one false positive was a legitimate package-delivery message that the model classified as Scam. These are results on a small project test set, not established real-world detection rates.

The saved archive also contains test_metrics.json, test_confusion_matrix.png, validation search records, and prediction/feature analysis files.

Intended use and limitations

This model is meant for educational decision support, not for automatically blocking messages or establishing whether a sender is legitimate. It may help users decide when to investigate a message further. Users should verify unexpected requests through independently obtained official contact information.

The dataset is small, English-language, partly synthetic/adapted, and drawn from several countries and communication channels.
Repeated patterns or similar templates could cause reported test performance to exceed performance on new real messages.
Training labels and source examples may contain errors; legitimate messages can be flagged and scams can be missed.
The score is an output of the statistical model, not a verified real-world probability that a message is fraudulent.
The classifier does not visit links, check telephone numbers, verify senders, or validate claims made by organizations.
In the application, Qwen-generated explanations may include unsupported statements. Rule-based output checks reduce certain mistakes but do not prove semantic accuracy.
Original datasets can contain personal or contact details and have not been fully redacted. Take care when viewing, sharing, or reusing them.

Reproduction: three Colab notebooks

These are three stages of one project, not three separately trained models. The model described on this card is the Logistic Regression classifier; Qwen has its own official model card.

Data preparation and augmentation — Upload the original Scam and Normal spreadsheets (using the expected filenames), check labels and duplicates, split the 500 messages, and augment training only. Outputs include train-850.csv, validation-75.csv, and test-75.csv.
Model training and evaluation — Upload the three processed CSVs, compare five classifier configurations, select the validation threshold, evaluate the test set, and export sms_scam_model_artifacts.zip containing the fitted .joblib pipeline and metadata.
Sift Gradio application — Load the saved classifier, load the published Qwen2.5 model, and launch the Gradio interface. The app notebook also contains a test-set explanation-check cell that reads the bundled test_predictions.csv, and an optional deployment-export cell.

Current application reproduction (October 2026):

The app notebook downloads sms_scam_model_artifacts.zip directly from this Hugging Face model repository into /content/sms_scam_model_artifacts/. The included Qwen test cell reads test_predictions.csv from the same extracted archive.

To launch the interface with the saved classifier, users do not need to rerun the data-preparation or training notebooks.

Run the notebook from the first cell.
If the dependency-installation cell prompts a session restart, select Runtime → Restart session.
After restarting, rerun the dependency-installation cell and all cells below it. There is no need to download the model again.
Load Qwen and launch Gradio.

The public gradio.live link is temporary and must be regenerated by rerunning the app. No Hugging Face login is required when the model repository is public.

To reproduce model development from raw data, obtain the original datasets through the cited sources and run the data-preparation and model-training notebooks in that order.

Load the saved classifier (trusted artifact only)

Install the packages listed in the archive's requirements.txt (the training notebook used scikit-learn 1.6.1). A simple inference example is:

from huggingface_hub import hf_hub_download
from pathlib import Path
import zipfile, json, joblib

archive = hf_hub_download(
    repo_id="Gwen1220/sift-sms-scam-classifier",
    filename="sms_scam_model_artifacts.zip",
)
folder = Path("sift_model")
folder.mkdir(exist_ok=True)
with zipfile.ZipFile(archive) as z:
    z.extractall(folder)

model = joblib.load(folder / "sms_scam_tfidf_logistic_pipeline.joblib")
config = json.loads((folder / "best_configuration.json").read_text())
text = "Your study group meets in the library tomorrow."
score = float(model.predict_proba([text])[0, 1])
print("Scam" if score >= config["threshold"] else "Normal", score)

Security: .joblib uses Python serialization, which can execute code on load; do not load model artifacts from untrusted sources. The ZIP also contains evaluation files, and should be reviewed before public redistribution.

Downloads last month: -; Downloads are not tracked for this model. How to track