Skibidibois2000/spam-malware-detection-model

Spam, Malware and Process Detection

Four models in one scikit-learn bundle, from the COS30049 (AI4Cyber) group project on spam and malware detection.

task	method	question	score
spam classification	TF-IDF + Logistic Regression	is this message spam?	F1 0.967 test, 0.902 on an unseen source
malware classification	HistGradientBoosting, one model per platform	is this file malware?	F1 0.886 to 0.992 per platform
spam clustering	K-Means on SVD of the TF-IDF	do the messages group on their own?	purity 0.732, ARI 0.215
process regression	Ridge on 31 process counters	how many context switches?	R2 0.962, MAE 954

The bundle

combined_model.joblib (about 3 MB) holds three keys:

key	contents
spam	TF-IDF vectorizer (20,000 unigrams) + Logistic Regression, threshold 0.5
malware	one pipeline per platform: pe_windows, android_drebin, android_permissions, android_tuandromd, each with median imputation and a classifier
regression	Ridge pipeline predicting nvcsw (voluntary context switches) from 31 Linux process counters

import joblib

bundle = joblib.load("combined_model.joblib")

# spam: probability that a message is spam
spam = bundle["spam"]
prob = spam["pipeline"].predict_proba(["WIN a FREE prize now!"])[0][1]

# malware: name the platform, because each one has its own columns
model = bundle["malware"]["platforms"]["pe_windows"]
model["pipeline"].predict_proba([{"NumberOfSections": 4, "SizeOfCode": 100000}])

# regression
bundle["regression"]["pipeline"].predict([{"utime": 350000, "map_count": 24}])

Columns left out of a malware or process sample are filled with the training median. The reply in the project's own API also reports how many columns were filled, so a weak answer is visible as a weak answer.

How it was trained and measured

Everything is rebuilt from the raw Kaggle downloads by one script. The full dataset is in Skibidibois2000/spam-malware-detection-dataset; the code and the report are in github.com/CodingNuz/Innovation-G2.

Each model is scored on data it never trained on. The spam model is also scored leave-one-source-out: train on seven corpora and test on the eighth, which is what happens when a user pastes a message from somewhere new. That is why the honest number is 0.902 while the ordinary cross-validation score is 0.966.

The regression split is grouped by process. All 1,000 snapshots of one process carry the same label, so a random split lets the model memorise the process: R2 falls from 0.980 (random) to 0.962 (grouped). The grouped number is the one reported.

Results

Spam, 88,077 messages from 8 corpora.

evaluation	F1
20% hold-out split	0.967
5-fold cross-validation	0.966 ± 0.001
mean over unseen sources	0.902

Per unseen source, worst first: sms_smishing_kaggle 0.749, sms_master 0.800, email_spamassassin 0.932, email_enron 0.941, email_phishing 0.962, email_ceas 0.964, email_ling 0.968.

Malware, 5-fold cross-validation F1, best model per platform:

platform	rows	Logistic Regression	HistGradientBoosting
pe_windows	39,669	0.952	0.992
android_drebin	7,257	0.918	0.954
android_permissions	7,379	0.917	0.936
android_tuandromd	660	0.837	0.886

Regression: R2 0.962, MAE 954, against R2 0.000 and MAE 6,390 for a model that always answers the average.

Files

path	contents
combined_model.joblib	all four models
bundle_manifest.json	what is inside the bundle
results/spam_cv_results.json	spam test split, 5-fold CV, leave-one-source-out, K-Means
results/spam_feature_importance.json	the 20 strongest spam and ham words
results/malware_results.json	both candidates scored on all four platforms
results/regression_results.json	regression scores, grouped and random split
figures/	confusion matrices and the five data figures

Known limits

• SMS is the weak spot: F1 is 0.749 to 0.800 when an SMS source is held out, against 0.932 to 0.968 for email. SMS messages are short, use slang, and are the smallest part of the training data.
• Each malware model only understands its own platform's columns, and the caller has to say which platform a sample comes from.
• The word list still carries a little source bias: enron and wrote are the two strongest "ham" words. Removing them by hand was measured and made the model worse, so they were left in.
• TUANDROMD has 660 rows, so its score is much less certain than the others (cross-validation standard deviation 0.046 against 0.001 to 0.008).
• All text data is English. The regression model describes a running process from one snapshot rather than predicting its future.

Citation

Student group project for COS30049 (AI4Cyber), Swinburne University of Technology. Dataset licences and authors are listed in Kaggle datasets source.md in the dataset repository.