presentation/reports/error_analysis_test3.md

Error analysis — test3 split (three-way holdout), run ft-focal (2026-10-09 10:46)

All quoted emails are rows of public corpora (source Assassin, CEAS-08, Enron, Nazario, Nigerian_Fraud, TREC-07); nothing here is private mail. Threshold 0.5. Binary positive class = Unwanted (phishing or spam), P(unwanted) = P(phishing) + P(spam).

Counts (binary roll-up)

• rows 3,220 (positives 1,221, 37.9%); label3 counts legit 1,999, phish 521, spam 700
• recall (unwanted) 0.9836 — 20 missed unwanted emails (false negatives) out of 1,221
• precision 0.9893 — 13 false alarms out of 1,214 flagged
• F1 0.9864, accuracy 0.9898, specificity 0.9935
• ROC-AUC 0.9993, PR-AUC 0.9989; recall at 1% FPR 0.9869, at 0.1% FPR 0.8788
• credential-lure slice: 275 rows, 268 positive (97.5% label purity: slice precision cannot exceed what the phrase list selects), recall 1.0000, precision 1.0000, 0 missed

Top-25 false negatives (missed unwanted mail, most confident first)

#	doc_id	P(unwanted)	tokens (full)	source	label_set	preview (subject │ body, 160 chars)
1	primary:191966	0.0224	86	Enron	spam	re : saturday . here new and │ good day : i tried to call your 3 three time but your phone is not available . i think you did a mistake during filling the form ...
2	primary:148887	0.0467	1097	Enron	spam	southern california demographic analysis workshop announcement ( january charset = windows - 1252 " > │ analyzing southern california communities : demographic ...
3	aura:Nazario:000097	0.1782	39	Nazario	phish	Oil Payment Approval │ Find the attached document via Google secure to view closing statement. All parties have completed Documents. please sign these at your e...
4	primary:010013	0.2112	135	Enron	spam	new birthday calendar │ hi i ' m setting up a birthday calendar and need your help . just click the link below and enter your birthday details . ( it ' s quick ...
5	primary:007176	0.2300	281	Assassin	spam	TOMORROW -- Password: Soleil -- TV Casting │ Tomorrow will be huge. Password: Soleil has become the premiere Thursday night in New York - with a chill after-wor...
6	aura:Nazario:000643	0.2584	368	Nazario	phish	Phishing Email Corpus │ Good Morning, Jose, My name is Michael Kotson, and I am a Cybersecurity researcher at MIT Lincoln Laboratory. I am developing a lecture ...
7	primary:181246	0.2858	1051	Assassin	spam	[ILUG] (no subject) │ Suppliers of Computers, Printers, etc. & Consumables Central Data Supplies 24, Middleton Road, Banbury, Oxon, OX16 4QJ. Tel: 01295 271666 ...
8	primary:095405	0.2967	40	Enron	spam	pocket micro scope id : sxo 683 │ re : ppql 308243023 not interested , please let us know : powerlinepromo @ yahoo . com
9	primary:014300	0.3203	301	Enron	spam	cecil county networking lunch │ networking lunch chesapeake bay golf club in north east wednesday , november 17 th 11 : 15 am smallcompanies . com in conjunctio...
10	primary:001939	0.3348	1153	Assassin	spam	Invite: Content Management Summit, Oct. 10th New York City │ ![CMS](http://www.cmsummit.com/images/cms_title...
11	primary:033220	0.3678	28444	Assassin	spam	Wir suchen Erotikdarsteller! │ This is a Multipart MIME message. Since your mail reader does not understand this format, some or all of this message may not be ...
12	primary:025244	0.3789	8	Enron	spam	message subject │ goodbye
13	primary:133074	0.3791	86	Enron	spam	his lips , his nose , │ good day : i tried to call your 3 three time but your phone is not available . i think you did a mistake during filling the form . anywa...
14	aura:Nigerian_Fraud:003049	0.3823	516	Nigerian_Fraud	phish	Invitation from aditya monu │ This is a multi-part message in MIME format. --incrowd_6506e2e6636163656626f7892e636f26d05b3743 Content-Type: text/plain; charset=...
15	aura:Nazario:000299	0.3973	46	Nazario	phish	RE: IT Administrator │ IT Administrator desk: has currently upgraded our mailbox service quote to 150GB space Faculty/Staff are hereby advice to click on WEBMAI...
16	primary:046569	0.4039	1078	Enron	spam	invite : content management summit , oct . 10 th new york city │ dear friends , the online content business is doing fantastic . sound crazy ? perhaps . with ad...
17	primary:135631	0.4338	18	Enron	spam	reply : shy mom wants to date you │ hello there ! sode
18	primary:148857	0.4750	180	Assassin	spam	send REAL PAPER GREETING CARDS on-line! │ Cards In Advance Birthday Anniversary Get Well Soon Thinking of you Congratulations Sympathy Engagement Good Luck New ...
19	primary:129677	0.4929	510	Assassin	spam	Proposal for cooperation │ Hello. I meant to contact http://taint.org webmaster, Am I correct, I hope? Actually, I am not quite sure... Did I or Natasha Fedorov...
20	primary:075760	0.4969	719	Enron	spam	greece based professionals and organizations │ you areinvited to register on training consortiumjoin us by registering for free as a trainer , consultant , or s...

Top-25 false positives (legitimate flagged, most confident first)

#	doc_id	P(unwanted)	tokens (full)	source	label_set	preview (subject │ body, 160 chars)
1	primary:144048	0.9853	46	CEAS-08	legit	Whiskas Scratching Post │ HTML Message - =?windows-1252?B?V2hpc2thcyBTY3JhdGNoaW5nIFBvc3QA?=
2	primary:199386	0.9471	2206	Assassin	legit	Back to School with Buy4Now │ This is a multi-part message in MIME format. ------=_NextPart_000_22A2C_01C23F09.09760420 Content-Type: text/plain; charset="iso-8...
3	primary:091801	0.9436	66	CEAS-08	legit	Lots Of SPAM │ Hi List, i have posted my RAW email in http://pastebin.ca/918849 , i'm receiving 1000 to 4000 per day this king of mesages. SA also skipping this...
4	primary:152763	0.8427	1228	CEAS-08	legit	SAVE 30% on Best Recipe Cookbooks │ Untitled Document We don't want you to miss out on our e-mail offers! Please add our e-mail address, arfldpykmjrzc@americast...
5	primary:165228	0.8246	4917	Assassin	legit	United We Fall? │ This is a multi-part message in MIME format... ------------=_1029244856-11739-4 Content-Type: text/plain; charset="iso-8859-1" Content-Disposi...
6	primary:041713	0.7668	340	TREC-07	legit	Electric Transmission Week complimentary trial │ 1105_ETW_Trial View this message as a Web page. FREE Trial Offer You're invited to receive a FREE trial subscri...
7	primary:103927	0.7592	1587	CEAS-08	legit	FREE Shipping on "America's Best Lost Recipes" Cookbook - Plus Free OXO(R) Tongs │ Untitled Document We don't want you to miss out on our e-mail offers! Please ...
8	primary:055965	0.7247	5197	Assassin	legit	Brother, Can You Spare a Jet? │ This is a multi-part message in MIME format... ------------=_1029157891-23956-4 Content-Type: text/plain; charset="iso-8859-1" C...
9	primary:170237	0.6660	88	CEAS-08	legit	The jet set │ Well, it appears you have all taken an idea and run with it. My point was that "Knocked Up" appeared quite the good film, and the Williams one so ...
10	primary:117398	0.6510	5959	Assassin	legit	AOL Keyword: Fraud Probe │ This is a multi-part message in MIME format... ------------=_1033046883-23957-4 Content-Type: text/plain; charset="iso-8859-1" Conten...
11	primary:168120	0.5796	268	TREC-07	legit	Please confirm your request to join Deficiv │ Hello deficient@speedy.uwaterloo.ca, We have received your request to join the Deficiv group hosted by Yahoo! Grou...
12	primary:057514	0.5173	809	CEAS-08	legit	WIN a $6,000 Cook Islands getaway │ HotelClub Newsletter Dear Tony, Subscribe to any free NEWS.com.au newsletter for your chance to WIN a $6,000 Cook Islands ge...
13	primary:021911	0.5051	875	CEAS-08	legit	It's here. Dell's first tablet PC. │ Dell would like to introduce our first tablet PC, the Latitude XT. Part of the trusted Latitude line, the XT gives you all ...

Three-way (legitimate / phishing / spam)

Decision rule: unwanted iff P(phishing)+P(spam) >= threshold, then kind = argmax(P(phishing), P(spam)); the 3x3 matrix therefore collapses exactly onto the binary confusion matrix. argmax-of-3 reported alongside.

Rows with a full label: 3,220; unwanted-but-unsplit rows (binary roll-up only): 0.

true \ predicted	Legitimate	Phishing	Spam
Legitimate	1,986	1	12
Phishing	4	514	3
Spam	16	1	683

• accuracy 0.9885, macro-F1 0.9867 (over 3 classes; over the 3 present: 0.9867); argmax-of-3 rule: macro-F1 0.9858, 6 rows differ between the rules

class	precision	recall	F1	support
Legitimate	0.9900	0.9935	0.9918	1,999
Phishing	0.9961	0.9866	0.9913	521
Spam	0.9785	0.9757	0.9771	700

Phishing vs spam on the 1,221 true unwanted rows

Kind decision alone (given the row is unwanted, argmax(P(phishing), P(spam))): accuracy 0.9959; phishing read as spam 4, spam read as phishing 1.

true \ predicted (roll-up rule)	Legitimate (missed)	Phishing	Spam
Phishing	4	514	3
Spam	16	1	683

Per source (three-way rows)

source	n	Legitimate	Phishing	Spam	accuracy	macro-F1 (present classes)
Assassin	236	164	0	72	0.9534	0.9458
CEAS-08	736	736	0	0	0.9905	0.9952
Enron	628	0	0	628	0.9841	0.9920
Nazario	222	0	222	0	0.9865	0.9932
Nigerian_Fraud	299	0	299	0	0.9866	0.9933
TREC-07	1,099	1,099	0	0	0.9982	0.9991

Top-25 phishing -> spam confusions (3 total, most confident first)

#	doc_id	P(legit)	P(phish)	P(spam)	tokens (full)	source	preview (subject │ body, 160 chars)
1	aura:Nigerian_Fraud:001698	0.0949	0.0320	0.8730	2061	Nigerian_Fraud	" Please add me to your mailing list. " │ "How to turn 3 into 10,000" This is not a get rich quick scheme, but with a little effort will make a nice little retu...
2	aura:Nigerian_Fraud:000337	0.1620	0.3029	0.5352	330	Nigerian_Fraud	Dddddddd R. Rrrrr Message from Vasil Nanchev. │ Hi, This is Vasil Nanchev. I just wanted to inform you about something which is really time sensitive. For the f...
3	aura:Nigerian_Fraud:000824	0.1349	0.3997	0.4654	1089	Nigerian_Fraud	Seeking joint-venture partners for ERW oil casing project │ Dear Sir or Madam: We are keenly interested in setting up a new factory to produce 120,000 ERW oil c...

Top-25 spam -> phishing confusions (1 total, most confident first)

#	doc_id	P(legit)	P(phish)	P(spam)	tokens (full)	source	preview (subject │ body, 160 chars)
1	primary:077835	0.0080	0.8669	0.1251	1179	Assassin	URGENT BUSINESS ASSISTANCE. │ --0-1118764524-1030968705=:44128 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit KEVIN GEORGE ABIDJAN...

Tokens over-represented in phishing read as spam vs phishing read correctly

token	share of phish->spam docs	share of phish->phish docs	ratio
join	1.000	0.019	49.00
could	1.000	0.076	13.03
www	1.000	0.150	6.64
new	1.000	0.160	6.24
http	1.000	0.193	5.17
years	1.000	0.196	5.07
get	1.000	0.202	4.92
about	1.000	0.239	4.17
after	1.000	0.278	3.59
who	1.000	0.317	3.15
into	1.000	0.323	3.09
com	1.000	0.339	2.95
which	1.000	0.362	2.76
now	1.000	0.379	2.63
can	1.000	0.467	2.14
are	1.000	0.562	1.78
from	1.000	0.644	1.55
will	1.000	0.714	1.40
for	1.000	0.800	1.25
this	1.000	0.815	1.23

Tokens over-represented in spam read as phishing vs spam read correctly

(too few rows in one of the groups)

Predicted kind of the flagged rows (informational — includes the unsplit rows, which have no kind label)

group	n	predicted Phishing	predicted Spam
true positives flagged	1,201	515	686
false positives flagged	13	1	12
missed positives (kind the model would have given)	20	3	17

Patterns (simple lexical statistics, not a model explanation)

Tokens over-represented in false negatives vs true positives (document frequency, smoothed ratio)

token	share of FN docs	share of TP docs	ratio
invite	0.250	0.002	128.94
cut	0.150	0.002	82.05
practical	0.150	0.002	82.05
networking	0.200	0.002	75.36
audience	0.150	0.003	45.59
oct	0.150	0.003	45.59
innovative	0.150	0.003	45.59
event	0.200	0.005	40.58
angeles	0.150	0.004	37.30
anyway	0.150	0.004	37.30
feature	0.150	0.005	31.56
york	0.150	0.005	31.56
subscription	0.200	0.007	27.76
technologies	0.150	0.006	27.35
entertainment	0.150	0.007	24.13
staff	0.200	0.011	19.54
association	0.150	0.008	19.54
markets	0.150	0.009	17.84
hosting	0.150	0.009	17.84
mailman	0.150	0.009	17.84

Tokens over-represented in false positives vs true negatives

token	share of FP docs	share of TN docs	ratio
outlets	0.231	0.001	206.01
zzz-unspun	0.231	0.001	206.01
czzz-unspun	0.231	0.001	206.01
cook	0.231	0.001	206.01
full-color	0.385	0.002	179.85
muehlbauer	0.231	0.002	147.15
vanderkolk	0.231	0.002	147.15
eletra	0.231	0.002	147.15
loripatel	0.231	0.002	147.15
copyedited	0.231	0.002	147.15
jimduffy86	0.231	0.002	147.15
asbrand	0.231	0.002	147.15
patel	0.231	0.002	147.15
authoritatively	0.231	0.002	147.15
four-week	0.231	0.002	147.15
duffy	0.231	0.002	147.15
mediaunspun	0.231	0.002	147.15
savviest	0.231	0.002	147.15
dasbrand	0.231	0.002	147.15
sendashtml	0.231	0.002	147.15

Composition of the error groups (share of rows)

group	n	mailing-list / digest	HTML-heavy	very short (<20 content tokens)	truncated (>512)	lure phrase
FP	13	46.2%	0.0%	0.0%	61.5%	0.0%
TN	1,986	56.3%	0.0%	0.0%	38.3%	0.4%
FN	20	30.0%	0.0%	10.0%	35.0%	0.0%
TP	1,201	8.3%	0.0%	0.6%	26.6%	22.3%

Length confound (RUN_LOG entry 2)

group	n	mean tokens (full)	mean tokens fed	median (full)	% truncated
TP	1,201	978.0	279.8	243	26.6
TN	1,986	732.6	358.4	392	38.3
FP	13	1814.3	377.2	875	61.5
FN	20	1807.8	284.1	291	35.0
CORRECT	3,187	825.1	328.8	339	33.9
WRONG	33	1810.4	320.8	368	45.5
all legitimate	1,999	739.7	358.6	393	38.5
all unwanted	1,221	991.6	279.9	243	26.7
all label3=phish	521	1349.4	355.1	412	39.9
all label3=spam	700	725.3	223.9	159	16.9

length bucket (full tokens)	n	positive rate	recall	FPR	accuracy
<=64	129	0.767	0.9495	0.0333	0.9535
65-128	427	0.590	0.9921	0.0114	0.9906
129-256	699	0.395	0.9928	0.0000	0.9971
257-512	870	0.308	0.9851	0.0033	0.9931
>512	1,095	0.298	0.9785	0.0104	0.9863

Summary

• The classifier catches 98.36% of unwanted mail (phishing or spam) at threshold 0.5 (20 of 1,221 missed) while flagging 0.65% of legitimate mail (13 false alarms). Recall is the headline because a missed phish is the costly error for a small-business inbox.
• Holding false alarms to 1 in 100 legitimate emails keeps recall at 98.69% (threshold 0.434, flag when p >= threshold); at 1 in 1,000 it is 87.88%. The best-F1 threshold is 0.475 (F1 0.9877).
• Three-way: macro-F1 0.9867 over 3,220 rows with a full label (recall legitimate 99.3%, phishing 98.7%, spam 97.6%). Given that an email is unwanted, the model tells phishing from spam 99.6% of the time (4 phishing read as spam, 1 spam read as phishing). The phishing/spam labels follow corpus provenance (Nazario + Nigerian fraud + synthetic lures = phishing; SpamAssassin + Enron-Spam spam folders = spam), so part of this confusion is corpus style, not lure content.
• On the 275-row credential-lure slice (37 documented phrases, 97.5% of its rows are labelled positive) recall is 100.00% with precision 100.00%: the highest-cost phish are caught at least as well as the average.
• Length confound: missed unwanted mail averages 1808 tokens vs 978 for caught; false positives average 1814 tokens vs 733 for correct legitimate mail (all legitimate 740, all unwanted 992). Legitimate mail is ~0.7x longer than unwanted mail on average; the error groups do not follow the length gap, so length is not the main shortcut. Phishing rows average 1349 tokens vs spam 725: a length gap between the two unwanted kinds is a second shortcut to watch in the phishing-vs-spam confusion.
• Recall by length is lowest in the <=64-token bucket (94.9%, 99 positives).
• Of the 13 false positives, 46.2% look like mailing-list or digest traffic, 0.0% are HTML-heavy and 0.0% are very short (TN reference: 56.3% / 0.0% / 0.0%).
• Tokens most over-represented in missed unwanted mail vs caught: invite, cut, practical, networking, audience, oct, innovative, event, angeles, anyway.
• Tokens most over-represented in false positives vs correct legitimate mail: outlets, zzz-unspun, czzz-unspun, cook, full-color, muehlbauer, vanderkolk, eletra, loripatel, copyedited.