abidlabs/jev-spam-decision-2b

Jev-style spam decision model (Qwen3.5-2B + LoRA)

A Jev-like binary decision model: given a forum message and a noul (yes/no) question — "Is this forum message spam?" — it returns a calibrated P(yes) = P(spam) from the next-token distribution over yes / no. Fine-tuned with TRL SFTTrainer (LoRA r=32, completion-only loss on the single answer token, 2 epochs) on Deysi/spam-detection-dataset (8,175 train rows).

Results — all three models on the same held-out test split

Full 2,725-row test split of Deysi/spam-detection-dataset, identical protocol for every model (same question, P(yes) >= 0.5, texts truncated at 1,500 chars):

Model	Accuracy	F1 (spam)	ROC-AUC	ECE	TP / FP / FN / TN
abidlabs/jev-spam-decision-2b (this model)	0.9993	0.9993	1.000	0.001	1375 / 2 / 0 / 1348
akhilaaa3/Jev-Omni (12B)	0.9736	0.9737	0.996	0.010	1335 / 32 / 40 / 1318
feder-cr/jev (jevos-v4)	0.6415	0.4508	0.937	0.297	401 / 3 / 974 / 1347

Reading the table:

• This model beats both open Jev models on its fine-tuning domain — expected, since it was trained on this dataset's train split while the baselines are general-purpose decision models.
• jevos-v4's low accuracy is mostly a calibration problem, not a ranking problem: its ROC-AUC is 0.937 (excellent), but its P(yes) almost never crosses 0.5 (ECE 0.30), so it catches only 401 of 1,375 spam messages. With a tuned threshold it would do much better; at the shared 0.5 protocol it does not.
• Caveat: the Deysi dataset's spam rows are largely synthetic/obvious promotions and its not_spam rows are real forum posts, so the fine-tuned score is optimistic for real-world spam. The fair cross-model conclusion is the gap under identical conditions, not the absolute 0.9993.

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoTokenizer

model = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.5-2B", dtype=torch.bfloat16)
model = PeftModel.from_pretrained(model, "abidlabs/jev-spam-decision-2b").eval()
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-2B")
tok.padding_side = "left"

yes_id, no_id = tok.encode(" yes")[-1], tok.encode(" no")[-1]

@torch.inference_mode()
def p_spam(text):
    prompt = tok.apply_chat_template(
        [{"role": "user", "content": f"Is this forum message spam? Answer with yes or no.\n\n{text[:1500]}"}],
        add_generation_prompt=True, tokenize=False,
    )
    enc = tok([prompt], return_tensors="pt").to(model.device)
    logits = model(**enc, logits_to_keep=1).logits[0, -1]
    return torch.softmax(logits[[yes_id, no_id]].float(), dim=-1)[0].item()

print(p_spam("Get rich quick! Make millions in just days!"))  # ~1.0

Reproduction

Training + eval ran in a single Hugging Face Jobs run (a10g-small, ~39 min): TRL 1.17.1 SFTTrainer, transformers 5.22.1, peft 0.21.2; LoRA r=32/alpha=32 on all attention+MLP projections, lr 1e-4 cosine, effective batch 64, max_length=640, bf16, adamw_8bit; conversational prompt-completion format with completion_only_loss=True so loss lands only on the answer token. Baselines: Jev-Omni via its public loader (A100), jevos-v4 via the official jev binary serve API (CPU). Training loss logs: abidlabs/jev-spam-decision-finetune-trackio.