Model-family provenance classifiers (code domain)

Six one-vs-rest logistic-regression classifiers that guess which model family produced a reasoning trace, from TF-IDF features over a vocabulary selected by n-gram analytics. Trained on the code domain only.

Given a trace, each classifier answers one binary question ("is this family X or not"), so an argmax over the six gives a 6-way family prediction.

Families

family parent models folded into it
openai gpt_oss_20b, gpt_oss_120b
qwen qwen3_235B, qwq
deepseek deepseekv3_2, deepseek_distill_qwen_25, deepseek_distill_llama3_70b
olmo olmo31
glm glm
exaone exaone4_32b

Usage

preprocessing.py must be importable β€” the vectorizers were pickled holding a reference to preprocessing.simple_tokenizer, and loading them without it raises ModuleNotFoundError: No module named 'preprocessing'.

from huggingface_hub import snapshot_download
import sys, joblib, numpy as np

d = snapshot_download("SupritiVijay/classifiers_model_provenance")
sys.path.insert(0, d)                      # required, see above
from preprocessing import preprocess_text

FAMILIES = ["openai", "qwen", "deepseek", "olmo", "glm", "exaone"]
clfs = {f: joblib.load(f"{d}/tf_idf/classifier_{f}.joblib") for f in FAMILIES}
vecs = {f: joblib.load(f"{d}/tf_idf/vectorizer_{f}.joblib") for f in FAMILIES}

trace = "<think>\nWe need to compute dp[i]...\n</think>\n```python\n...\n```"
pre = [preprocess_text(trace)]
probs = np.array([clfs[f].predict_proba(vecs[f].transform(pre))[0, 1]
                  for f in FAMILIES])
print(FAMILIES[probs.argmax()], dict(zip(FAMILIES, probs.round(3))))

load_classifiers.py in this repo wraps the above. The six probabilities are independent one-vs-rest outputs and do not sum to 1 β€” argmax is a sensible combination rule, not a calibrated posterior.

Training data

micdun/classifier_training_traces, code domain, train split, rows flagged throw_away dropped.

10,000 rows per family, with each family's budget split evenly across its parents so that parent count does not become class weight:

family parents rows per parent total
openai 2 5,000 each 10,000
qwen 2 5,000 each 10,000
deepseek 3 3,334 / 3,333 / 3,333 10,000
olmo 1 10,000 10,000
glm 1 10,000 10,000
exaone 1 10,000 10,000

Each one-vs-rest classifier then trains on 10,000 positive + 10,000 negative rows, the negatives drawn from the other five (already equal-sized) families. No class_weight="balanced" is used β€” the balance is in the sample. This matters: an earlier version of these classifiers used class_weight="balanced" over the natural family sizes, and because deepseek had 3 parents against single-parent olmo/glm/exaone, the small families became attractor classes (exaone reached recall 0.53 at precision 0.28 while deepseek sat at precision 0.59, recall 0.32).

Features

  1. n-gram TF and document frequency counted per family for n ∈ {1, 2, 3}.
  2. Per family and per n, the top 10,000 n-grams by tf * idf are kept β†’ 180,000 ranked entries (60,000 each of uni-, bi- and trigrams).
  3. Those entries are split into component words and unioned, giving 17,834 unique tokens, which become the fixed TfidfVectorizer vocabulary.
  4. LogisticRegression(max_iter=1000, random_state=42) on that.

Text preprocessing: lowercase, all digits deleted, all punctuation except - mapped to space, whitespace collapsed.

Results β€” held out

Scored by downloading this repo and running it against the test split, 3,000 rows (500 per family) that no classifier saw during training. 6-way argmax over the six one-vs-rest heads, chance = 0.1667:

family recall precision predicted-as (of 500 true)
openai 0.9800 0.9608 510
olmo 0.9660 0.8285 583
exaone 0.9480 0.6583 720
glm 0.8780 0.8762 501
deepseek 0.6080 0.8444 360
qwen 0.5020 0.7699 326
overall accuracy 0.8137

exaone is the residual attractor β€” it absorbs 720 predictions for 500 true rows, so its high recall comes at precision 0.658. qwen and deepseek are the ones it mostly steals from, and they are exactly the two multi-parent / Qwen-entangled families: qwen merges qwen3_235B with qwq, while deepseek includes deepseek_distill_qwen_25. Sample balancing removed the severe attractor behaviour (see below) but did not fully eliminate it.

In-sample, for comparison

The training pipeline reports clf.predict(x_train), so these are training-set numbers. Per-family binary one-vs-rest (chance = 0.50, balanced 10k/10k):

family accuracy F1
openai 0.9855 0.9855
glm 0.9628 0.9636
olmo 0.9435 0.9450
exaone 0.9423 0.9441
deepseek 0.8934 0.8921
qwen 0.8655 0.8700

Combined 6-way argmax on 1,800 held-in rows (300 per family): overall 0.8589, versus 0.8137 held out β€” a 4.5-point generalization gap.

family in-sample 6-way recall held-out 6-way recall
glm 0.9800 0.8780
openai 0.9667 0.9800
olmo 0.9300 0.9660
exaone 0.9267 0.9480
deepseek 0.7467 0.6080
qwen 0.6033 0.5020

qwen is hardest in both.

Files

tf_idf/classifier_<family>.joblib    LogisticRegression, one per family
tf_idf/vectorizer_<family>.joblib    TfidfVectorizer, fixed 17,834-token vocabulary
preprocessing.py                     REQUIRED to unpickle the vectorizers
load_classifiers.py                  loading + prediction helper
Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support