Instructions to use SupritiVijay/classifiers_model_provenance with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Scikit-learn
How to use SupritiVijay/classifiers_model_provenance with Scikit-learn:
from huggingface_hub import hf_hub_download import joblib model = joblib.load( hf_hub_download("SupritiVijay/classifiers_model_provenance", "sklearn_model.joblib") ) # only load pickle files from sources you trust # read more about it here https://skops.readthedocs.io/en/stable/persistence.html - Notebooks
- Google Colab
- Kaggle
Model-family provenance classifiers (code domain)
Six one-vs-rest logistic-regression classifiers that guess which model family
produced a reasoning trace, from TF-IDF features over a vocabulary selected by
n-gram analytics. Trained on the code domain only.
Given a trace, each classifier answers one binary question ("is this family X or
not"), so an argmax over the six gives a 6-way family prediction.
Families
| family | parent models folded into it |
|---|---|
openai |
gpt_oss_20b, gpt_oss_120b |
qwen |
qwen3_235B, qwq |
deepseek |
deepseekv3_2, deepseek_distill_qwen_25, deepseek_distill_llama3_70b |
olmo |
olmo31 |
glm |
glm |
exaone |
exaone4_32b |
Usage
preprocessing.py must be importable β the vectorizers were pickled holding a
reference to preprocessing.simple_tokenizer, and loading them without it raises
ModuleNotFoundError: No module named 'preprocessing'.
from huggingface_hub import snapshot_download
import sys, joblib, numpy as np
d = snapshot_download("SupritiVijay/classifiers_model_provenance")
sys.path.insert(0, d) # required, see above
from preprocessing import preprocess_text
FAMILIES = ["openai", "qwen", "deepseek", "olmo", "glm", "exaone"]
clfs = {f: joblib.load(f"{d}/tf_idf/classifier_{f}.joblib") for f in FAMILIES}
vecs = {f: joblib.load(f"{d}/tf_idf/vectorizer_{f}.joblib") for f in FAMILIES}
trace = "<think>\nWe need to compute dp[i]...\n</think>\n```python\n...\n```"
pre = [preprocess_text(trace)]
probs = np.array([clfs[f].predict_proba(vecs[f].transform(pre))[0, 1]
for f in FAMILIES])
print(FAMILIES[probs.argmax()], dict(zip(FAMILIES, probs.round(3))))
load_classifiers.py in this repo wraps the above. The six probabilities are
independent one-vs-rest outputs and do not sum to 1 β argmax is a sensible
combination rule, not a calibrated posterior.
Training data
micdun/classifier_training_traces,
code domain, train split, rows flagged throw_away dropped.
10,000 rows per family, with each family's budget split evenly across its parents so that parent count does not become class weight:
| family | parents | rows per parent | total |
|---|---|---|---|
openai |
2 | 5,000 each | 10,000 |
qwen |
2 | 5,000 each | 10,000 |
deepseek |
3 | 3,334 / 3,333 / 3,333 | 10,000 |
olmo |
1 | 10,000 | 10,000 |
glm |
1 | 10,000 | 10,000 |
exaone |
1 | 10,000 | 10,000 |
Each one-vs-rest classifier then trains on 10,000 positive + 10,000 negative
rows, the negatives drawn from the other five (already equal-sized) families. No
class_weight="balanced" is used β the balance is in the sample. This matters:
an earlier version of these classifiers used class_weight="balanced" over the
natural family sizes, and because deepseek had 3 parents against single-parent
olmo/glm/exaone, the small families became attractor classes (exaone
reached recall 0.53 at precision 0.28 while deepseek sat at precision 0.59,
recall 0.32).
Features
- n-gram TF and document frequency counted per family for n β {1, 2, 3}.
- Per family and per n, the top 10,000 n-grams by
tf * idfare kept β 180,000 ranked entries (60,000 each of uni-, bi- and trigrams). - Those entries are split into component words and unioned, giving
17,834 unique tokens, which become the fixed
TfidfVectorizervocabulary. LogisticRegression(max_iter=1000, random_state=42)on that.
Text preprocessing: lowercase, all digits deleted, all punctuation except -
mapped to space, whitespace collapsed.
Results β held out
Scored by downloading this repo and running it against the test split, 3,000
rows (500 per family) that no classifier saw during training. 6-way argmax
over the six one-vs-rest heads, chance = 0.1667:
| family | recall | precision | predicted-as (of 500 true) |
|---|---|---|---|
openai |
0.9800 | 0.9608 | 510 |
olmo |
0.9660 | 0.8285 | 583 |
exaone |
0.9480 | 0.6583 | 720 |
glm |
0.8780 | 0.8762 | 501 |
deepseek |
0.6080 | 0.8444 | 360 |
qwen |
0.5020 | 0.7699 | 326 |
| overall accuracy | 0.8137 |
exaone is the residual attractor β it absorbs 720 predictions for 500 true rows,
so its high recall comes at precision 0.658. qwen and deepseek are the ones it
mostly steals from, and they are exactly the two multi-parent / Qwen-entangled
families: qwen merges qwen3_235B with qwq, while deepseek includes
deepseek_distill_qwen_25. Sample balancing removed the severe attractor
behaviour (see below) but did not fully eliminate it.
In-sample, for comparison
The training pipeline reports clf.predict(x_train), so these are training-set
numbers. Per-family binary one-vs-rest (chance = 0.50, balanced 10k/10k):
| family | accuracy | F1 |
|---|---|---|
openai |
0.9855 | 0.9855 |
glm |
0.9628 | 0.9636 |
olmo |
0.9435 | 0.9450 |
exaone |
0.9423 | 0.9441 |
deepseek |
0.8934 | 0.8921 |
qwen |
0.8655 | 0.8700 |
Combined 6-way argmax on 1,800 held-in rows (300 per family): overall
0.8589, versus 0.8137 held out β a 4.5-point generalization gap.
| family | in-sample 6-way recall | held-out 6-way recall |
|---|---|---|
glm |
0.9800 | 0.8780 |
openai |
0.9667 | 0.9800 |
olmo |
0.9300 | 0.9660 |
exaone |
0.9267 | 0.9480 |
deepseek |
0.7467 | 0.6080 |
qwen |
0.6033 | 0.5020 |
qwen is hardest in both.
Files
tf_idf/classifier_<family>.joblib LogisticRegression, one per family
tf_idf/vectorizer_<family>.joblib TfidfVectorizer, fixed 17,834-token vocabulary
preprocessing.py REQUIRED to unpickle the vectorizers
load_classifiers.py loading + prediction helper
- Downloads last month
- -