rasulkhanbayov/matpcr-verifier

MatPCR physical-consistency verifier (LoRA adapters)

About these weights. The training runs reported in the paper did not save their adapters. The script was corrected and all 13 adapters were retrained with the same data, splits and seed (seed 0), in float32 on an NVIDIA P100 because that GPU lacks bfloat16 support. These retrained adapters are the ones released here. Their metrics are listed below next to the paper's.

This is a binary classifier fine-tuned on oracle labels only (no human annotation). From the input description and a model's reasoning-chain text (no image), it predicts whether the chain contains a physical-consistency violation.

• Code: https://github.com/KurbanIntelligenceLab/matpcr (experiments/exp5_train_verifier.py)
• Training data: experiments/exp5_verifier_dataset.jsonl in the GitHub repo (2,152 records, 6 oracle families)

Training setup

• Base model: Qwen/Qwen2.5-0.5B-Instruct with a sequence-classification head (2 labels).
• LoRA: r=16, alpha=32, dropout 0.05, on q/k/v/o_proj. 4 epochs, lr 2e-4, batch size 8, max length 512; paper runs in bfloat16 (A100), released adapters in float32 (P100).
• Splits: splits are disjoint by source item.

Layout (13 adapters)

Folder	Train	Test
random/	80% of source items, all 6 families	20% held-out source items
modality_heldout_<m>/	5 families	the unseen family <m>
constraint_heldout_<m>/	same partition as modality_heldout_* (in MatPCR, modality and constraint type coincide)

<m> ∈ {xrd, sem, spectrum, stability, metallicity, magnetism}.

Performance (source-disjoint splits; ROC AUC)

Split	Paper run	Released adapters (95% CI)
In-distribution (random)	0.850	0.850 [0.814, 0.884]
Held-out diffraction	0.497	0.532 [0.451, 0.614]
Held-out microscopy	0.527	0.602 [0.542, 0.662]
Held-out spectra	0.558	0.536 [0.465, 0.604]
Held-out stability	0.436	0.383 [0.322, 0.445]
Held-out metallicity	0.582	0.542 [0.470, 0.615]
Held-out magnetism	0.503	0.502 [0.429, 0.578]

Within the in-distribution split, discrimination differs a lot by family. It is strong for microscopy (paper run AUC 0.92) and stability (0.80), moderate for magnetism (0.73) and diffraction (0.70), and near chance for metallicity and spectra. A baseline that knows only the item's family already reaches AUC 0.76.

Intended use: an in-distribution triage signal for microscopy and stability claims (weaker for magnetism and diffraction). It does not generalize to unseen constraint types (it is at chance on all six held-out families), so do not use it as a modality-agnostic detector.

Loading

from transformers import AutoModelForSequenceClassification, AutoTokenizer
from peft import PeftModel
base = AutoModelForSequenceClassification.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct", num_labels=2)
tok = AutoTokenizer.from_pretrained("rasulkhanbayov/matpcr-verifier", subfolder="random")
model = PeftModel.from_pretrained(base, "rasulkhanbayov/matpcr-verifier", subfolder="random")