mednli_laya

Laya decision model fine-tuned as a three-way medical claim checker. Given a claim and a piece of evidence, it answers supported, unsupported or contradicted.

Fine-tuned by Pavan Kumar Mantha from convaiinnovations/laya (multilingual checkpoint) on MedNLI (presencesw/mednli), with entailment, neutral and contradiction mapped to the three labels.

Usage

import laya

agent = laya.load("pavanmantha/mednli_laya")
questions = {
  "verdict": {
    "type": "choice",
    "instructions": "Decide whether the cited text supports the claim.",
    "criteria": {
      "supported": "the cited text confirms the claim",
      "unsupported": "the cited text does not say enough to confirm or deny the claim",
      "contradicted": "the cited text states the opposite of the claim"
    }
  }
}
state = {"claim": "Patient has elevated Cr", "evidence": "Labs were notable for Cr 1.7 (baseline 0.5)."}
print(agent.predict(state, questions)["answers"]["verdict"])

Put the claim first and the evidence second, as in training.

Benchmark

MedNLI test split, n=1422, chance accuracy 0.3333. The question wording used here was never seen in training.

Main results

model	accuracy (95% CI)	macro-F1	NLL	Brier	ECE	mean confidence	ms/decision
original	0.4494 (0.4226-0.4761)	0.4156	1.3686	0.7485	0.2610	0.7103	0.84
this model	0.7904 (0.7693-0.8129)	0.7909	0.5039	0.2863	0.0424	0.8261	0.92

Robustness tests

model	option-order flip rate	claim-only accuracy (95% CI)	mismatched evidence answered unsupported	accuracy with a training wording
original	0.0816	0.3361 (0.3101-0.3608)	0.8509	0.4142
this model	0.0155	0.4114 (0.3861-0.4367)	0.7398	0.7996

• Option-order flip rate: share of answers that change when the options are listed in reverse. Lower is better.
• Claim-only accuracy: accuracy with the evidence blanked. Values well above chance mean the model can exploit artefacts in the claims alone, a known property of MedNLI.
• Mismatched evidence: each claim is paired with evidence from an unrelated premise; the share answered unsupported. Higher is better.
• Training wording: accuracy when asked with one of the four wordings used in training.

Significance

Exact McNemar test against the original model on the same items: this model fixed 581 errors and introduced 96 (p = 1.8e-85).

Per-class results (this model)

label	precision	recall	F1
supported	0.7810	0.7300	0.7546
unsupported	0.7215	0.7869	0.7528
contradicted	0.8766	0.8544	0.8654

Confusion matrix (this model)

Rows are gold labels, columns are predictions.

	supported	unsupported	contradicted
supported	346	98	30
unsupported	74	373	27
contradicted	23	46	405

Seed variance

Across 2 fine-tuning seeds: accuracy 0.7904 +/- 0.0000, macro-F1 0.7911 +/- 0.0001, ECE 0.0367 +/- 0.0057. This repository holds one of those runs.

Training

Base checkpoint	multilingual
Train / calibration / validation items	10819 / 401 / 1395
Train rows dropped (conflicting labels / duplicates)	9 / 3
De-identification markers	kept as in source
Best epoch	2 of 6
Effective batch	64
Learning rates (encoder / head)	2.5e-05 / 0.0001
Label smoothing	0
Seed	0
Loss	cross-entropy on hard labels, option order shuffled, 4 rotating question wordings
Mismatched-evidence augmentation	15% of training rows per epoch

Validation (MedNLI validation split, averaged over the training wordings):

	accuracy	macro-F1	NLL
before fine-tuning	0.4317	0.3888	1.3586
best epoch	0.8075	0.8074	0.4914

Per-epoch history:

epoch	train loss	val accuracy	val macro-F1	val NLL
1	0.6777	0.7905	0.7929	0.5148
2	0.3982	0.8075	0.8074	0.4914
3	0.2376	0.7885	0.7899	0.6095
4	0.1328	0.7753	0.7776	0.8555

Calibration: the choice temperature was fitted to 1.256 on 1604 held-out calibration items (split by premise), moving NLL from 0.4458 to 0.433 on that slice.

Limitations

• Trained on short clinical-note sentences; accuracy on long guideline passages is untested.
• English only. Other Laya question types were not re-evaluated after fine-tuning.
• Results are on MedNLI only; no out-of-domain benchmark has been run yet.
• Not a medical device and not for clinical decisions.