academic-rhetoric-lab / EXTERNAL_VALIDATION.md
Hugdrigo's picture
Upload 27 files
be1435c verified
|
Raw History Blame Contribute Delete
2.31 kB

External Validation Protocol

Purpose

The original Academic Rhetoric Classifier was trained on a synthetic-curated multilingual corpus. A controlled IID score and a synthetic novel-pattern challenge score are not sufficient evidence of real scholarly-domain transfer.

This repository therefore treats external evaluation as a separate stage.

External corpus

PubMed RCT 20k, derived from the PubMed 200k RCT benchmark by Dernoncourt & Lee, provides real biomedical abstract sentences with five section/rhetorical labels:

  • background
  • objective
  • methods
  • results
  • conclusions

The external evaluator maps those labels to the closest portfolio-model labels:

PubMed RCT Portfolio model
background background
objective objective
methods method
results result
conclusions implication

limitation has no direct PubMed RCT counterpart.

Development smoke test

A small 28-sentence sample visible in the public dataset viewer was used only as a development smoke test.

  • Accuracy: 0.357
  • Macro F1 across the six model labels: 0.293
  • Background recall: 0.000
  • Method recall: 0.833

This exposes a major biomedical domain-transfer gap.

The viewer sample is not treated as a final benchmark and no source sentence text is redistributed in this repository.

Full external evaluation

Run:

pip install datasets pandas scikit-learn joblib
python external_validation_pubmed.py

The script downloads the source at runtime and writes only aggregate metrics.

Domain adaptation

A separate script is provided:

python train_pubmed_adapted.py

It creates a new adapted model rather than overwriting the original baseline.

Important controls:

  • PubMed train may be used for adaptation.
  • PubMed test must remain untouched until final evaluation.
  • The synthetic limitation class is retained.
  • Adaptation and external evaluation results must be reported separately.
  • Raw PubMed text should not be copied into this repository merely for convenience.

Interpretation

The objective is not to manufacture a higher score. The objective is to quantify the gap between controlled synthetic evaluation and genuine scientific prose, then improve the model using a reproducible, non-leaky protocol.