# External Validation Protocol ## Purpose The original Academic Rhetoric Classifier was trained on a synthetic-curated multilingual corpus. A controlled IID score and a synthetic novel-pattern challenge score are not sufficient evidence of real scholarly-domain transfer. This repository therefore treats external evaluation as a separate stage. ## External corpus **PubMed RCT 20k**, derived from the PubMed 200k RCT benchmark by Dernoncourt & Lee, provides real biomedical abstract sentences with five section/rhetorical labels: - background - objective - methods - results - conclusions The external evaluator maps those labels to the closest portfolio-model labels: | PubMed RCT | Portfolio model | |---|---| | background | background | | objective | objective | | methods | method | | results | result | | conclusions | implication | `limitation` has no direct PubMed RCT counterpart. ## Development smoke test A small 28-sentence sample visible in the public dataset viewer was used only as a **development smoke test**. - Accuracy: **0.357** - Macro F1 across the six model labels: **0.293** - Background recall: **0.000** - Method recall: **0.833** This exposes a major biomedical domain-transfer gap. The viewer sample is not treated as a final benchmark and no source sentence text is redistributed in this repository. ## Full external evaluation Run: ```bash pip install datasets pandas scikit-learn joblib python external_validation_pubmed.py ``` The script downloads the source at runtime and writes only aggregate metrics. ## Domain adaptation A separate script is provided: ```bash python train_pubmed_adapted.py ``` It creates a new adapted model rather than overwriting the original baseline. Important controls: - PubMed `train` may be used for adaptation. - PubMed `test` must remain untouched until final evaluation. - The synthetic `limitation` class is retained. - Adaptation and external evaluation results must be reported separately. - Raw PubMed text should not be copied into this repository merely for convenience. ## Interpretation The objective is not to manufacture a higher score. The objective is to quantify the gap between controlled synthetic evaluation and genuine scientific prose, then improve the model using a reproducible, non-leaky protocol.