Spaces:
Running
Download EXTERNAL_VALIDATION.md from Hugdrigo/academic-rhetoric-lab: direct link, hf CLI and curl.
- Browser
- Download file 2.31 kB
-
https://huggingface.co/spaces/Hugdrigo/academic-rhetoric-lab/resolve/main/EXTERNAL_VALIDATION.md
- Command line
-
hf download hf://spaces/Hugdrigo/academic-rhetoric-lab/EXTERNAL_VALIDATION.md
-
curl -L -o EXTERNAL_VALIDATION.md https://huggingface.co/spaces/Hugdrigo/academic-rhetoric-lab/resolve/main/EXTERNAL_VALIDATION.md
External Validation Protocol
Purpose
The original Academic Rhetoric Classifier was trained on a synthetic-curated multilingual corpus. A controlled IID score and a synthetic novel-pattern challenge score are not sufficient evidence of real scholarly-domain transfer.
This repository therefore treats external evaluation as a separate stage.
External corpus
PubMed RCT 20k, derived from the PubMed 200k RCT benchmark by Dernoncourt & Lee, provides real biomedical abstract sentences with five section/rhetorical labels:
- background
- objective
- methods
- results
- conclusions
The external evaluator maps those labels to the closest portfolio-model labels:
| PubMed RCT | Portfolio model |
|---|---|
| background | background |
| objective | objective |
| methods | method |
| results | result |
| conclusions | implication |
limitation has no direct PubMed RCT counterpart.
Development smoke test
A small 28-sentence sample visible in the public dataset viewer was used only as a development smoke test.
- Accuracy: 0.357
- Macro F1 across the six model labels: 0.293
- Background recall: 0.000
- Method recall: 0.833
This exposes a major biomedical domain-transfer gap.
The viewer sample is not treated as a final benchmark and no source sentence text is redistributed in this repository.
Full external evaluation
Run:
pip install datasets pandas scikit-learn joblib
python external_validation_pubmed.py
The script downloads the source at runtime and writes only aggregate metrics.
Domain adaptation
A separate script is provided:
python train_pubmed_adapted.py
It creates a new adapted model rather than overwriting the original baseline.
Important controls:
- PubMed
trainmay be used for adaptation. - PubMed
testmust remain untouched until final evaluation. - The synthetic
limitationclass is retained. - Adaptation and external evaluation results must be reported separately.
- Raw PubMed text should not be copied into this repository merely for convenience.
Interpretation
The objective is not to manufacture a higher score. The objective is to quantify the gap between controlled synthetic evaluation and genuine scientific prose, then improve the model using a reproducible, non-leaky protocol.