academic-rhetoric-lab / EXTERNAL_VALIDATION.md
Hugdrigo's picture
Upload 27 files
be1435c verified
|
Raw History Blame Contribute Delete
2.31 kB
# External Validation Protocol
## Purpose
The original Academic Rhetoric Classifier was trained on a synthetic-curated multilingual corpus. A controlled IID score and a synthetic novel-pattern challenge score are not sufficient evidence of real scholarly-domain transfer.
This repository therefore treats external evaluation as a separate stage.
## External corpus
**PubMed RCT 20k**, derived from the PubMed 200k RCT benchmark by Dernoncourt & Lee, provides real biomedical abstract sentences with five section/rhetorical labels:
- background
- objective
- methods
- results
- conclusions
The external evaluator maps those labels to the closest portfolio-model labels:
| PubMed RCT | Portfolio model |
|---|---|
| background | background |
| objective | objective |
| methods | method |
| results | result |
| conclusions | implication |
`limitation` has no direct PubMed RCT counterpart.
## Development smoke test
A small 28-sentence sample visible in the public dataset viewer was used only as a **development smoke test**.
- Accuracy: **0.357**
- Macro F1 across the six model labels: **0.293**
- Background recall: **0.000**
- Method recall: **0.833**
This exposes a major biomedical domain-transfer gap.
The viewer sample is not treated as a final benchmark and no source sentence text is redistributed in this repository.
## Full external evaluation
Run:
```bash
pip install datasets pandas scikit-learn joblib
python external_validation_pubmed.py
```
The script downloads the source at runtime and writes only aggregate metrics.
## Domain adaptation
A separate script is provided:
```bash
python train_pubmed_adapted.py
```
It creates a new adapted model rather than overwriting the original baseline.
Important controls:
- PubMed `train` may be used for adaptation.
- PubMed `test` must remain untouched until final evaluation.
- The synthetic `limitation` class is retained.
- Adaptation and external evaluation results must be reported separately.
- Raw PubMed text should not be copied into this repository merely for convenience.
## Interpretation
The objective is not to manufacture a higher score. The objective is to quantify the gap between controlled synthetic evaluation and genuine scientific prose, then improve the model using a reproducible, non-leaky protocol.