Spaces:
Running
Running
|
Download EXTERNAL_VALIDATION.md from Hugdrigo/academic-rhetoric-lab: direct link, hf CLI and curl.
- Browser
- Download file 2.31 kB
-
https://huggingface.co/spaces/Hugdrigo/academic-rhetoric-lab/resolve/main/EXTERNAL_VALIDATION.md
- Command line
-
hf download hf://spaces/Hugdrigo/academic-rhetoric-lab/EXTERNAL_VALIDATION.md
-
curl -L -o EXTERNAL_VALIDATION.md https://huggingface.co/spaces/Hugdrigo/academic-rhetoric-lab/resolve/main/EXTERNAL_VALIDATION.md
2.31 kB
| # External Validation Protocol | |
| ## Purpose | |
| The original Academic Rhetoric Classifier was trained on a synthetic-curated multilingual corpus. A controlled IID score and a synthetic novel-pattern challenge score are not sufficient evidence of real scholarly-domain transfer. | |
| This repository therefore treats external evaluation as a separate stage. | |
| ## External corpus | |
| **PubMed RCT 20k**, derived from the PubMed 200k RCT benchmark by Dernoncourt & Lee, provides real biomedical abstract sentences with five section/rhetorical labels: | |
| - background | |
| - objective | |
| - methods | |
| - results | |
| - conclusions | |
| The external evaluator maps those labels to the closest portfolio-model labels: | |
| | PubMed RCT | Portfolio model | | |
| |---|---| | |
| | background | background | | |
| | objective | objective | | |
| | methods | method | | |
| | results | result | | |
| | conclusions | implication | | |
| `limitation` has no direct PubMed RCT counterpart. | |
| ## Development smoke test | |
| A small 28-sentence sample visible in the public dataset viewer was used only as a **development smoke test**. | |
| - Accuracy: **0.357** | |
| - Macro F1 across the six model labels: **0.293** | |
| - Background recall: **0.000** | |
| - Method recall: **0.833** | |
| This exposes a major biomedical domain-transfer gap. | |
| The viewer sample is not treated as a final benchmark and no source sentence text is redistributed in this repository. | |
| ## Full external evaluation | |
| Run: | |
| ```bash | |
| pip install datasets pandas scikit-learn joblib | |
| python external_validation_pubmed.py | |
| ``` | |
| The script downloads the source at runtime and writes only aggregate metrics. | |
| ## Domain adaptation | |
| A separate script is provided: | |
| ```bash | |
| python train_pubmed_adapted.py | |
| ``` | |
| It creates a new adapted model rather than overwriting the original baseline. | |
| Important controls: | |
| - PubMed `train` may be used for adaptation. | |
| - PubMed `test` must remain untouched until final evaluation. | |
| - The synthetic `limitation` class is retained. | |
| - Adaptation and external evaluation results must be reported separately. | |
| - Raw PubMed text should not be copied into this repository merely for convenience. | |
| ## Interpretation | |
| The objective is not to manufacture a higher score. The objective is to quantify the gap between controlled synthetic evaluation and genuine scientific prose, then improve the model using a reproducible, non-leaky protocol. | |