Upload evaluation/EVALUATION_REPORT.md with huggingface_hub
Browse files
evaluation/EVALUATION_REPORT.md
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
## Published evaluation results
|
| 2 |
+
|
| 3 |
+
> These values are generated by `notebooks/complete_longformer_training_evaluation_pipeline.ipynb`. They should not be edited manually.
|
| 4 |
+
|
| 5 |
+
**Dataset:** QASPER v0.3 — contiguous extractive answers only
|
| 6 |
+
|
| 7 |
+
| Model / approach | Examples | Exact Match | Token F1 | Evidence recovered | Evidence token recall | Avg latency |
|
| 8 |
+
|---|---:|---:|---:|---:|---:|---:|
|
| 9 |
+
| BERT truncated 512 | 200 | 1.50% | 7.37% | 26.00% | 41.34% | 0.029 s |
|
| 10 |
+
| Longformer SQuAD sliding windows | 200 | 6.00% | 16.16% | 30.00% | 45.88% | 0.256 s |
|
| 11 |
+
| Longformer QASPER fine-tuned | 200 | 12.50% | 26.66% | 49.00% | 60.14% | 0.265 s |
|
| 12 |
+
|
| 13 |
+
### Fine-tuning status
|
| 14 |
+
|
| 15 |
+
The Longformer checkpoint was further fine-tuned by this project using the **high-vram** profile.
|
| 16 |
+
|
| 17 |
+
- Training loss: `4.296871501620453`
|
| 18 |
+
- Global steps: `202`
|
| 19 |
+
- Training duration: `432.2` seconds
|
| 20 |
+
- Saved model: `C:\Users\atripathi\OneDrive - Veralto\Desktop\AI Codes\GIT Projects\transformer-projects\04-long-document-question-answering-longformer\models\qasper-longformer`
|
| 21 |
+
|
| 22 |
+
### Interpretation guardrails
|
| 23 |
+
|
| 24 |
+
- Exact Match and Token F1 are calculated against all available contiguous extractive references.
|
| 25 |
+
- Evidence recovery is reported both as a binary thresholded rate and continuous token recall.
|
| 26 |
+
- The confidence value remains an uncalibrated model proxy and is not a probability of correctness.
|
| 27 |
+
- QASPER includes answer types outside contiguous extractive QA; those are excluded and documented.
|