| --- |
| license: apache-2.0 |
| pipeline_tag: token-classification |
| tags: |
| - ner |
| - gliner |
| - data-use |
| --- |
| |
| # gliner_datause_extended |
|
|
| Fine-tune of `urchade/gliner_large-v2.1` for data-use mention extraction |
| (dataset / survey / census / registry mentions in economics research papers). |
|
|
| ## Labels |
| - `NAMED_DATA` — a proper name, title, or acronym of a specific data source |
| - `DESCRIPTIVE_DATA` — a source described in words but not named |
| - `VAGUE_DATA` — generic data wording with no identifiable source |
|
|
| ## Training |
| - base model: `urchade/gliner_large-v2.1` |
| - dataset: `rafmacalaba/data-use-mentions-extended` (gliner config) |
| - epochs: 5 |
| - learning rate: 5e-06 |
| - batch size: 16 |
| - precision: bf16 |
|
|
| ## Evaluation (holdout) |
|
|
| | thr | tp | fp | fn | precision | recall | f0.5 | f1 | |
| | --- | --- | --- | --- | --- | --- | --- | --- | |
| | 0.10 | 12283 | 8280 | 281 | 0.5973 | 0.9776 | 0.6477 | 0.7416 | |
| | 0.20 | 12192 | 6065 | 372 | 0.6678 | 0.9704 | 0.7122 | 0.7911 | |
| | 0.30 | 12062 | 4883 | 502 | 0.7118 | 0.9600 | 0.7506 | 0.8175 | |
| | 0.40 | 11845 | 3857 | 719 | 0.7544 | 0.9428 | 0.7858 | 0.8381 | |
| | 0.50 | 11498 | 2837 | 1066 | 0.8021 | 0.9152 | 0.8224 | 0.8549 | |
| | 0.60 | 10519 | 1798 | 2045 | 0.8540 | 0.8372 | 0.8506 | 0.8455 | |
| | 0.70 | 8328 | 892 | 4236 | 0.9033 | 0.6628 | 0.8422 | 0.7646 | |
|
|
| **Best F0.5**: 0.8506 (thr=0.6) |
| **Best F1**: 0.8549 (thr=0.5) |
|
|
| <!-- NER_COMPARISON_START --> |
| ## NER holdout comparison |
| device: NVIDIA H100 NVL |
|
|
| ### rafmacalaba/data-use-mentions-extended (n=9249) |
|
|
| | model | backend | best F0.5 | thr | best F1 | thr | wall-clock (s) | texts/s | |
| | --- | --- | --- | --- | --- | --- | --- | --- | |
| | `rafmacalaba/gliner_datause_extended` | gliner | 0.8506 | 0.6 | 0.8549 | 0.5 | 202.7 | 45.6 | |
| | `ai4data/gliner2_datause` | gliner2 | 0.8634 | 0.7 | 0.8624 | 0.6 | 180.8 | 51.1 | |
|
|
| F0.5 by threshold (sweet spots side-by-side): |
|
|
| | thr | `rafmacalaba/gliner_datause_extended` | `ai4data/gliner2_datause` | |
| | --- | --- | --- | |
| | 0.1 | 0.6476 | 0.7321 | |
| | 0.2 | 0.7121 | 0.7712 | |
| | 0.3 | 0.7505 | 0.7979 | |
| | 0.4 | 0.7858 | 0.8201 | |
| | 0.5 | 0.8224 | 0.8363 | |
| | 0.6 | 0.8506 | 0.8523 | |
| | 0.7 | 0.8422 | 0.8634 | |
| <!-- NER_COMPARISON_END --> |
|
|
|
|
| ## FCV cross-corpus validation (precision) |
|
|
| Validated on the World Bank Fragility, Conflict & Violence (FCV) document corpus with |
| the `gpt-5.6-luna` judge (Luna validation). Precision-only: there is no gold set of all |
| true mentions, so recall / F1 are not reported here. |
|
|
| ### Precision by config × label |
|
|
| | config | NAMED_DATA | DESCRIPTIVE_DATA | VAGUE_DATA | overall | |
| | --- | --- | --- | --- | --- | |
| | fcv_pads_east_asia | 80.1% | 81.6% | 91.8% | 81.9% | |
| | jdc_operational | 86.8% | 91.3% | 91.5% | 89.1% | |
| | refugee_pads | 83.8% | 92.8% | 91.9% | 89.1% | |
| | reliefweb | 83.7% | 85.4% | 95.5% | 85.6% | |
|
|
| ### Specificity agreement (extractor label == judge specificity) |
|
|
| | config | NAMED_DATA | DESCRIPTIVE_DATA | VAGUE_DATA | |
| | --- | --- | --- | --- | |
| | fcv_pads_east_asia | 82.8% | 88.3% | 82.1% | |
| | jdc_operational | 90.2% | 83.2% | 71.7% | |
| | refugee_pads | 89.4% | 86.5% | 77.5% | |
| | reliefweb | 92.6% | 82.5% | 73.4% | |
|
|
| Notes: |
| - `VAGUE_DATA` is the safest label (91–95% precision); `NAMED_DATA` is the weakest (80–87%). |
| - The extractor under-specifies: `VAGUE_DATA` is often judged `descriptive`, and |
| `DESCRIPTIVE_DATA` → `named` leaks occur. |
| - `reliefweb` carries more bad-JSON judge errors (OCR/HTML artifacts) than the other configs. |
|
|
