| --- |
| license: apache-2.0 |
| pipeline_tag: token-classification |
| tags: |
| - ner |
| - gliner |
| - data-use |
| --- |
| |
| # gliner_datause |
| |
| Fine-tune of `urchade/gliner_large-v2.1` for data-use mention extraction |
| (dataset / survey / census / registry mentions in economics research papers). |
|
|
| ## Labels |
| - `NAMED_DATA` — a proper name, title, or acronym of a specific data source |
| - `DESCRIPTIVE_DATA` — a source described in words but not named |
| - `VAGUE_DATA` — generic data wording with no identifiable source |
|
|
| ## Training |
| - base model: `urchade/gliner_large-v2.1` |
| - dataset: `rafmacalaba/data-use-mentions` (gliner config) |
| - corpus: `all` |
| - epochs: 5 |
| - learning rate: 5e-06 |
| - batch size: 8 |
| - precision: bf16 |
|
|
| ## Evaluation (holdout) |
|
|
| | thr | tp | fp | fn | precision | recall | f0.5 | f1 | |
| | --- | --- | --- | --- | --- | --- | --- | --- | |
| | 0.10 | 16046 | 9301 | 123 | 0.6331 | 0.9924 | 0.6825 | 0.7730 | |
| | 0.20 | 16016 | 6724 | 153 | 0.7043 | 0.9905 | 0.7475 | 0.8233 | |
| | 0.30 | 15971 | 5288 | 198 | 0.7513 | 0.9878 | 0.7890 | 0.8534 | |
| | 0.40 | 15895 | 4053 | 274 | 0.7968 | 0.9831 | 0.8282 | 0.8802 | |
| | 0.50 | 15694 | 2871 | 475 | 0.8454 | 0.9706 | 0.8678 | 0.9037 | |
| | 0.60 | 14185 | 1753 | 1984 | 0.8900 | 0.8773 | 0.8874 | 0.8836 | |
| | 0.70 | 11161 | 827 | 5008 | 0.9310 | 0.6903 | 0.8703 | 0.7928 | |
|
|
| **Best F0.5**: 0.8874 (thr=0.6) |
| **Best F1**: 0.9037 (thr=0.5) |
|
|
| ## Evaluation breakdown (holdout) |
|
|
| | group | examples | spans | thr | precision | recall | f0.5 | f1 | |
| | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | |
| | overall | 12531 | 16417 | 0.60 | 0.8900 | 0.8773 | 0.8874 | 0.8836 | |
| | prwp | 9079 | 12380 | 0.60 | 0.8886 | 0.8935 | 0.8896 | 0.8911 | |
| | fcv | 3452 | 4037 | 0.50 | 0.8670 | 1.0000 | 0.8907 | 0.9287 | |
| | general_prwp | 9079 | 12380 | 0.60 | 0.8886 | 0.8935 | 0.8896 | 0.8911 | |
| | fcv_pads_east_asia | 784 | 863 | 0.50 | 0.8877 | 1.0000 | 0.9081 | 0.9405 | |
| | jdc_operational | 163 | 178 | 0.60 | 0.9193 | 0.8555 | 0.9058 | 0.8862 | |
| | refugee_pads | 803 | 875 | 0.50 | 0.8475 | 1.0000 | 0.8742 | 0.9175 | |
|
|
