| --- |
| language: |
| - hu |
| license: apache-2.0 |
| base_model: SZTAKI-HLT/hubert-base-cc |
| pipeline_tag: token-classification |
| tags: |
| - hungarian |
| - ner |
| - corruption |
| datasets: |
| - K-Monitor/kmdb_base |
| metrics: |
| - precision |
| - recall |
| - f1 |
| --- |
| |
| # kmdb_ner_v5 |
|
|
| Hungarian named entity recognition for the |
| [K-Monitor press database](https://adatbazis.k-monitor.hu/). It does two jobs |
| at once: it finds people, organisations and places in an article, and it marks |
| which of them are the ones an editor would actually record for that article. |
|
|
| Labels are `B-/I-` + `{POS,NEG}_{PER,ORG,LOC}`. `POS` means the entity is |
| relevant to the article's case and belongs in the database; `NEG` means it is |
| merely mentioned. |
|
|
| Finetuned from `SZTAKI-HLT/hubert-base-cc` on 58,524 articles from |
| [`K-Monitor/kmdb_base`](https://huggingface.co/datasets/K-Monitor/kmdb_base): |
| spaCy (`hu_core_news_lg`) supplies the entity spans, and an entity is labelled |
| `POS` when it matches one of the names an editor tagged on that article. |
|
|
| ## Results |
|
|
| Entity-level seqeval scores on 4,161 held-out articles. The honest comparison |
| against the previous `boapps/kmdb_ner_model` is the second column: that |
| checkpoint was trained with an unseeded split, so most of the general test set |
| was in its training data, whereas these articles were added to the database |
| after it was published. |
|
|
| | model | test F1 | F1 on articles the old model never saw | |
| | --- | --- | --- | |
| | `boapps/kmdb_ner_model` | 0.8244\* | 0.6684 | |
| | **`kmdb_ner_v5`** | 0.8171 | **0.7299** | |
|
|
| \* inflated by train/test leakage in the earlier checkpoint. |
|
|
| Per class, on those unseen articles: |
|
|
| | label | precision | recall | F1 | previous F1 | |
| | --- | --- | --- | --- | --- | |
| | NEG_LOC | 0.7817 | 0.7075 | 0.7427 | 0.7208 | |
| | NEG_ORG | 0.7530 | 0.7587 | 0.7558 | 0.7176 | |
| | NEG_PER | 0.7298 | 0.6752 | 0.7014 | 0.6505 | |
| | POS_LOC | 0.6219 | 0.7128 | 0.6642 | 0.5729 | |
| | POS_ORG | 0.6469 | 0.7130 | 0.6784 | 0.5270 | |
| | POS_PER | 0.6971 | 0.8029 | 0.7463 | 0.6484 | |
| | **micro avg** | 0.7242 | 0.7356 | **0.7299** | 0.6684 | |
|
|
| The largest gain is on `POS_ORG` (0.5270 → 0.6784): the previous training data |
| matched gold names by exact substring, so tagged organisations whose database |
| name carries a suffix — `Hunguest Hotels Zrt.`, `Honvédelmi Minisztérium (HM)` — |
| were labelled `NEG` and the model learned to ignore them. |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoTokenizer, pipeline |
| |
| classifier = pipeline( |
| "ner", |
| model="K-Monitor/kmdb_ner_v5", |
| aggregation_strategy="first", |
| stride=8, |
| tokenizer=AutoTokenizer.from_pretrained( |
| "SZTAKI-HLT/hubert-base-cc", model_max_length=512 |
| ), |
| ) |
| classifier(article_text) |
| ``` |
|
|
| Adjacent fragments of one entity have to be merged afterwards (huBERT splits |
| detached Hungarian suffixes, e.g. `Budapest` + `-en`); when merging, keep the |
| entity type and treat the result as `POS` if either fragment is `POS`. The |
| reference implementation is `join_entities` in |
| [`sajtoadatbazis-automat`](https://github.com/k-monitor/sajtoadatbazis-automat) |
| (`webapp/auto_kmdb/processors/NERProcessor.py`). |
|
|
| ## Limitations |
|
|
| The `POS`/`NEG` supervision is derived automatically by matching spaCy entities |
| against editor-tagged names, so it inherits spaCy's boundary errors and the |
| matching heuristic's mistakes; it is not hand-annotated. Scores on articles |
| from after the training window (~0.73 F1) are well below in-distribution scores |
| (~0.82), so expect the real-world quality to track the former. |
|
|
