Token Classification
Transformers
Safetensors
Arabic
bert
nested-ner
multi-label
arabic
wojood
konooz
knowledgegrapheval-2026
arabicnlp-2026
adaptner
Eval Results (legacy)
Instructions to use NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-MARBERTv2-Nested with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-MARBERTv2-Nested with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("token-classification", model="NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-MARBERTv2-Nested")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForTokenClassification tokenizer = AutoTokenizer.from_pretrained("NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-MARBERTv2-Nested") model = AutoModelForTokenClassification.from_pretrained("NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-MARBERTv2-Nested", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Align card with camera-ready paper: title, authors, official test results; mark as development-phase checkpoint
144187c verified |
Download README.md from NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-MARBERTv2-Nested: direct link, hf CLI and curl.
- Browser
- Download file 25.3 kB
-
https://huggingface.co/NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-MARBERTv2-Nested/resolve/main/README.md
- Command line
-
hf download hf://NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-MARBERTv2-Nested/README.md
-
curl -L -o README.md https://huggingface.co/NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-MARBERTv2-Nested/resolve/main/README.md
25.3 kB
| language: ar | |
| license: apache-2.0 | |
| library_name: transformers | |
| pipeline_tag: token-classification | |
| base_model: UBC-NLP/MARBERTv2 | |
| tags: | |
| - token-classification | |
| - nested-ner | |
| - multi-label | |
| - arabic | |
| - wojood | |
| - konooz | |
| - knowledgegrapheval-2026 | |
| - arabicnlp-2026 | |
| - adaptner | |
| metrics: | |
| - precision | |
| - recall | |
| - f1 | |
| model-index: | |
| - name: KnowledgeGraphEval-2026-AdaptNER-MARBERTv2-Nested | |
| results: | |
| - task: | |
| type: token-classification | |
| name: Nested Named Entity Recognition | |
| dataset: | |
| name: Wojood 1.1 (nested) - test split | |
| type: wojood | |
| config: nested | |
| split: test | |
| metrics: | |
| - type: precision | |
| value: 0.9058 | |
| name: Entity-level micro precision | |
| - type: recall | |
| value: 0.9127 | |
| name: Entity-level micro recall | |
| - type: f1 | |
| value: 0.9092 | |
| name: Entity-level micro F1 | |
| - type: f1 | |
| value: 0.5877 | |
| name: Macro F1 over 21 entity types | |
| - task: | |
| type: token-classification | |
| name: Cross-domain Nested NER (AdaptNER) | |
| dataset: | |
| name: Konooz - KnowledgeGraphEval 2026 development set (development phase) | |
| type: konooz | |
| split: validation | |
| metrics: | |
| - type: f1 | |
| value: 0.67 | |
| name: Entity-level micro F1 (development-phase leaderboard) | |
| # NAMAA Community โ AdaptNER (MARBERTv2, nested) | |
| **KnowledgeGraphEval 2026 ยท Subtask 1 (AdaptNER) ยท ArabicNLP 2026** | |
| Multi-label nested Arabic Named Entity Recognition. This checkpoint fine-tunes | |
| [`UBC-NLP/MARBERTv2`](https://huggingface.co/UBC-NLP/MARBERTv2) on the nested variant of the | |
| Wojood corpus, predicting 21 entity types simultaneously so that overlapping mentions are | |
| preserved. | |
| It is released as the **dialect-oriented backbone comparison point** from the team's | |
| development-phase encoder sweep. It is not the team's best system. | |
| > **How this checkpoint relates to the paper.** This is a **development-phase** checkpoint from | |
| > the NAMAA submission to KnowledgeGraphEval 2026. It is **not** the system described in the | |
| > camera-ready paper. The submitted AdaptNER system, which ranked **1st** with a test micro-F1 of | |
| > **75.96**, is an exact-span majority vote over three models: ARBERTv2 + CRF, XLM-R-large + CRF, | |
| > and ARBERTv2 + GlobalPointer. See [ยง6](#6-official-shared-task-results). We release this | |
| > checkpoint because the team's in-domain Wojood experiments were built on it. | |
| > ### โ ๏ธ Read this before using the model | |
| > | |
| > This checkpoint **never predicts four of the 21 entity types**. `PERCENT`, `PRODUCT`, `UNIT` | |
| > and `QUANTITY` all score exactly **F1 = 0.0000** on the Wojood test set, and `CURR` is | |
| > effectively dead at 0.0465 (1 of 41 mentions recovered). Its headline micro-F1 of 0.9092 | |
| > looks close to the AraBERTv02 sibling's 0.9284, but macro-F1 over the 21 types is **0.5877 | |
| > vs 0.8411** โ a 25-point gap. | |
| > | |
| > For general-purpose nested Arabic NER, use | |
| > [`KnowledgeGraphEval-2026-AdaptNER-AraBERTv02-Nested`](https://huggingface.co/NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-AraBERTv02-Nested) | |
| > instead: it beats this model on **all 21 types**. Full breakdown in ยง5.3. | |
| --- | |
| ## 1. Model details | |
| | | | | |
| |---|---| | |
| | **Developed by** | NAMAA Community (see [Citation](#11-citation)) | | |
| | **Team / submission** | NAMAA at KnowledgeGraphEval 2026 (development-phase checkpoint) | | |
| | **Experiment ID** | EXP-002 (nested) | | |
| | **Model type** | BERT encoder + multi-label token classification head | | |
| | **Base encoder** | `UBC-NLP/MARBERTv2` | | |
| | **Language** | Arabic (dialect-heavy pre-training; MSA + Levantine fine-tuning) | | |
| | **Parameters** | 162,283,050 (F32) | | |
| | **Architecture** | 12 layers, hidden 768, 12 heads, vocab 100,000, max positions 512 | | |
| | **Label space** | 42 outputs = 21 entity types x {B, I}; `O` is implicit | | |
| | **Decision rule** | Independent sigmoid per label, threshold **0.5** | | |
| | **Max sequence length** | 256 sub-words | | |
| | **License** | Apache-2.0 | | |
| | **Finetuned from** | MARBERTv2 ([Abdul-Mageed et al., 2021](https://aclanthology.org/2021.acl-long.551/)) | | |
| The larger parameter count relative to the AraBERTv02 sibling (162M vs 135M) is entirely the | |
| 100,000-entry vocabulary versus 64,000; the transformer body is the same size. | |
| ### Task head | |
| The nested formulation treats each token as carrying a **set** of tags rather than one tag. | |
| The head emits 42 independent logits (`B-<TYPE>` and `I-<TYPE>` for each of the 21 types); | |
| every logit is squashed with a sigmoid and thresholded at 0.5. A token with no label above | |
| threshold is `O`. Entity spans are then BIO-decoded **per type channel**. | |
| --- | |
| ## 2. Intended use | |
| **In scope** | |
| - Backbone-comparison and ablation studies for Arabic nested NER. | |
| - Reproducing the team's development-phase encoder comparison. | |
| - Ensemble membership, where its errors are decorrelated from AraBERT-family members. | |
| **Out of scope** | |
| - **Any application needing `PERCENT`, `PRODUCT`, `UNIT`, `QUANTITY` or `CURR`.** The model | |
| does not predict them. Numeric/measurement extraction will silently return nothing. | |
| - General-purpose nested Arabic NER โ the AraBERTv02 sibling dominates it on every type. | |
| - Same-type nested mentions (an `ORG` inside another `ORG`) โ see [Limitations](#8-limitations-and-known-issues). | |
| - Languages other than Arabic. | |
| --- | |
| ## 3. Task and data | |
| KnowledgeGraphEval 2026 ([Aljabari et al., 2026](https://sina.birzeit.edu/knowledgegraph_sharedtask2026/)) | |
| is the first ArabicNLP shared task on Arabic knowledge-graph construction. It splits the problem | |
| into **Subtask 1 (AdaptNER)** โ cross-domain nested NER โ and **Subtask 2 (RE)** โ relation | |
| extraction. Both are scored with strict entity-level micro-F1. | |
| | Role | Dataset | Size | Notes | | |
| |---|---|---|---| | |
| | **Training / model selection** | Wojood 1.1, nested variant ([Jarrar et al., 2022](https://aclanthology.org/2022.lrec-1.387/)) | 23,125 train / 3,304 dev / 6,606 test sentences | ~448K tokens, ~72K mentions, 21 entity types, 22.5% of mentions nested | | |
| | **Official cross-domain evaluation** | Konooz ([Hamad et al., 2025](https://aclanthology.org/2025.findings-acl.382/)) | ~50K tokens, 10 unseen MSA domains | Dev release: 100 sentences (10 per domain), 3,166 tokens, unlabelled | | |
| The model **never sees Konooz during training** โ that domain shift is the challenge the task targets. | |
| Konooz domains: Agriculture, Art, Economics, Finance, Health, History, Law, Politics, Science, Sport. | |
| ### Label space | |
| 21 entity types, in the **official submission column order** (a mis-ordered submission is scored | |
| against the wrong type channels and collapses the score): | |
| ``` | |
| CARDINAL, CURR, DATE, EVENT, FAC, GPE, LANGUAGE, LAW, LOC, MONEY, NORP, | |
| OCC, ORDINAL, ORG, PERCENT, PERS, PRODUCT, QUANTITY, TIME, UNIT, WEBSITE | |
| ``` | |
| --- | |
| ## 4. Training configuration | |
| Recorded verbatim in `repro_bundle.json` shipped with this repo. Identical to the AraBERTv02 | |
| sibling โ the two differ only in base encoder, so the comparison in ยง7 is controlled. | |
| | Hyperparameter | Value | | |
| |---|---| | |
| | Epochs | 8 | | |
| | Learning rate | 3e-5 | | |
| | Batch size (train / eval) | 32 / 64 | | |
| | Gradient accumulation | 1 | | |
| | Optimizer | AdamW | | |
| | Weight decay | 0.01 | | |
| | Warmup ratio | 0.1 | | |
| | Max length | 256 | | |
| | Prediction threshold | 0.5 | | |
| | Precision | bf16 (+ TF32) | | |
| | Early stopping | patience 3, monitor dev micro-F1 | | |
| | Seed | 42 | | |
| | Sub-word alignment | label on first sub-word; continuations masked (`-100`) | | |
| **Environment** โ Python 3.12.13, PyTorch 2.11.0+cu128, Transformers 5.10.2, Datasets 4.0.0, | |
| CUDA 12.8, **Tesla T4**. Arabic normalisation via CAMeL Tools | |
| ([Obeid et al., 2020](https://aclanthology.org/2020.lrec-1.868/)). | |
| > The AraBERTv02 sibling trained on an A100-40GB under Transformers 5.13.1; this run used a | |
| > T4 under 5.10.2. Hyperparameters match, but the environments are not bit-identical. | |
| --- | |
| ## 5. Evaluation | |
| ### 5.1 Methodology | |
| Entity-level **strict micro-F1**: a prediction counts as correct only if the span boundaries | |
| *and* the entity type match the gold mention exactly. | |
| ``` | |
| micro-F1 = 2TP / (2TP + FP + FN) | |
| ``` | |
| Spans are decoded independently per type channel, then compared as a set of | |
| `(sentence, type, start, end)` tuples. | |
| ### 5.2 Overall results | |
| | Split | Precision | Recall | micro-F1 | Macro-F1 (21 types) | | |
| |---|---:|---:|---:|---:| | |
| | Wojood dev (in-domain) | 0.9042 | 0.9100 | 0.9071 | โ | | |
| | **Wojood test (in-domain)** | **0.9058** | **0.9127** | **0.9092** | **0.5877** | | |
| | Konooz dev (cross-domain, official) | โ | โ | **0.67** | โ | | |
| Test-set totals: 18,210 gold entities, 18,316 predicted, TP 16,590 / FP 1,726 / FN 1,620. | |
| Two independent gaps are visible here and both matter: | |
| - **Micro vs macro (0.9092 vs 0.5877).** A class-coverage failure, detailed in ยง5.3. | |
| - **In-domain vs cross-domain (0.9092 vs 0.67).** The โ0.24 F1 domain-shift drop the task targets. | |
| ### 5.3 Per-entity-type breakdown (Wojood test) | |
| Support = gold mention count. Sorted by frequency. The rightmost column is the AraBERTv02 | |
| sibling trained on identical data and hyperparameters. | |
| | Type | Support | Precision | Recall | F1 | AraBERTv02 F1 | | |
| |---|---:|---:|---:|---:|---:| | |
| | GPE | 4,344 | 0.9609 | 0.9620 | 0.9615 | 0.9657 | | |
| | ORG | 3,928 | 0.9098 | 0.9374 | 0.9234 | 0.9341 | | |
| | DATE | 3,206 | 0.9509 | 0.9604 | 0.9556 | 0.9576 | | |
| | PERS | 1,571 | 0.9195 | 0.9313 | 0.9254 | 0.9369 | | |
| | OCC | 1,126 | 0.8798 | 0.9032 | 0.8913 | 0.9074 | | |
| | NORP | 1,095 | 0.7736 | 0.8237 | 0.7979 | 0.8127 | | |
| | ORDINAL | 1,083 | 0.9704 | 0.9391 | 0.9545 | 0.9562 | | |
| | EVENT | 570 | 0.7141 | 0.7667 | 0.7394 | 0.8258 | | |
| | CARDINAL | 342 | 0.8475 | 0.8450 | 0.8463 | 0.8882 | | |
| | FAC | 246 | 0.6883 | 0.6911 | 0.6897 | 0.8200 | | |
| | LOC | 219 | 0.8010 | 0.7352 | 0.7667 | 0.8578 | | |
| | WEBSITE | 116 | 0.5644 | 0.4914 | 0.5253 | 0.7016 | | |
| | LAW | 90 | 0.7400 | 0.8222 | 0.7789 | 0.8342 | | |
| | TIME | 84 | 0.6341 | 0.3095 | 0.4160 | 0.7241 | | |
| | LANGUAGE | 44 | 1.0000 | 0.2273 | 0.3704 | 0.9048 | | |
| | CURR | 41 | 0.5000 | 0.0244 | **0.0465** | 0.9136 | | |
| | MONEY | 33 | 0.7222 | 0.7879 | 0.7536 | 0.8493 | | |
| | PERCENT | 33 | 0.0000 | 0.0000 | **0.0000** | 0.9697 | | |
| | PRODUCT | 19 | 0.0000 | 0.0000 | **0.0000** | 0.7368 | | |
| | UNIT | 11 | 0.0000 | 0.0000 | **0.0000** | 0.6667 | | |
| | QUANTITY | 9 | 0.0000 | 0.0000 | **0.0000** | 0.5000 | | |
| | **Macro average** | | | | **0.5877** | **0.8411** | | |
| **This is the most important table in the card.** Four types are never predicted at all | |
| (`PERCENT`, `PRODUCT`, `UNIT`, `QUANTITY` โ zero precision *and* zero recall, so no | |
| predictions were emitted on those channels). Three more are severely degraded: | |
| | Type | Symptom | | |
| |---|---| | |
| | `CURR` | recall 0.0244 โ 1 of 41 mentions found | | |
| | `LANGUAGE` | precision 1.0000, recall 0.2273 โ extremely conservative, 10 of 44 found | | |
| | `TIME` | recall 0.3095 โ under half the AraBERTv02 recall | | |
| The pattern is consistent: **the sigmoid head at a fixed 0.5 threshold collapses on | |
| low-support numeric and measurement classes.** Those five types (`PERCENT`, `PRODUCT`, `UNIT`, | |
| `QUANTITY`, `CURR`) total only 113 of 18,210 mentions (0.6%), so micro-F1 barely registers | |
| their loss โ which is exactly why the macro average, not the headline number, is the honest | |
| summary for this checkpoint. | |
| The AraBERTv02 sibling, trained identically, keeps all 21 channels alive. The difference is | |
| attributable to the encoder, not the recipe. | |
| ### 5.4 Per-domain breakdown (Wojood test) | |
| F1 is the published figure from `test_metrics.json`. Precision and recall are recomputed from | |
| the shipped `test_predictions_conll.txt` against gold; they reproduce the published F1 to | |
| within ยฑ0.02 (small residual from duplicate sentences during domain re-alignment). | |
| | Sub-corpus | Sentences | Gold entities | Precision | Recall | F1 | | |
| |---|---:|---:|---:|---:|---:| | |
| | Awraq_G3 | 1,430 | 4,503 | 0.9684 | 0.9740 | 0.9706 | | |
| | Awraq_G2 | 979 | 3,493 | 0.9409 | 0.9530 | 0.9463 | | |
| | Awraq_G1 | 931 | 3,264 | 0.9369 | 0.9513 | 0.9426 | | |
| | PCMA | 209 | 1,181 | 0.8788 | 0.8899 | 0.8841 | | |
| | Curras | 1,045 | 916 | 0.7898 | 0.7631 | 0.7898 | | |
| | T2: Foreign policy | 288 | 716 | 0.7858 | 0.7891 | 0.7836 | | |
| | T1: Terrorism | 276 | 628 | 0.8775 | 0.9124 | 0.8865 | | |
| | T4: Elections | 322 | 620 | 0.9172 | 0.9290 | 0.9231 | | |
| | T6: Health policy | 216 | 516 | 0.8484 | 0.8353 | 0.8411 | | |
| | T3: Security | 166 | 382 | 0.8016 | 0.7827 | 0.7953 | | |
| | QUORA | 530 | 376 | 0.6580 | 0.6090 | 0.6267 | | |
| | T5: Migration | 129 | 268 | 0.8346 | 0.7910 | 0.8054 | | |
| | Lebanese | 85 | 60 | 0.5660 | 0.5000 | 0.5426 | | |
| Notably, MARBERTv2's dialect-heavy pre-training does **not** translate into an advantage on the | |
| dialect sub-corpora: it trails AraBERTv02 on `Curras` (0.7898 vs 0.8439), `Lebanese` | |
| (0.5426 vs 0.6074) and `QUORA` (0.6267 vs 0.6877). The expected dialect benefit does not | |
| materialise under this task formulation. | |
| ### 5.5 Per-domain breakdown (Wojood dev) | |
| | Sub-corpus | Sentences | F1 | | |
| |---|---:|---:| | |
| | Awraq_G3 | 714 | 0.9772 | | |
| | Awraq_G2 | 490 | 0.9539 | | |
| | Awraq_G1 | 466 | 0.9410 | | |
| | T4: Elections | 161 | 0.9214 | | |
| | T1: Terrorism | 138 | 0.8729 | | |
| | PCMA | 105 | 0.8645 | | |
| | T5: Migration | 65 | 0.8547 | | |
| | T3: Security | 83 | 0.8367 | | |
| | T6: Health policy | 108 | 0.8013 | | |
| | Curras | 523 | 0.7801 | | |
| | T2: Foreign policy | 144 | 0.7714 | | |
| | Lebanese | 42 | 0.5970 | | |
| | QUORA | 265 | 0.5567 | | |
| --- | |
| ## 6. Official shared-task results | |
| These are the official **test** results for team **NAMAA**, as reported in the camera-ready paper. | |
| All scores are entity-level micro-averages under exact span-and-type matching, in percent. | |
| **This checkpoint is not one of the submitted systems.** | |
| **Subtask 1 โ AdaptNER** (Konooz test, 10 unseen domains). **Ranked 1st.** | |
| | System | Recall | Precision | Micro-F1 | | |
| |---|---:|---:|---:| | |
| | Shared-task baseline | 63.87 | 62.30 | 63.00 | | |
| | ARBERTv2 + CRF | 70.08 | 80.69 | 75.01 | | |
| | XLM-R-large + CRF | 71.12 | 79.41 | 75.04 | | |
| | ARBERTv2 + GlobalPointer | 66.17 | 84.02 | 74.03 | | |
| | **Exact-span ensemble (โฅ2 of 3 votes)** | 69.44 | 83.83 | **75.96** | | |
| **Subtask 2 โ Relation Extraction** (WojoodRelations, 41 labels). **Ranked 2nd.** | |
| | System | Recall | Precision | Micro-F1 | | |
| |---|---:|---:|---:| | |
| | Shared-task baseline | โ | โ | 93.09 | | |
| | Qwen3-8B (LoRA) | 89.35 | 95.59 | 92.36 | | |
| | Qwen3.5-4B (LoRA) | 89.58 | 96.20 | 92.77 | | |
| | Llama-3.1-8B-Instruct (LoRA) | 89.03 | 96.21 | 92.48 | | |
| | **Majority-vote ensemble** | 90.38 | 96.80 | **93.48** | | |
| ### Development-phase Konooz results | |
| During development the team scored its candidate encoders on the Konooz **development** set | |
| (100 sentences, entity-level micro-F1). These scores are not in the camera-ready paper. We | |
| include them here because they are the only cross-domain numbers available for this checkpoint. | |
| | System | Konooz dev F1 | System | Konooz dev F1 | | |
| |---|---:|---|---:| | |
| | ARBERTv2 (unweighted) | 0.74 | MARBERT | 0.68 | | |
| | Top-3 weighted-vote ensemble | 0.74 | mDeBERTa-v3 base | 0.67 | | |
| | AraBERTv02, nested | 0.73 | MARBERTv2 โก | 0.67 | | |
| | ARBERTv2 (class-weighted) | 0.72 | ARBERTv2 + Konooz-MLM (last-3) | 0.65 | | |
| | XLM-R large | 0.71 | AraBERTv02 organiser baseline | 0.61 | | |
| | AraBERTv02, flat | 0.69 | Qwen3-235B (zero-shot) | 0.26 | | |
| โก This MARBERTv2 entry may be a different run from the released MARBERTv2-Nested checkpoint. | |
| That entry's in-domain validation F1 was 0.873, while the released checkpoint's is 0.9071. | |
| --- | |
| ## 7. Sibling checkpoints | |
| Three checkpoints from this submission are released. All share the same data, seed, | |
| hyperparameters and label space. | |
| | Checkpoint | Formulation | Wojood dev F1 | Wojood test F1 | Macro-F1 (test) | Konooz dev F1 | | |
| |---|---|---:|---:|---:|---:| | |
| | AraBERTv02 nested | nested multi-label | 0.9245 | **0.9284** | **0.8411** | **0.73** | | |
| | **MARBERTv2 nested** (this model) | nested multi-label | 0.9071 | 0.9092 | 0.5877 | 0.67 | | |
| | AraBERTv02 flat | flat single-label | 0.9003 | n/a | n/a | 0.69 | | |
| The controlled AraBERTv02-vs-MARBERTv2 comparison is the reason this checkpoint is published: | |
| same data, same seed, same recipe, different encoder. AraBERTv02 wins on **all 21 entity types** | |
| and on **all 13 sub-corpora**. | |
| --- | |
| ## 8. Limitations and known issues | |
| - **Four entity types are never predicted.** `PERCENT`, `PRODUCT`, `UNIT`, `QUANTITY` return | |
| nothing. This is a silent failure: the model emits no error, just no entities on those | |
| channels. Do not deploy it where those types matter. | |
| - **`CURR`, `LANGUAGE` and `TIME` are severely under-recalled** (0.0244 / 0.2273 / 0.3095). | |
| - **Threshold is untuned.** 0.5 was fixed a priori. Because the dead classes fail on *recall* | |
| with zero predictions, lowering the per-type threshold is the first thing to try before | |
| concluding the capability is absent from the weights โ the signal may be present below 0.5. | |
| - **Same-type nesting is unrecoverable.** One BIO channel per type cannot encode an `ORG` | |
| nested inside another `ORG`. A biaffine span head ([Yu et al., 2020](https://aclanthology.org/2020.acl-main.577/)) | |
| would remove this limitation. | |
| - **Cross-domain degradation.** โ0.24 F1 from Wojood to Konooz, worse than the AraBERTv02 | |
| sibling's โ0.20. | |
| - **No dialect advantage observed**, despite MARBERTv2's dialect-heavy pre-training (ยง5.4). | |
| - **Rare-type estimates are noisy.** Types with fewer than ~40 gold mentions cannot support a | |
| stable F1 estimate; treat those rows as indicative. | |
| - **Column order matters.** Submissions are token + 21 space-separated B/I/O columns in the | |
| exact order listed in ยง3. | |
| ### Head architecture | |
| The released head is a **multi-label sigmoid + BCE** head. `config.json` has exactly 42 labels | |
| (21 ร {B, I}, no `O` class), and `repro_bundle.json` records `pred_threshold: 0.5`, so each | |
| B/I channel is an independent sigmoid. This differs from the BIO-CRF and GlobalPointer heads | |
| of the submitted system described in the camera-ready paper. | |
| --- | |
| ## 9. Usage | |
| ```python | |
| from transformers import AutoTokenizer, AutoModelForTokenClassification | |
| import torch | |
| REPO = "NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-MARBERTv2-Nested" | |
| tok = AutoTokenizer.from_pretrained(REPO) | |
| model = AutoModelForTokenClassification.from_pretrained(REPO).eval() | |
| def predict(words, threshold=0.5): | |
| """Nested NER: returns one tag-set per input word.""" | |
| enc = tok(words, is_split_into_words=True, truncation=True, | |
| max_length=256, return_tensors="pt") | |
| with torch.no_grad(): | |
| probs = torch.sigmoid(model(**enc).logits)[0] | |
| out, seen = [], set() | |
| for pos, wid in enumerate(enc.word_ids(0)): # first sub-word of each word only | |
| if wid is None or wid in seen: | |
| continue | |
| seen.add(wid) | |
| tags = {model.config.id2label[i] | |
| for i in (probs[pos] > threshold).nonzero().flatten().tolist()} | |
| out.append((words[wid], tags or {"O"})) | |
| return out | |
| sentence = "ุญุฑุจ 1948 ( ุงูููุจุฉ ) ูุฌุฑุช ุงูููุณุทููููู ู ู ููุณุทูู".split() | |
| for word, tags in predict(sentence): | |
| print(f"{word}\t{'|'.join(sorted(tags))}") | |
| ``` | |
| Lower `threshold` if you need any recall at all on `PERCENT`, `PRODUCT`, `UNIT`, `QUANTITY` | |
| or `CURR`; at the default 0.5 those channels are empty. | |
| --- | |
| ## 10. Reproducibility | |
| Files shipped with this repo: | |
| | File | Contents | | |
| |---|---| | |
| | `repro_bundle.json` | Full config, resolved data paths, label set, seed, library versions | | |
| | `final_metrics.json` | Wojood **dev** overall + per-domain F1 | | |
| | `test_metrics.json` | Wojood **test** overall + per-domain F1 | | |
| | `test_predictions_conll.txt` | Test-set predictions, CoNLL format (token, tab, pipe-joined tags) | | |
| The per-type table in ยง5.3 was computed by BIO-decoding `test_predictions_conll.txt` per type | |
| channel and comparing against gold as a set of `(sentence, type, start, end)` tuples; it | |
| reproduces the published precision exactly and micro-F1 to within 0.001. | |
| Training notebooks, per-run configs, development-phase Konooz predictions and the evaluation scripts are in | |
| [`KnowledgeGraphEval-2026-AdaptNER-Experiments`](https://huggingface.co/NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-Experiments). | |
| --- | |
| ## 11. Citation | |
| ### This system | |
| ```bibtex | |
| @inproceedings{namaa-knowledgegrapheval-2026, | |
| title = {{NAMAA} at {KnowledgeGraphEval} 2026: Entity-Span Voting and Constrained | |
| {LLM} Ensembles for {A}rabic Knowledge Graph Construction}, | |
| author = {Fathallah, Mahmoud and Zaytoon, Mohamed and Emad Eldin, Fatimah and | |
| A.azim, Mona and Nacar, Omer}, | |
| booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Conference | |
| (ArabicNLP 2026)}, | |
| year = {2026}, | |
| address = {Budapest, Hungary}, | |
| publisher = {Association for Computational Linguistics} | |
| } | |
| ``` | |
| Affiliations: Alexandria University; Trouve Labs; Ain Shams University; Tuwaiq Academy; | |
| NAMAA Community. | |
| ### Shared task | |
| ```bibtex | |
| @inproceedings{aljabari-etal-2026-knowledgegrapheval, | |
| title = {{KnowledgeGraphEval} 2026: The First Shared Task on {A}rabic Knowledge Graphs}, | |
| author = {Aljabari, Alaa and Hamad, Nagham and El Mekki, Abdellah and | |
| Abdul-Mageed, Muhammad and Zitouni, Imed and Chawla, Sanjay and Jarrar, Mustafa}, | |
| booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Conference | |
| (ArabicNLP 2026)}, | |
| year = {2026}, | |
| publisher = {Association for Computational Linguistics} | |
| } | |
| ``` | |
| ### Training corpus โ Wojood | |
| ```bibtex | |
| @inproceedings{jarrar-etal-2022-wojood, | |
| title = {{W}ojood: Nested {A}rabic Named Entity Corpus and Recognition using {BERT}}, | |
| author = {Jarrar, Mustafa and Khalilia, Mohammed and Ghanem, Sana}, | |
| booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference (LREC)}, | |
| year = {2022}, | |
| address = {Marseille, France}, | |
| pages = {3626--3636}, | |
| publisher = {European Language Resources Association}, | |
| url = {https://aclanthology.org/2022.lrec-1.387/} | |
| } | |
| ``` | |
| ### Evaluation corpus โ Konooz | |
| ```bibtex | |
| @inproceedings{hamad-etal-2025-konooz, | |
| title = {{K}onooz: Multi-domain Multi-dialect Corpus for Named Entity Recognition}, | |
| author = {Hamad, Nagham and Khalilia, Mohammed and Jarrar, Mustafa}, | |
| booktitle = {Findings of the Association for Computational Linguistics: ACL 2025}, | |
| year = {2025}, | |
| pages = {7316--7331}, | |
| publisher = {Association for Computational Linguistics}, | |
| url = {https://aclanthology.org/2025.findings-acl.382/} | |
| } | |
| ``` | |
| ### Base encoder โ MARBERTv2 | |
| ```bibtex | |
| @inproceedings{abdul-mageed-etal-2021-arbert, | |
| title = {{ARBERT} \& {MARBERT}: Deep Bidirectional Transformers for {A}rabic}, | |
| author = {Abdul-Mageed, Muhammad and Elmadany, AbdelRahim and | |
| Nagoudi, El Moatez Billah}, | |
| booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational | |
| Linguistics and the 11th International Joint Conference on Natural Language | |
| Processing (Volume 1: Long Papers)}, | |
| year = {2021}, | |
| pages = {7088--7105}, | |
| publisher = {Association for Computational Linguistics}, | |
| url = {https://aclanthology.org/2021.acl-long.551/} | |
| } | |
| ``` | |
| ### Also referenced | |
| ```bibtex | |
| @inproceedings{obeid-etal-2020-camel, | |
| title = {{CAM}e{L} Tools: An Open Source Python Toolkit for {A}rabic | |
| Natural Language Processing}, | |
| author = {Obeid, Ossama and Zalmout, Nasser and Khalifa, Salam and Taji, Dima and | |
| Oudah, Mai and Alhafni, Bashar and Inoue, Go and Eryani, Fadhl and | |
| Erdmann, Alexander and Habash, Nizar}, | |
| booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference (LREC)}, | |
| year = {2020}, | |
| pages = {7022--7032}, | |
| url = {https://aclanthology.org/2020.lrec-1.868/} | |
| } | |
| @inproceedings{yu-etal-2020-named, | |
| title = {Named Entity Recognition as Dependency Parsing}, | |
| author = {Yu, Juntao and Bohnet, Bernd and Poesio, Massimo}, | |
| booktitle = {Proceedings of the 58th Annual Meeting of the Association for | |
| Computational Linguistics (ACL)}, | |
| year = {2020}, | |
| pages = {6470--6476}, | |
| url = {https://aclanthology.org/2020.acl-main.577/} | |
| } | |
| ``` | |
| --- | |
| ## 12. Acknowledgements | |
| We thank Tuwaiq Academy for its continued support of this work, and the organisers of KnowledgeGraphEval 2026 for providing the Wojood, Konooz and | |
| WojoodRelations resources and the evaluation framework. | |
| Wojood and Konooz are distributed by [SinaLab, Birzeit University](https://sina.birzeit.edu/) | |
| under their own terms. The Apache-2.0 license here covers the fine-tuned weights and code, | |
| **not** the underlying corpora โ users must obtain those under the original licenses. | |