| <details> |
| <summary>TabuLM model card content (click to expand)</summary> |
|
|
| --- |
| language: |
| - rw |
| license: mit |
| tags: |
| - kinyarwanda |
| - table-question-answering |
| - tabular |
| - low-resource-nlp |
| - morphological-modeling |
| - pretraining |
| datasets: |
| - TabuLM-Research/tabulm |
| base_model: kinyabert-large |
| pipeline_tag: table-question-answering |
| --- |
|
|
| # TabuLM: Morphology-Aware Tabular Pre-training for Low-Resource Languages |
|
|
| **Ireddi Rakshitha** (Software Engineer, Barclays) · **Devavarapu Yashwanth** (Software Engineer, Barclays) · **Pierre Ntakirutimana** (Research Associate, Carnegie Mellon University) |
|
|
| > **arXiv preprint · August 2026** · [Paper](https://arxiv.org/abs/XXXX.XXXXX) · [Code](https://github.com/TabuLM-Research/tabulm) |
|
|
| --- |
|
|
| ## Model Description |
|
|
| **TabuLM** is the first language model pre-trained on Kinyarwanda tabular data. It extends KinyaBERT-large's two-tier morphological transformer with: |
|
|
| - **Row, column, and cell-type embeddings** — additive structural embeddings injected at every Tier 2 layer input |
| - **Table-structure attention bias** — learned per-head scalars for same-row, same-column, and header signals |
| - **Masked Cell Recovery (MCR)** — masks entire cells; model reconstructs from row and column context |
| - **Column Type Prediction (CTP)** — masks column headers; model predicts numeric/text/categorical/date type |
|
|
| Pre-training corpus: **172 Rwandan government tables** (~35,000 cells) from NISR, RAB, REB, and MoH open-data portals. |
|
|
| **Parameters:** ~65M · **GPU:** RTX 3090 · **Training:** 10,000 iterations, ~22 hours |
|
|
| --- |
|
|
| ## Included Files |
|
|
| | File | Description | |
| |------|-------------| |
| | `tabulm_checkpoint.pt` | Full TabuLM pre-trained checkpoint (751 MB) | |
| | `tabqa_kin.json` | TabQA-kin benchmark — 526 QA pairs, 31 tables | |
| | `tabqa_kin_train.json` | Training split (420 pairs) | |
| | `tabqa_kin_dev.json` | Dev split (106 pairs) | |
|
|
| --- |
|
|
| ## Results on TabQA-kin |
|
|
| | Model | Overall EM | Lookup | Comparison | Aggregation | |
| |---|---|---|---|---| |
| | GPT-4o (zero-shot) | 64.0% | 82.9% | 79.2% | 25.9% | |
| | mBERT | 49.3% | 16.7% | 50.0% | 80.8% | |
| | XLM-R | 50.0% | 19.2% | 44.4% | 85.2% | |
| | KinyaBERT-large | 56.3% | 26.7% | 59.1% | 88.9% | |
| | **TabuLM (ours)** | **62.0%** | **28.6%** | **66.7%** | **79.2%** | |
|
|
| --- |
|
|
| ## Usage |
|
|
| ```python |
| import torch |
| from kinyabert_utils import load_tabulm |
| |
| model, tokenizer = load_tabulm("TabuLM-Research/tabulm") |
| |
| table = {"headers": ["Akarere", "Umusaruro (t)"], |
| "rows": [["Gasabo", "4,203"], ["Bugesera", "38,145"]]} |
| question = "Ni ikihe karere gifite umusaruro munini?" |
| |
| answer = model.predict(question, table) |
| |
| --- |
| |
| Citation |
| |
| bibtex |
| @article{ireddi2026tabulm, |
| title = {{TabuLM}: Morphology-Aware Tabular Pre-training for Low-Resource Languages}, |
| author = {Ireddi, Rakshitha and Devavarapu, Yashwanth and Ntakirutimana, Pierre}, |
| journal = {arXiv preprint arXiv:XXXX.XXXXX}, |
| year = {2026} |
| } |
| |
| </details> |
