TabuLM model card content (click to expand)

---

language: - rw license: mit tags: - kinyarwanda - table-question-answering - tabular - low-resource-nlp - morphological-modeling - pretraining datasets: - TabuLM-Research/tabulm base_model: kinyabert-large pipeline_tag: table-question-answering

TabuLM: Morphology-Aware Tabular Pre-training for Low-Resource Languages

Ireddi Rakshitha (Software Engineer, Barclays) · Devavarapu Yashwanth (Software Engineer, Barclays) · Pierre Ntakirutimana (Research Associate, Carnegie Mellon University)

arXiv preprint · August 2026 · Paper · Code

---

Model Description

TabuLM is the first language model pre-trained on Kinyarwanda tabular data. It extends KinyaBERT-large's two-tier morphological transformer with:

• Row, column, and cell-type embeddings — additive structural embeddings injected at every Tier 2 layer input
• Table-structure attention bias — learned per-head scalars for same-row, same-column, and header signals
• Masked Cell Recovery (MCR) — masks entire cells; model reconstructs from row and column context
• Column Type Prediction (CTP) — masks column headers; model predicts numeric/text/categorical/date type

Pre-training corpus: 172 Rwandan government tables (~35,000 cells) from NISR, RAB, REB, and MoH open-data portals.

Parameters: ~65M · GPU: RTX 3090 · Training: 10,000 iterations, ~22 hours

---

Included Files

File	Description
tabulm_checkpoint.pt	Full TabuLM pre-trained checkpoint (751 MB)
tabqa_kin.json	TabQA-kin benchmark — 526 QA pairs, 31 tables
tabqa_kin_train.json	Training split (420 pairs)
tabqa_kin_dev.json	Dev split (106 pairs)

---

Results on TabQA-kin

Model	Overall EM	Lookup	Comparison	Aggregation
GPT-4o (zero-shot)	64.0%	82.9%	79.2%	25.9%
mBERT	49.3%	16.7%	50.0%	80.8%
XLM-R	50.0%	19.2%	44.4%	85.2%
KinyaBERT-large	56.3%	26.7%	59.1%	88.9%
TabuLM (ours)	62.0%	28.6%	66.7%	79.2%

---

Usage

import torch
from kinyabert_utils import load_tabulm

model, tokenizer = load_tabulm("TabuLM-Research/tabulm")

table = {"headers": ["Akarere", "Umusaruro (t)"],
         "rows": [["Gasabo", "4,203"], ["Bugesera", "38,145"]]}
question = "Ni ikihe karere gifite umusaruro munini?"

answer = model.predict(question, table)

---

Citation

bibtex
@article{ireddi2026tabulm,
  title   = {{TabuLM}: Morphology-Aware Tabular Pre-training for Low-Resource Languages},
  author  = {Ireddi, Rakshitha and Devavarapu, Yashwanth and Ntakirutimana, Pierre},
  journal = {arXiv preprint arXiv:XXXX.XXXXX},
  year    = {2026}
}

</details>