TabuLM model card content (click to expand)

--- language: - rw license: mit tags: - kinyarwanda - table-question-answering - tabular - low-resource-nlp - morphological-modeling - pretraining datasets: - TabuLM-Research/tabulm base_model: kinyabert-large pipeline_tag: table-question-answering --- # TabuLM: Morphology-Aware Tabular Pre-training for Low-Resource Languages **Ireddi Rakshitha** (Software Engineer, Barclays) · **Devavarapu Yashwanth** (Software Engineer, Barclays) · **Pierre Ntakirutimana** (Research Associate, Carnegie Mellon University) > **arXiv preprint · August 2026** · [Paper](https://arxiv.org/abs/XXXX.XXXXX) · [Code](https://github.com/TabuLM-Research/tabulm) --- ## Model Description **TabuLM** is the first language model pre-trained on Kinyarwanda tabular data. It extends KinyaBERT-large's two-tier morphological transformer with: - **Row, column, and cell-type embeddings** — additive structural embeddings injected at every Tier 2 layer input - **Table-structure attention bias** — learned per-head scalars for same-row, same-column, and header signals - **Masked Cell Recovery (MCR)** — masks entire cells; model reconstructs from row and column context - **Column Type Prediction (CTP)** — masks column headers; model predicts numeric/text/categorical/date type Pre-training corpus: **172 Rwandan government tables** (~35,000 cells) from NISR, RAB, REB, and MoH open-data portals. **Parameters:** ~65M · **GPU:** RTX 3090 · **Training:** 10,000 iterations, ~22 hours --- ## Included Files | File | Description | |------|-------------| | `tabulm_checkpoint.pt` | Full TabuLM pre-trained checkpoint (751 MB) | | `tabqa_kin.json` | TabQA-kin benchmark — 526 QA pairs, 31 tables | | `tabqa_kin_train.json` | Training split (420 pairs) | | `tabqa_kin_dev.json` | Dev split (106 pairs) | --- ## Results on TabQA-kin | Model | Overall EM | Lookup | Comparison | Aggregation | |---|---|---|---|---| | GPT-4o (zero-shot) | 64.0% | 82.9% | 79.2% | 25.9% | | mBERT | 49.3% | 16.7% | 50.0% | 80.8% | | XLM-R | 50.0% | 19.2% | 44.4% | 85.2% | | KinyaBERT-large | 56.3% | 26.7% | 59.1% | 88.9% | | **TabuLM (ours)** | **62.0%** | **28.6%** | **66.7%** | **79.2%** | --- ## Usage ```python import torch from kinyabert_utils import load_tabulm model, tokenizer = load_tabulm("TabuLM-Research/tabulm") table = {"headers": ["Akarere", "Umusaruro (t)"], "rows": [["Gasabo", "4,203"], ["Bugesera", "38,145"]]} question = "Ni ikihe karere gifite umusaruro munini?" answer = model.predict(question, table) --- Citation bibtex @article{ireddi2026tabulm, title = {{TabuLM}: Morphology-Aware Tabular Pre-training for Low-Resource Languages}, author = {Ireddi, Rakshitha and Devavarapu, Yashwanth and Ntakirutimana, Pierre}, journal = {arXiv preprint arXiv:XXXX.XXXXX}, year = {2026} }