README.md

metadata

language:
  - ckb
license: other
base_model: cis-lmu/glot500-base
library_name: transformers
tags:
  - text-correction
  - spelling-correction
  - asr-post-correction
  - punctuation
  - sorani
  - kurdish
extra_gated_prompt: >-
  Access to this model is granted by RegaLabs on request. Tell us who you are
  and how you plan to use it.
extra_gated_fields:
  Name: text
  Organization: text
  Intended use: text

ckb-corrector v2 (Central Kurdish / Sorani text corrector)

A non-autoregressive corrector for Central Kurdish (Sorani, ckb) text: human-typed text and ASR output. One encoder forward pass per line, then find → propose → pick:

1. Normalize (deterministic): legacy encodings, ي/ك, ZWNJ, punctuation spacing, spacing table.
2. Find: per-word label KEEP / REPLACE / DELETE / SPLIT / MERGE_NEXT, plus a punctuation-after label (none ، . ؟ ! ؛ :).
3. Propose: candidates for flagged words only (lexicon by edit distance and sound, real-ASR confusion table, mined misspellings, split/merge options).
4. Pick: the same encoder scores candidates in context; a GBDT pick head combines MLM and channel features. A precision guard keeps the original word unless the best candidate wins by a tuned margin.

Design priority is precision: when unsure, the text is left unchanged.

Files

File	What
model.safetensors, config.json, tokenizer files	Encoder: XLM-R architecture (119M), glot500-ckb-mlm (Glot500-base with Sorani MLM continued pretraining), fine-tuned
heads.pt	Find and punctuation heads (PyTorch pickle)
pick.pkl	GBDT pick head (Python pickle)
confusion.tsv	Real-ASR confusion table (training rows only)
corrector_config.json	Guard thresholds used at inference (chosen by the dev sweep)
corrector_config.tune.json	Config used during tuning (flag floor 0.1), kept for reference

heads.pt and pick.pkl are pickles: load them only from this repo. The inference code (correct.Corrector) and lexicon resources are not in this repo.

Training (summary)

• Encoder start: glot500-ckb-mlm (Sorani MLM continued pretraining of cis-lmu/glot500-base).
• 662,172 training rows: real ASR error pairs (×2), synthetic human/ASR noise on a gold pool, clean identity lines (incl. rudaw / anfsorani), 28,353 legacy-script normalization pairs.
• 6,200 steps (1,602 on a Quadro M2200, 4,598 on a Kaggle T4 with fp16 AMP).
• Guard chosen by a 255-setting sweep on dev with the gate dev clean word false-change rate ≤ 0.05%: τ_find 0.5, margin 7, valid-word margin 8, punctuation bare 0.9 / punctuated 0.98.

Evaluation (held-out, measured)

All numbers come from the project's eval scripts on held-out sets. Latency was not measured on production hardware.

Set	Metric	Input	v2
test_clean (11,032 lines)	word false-change rate	–	0.045% (5 lines: 3 from the normalizer, 2 correct fixes of a gold error; 0 wrong model word changes)
test_clean	whole-string change rate	–	0.199% (17 of 22 punctuation only)
dev_mix (6,550)	WER, no punctuation	16.85	15.21 (edit P 0.689 / R 0.087 / F0.5 0.289)
FLEURS-female test (real ASR)	WER	37.66	37.44 (normalize only: 37.45)
zainab test (real ASR)	WER	40.44	40.28 (normalize only: 40.28)
human700-tune (real ASR)	WER	37.54	37.54
FLEURS-female test	punctuation P / R / F1	–	0.955 / 0.386 / 0.550

On real ASR output v2 currently does about what normalization alone does; its value there is safety (it does not damage correct text) and punctuation.

Latency (laptop, not production)

Quadro M2200, fp32, full pipeline: batch 1 p50 44.8 ms / p99 368 ms; batch 32 2.11 s per batch (66 ms/line). CPU, 4 threads: batch 1 p50 98 ms / p99 1,254 ms. ONNX / TensorRT export not done yet.

Limitations

• Low recall on real ASR errors (see table).
• Text convention: verbatim colloquial verb forms are kept; numbers are expected as words (convert with an ITN afterwards).
• Not tested on production hardware.

Base model

Built on cis-lmu/glot500-base (Apache-2.0).