corrector v2 (glot500-ckb-mlm, guard tau_find 0.5 / margin 7 / valid 8)

c721885 verified 19 days ago

4.19 kB

---
language:
- ckb
license: other
base_model: cis-lmu/glot500-base
library_name: transformers
tags:
- text-correction
- spelling-correction
- asr-post-correction
- punctuation
- sorani
- kurdish
extra_gated_prompt: "Access to this model is granted by RegaLabs on request. Tell us who you are and how you plan to use it."
extra_gated_fields:
  Name: text
  Organization: text
  Intended use: text
---
# ckb-corrector v2 (Central Kurdish / Sorani text corrector)

A non-autoregressive corrector for Central Kurdish (Sorani, `ckb`) text: human-typed text and ASR output.
One encoder forward pass per line, then **find → propose → pick**:

1. **Normalize** (deterministic): legacy encodings, ي/ك, ZWNJ, punctuation spacing, spacing table.
2. **Find**: per-word label `KEEP / REPLACE / DELETE / SPLIT / MERGE_NEXT`, plus a punctuation-after label
   (`none ، . ؟ ! ؛ :`).
3. **Propose**: candidates for flagged words only (lexicon by edit distance and sound, real-ASR confusion table,
   mined misspellings, split/merge options).
4. **Pick**: the same encoder scores candidates in context; a GBDT pick head combines MLM and channel features.
   A **precision guard** keeps the original word unless the best candidate wins by a tuned margin.

Design priority is precision: when unsure, the text is left unchanged.

## Files

| File | What |
|---|---|
| `model.safetensors`, `config.json`, tokenizer files | Encoder: XLM-R architecture (119M), `glot500-ckb-mlm` (Glot500-base with Sorani MLM continued pretraining), fine-tuned |
| `heads.pt` | Find and punctuation heads (PyTorch pickle) |
| `pick.pkl` | GBDT pick head (Python pickle) |
| `confusion.tsv` | Real-ASR confusion table (training rows only) |
| `corrector_config.json` | Guard thresholds used at inference (chosen by the dev sweep) |
| `corrector_config.tune.json` | Config used during tuning (flag floor 0.1), kept for reference |

`heads.pt` and `pick.pkl` are pickles: load them only from this repo.
The inference code (`correct.Corrector`) and lexicon resources are not in this repo.

## Training (summary)

- Encoder start: `glot500-ckb-mlm` (Sorani MLM continued pretraining of `cis-lmu/glot500-base`).
- 662,172 training rows: real ASR error pairs (×2), synthetic human/ASR noise on a gold pool, clean identity lines
  (incl. rudaw / anfsorani), 28,353 legacy-script normalization pairs.
- 6,200 steps (1,602 on a Quadro M2200, 4,598 on a Kaggle T4 with fp16 AMP).
- Guard chosen by a 255-setting sweep on dev with the gate **dev clean word false-change rate ≤ 0.05%**:
  τ_find 0.5, margin 7, valid-word margin 8, punctuation bare 0.9 / punctuated 0.98.
## Evaluation (held-out, measured)
All numbers come from the project's eval scripts on held-out sets. Latency was **not** measured on production hardware.
| Set | Metric | Input | v2 |
|---|---|---|---|
| test_clean (11,032 lines) | word false-change rate | – | **0.045%** (5 lines: 3 from the normalizer, 2 correct fixes of a gold error; 0 wrong model word changes) |
| test_clean | whole-string change rate | – | 0.199% (17 of 22 punctuation only) |
| dev_mix (6,550) | WER, no punctuation | 16.85 | 15.21 (edit P 0.689 / R 0.087 / F0.5 0.289) |
| FLEURS-female test (real ASR) | WER | 37.66 | 37.44 (normalize only: 37.45) |
| zainab test (real ASR) | WER | 40.44 | 40.28 (normalize only: 40.28) |
| human700-tune (real ASR) | WER | 37.54 | 37.54 |
| FLEURS-female test | punctuation P / R / F1 | – | 0.955 / 0.386 / 0.550 |

On real ASR output v2 currently does about what normalization alone does; its value there is safety (it does not
damage correct text) and punctuation.

### Latency (laptop, not production)

Quadro M2200, fp32, full pipeline: batch 1 p50 44.8 ms / p99 368 ms; batch 32 2.11 s per batch (66 ms/line).
CPU, 4 threads: batch 1 p50 98 ms / p99 1,254 ms. ONNX / TensorRT export not done yet.

## Limitations

- Low recall on real ASR errors (see table).
- Text convention: verbatim colloquial verb forms are kept; numbers are expected as words (convert with an ITN afterwards).
- Not tested on production hardware.

## Base model

Built on [cis-lmu/glot500-base](https://huggingface.co/cis-lmu/glot500-base) (Apache-2.0).

	---
	language:
	- ckb
	license: other
	base_model: cis-lmu/glot500-base
	library_name: transformers
	tags:
	- text-correction
	- spelling-correction
	- asr-post-correction
	- punctuation
	- sorani
	- kurdish
	extra_gated_prompt: "Access to this model is granted by RegaLabs on request. Tell us who you are and how you plan to use it."
	extra_gated_fields:
	Name: text
	Organization: text
	Intended use: text
	---

	# ckb-corrector v2 (Central Kurdish / Sorani text corrector)

	A non-autoregressive corrector for Central Kurdish (Sorani, `ckb`) text: human-typed text and ASR output.
	One encoder forward pass per line, then find → propose → pick:

	1. Normalize (deterministic): legacy encodings, ي/ك, ZWNJ, punctuation spacing, spacing table.
	2. Find: per-word label `KEEP / REPLACE / DELETE / SPLIT / MERGE_NEXT`, plus a punctuation-after label
	(`none ، . ؟ ! ؛ :`).
	3. Propose: candidates for flagged words only (lexicon by edit distance and sound, real-ASR confusion table,
	mined misspellings, split/merge options).
	4. Pick: the same encoder scores candidates in context; a GBDT pick head combines MLM and channel features.
	A precision guard keeps the original word unless the best candidate wins by a tuned margin.

	Design priority is precision: when unsure, the text is left unchanged.

	## Files

	\| File \| What \|
	\|---\|---\|
	\| `model.safetensors`, `config.json`, tokenizer files \| Encoder: XLM-R architecture (119M), `glot500-ckb-mlm` (Glot500-base with Sorani MLM continued pretraining), fine-tuned \|
	\| `heads.pt` \| Find and punctuation heads (PyTorch pickle) \|
	\| `pick.pkl` \| GBDT pick head (Python pickle) \|
	\| `confusion.tsv` \| Real-ASR confusion table (training rows only) \|
	\| `corrector_config.json` \| Guard thresholds used at inference (chosen by the dev sweep) \|
	\| `corrector_config.tune.json` \| Config used during tuning (flag floor 0.1), kept for reference \|

	`heads.pt` and `pick.pkl` are pickles: load them only from this repo.
	The inference code (`correct.Corrector`) and lexicon resources are not in this repo.

	## Training (summary)

	- Encoder start: `glot500-ckb-mlm` (Sorani MLM continued pretraining of `cis-lmu/glot500-base`).
	- 662,172 training rows: real ASR error pairs (×2), synthetic human/ASR noise on a gold pool, clean identity lines
	(incl. rudaw / anfsorani), 28,353 legacy-script normalization pairs.
	- 6,200 steps (1,602 on a Quadro M2200, 4,598 on a Kaggle T4 with fp16 AMP).
	- Guard chosen by a 255-setting sweep on dev with the gate dev clean word false-change rate ≤ 0.05%:
	τ_find 0.5, margin 7, valid-word margin 8, punctuation bare 0.9 / punctuated 0.98.

	## Evaluation (held-out, measured)

	All numbers come from the project's eval scripts on held-out sets. Latency was not measured on production hardware.

	\| Set \| Metric \| Input \| v2 \|
	\|---\|---\|---\|---\|
	\| test_clean (11,032 lines) \| word false-change rate \| – \| 0.045% (5 lines: 3 from the normalizer, 2 correct fixes of a gold error; 0 wrong model word changes) \|
	\| test_clean \| whole-string change rate \| – \| 0.199% (17 of 22 punctuation only) \|
	\| dev_mix (6,550) \| WER, no punctuation \| 16.85 \| 15.21 (edit P 0.689 / R 0.087 / F0.5 0.289) \|
	\| FLEURS-female test (real ASR) \| WER \| 37.66 \| 37.44 (normalize only: 37.45) \|
	\| zainab test (real ASR) \| WER \| 40.44 \| 40.28 (normalize only: 40.28) \|
	\| human700-tune (real ASR) \| WER \| 37.54 \| 37.54 \|
	\| FLEURS-female test \| punctuation P / R / F1 \| – \| 0.955 / 0.386 / 0.550 \|

	On real ASR output v2 currently does about what normalization alone does; its value there is safety (it does not
	damage correct text) and punctuation.

	### Latency (laptop, not production)

	Quadro M2200, fp32, full pipeline: batch 1 p50 44.8 ms / p99 368 ms; batch 32 2.11 s per batch (66 ms/line).
	CPU, 4 threads: batch 1 p50 98 ms / p99 1,254 ms. ONNX / TensorRT export not done yet.

	## Limitations

	- Low recall on real ASR errors (see table).
	- Text convention: verbatim colloquial verb forms are kept; numbers are expected as words (convert with an ITN afterwards).
	- Not tested on production hardware.

	## Base model

	Built on [cis-lmu/glot500-base](https://huggingface.co/cis-lmu/glot500-base) (Apache-2.0).