Vatho/koelsch-w2vbert-orthography

Kölsch orthographic ASR — w2v-BERT 2.0 + CTC

Character-level CTC model that transcribes Kölsch (Ripuarian, spoken in Cologne) in Kölsch spelling rather than Standard German. Fine-tuned from facebook/w2v-bert-2.0 on 3.97 hours of labelled Kölsch speech, tripled to ~11.9 hours by 3× speed perturbation.

Results

Held-out test split: 652 utterances, 0.51 h.

decoding	WER	CER
greedy	31.51 %	12.73 %
+ 4-gram KenLM beam search (beam 100, α=0.7, β=1.0)	30.02 %	12.63 %
+ KenLM, spelling-variant-folded (secondary metric, see below)	27.25 %	12.14 %

The n-gram LM is trained on the training-split transcripts only (37 k tokens). Raising the order does not help — measured WER is 28.59 / 28.46 / 28.42 / 28.44 / 28.44 for orders 2–6 on validation, because 37 k tokens contain no higher-order evidence.

⚠️ Read before quoting these numbers

The evaluation split is not recording-disjoint. All 126 test recordings also appear in the training split — the split is random at the utterance level over shared source tracks. The model has therefore heard these speakers and rooms. Every figure above is an optimistic bound on performance with a new speaker, and should not be read as a speaker-independent result. A recording-disjoint split would score worse; how much worse has not been measured.

Kölsch has no standardised orthography

This bounds what WER can mean here. Measured on this model's output, ~2.8 WER points come from cases where the model wrote a legitimate, attested Kölsch spelling and the transcriber wrote a different legitimate spelling of the same word — dä/där/der, do/da, so/su. The audio does not determine which form a human will type, so WER charges a full word error for an undecidable choice.

The "variant-folded" row above folds a conservative, closed list of such variants in both reference and hypothesis before scoring (1.0 % vocabulary collapse; minimal pairs such as mir/mer and kamm/kam are explicitly protected). Treat raw WER/CER as primary and the folded figure as a clearly-labelled secondary metric. CER is the more stable headline for this task, since it charges proportionally for a one-character disagreement instead of treating it as an entirely wrong word.

Training

init	facebook/w2v-bert-2.0 (CTC head + conv adapter randomly initialised)
data	5,207 utterances / 3.97 h, ×3 speed perturbation (0.9/1.0/1.1) → 15,621 / ~11.9 h
vocab	35 tokens — a–z, ä ö ü ß, word delimiter, [UNK], [PAD], bos/eos
optimiser	LR 5e-5 cosine, 500 warmup, batch 16, bf16
masking	mask_time_prob=0.30, mask_feature_prob=0.15, layerdrop=0.05
schedule	20 epochs / 19,540 steps; best validation CER 0.1156 at step 19,000

Speed perturbation resamples the waveform, so it shifts both tempo and pitch (the Kaldi recipe) — pitch movement is what simulates an unseen vocal tract.

Known limitations

• Data-limited. With 3.97 h of labels, an oracle that combines perfect N-best rescoring, free out-of-vocabulary words, and folded spelling still only reaches 19.6 % WER on this acoustic model. Substantially lower error needs more labelled audio or a stronger pretrained model, not more tuning.
• Overfitting persists. Training loss fell 3.47 → 0.17 while validation loss rose from 0.64 to 1.02. Tripling the data via augmentation did not close that gap.
• 10.4 % of test tokens never occur in the training transcripts; recall on those is far below in-vocabulary recall, and an n-gram LM cannot invent them.
• Trained on read/spontaneous speech from a specific Kölsch corpus. Expect degradation on other Ripuarian varieties, other recording conditions, or spontaneous conversation.
• Not suitable for production transcription of unseen speakers without revalidation on a recording-disjoint split.
• The training corpus derives from published Kölsch source material. Downstream users should satisfy themselves about the provenance and terms of that material.

Usage

import torch, librosa
from transformers import Wav2Vec2BertForCTC, Wav2Vec2BertProcessor

proc = Wav2Vec2BertProcessor.from_pretrained("Vatho/koelsch-w2vbert-orthography")
model = Wav2Vec2BertForCTC.from_pretrained("Vatho/koelsch-w2vbert-orthography").eval()

audio, _ = librosa.load("clip.wav", sr=16000)
feats = proc(audio, sampling_rate=16000, return_tensors="pt").input_features
with torch.inference_mode():
    ids = model(input_features=feats).logits.argmax(-1)
print(proc.batch_decode(ids)[0])

For the 30.02 % WER figure, decode with the bundled kolsch_4gram.arpa via pyctcdecode at α=0.7, β=1.0, beam 100. Note pyctcdecode needs the blank token mapped to "" and the word delimiter to " ".

Citation

If you use this model, please cite the accompanying Kölsch ASR work and note the evaluation caveat above.