derkar00/whisper-base-turkmen

Whisper Base — Turkmen, checkpoint 8,500

Original full fine-tune used by ses.; unchanged FP32 SafeTensors and Transformers configuration. Optimized counterpart: derkar00/whisper-base-turkmen-ct2.

Recognition results: corrected inference settings

Checkpoint 8,500 measured 25.27% WER with decoding matched to its training prefix; its production CT2 INT8/FP16 export measured 25.49%. The 39.63% figure in the training-history table is the older checkpoint-selection result under different decoding settings, not a deterioration after publication.

Recorded evaluation	Runtime / decoding	Clips	WER% ↓	CER% ↓
Historical training selection	Transformers BF16, standard four-token forced-Turkmen/transcribe prefix	256	39.6336	13.7019
Later matched-prefix evaluation	Same checkpoint, Transformers BF16, matching two-token prefix	256	25.2652	6.6570
Production conversion evaluation	Same checkpoint converted to CT2 INT8/FP16, matching two-token prefix	256	25.4902	6.7065

All rows use the same 256 Mamedov validation/selection recordings. The later evaluations use [50258,50363], matching the actual training-label prefix, instead of the historical [50258,50341,50359,50363]. A separate paired fresh-load prompt comparison recorded 39.41% with the four-token prefix and 25.27% with the matching two-token prefix; it should not be confused with the original Trainer's 39.63% result. Model weights were unchanged by this inference correction.

These are retained GPU evaluation results, not newly rerun release benchmarks. The 25.27% result is for Transformers BF16 and the 25.49% result is for the production direct-CT2 GPU path with its retained processor. The supplied examples preserve the matching prompt, but their CPU/FP16 arithmetic and Faster-Whisper frontend were smoke-tested on only three clips; no full-set CPU or Faster-Whisper-wrapper WER is claimed. The small original-to-INT8 difference does not establish a general accuracy loss or gain.

This validation set was repeatedly used for model and decoder selection. These scores are not an independent test, not results on all 7,470 held-out clips, and not guaranteed error rates for phone recordings. No retraining, beam increase, external language model or ses. correction postprocessing was used for the matching-prefix results. See provenance.json for numerical records and evaluation scopes.

Installation and inference

Use Python 3.12. For CPU-only PyTorch, install its CPU wheel first:

python -m venv .venv
source .venv/bin/activate
python -m pip install "torch==2.11.0" --index-url https://download.pytorch.org/whl/cpu
python -m pip install "transformers==4.57.1" "numpy==1.26.4" "soundfile==0.13.1" "scipy==1.15.3"
hf download derkar00/whisper-base-turkmen infer.py --local-dir turkmen-whisper
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 python turkmen-whisper/infer.py speech.wav --device cpu

The script downloads the processor and weights from this repository ID, not a local project. WAV/FLAC input is mixed to mono and resampled to 16 kHz; nonempty finite audio is required. Thirty-second windows include the tail of longer files. Output preserves Unicode. To inspect timing/options add --json; pin a repository commit with --revision COMMIT and disable token use with --anonymous.

Python usage: first change into the downloaded turkmen-whisper directory, then run:

from infer import Transcriber
asr = Transcriber("derkar00/whisper-base-turkmen", device="cpu", threads=1)
print(asr.transcribe("speech.wav")["text"])

Transcriber loads WhisperProcessor and WhisperForConditionalGeneration from Transformers. It calls generate(language="tk", task="transcribe", decoder_input_ids=[[50258,50363]], num_beams=1, do_sample=False, use_cache=True). The serialized training configuration is unchanged; caching is enabled at inference through call arguments.

For CUDA, install an appropriate PyTorch CUDA build instead of the CPU wheel and select --device cuda; the script uses FP16 on CUDA. See PyTorch installation for your platform. GPU requirements include weights, encoder/decoder workspace and framework allocations; there is no model-file-size-only VRAM guarantee. CPU FP32 is supported and tested, one thread by default.

Minimal reproducible smoke test: use your own valid speech file, run the command above with --json --anonymous, and require a nonempty Turkmen text field. Near-zero audio intentionally returns empty text. CPU smoke checks of this packaged script are documented in VALIDATION.md; CPU output may differ from historical BF16/GPU decoding.

Purpose and language

This is an automatic speech recognition (ASR) model: Turkmen speech in, Turkmen Latin-script transcription out. It is not a speech-translation or text-to-speech model. In ses., translation and speech generation are separate pipeline stages. English, Romanian, Russian and other multilingual recognition capabilities were not preserved/evaluated as a goal of this fine-tune; use the original multilingual model for those languages.

Architecture and provenance

OpenAI multilingual Whisper Base, encoder–decoder Transformer: 6 encoder layers, 6 decoder layers, hidden size 512, vocabulary 51,865; 72,593,920 parameters, all fine-tuned (no LoRA/adapters). Base: openai/whisper-base, pinned revision e37978b90ca9030d5170a5c07aadb050351a65bb.

The selected original weight SHA256 is 2a299a87e3ce43fb75facb92d20f132dbea2a0080ed838d92c6f6aa6ada2dac1. Both repositories refer to this same checkpoint, 8,500. The run finished at 10,320 optimizer steps; the final model is not the selected model. File-level provenance and training/evaluation configuration are recorded in provenance.json.

Training data and preprocessing

Fine-tuned on mamed0v/TurkmenSpeech, revision 62f2ed565b0bd669ca529e2b016d6805d442a392, attributed to mamed0v, CC BY-NC 4.0. Training used 110,076 clips /202.7714 hours. A source-recording-disjoint validation partition contains 7,470 clips /13.7229 hours; scheduled evaluation sampled the same 256 clips /0.4324 hours. These hours are sums of retained metadata durations, not independently measured human speech time.

The local deterministic split hashes source identifiers with seed 42; training/validation source sets do not overlap. Later video-ID audit found 390 training videos and25 held-out videos with no ID overlap. It also found source-URL/video-ID mismatches in 22 of256 evaluation rows, a provenance limitation. The dataset contains broadcast/news/narration and conversational material; the recordings and labels were not all manually re-audited by this release author.

Filtering rejected 121 rows with missing text/source and2,180 invalid/out-of-range durations; no row was rejected for excessive token count. Training accepted 0.2–30-second clips, mixed channels to mono, resampled to 16 kHz and used Whisper's80-bin log-mel frontend, FFT400/hop160, with a30-second padded window. Labels used Unicode NFC and trimmed outer whitespace. Turkmen characters ä, ç, ş, ž, ň, ö, ü, ý were retained; no ASCII transliteration or forced removal of diacritics. Metric scoring lowercased NFC text and collapsed whitespace, retaining punctuation.

Common Voice was not used to train checkpoint 8,500. Mozilla Common Voice 27.0 and other adaptation/correction experiments occurred later and are not merged into these weights. No private phone recordings, correction dictionary, language model, dataset audio or transcript collection is included in this release.

Training configuration and checkpoint selection

Setting	Recorded value
Method	Full parameter supervised fine-tuning, PyTorch / Transformers Seq2SeqTrainer
Framework versions	PyTorch 2.11.0+cu128; Transformers 4.57.1
Hardware	NVIDIA GeForce RTX 3050 Laptop GPU,4 GiB
Training arithmetic /saved weights	BF16 mixed precision /FP32 SafeTensors
Optimizer	AdamW (adamw_torch)
Peak learning rate /scheduler	1e-5 /linear decay
Warmup /weight decay	5% /0.01
Per-device batch /gradient accumulation	1 /32 (effective32 clips per optimizer step)
Schedule	3 epochs,10,320 optimizer steps total
Published checkpoint	8,500 optimizer steps
Memory method	Gradient checkpointing; training cache disabled
Evaluation	Every 500 steps, same 256 held-out clips, greedy decoding
Seeds	42

Selection was the lowest observed validation WER, with CER used for ties. Checkpoint 8,500 was also preferred in informal ses. listening tests; there was no scored/blinded listening study. Later training did not beat its recorded WER:

Checkpoint	Historical training-selection WER%	CER%
8,500	39.6336	13.7019
9,000	40.3086	14.9375
9,500	39.6657	14.5991
10,000	39.7300	14.6675

Checkpoint 8,500 validation loss was 0.386895. Step 10,320 was not separately evaluated. The small difference to 9,500 is not evidence of statistical significance; no confidence intervals were computed. The256-clip validation subset influenced selection and is not a fresh external test, nor a score over all 7,470 held-out clips.

Decoder policy: important for reproducing ses.

The serialized tokenizer's training prefix omitted language/task tokens: [50258,50363] = <|startoftranscript|><|notimestamps|>. The production implementation explicitly preserves it. This is not a missing Turkmen token: standard Whisper supports tk (<|tk|> =50341).

The provided infer.py sets Turkmen/transcription explicitly where supported and supplies the two-token prefix. It uses beam1, no sampling/fallback, no VAD, no previous-text conditioning, and independently processes each30-second window, including the tail. It skips near-zero windows (peak<1e-5), not general background noise. It does not apply ses. correction postprocessing.

The historical 39.63%/13.70% selection scores used the recorded standard forced-Turkmen/transcribe generation path, BF16 autocast and max length 448. A 32-clip prompt probe was followed by a full 256-clip Mamedov comparison: matching the actual training prefix measured 25.27% WER /6.66% CER. The production CT2 INT8/FP16 conversion later measured 25.49% WER /6.71% CER on those same clips. These are distinct retained evaluations, not newly measured scores for the packaged CPU or Faster-Whisper script; see the recognition-results table above.

Limitations

Word boundaries, repeated words, names, inflections and diacritics can still be wrong. Broadcast-heavy training does not establish accuracy on everyday phone speech, noise, accents or every speaker. Silence handling is only a near-zero check, not robust endpoint detection. Fixed windows have no overlap/stitching and can lose context or duplicate words at boundaries. Long/noisy speech can reach the decoding ceiling; this is not streaming ASR or word-aligned timestamp support. Review transcriptions where errors matter.

Historical evaluation is validation, not an unbiased test. The fresh release smoke check uses a few already-held-out validation clips; it verifies packaging/Unicode/major regressions, not population accuracy. No new full-corpus WER, non-Turkmen benchmark, mobile/WAN latency or calibrated service SLA is claimed. See VALIDATION.md for the actual release checks and timing scope.

License and attribution

Published fine-tuned weights: CC BY-NC 4.0, noncommercial use, attribution and indication of changes required. LICENSE contains the full terms. This release does not grant commercial rights to TurkmenSpeech or derivative fine-tuned weights.

The pinned Hugging Face Whisper Base card identifies Apache-2.0; OpenAI's original Whisper repository publishes code/weights under MIT. Both upstream notices are retained under licenses/. NOTICE.md attributes OpenAI, the upstream Hugging Face distribution and mamed0v, and describes the fine-tuning/conversion changes. Example code is Apache-2.0; this does not change the weights' noncommercial terms. No affiliation or endorsement is implied. Common Voice's separate CC0 status does not remove TurkmenSpeech's restrictions.