Fleet-2 / UPSTREAM_MODEL_CARD.md
Lucasgitguy-co's picture
Fleet-2: Whistle student of Whisper-large-v3-turbo
15e183b verified
|
Raw History Blame Contribute Delete
6.48 kB
metadata
library_name: cactus-needle
pipeline_tag: automatic-speech-recognition
license: apache-2.0
base_model: Cactus-Compute/whistle
base_model_relation: finetune
language:
  - ko
  - en
  - de
  - fr
  - es
  - it
  - nl
  - pl
datasets:
  - Bingsu/zeroth-korean
  - google/fleurs
metrics:
  - cer
  - wer
tags:
  - speech-recognition
  - speech-to-text
  - korean
  - on-device
  - edge
  - lora

Whistle fine-tuned for Korean

This is a fine-tune of Cactus-Compute/whistle that adds Korean to the original seven languages: English, German, French, Spanish, Italian, Dutch and Polish. It is still a single on-device file (whistle.cact, 30 MB, 4-bit) that runs on the same CPU engine as the original. The model detects all eight languages, including Korean, unless you set the language yourself.

Results

All scores come from the shipped 4-bit whistle.cact, run on the cactus-needle engine at its defaults. Lower is better.

  • Korean: CER (character error rate) with spaces removed.
  • Other languages: WER (word error rate) after the Whisper normalizer.
Test set Original Whistle This model
Zeroth-Korean test, 457 clips, ko CER โ€” (no Korean) 18.5%
FLEURS ko_kr test, 382 clips, ko CER 264% (answers in other languages) 32.2%
FLEURS en_us test, 100 clips, WER 16.1% 16.2%
FLEURS de_de test, 100 clips, WER 21.9% 22.2%
FLEURS fr_fr test, 100 clips, WER 23.4% 25.9%
FLEURS es_419 test, 100 clips, WER 10.6% 10.8%
FLEURS it_it test, 100 clips, WER 13.9% 13.9%
FLEURS nl_nl test, 100 clips, WER 27.0% 29.8%
FLEURS pl_pl test, 100 clips, WER 35.5% 35.8%

Language detection was 99% correct on Korean FLEURS clips and 97โ€“100% correct on the other seven languages.

Fine-tuning on Korean data alone pushed the other seven languages to 92โ€“214% WER and made the model label all speech as Korean. Mixing in some data from the original languages, as described below, avoided that.

Usage

pip install cactus-needle
import needle
from huggingface_hub import hf_hub_download

path = hf_hub_download("Sky-Kim/whistle-finetuned-ko", "whistle.cact")
whistle = needle.Whistle(path)

print(whistle.transcribe("clip.wav"))                 # detects the language, Korean included
print(whistle.transcribe("clip.wav", language="ko"))  # forces Korean

The input must be 16 kHz mono audio. The file also works with the needle runner and C API, the same way the original does:

needle --model whistle.cact --audio clip.wav

Training

  • Method: LoRA (rank 32, alpha 64) on the attention projections, with the base weights frozen. Training used 4-bit weights and 8-bit activations, matching the exported file. The run used needle whistle finetune from cactus-needle, plus a wrapper that saves the adapter after every epoch.
  • Vocabulary: the original tokenizer had no Hangul, so every syllable became three byte-fallback tokens. 1,202 Hangul syllables from the training transcripts were added (2,404 new pieces: each syllable with and without a leading space), and ko was added as a ninth language tag. This cut the median token count per Korean sentence from 133 to 40.
  • Data: a pool of 29,263 clips, of which 28,678 were trained on.
    • The full Zeroth-Korean train split: 22,263 clips, about 51 h.
    • A random 1,000 clips per original language from FLEURS train, using raw_transcription so casing and punctuation are kept. That is about a third of each language's train split, which holds 2,602โ€“3,193 clips; the slice is there only to keep the original languages from degrading.
    • A random 2% of the pool (585 clips, Korean and other languages mixed) was held out for validation and never trained on.
    • No test split was used for training: not Zeroth-Korean test, FLEURS ko_kr test, or any language's FLEURS test.
  • Schedule: AdamW with a learning rate of 5e-4 (10ร— for the new vocabulary rows), batch size 16, warmup and cosine decay over 3 epochs. This release is the epoch 2 checkpoint, which scored slightly better on Korean FLEURS than epoch 3 (32.2% vs 33.5% CER), while the other languages scored about the same.
  • Hardware: a single Apple-silicon CPU.

The LoRA adapter is in adapter/whistle_ko_lora.safetensors. To rebuild the .cact from it, for example for another platform folder or decoder depth, run the command below. It downloads the original checkpoint if it is missing, and the result is byte-identical to the shipped whistle.cact.

pip install "cactus-needle[train] @ git+https://github.com/cactus-compute/needle"   # whistle build/finetune are newer than the PyPI release used here
needle whistle build --lora adapter/whistle_ko_lora.safetensors --out whistle.cact

Limitations

  • Numbers: Zeroth writes numbers out in Hangul ("์‚ผ ์›”"), while FLEURS uses digits ("3์›”"), so the model often spells numbers in Hangul. This accounts for part of the FLEURS CER.
  • Proper nouns: foreign names and loanwords are often misspelled. For example, "๋ธ ํฌํŠธ๋กœ" comes out as "ํ…Œ๋ณดํŠธ๋กœ".
  • Training data size: the Korean data is about 51 h of read news-style speech. Accuracy on conversational speech, dialects or noisy audio has not been measured.
  • Other languages: on the 100-clip FLEURS samples, French and Dutch are 2โ€“3 WER points worse than the original. The other languages are within 0.3 points.

License and attribution

If you use Whistle, please cite the original work:

@misc{whistle_2026,
  title        = {Whistle: Speech Recognition for Tiny Devices},
  author       = {Mroz, Jakub and Ndubuaku, Henry and Mosoyan, Karen and Cylich, Noah and
                  Kumar, Satyajit and Sandhu, Parkirat and Shemet, Roman and Lee, Justin H.},
  year         = {2026},
  organization = {Cactus Compute, Inc.},
  howpublished = {\url{https://github.com/cactus-compute/needle}}
}