Fleet-2 / UPSTREAM_MODEL_CARD.md
Lucasgitguy-co's picture
Fleet-2: Whistle student of Whisper-large-v3-turbo
15e183b verified
|
Raw History Blame Contribute Delete
6.48 kB
---
library_name: cactus-needle
pipeline_tag: automatic-speech-recognition
license: apache-2.0
base_model: Cactus-Compute/whistle
base_model_relation: finetune
language:
- ko
- en
- de
- fr
- es
- it
- nl
- pl
datasets:
- Bingsu/zeroth-korean
- google/fleurs
metrics:
- cer
- wer
tags:
- speech-recognition
- speech-to-text
- korean
- on-device
- edge
- lora
---
# Whistle fine-tuned for Korean
This is a fine-tune of [Cactus-Compute/whistle](https://huggingface.co/Cactus-Compute/whistle) that adds **Korean** to the original seven languages: English, German, French, Spanish, Italian, Dutch and Polish. It is still a single on-device file (`whistle.cact`, 30 MB, 4-bit) that runs on the same CPU engine as the original. The model detects all eight languages, including Korean, unless you set the language yourself.
## Results
All scores come from the shipped 4-bit `whistle.cact`, run on the `cactus-needle` engine at its defaults. Lower is better.
- **Korean:** CER (character error rate) with spaces removed.
- **Other languages:** WER (word error rate) after the Whisper normalizer.
| Test set | Original Whistle | This model |
|---|---|---|
| Zeroth-Korean test, 457 clips, ko CER | โ€” (no Korean) | **18.5%** |
| FLEURS ko_kr test, 382 clips, ko CER | 264% (answers in other languages) | **32.2%** |
| FLEURS en_us test, 100 clips, WER | 16.1% | 16.2% |
| FLEURS de_de test, 100 clips, WER | 21.9% | 22.2% |
| FLEURS fr_fr test, 100 clips, WER | 23.4% | 25.9% |
| FLEURS es_419 test, 100 clips, WER | 10.6% | 10.8% |
| FLEURS it_it test, 100 clips, WER | 13.9% | 13.9% |
| FLEURS nl_nl test, 100 clips, WER | 27.0% | 29.8% |
| FLEURS pl_pl test, 100 clips, WER | 35.5% | 35.8% |
Language detection was 99% correct on Korean FLEURS clips and 97โ€“100% correct on the other seven languages.
Fine-tuning on Korean data alone pushed the other seven languages to 92โ€“214% WER and made the model label all speech as Korean. Mixing in some data from the original languages, as described below, avoided that.
## Usage
```sh
pip install cactus-needle
```
```python
import needle
from huggingface_hub import hf_hub_download
path = hf_hub_download("Sky-Kim/whistle-finetuned-ko", "whistle.cact")
whistle = needle.Whistle(path)
print(whistle.transcribe("clip.wav")) # detects the language, Korean included
print(whistle.transcribe("clip.wav", language="ko")) # forces Korean
```
The input must be 16 kHz mono audio. The file also works with the `needle` runner and C API, the same way the original does:
```sh
needle --model whistle.cact --audio clip.wav
```
## Training
- **Method:** LoRA (rank 32, alpha 64) on the attention projections, with the base weights frozen. Training used 4-bit weights and 8-bit activations, matching the exported file. The run used `needle whistle finetune` from [cactus-needle](https://github.com/cactus-compute/needle), plus a wrapper that saves the adapter after every epoch.
- **Vocabulary:** the original tokenizer had no Hangul, so every syllable became three byte-fallback tokens. 1,202 Hangul syllables from the training transcripts were added (2,404 new pieces: each syllable with and without a leading space), and `ko` was added as a ninth language tag. This cut the median token count per Korean sentence from 133 to 40.
- **Data:** a pool of 29,263 clips, of which 28,678 were trained on.
- The full [Zeroth-Korean](https://huggingface.co/datasets/Bingsu/zeroth-korean) train split: 22,263 clips, about 51 h.
- A random 1,000 clips per original language from [FLEURS](https://huggingface.co/datasets/google/fleurs) train, using `raw_transcription` so casing and punctuation are kept. That is about a third of each language's train split, which holds 2,602โ€“3,193 clips; the slice is there only to keep the original languages from degrading.
- A random 2% of the pool (585 clips, Korean and other languages mixed) was held out for validation and never trained on.
- No test split was used for training: not Zeroth-Korean test, FLEURS ko_kr test, or any language's FLEURS test.
- **Schedule:** AdamW with a learning rate of 5e-4 (10ร— for the new vocabulary rows), batch size 16, warmup and cosine decay over 3 epochs. This release is the **epoch 2** checkpoint, which scored slightly better on Korean FLEURS than epoch 3 (32.2% vs 33.5% CER), while the other languages scored about the same.
- **Hardware:** a single Apple-silicon CPU.
The LoRA adapter is in `adapter/whistle_ko_lora.safetensors`. To rebuild the `.cact` from it, for example for another platform folder or decoder depth, run the command below. It downloads the original checkpoint if it is missing, and the result is byte-identical to the shipped `whistle.cact`.
```sh
pip install "cactus-needle[train] @ git+https://github.com/cactus-compute/needle" # whistle build/finetune are newer than the PyPI release used here
needle whistle build --lora adapter/whistle_ko_lora.safetensors --out whistle.cact
```
## Limitations
- **Numbers:** Zeroth writes numbers out in Hangul ("์‚ผ ์›”"), while FLEURS uses digits ("3์›”"), so the model often spells numbers in Hangul. This accounts for part of the FLEURS CER.
- **Proper nouns:** foreign names and loanwords are often misspelled. For example, "๋ธ ํฌํŠธ๋กœ" comes out as "ํ…Œ๋ณดํŠธ๋กœ".
- **Training data size:** the Korean data is about 51 h of read news-style speech. Accuracy on conversational speech, dialects or noisy audio has not been measured.
- **Other languages:** on the 100-clip FLEURS samples, French and Dutch are 2โ€“3 WER points worse than the original. The other languages are within 0.3 points.
## License and attribution
- **Model:** a derivative of [Cactus-Compute/whistle](https://huggingface.co/Cactus-Compute/whistle) by Cactus Compute, Inc., released under the same Apache License 2.0 (`LICENSE`).
- **Training data:** [Zeroth-Korean](https://www.openslr.org/40/), CC BY 4.0, and [FLEURS](https://huggingface.co/datasets/google/fleurs), CC BY 4.0.
If you use Whistle, please cite the original work:
```bibtex
@misc{whistle_2026,
title = {Whistle: Speech Recognition for Tiny Devices},
author = {Mroz, Jakub and Ndubuaku, Henry and Mosoyan, Karen and Cylich, Noah and
Kumar, Satyajit and Sandhu, Parkirat and Shemet, Roman and Lee, Justin H.},
year = {2026},
organization = {Cactus Compute, Inc.},
howpublished = {\url{https://github.com/cactus-compute/needle}}
}
```