--- license: apache-2.0 language: [en, de, fr, es, it, nl, pl, ko] pipeline_tag: automatic-speech-recognition base_model: - Lucasgitguy-co/Fleet-2 - convaiinnovations/laya-multilingual datasets: - Lucasgitguy-co/fleet-clean-tags - amaai-lab/DisfluencySpeech - facebook/voxpopuli - Bingsu/zeroth-korean - google/fleurs tags: [speech-to-text, disfluency-removal, punctuation, token-classification, whistle, cactus, mmbert, laya] --- # Fleet-Clean-0.36B Speech in, clean text out. It is two open models with 363M parameters in total: 1. **[Fleet-2](https://huggingface.co/Lucasgitguy-co/Fleet-2)** (56M, Cactus Whistle) writes down what was said. 2. **A word tagger** (307M) cleans that transcript. It is the encoder of [convaiinnovations/laya-multilingual](https://huggingface.co/convaiinnovations/laya-multilingual) (mmBERT) with a new head that decides, for every word: - keep or delete it (fillers, discourse markers, repeats, false starts) - which mark follows it (none , . ? !) - whether its first letter changes case The tagger only removes Fleet-2's words and fixes punctuation and capitals. **It never writes a word of its own**, so it can't invent content or answer what was dictated. **GGUF version:** [Fleet-Clean-0.36B-GGUF](https://huggingface.co/Lucasgitguy-co/Fleet-Clean-0.36B-GGUF) · **Training data:** [fleet-clean-tags](https://huggingface.co/datasets/Lucasgitguy-co/fleet-clean-tags) · **Collection:** [Speech recognition](https://huggingface.co/collections/Lucasgitguy-co/speech-recognition-6aef2dcbe0ea208136b0ddc9) ## Usage ```bash pip install cactus-compute torch "transformers>=5" safetensors hf download Lucasgitguy-co/Fleet-Clean-0.36B --local-dir fleet-clean && cd fleet-clean python fleet_clean.py clip.wav # clean transcript python fleet_clean.py --json clip.wav # raw, clean and the per-word tags python fleet_clean.py --text "so I I think, you know, we should go" python fleet_clean.py --punct full clip.wav # also re-punctuate words away from deletions ``` Audio is 16 kHz WAV (`ffmpeg -i in.m4a -ar 16000 -ac 1 clip.wav`). Fleet-2 detects the language, or you can pass `--language`. The delete threshold is 0.5. It was calibrated on validation data so that fluent speech keeps its words, and `--threshold` changes it. ## Examples These are from the DisfluencySpeech test set (English): - **Fleet-2:** Yeah. Well, robots are darling. That would be fun if you could get them trained. Otherwise, they're pretty smelly. **Fleet-Clean:** Yeah, robots are darling. That would be fun if you could get them trained. Otherwise, they're pretty smelly. - **Fleet-2:** Well, McNew, New, and Lara, I guess it's leaving one of them is a leaving. **Fleet-Clean:** McNew, New, and Lara, I guess one of them is a leaving. - **Fleet-2:** Well, I don't want to. I don't want this to be a unilateral conversation, but yes, we do. **Fleet-Clean:** I don't want to. I don't want this to be a unilateral conversation, but yes, we do. ## Evaluation **Real disfluent English speech.** Fleet-2 transcribes the audio, and the output is compared with DisfluencySpeech's own cleaned transcript: | DisfluencySpeech test (250 clips) | Fleet-2 alone | Fleet-Clean | |---|---:|---:| | WER against the cleaned transcript (`transcript_c`), punctuation ignored | 18.8% | **14.6%** | | Punctuation F1 against `transcript_c` | **0.6700** | 0.6695 | **Fluent speech (FLEURS test).** These are the same 100 clips per language as Fleet-2's card. A cleanup model should leave fluent speech alone: | Language | Words deleted from fluent speech | WER before | WER after | |---|---:|---:|---:| | English | 0.47% | 14.5% | 14.3% | | German | 0.05% | 22.0% | 22.0% | | French | 0.04% | 24.9% | 25.0% | | Spanish | 0.05% | 11.9% | 11.9% | | Italian | 0.06% | 16.7% | 16.7% | | Dutch | 0.19% | 30.1% | 30.0% | | Polish | 0.29% | 41.6% | 41.6% | | Korean | 0.27% | 42.8% (CER) | 42.9% (CER) |
Synthetic disfluencies (held-out tag test rows) Fillers, discourse markers, repeats, restarts and self-corrections were inserted into Fleet-2 transcripts of held-out VoxPopuli / Zeroth / FLEURS clips. The table shows whether the tagger finds exactly the inserted words. "Weak" means F1 below 0.80. | Language | Delete F1 | Precision | Recall | |---|---:|---:|---:| | English | 0.981 | 0.967 | 0.996 | | German | 0.984 | 0.968 | 1.000 | | French | 0.970 | 0.946 | 0.995 | | Spanish | 0.971 | 0.978 | 0.964 | | Italian | 0.969 | 0.959 | 0.979 | | Dutch | 0.952 | 0.920 | 0.986 | | Polish | 0.990 | 0.980 | 1.000 | | Korean | 0.979 | 0.975 | 0.984 |
Limits: - **Real disfluent speech was only tested in English.** The other seven languages were trained and tested on synthetic disfluencies, so their numbers show how well the tagger removes those inserted patterns, not how it handles real conversation. - **Fleet-2 already leaves out most "uh/um".** It learned from Whisper transcripts. What is left to clean is mostly "you know", "I mean", "like", repeats and restarts. - **Recognition errors stay.** The tagger can only delete or re-punctuate words, so it can't fix a misheard word. - Clips are up to 30 s per call. The tagger reads up to 512 tokens and leaves anything past that unchanged. ## How it was built
Data, training and gates - **Transcripts:** Fleet-2 (`whistle.cact`, native Cactus engine, automatic language detection) transcribed: - DisfluencySpeech train/validation/test - VoxPopuli validation (7 languages, a split Fleet-2 never trained on) - Zeroth-Korean train minus Fleet-2's own training rows - FLEURS Korean train minus Fleet-2's rows - **Labels:** - **English deletions:** each Fleet-2 word is aligned to DisfluencySpeech's verbatim transcript, and that transcript's words are mapped to its cleaned version. - **Punctuation and case:** taken from the human reference wherever Fleet-2's word matches it. Zeroth has no punctuation, so its rows only teach deletion. - **Synthetic rows:** disfluencies are only ever inserted, so the clean text is always recoverable by deletion. - **Training:** - Laya-multilingual's encoder, with its 256k-token embedding table frozen, plus a new 768→10 head. - PyTorch on an Apple M4 (MPS), fp32. - Best validation checkpoint kept, then the delete threshold calibrated on validation only. - **Release gates:** - G1: cleaning lowers WER on real disfluent English, and punctuation F1 doesn't drop. WER dropped from 18.76% to 14.61%, but punctuation F1 went from 0.6700 to 0.6695 (−0.0005), so this part of the gate did not pass. It was published anyway by the owner's decision, and the number is reported as measured. - G2: at most 1.0% of words deleted from fluent FLEURS speech in every language. Passed in all 8. - **Punctuation mode:** by default, Fleet-2's own punctuation is kept, and the tagger's punctuation and case are used only next to removed words. This was chosen on DisfluencySpeech *validation*, where re-punctuating everything scored lower against the human clean transcripts. `--punct full` re-punctuates every word; on fluent validation speech in all 8 languages, full mode was slightly more accurate than Fleet-2's own marks.
## License and attribution Apache-2.0. See `NOTICE` for every component: - Fleet-2 / Cactus Whistle (Apache-2.0) - Laya-multilingual (Apache-2.0), which builds on mmBERT (MIT) - DisfluencySpeech (Apache-2.0) - VoxPopuli (CC0) - Zeroth-Korean and FLEURS (CC BY 4.0) No audio is included.