Fleet-Clean-0.36B / README.md
Lucasgitguy-co's picture
Fleet-Clean release
e7e5ec3 verified
|
Raw History Blame Contribute Delete
7.53 kB
---
license: apache-2.0
language: [en, de, fr, es, it, nl, pl, ko]
pipeline_tag: automatic-speech-recognition
base_model:
- Lucasgitguy-co/Fleet-2
- convaiinnovations/laya-multilingual
datasets:
- Lucasgitguy-co/fleet-clean-tags
- amaai-lab/DisfluencySpeech
- facebook/voxpopuli
- Bingsu/zeroth-korean
- google/fleurs
tags: [speech-to-text, disfluency-removal, punctuation, token-classification, whistle, cactus, mmbert, laya]
---
# Fleet-Clean-0.36B
Speech in, clean text out. It is two open models with 363M parameters in total:
1. **[Fleet-2](https://huggingface.co/Lucasgitguy-co/Fleet-2)** (56M, Cactus Whistle) writes down what was said.
2. **A word tagger** (307M) cleans that transcript. It is the encoder of [convaiinnovations/laya-multilingual](https://huggingface.co/convaiinnovations/laya-multilingual) (mmBERT) with a new head that decides, for every word:
- keep or delete it (fillers, discourse markers, repeats, false starts)
- which mark follows it (none , . ? !)
- whether its first letter changes case
The tagger only removes Fleet-2's words and fixes punctuation and capitals. **It never writes a word of its own**, so it can't invent content or answer what was dictated.
**GGUF version:** [Fleet-Clean-0.36B-GGUF](https://huggingface.co/Lucasgitguy-co/Fleet-Clean-0.36B-GGUF) · **Training data:** [fleet-clean-tags](https://huggingface.co/datasets/Lucasgitguy-co/fleet-clean-tags) · **Collection:** [Speech recognition](https://huggingface.co/collections/Lucasgitguy-co/speech-recognition-6aef2dcbe0ea208136b0ddc9)
## Usage
```bash
pip install cactus-compute torch "transformers>=5" safetensors
hf download Lucasgitguy-co/Fleet-Clean-0.36B --local-dir fleet-clean && cd fleet-clean
python fleet_clean.py clip.wav # clean transcript
python fleet_clean.py --json clip.wav # raw, clean and the per-word tags
python fleet_clean.py --text "so I I think, you know, we should go"
python fleet_clean.py --punct full clip.wav # also re-punctuate words away from deletions
```
Audio is 16 kHz WAV (`ffmpeg -i in.m4a -ar 16000 -ac 1 clip.wav`). Fleet-2 detects the language, or you can pass `--language`. The delete threshold is 0.5. It was calibrated on validation data so that fluent speech keeps its words, and `--threshold` changes it.
## Examples
These are from the DisfluencySpeech test set (English):
- **Fleet-2:** Yeah. Well, robots are darling. That would be fun if you could get them trained. Otherwise, they're pretty smelly.
**Fleet-Clean:** Yeah, robots are darling. That would be fun if you could get them trained. Otherwise, they're pretty smelly.
- **Fleet-2:** Well, McNew, New, and Lara, I guess it's leaving one of them is a leaving.
**Fleet-Clean:** McNew, New, and Lara, I guess one of them is a leaving.
- **Fleet-2:** Well, I don't want to. I don't want this to be a unilateral conversation, but yes, we do.
**Fleet-Clean:** I don't want to. I don't want this to be a unilateral conversation, but yes, we do.
## Evaluation
**Real disfluent English speech.** Fleet-2 transcribes the audio, and the output is compared with DisfluencySpeech's own cleaned transcript:
| DisfluencySpeech test (250 clips) | Fleet-2 alone | Fleet-Clean |
|---|---:|---:|
| WER against the cleaned transcript (`transcript_c`), punctuation ignored | 18.8% | **14.6%** |
| Punctuation F1 against `transcript_c` | **0.6700** | 0.6695 |
**Fluent speech (FLEURS test).** These are the same 100 clips per language as Fleet-2's card. A cleanup model should leave fluent speech alone:
| Language | Words deleted from fluent speech | WER before | WER after |
|---|---:|---:|---:|
| English | 0.47% | 14.5% | 14.3% |
| German | 0.05% | 22.0% | 22.0% |
| French | 0.04% | 24.9% | 25.0% |
| Spanish | 0.05% | 11.9% | 11.9% |
| Italian | 0.06% | 16.7% | 16.7% |
| Dutch | 0.19% | 30.1% | 30.0% |
| Polish | 0.29% | 41.6% | 41.6% |
| Korean | 0.27% | 42.8% (CER) | 42.9% (CER) |
<details>
<summary>Synthetic disfluencies (held-out tag test rows)</summary>
Fillers, discourse markers, repeats, restarts and self-corrections were inserted into Fleet-2 transcripts of held-out VoxPopuli / Zeroth / FLEURS clips. The table shows whether the tagger finds exactly the inserted words. "Weak" means F1 below 0.80.
| Language | Delete F1 | Precision | Recall |
|---|---:|---:|---:|
| English | 0.981 | 0.967 | 0.996 |
| German | 0.984 | 0.968 | 1.000 |
| French | 0.970 | 0.946 | 0.995 |
| Spanish | 0.971 | 0.978 | 0.964 |
| Italian | 0.969 | 0.959 | 0.979 |
| Dutch | 0.952 | 0.920 | 0.986 |
| Polish | 0.990 | 0.980 | 1.000 |
| Korean | 0.979 | 0.975 | 0.984 |
</details>
Limits:
- **Real disfluent speech was only tested in English.** The other seven languages were trained and tested on synthetic disfluencies, so their numbers show how well the tagger removes those inserted patterns, not how it handles real conversation.
- **Fleet-2 already leaves out most "uh/um".** It learned from Whisper transcripts. What is left to clean is mostly "you know", "I mean", "like", repeats and restarts.
- **Recognition errors stay.** The tagger can only delete or re-punctuate words, so it can't fix a misheard word.
- Clips are up to 30 s per call. The tagger reads up to 512 tokens and leaves anything past that unchanged.
## How it was built
<details>
<summary>Data, training and gates</summary>
- **Transcripts:** Fleet-2 (`whistle.cact`, native Cactus engine, automatic language detection) transcribed:
- DisfluencySpeech train/validation/test
- VoxPopuli validation (7 languages, a split Fleet-2 never trained on)
- Zeroth-Korean train minus Fleet-2's own training rows
- FLEURS Korean train minus Fleet-2's rows
- **Labels:**
- **English deletions:** each Fleet-2 word is aligned to DisfluencySpeech's verbatim transcript, and that transcript's words are mapped to its cleaned version.
- **Punctuation and case:** taken from the human reference wherever Fleet-2's word matches it. Zeroth has no punctuation, so its rows only teach deletion.
- **Synthetic rows:** disfluencies are only ever inserted, so the clean text is always recoverable by deletion.
- **Training:**
- Laya-multilingual's encoder, with its 256k-token embedding table frozen, plus a new 768→10 head.
- PyTorch on an Apple M4 (MPS), fp32.
- Best validation checkpoint kept, then the delete threshold calibrated on validation only.
- **Release gates:**
- G1: cleaning lowers WER on real disfluent English, and punctuation F1 doesn't drop. WER dropped from 18.76% to 14.61%, but punctuation F1 went from 0.6700 to 0.6695 (−0.0005), so this part of the gate did not pass. It was published anyway by the owner's decision, and the number is reported as measured.
- G2: at most 1.0% of words deleted from fluent FLEURS speech in every language. Passed in all 8.
- **Punctuation mode:** by default, Fleet-2's own punctuation is kept, and the tagger's punctuation and case are used only next to removed words. This was chosen on DisfluencySpeech *validation*, where re-punctuating everything scored lower against the human clean transcripts. `--punct full` re-punctuates every word; on fluent validation speech in all 8 languages, full mode was slightly more accurate than Fleet-2's own marks.
</details>
## License and attribution
Apache-2.0. See `NOTICE` for every component:
- Fleet-2 / Cactus Whistle (Apache-2.0)
- Laya-multilingual (Apache-2.0), which builds on mmBERT (MIT)
- DisfluencySpeech (Apache-2.0)
- VoxPopuli (CC0)
- Zeroth-Korean and FLEURS (CC BY 4.0)
No audio is included.