Fleet-Clean-0.36B / README.md
Lucasgitguy-co's picture
Fleet-Clean release
e7e5ec3 verified
|
Raw History Blame Contribute Delete
7.53 kB
metadata
license: apache-2.0
language:
  - en
  - de
  - fr
  - es
  - it
  - nl
  - pl
  - ko
pipeline_tag: automatic-speech-recognition
base_model:
  - Lucasgitguy-co/Fleet-2
  - convaiinnovations/laya-multilingual
datasets:
  - Lucasgitguy-co/fleet-clean-tags
  - amaai-lab/DisfluencySpeech
  - facebook/voxpopuli
  - Bingsu/zeroth-korean
  - google/fleurs
tags:
  - speech-to-text
  - disfluency-removal
  - punctuation
  - token-classification
  - whistle
  - cactus
  - mmbert
  - laya

Fleet-Clean-0.36B

Speech in, clean text out. It is two open models with 363M parameters in total:

  1. Fleet-2 (56M, Cactus Whistle) writes down what was said.
  2. A word tagger (307M) cleans that transcript. It is the encoder of convaiinnovations/laya-multilingual (mmBERT) with a new head that decides, for every word:
    • keep or delete it (fillers, discourse markers, repeats, false starts)
    • which mark follows it (none , . ? !)
    • whether its first letter changes case

The tagger only removes Fleet-2's words and fixes punctuation and capitals. It never writes a word of its own, so it can't invent content or answer what was dictated.

GGUF version: Fleet-Clean-0.36B-GGUF · Training data: fleet-clean-tags · Collection: Speech recognition

Usage

pip install cactus-compute torch "transformers>=5" safetensors
hf download Lucasgitguy-co/Fleet-Clean-0.36B --local-dir fleet-clean && cd fleet-clean
python fleet_clean.py clip.wav                  # clean transcript
python fleet_clean.py --json clip.wav           # raw, clean and the per-word tags
python fleet_clean.py --text "so I I think, you know, we should go"
python fleet_clean.py --punct full clip.wav    # also re-punctuate words away from deletions

Audio is 16 kHz WAV (ffmpeg -i in.m4a -ar 16000 -ac 1 clip.wav). Fleet-2 detects the language, or you can pass --language. The delete threshold is 0.5. It was calibrated on validation data so that fluent speech keeps its words, and --threshold changes it.

Examples

These are from the DisfluencySpeech test set (English):

  • Fleet-2: Yeah. Well, robots are darling. That would be fun if you could get them trained. Otherwise, they're pretty smelly. Fleet-Clean: Yeah, robots are darling. That would be fun if you could get them trained. Otherwise, they're pretty smelly.
  • Fleet-2: Well, McNew, New, and Lara, I guess it's leaving one of them is a leaving. Fleet-Clean: McNew, New, and Lara, I guess one of them is a leaving.
  • Fleet-2: Well, I don't want to. I don't want this to be a unilateral conversation, but yes, we do. Fleet-Clean: I don't want to. I don't want this to be a unilateral conversation, but yes, we do.

Evaluation

Real disfluent English speech. Fleet-2 transcribes the audio, and the output is compared with DisfluencySpeech's own cleaned transcript:

DisfluencySpeech test (250 clips) Fleet-2 alone Fleet-Clean
WER against the cleaned transcript (transcript_c), punctuation ignored 18.8% 14.6%
Punctuation F1 against transcript_c 0.6700 0.6695

Fluent speech (FLEURS test). These are the same 100 clips per language as Fleet-2's card. A cleanup model should leave fluent speech alone:

Language Words deleted from fluent speech WER before WER after
English 0.47% 14.5% 14.3%
German 0.05% 22.0% 22.0%
French 0.04% 24.9% 25.0%
Spanish 0.05% 11.9% 11.9%
Italian 0.06% 16.7% 16.7%
Dutch 0.19% 30.1% 30.0%
Polish 0.29% 41.6% 41.6%
Korean 0.27% 42.8% (CER) 42.9% (CER)
Synthetic disfluencies (held-out tag test rows)

Fillers, discourse markers, repeats, restarts and self-corrections were inserted into Fleet-2 transcripts of held-out VoxPopuli / Zeroth / FLEURS clips. The table shows whether the tagger finds exactly the inserted words. "Weak" means F1 below 0.80.

Language Delete F1 Precision Recall
English 0.981 0.967 0.996
German 0.984 0.968 1.000
French 0.970 0.946 0.995
Spanish 0.971 0.978 0.964
Italian 0.969 0.959 0.979
Dutch 0.952 0.920 0.986
Polish 0.990 0.980 1.000
Korean 0.979 0.975 0.984

Limits:

  • Real disfluent speech was only tested in English. The other seven languages were trained and tested on synthetic disfluencies, so their numbers show how well the tagger removes those inserted patterns, not how it handles real conversation.
  • Fleet-2 already leaves out most "uh/um". It learned from Whisper transcripts. What is left to clean is mostly "you know", "I mean", "like", repeats and restarts.
  • Recognition errors stay. The tagger can only delete or re-punctuate words, so it can't fix a misheard word.
  • Clips are up to 30 s per call. The tagger reads up to 512 tokens and leaves anything past that unchanged.

How it was built

Data, training and gates
  • Transcripts: Fleet-2 (whistle.cact, native Cactus engine, automatic language detection) transcribed:
    • DisfluencySpeech train/validation/test
    • VoxPopuli validation (7 languages, a split Fleet-2 never trained on)
    • Zeroth-Korean train minus Fleet-2's own training rows
    • FLEURS Korean train minus Fleet-2's rows
  • Labels:
    • English deletions: each Fleet-2 word is aligned to DisfluencySpeech's verbatim transcript, and that transcript's words are mapped to its cleaned version.
    • Punctuation and case: taken from the human reference wherever Fleet-2's word matches it. Zeroth has no punctuation, so its rows only teach deletion.
    • Synthetic rows: disfluencies are only ever inserted, so the clean text is always recoverable by deletion.
  • Training:
    • Laya-multilingual's encoder, with its 256k-token embedding table frozen, plus a new 768→10 head.
    • PyTorch on an Apple M4 (MPS), fp32.
    • Best validation checkpoint kept, then the delete threshold calibrated on validation only.
  • Release gates:
    • G1: cleaning lowers WER on real disfluent English, and punctuation F1 doesn't drop. WER dropped from 18.76% to 14.61%, but punctuation F1 went from 0.6700 to 0.6695 (−0.0005), so this part of the gate did not pass. It was published anyway by the owner's decision, and the number is reported as measured.
    • G2: at most 1.0% of words deleted from fluent FLEURS speech in every language. Passed in all 8.
  • Punctuation mode: by default, Fleet-2's own punctuation is kept, and the tagger's punctuation and case are used only next to removed words. This was chosen on DisfluencySpeech validation, where re-punctuating everything scored lower against the human clean transcripts. --punct full re-punctuates every word; on fluent validation speech in all 8 languages, full mode was slightly more accurate than Fleet-2's own marks.

License and attribution

Apache-2.0. See NOTICE for every component:

  • Fleet-2 / Cactus Whistle (Apache-2.0)
  • Laya-multilingual (Apache-2.0), which builds on mmBERT (MIT)
  • DisfluencySpeech (Apache-2.0)
  • VoxPopuli (CC0)
  • Zeroth-Korean and FLEURS (CC BY 4.0)

No audio is included.