Hebrish/Whisper-Hebrish-Small-ACFT

Whisper Hebrish Small ACFT - Whisper Fine Tuned For Code-Switching English With Hebrew (for FUTO Voice Input)

The phone-sized version of Whisper Hebrish. It is a Whisper model fine-tuned for English speech that contains Hebrew words, built to run inside FUTO Keyboard (and FUTO Voice Input) on Android.

• Output: English, with Hebrew words in a consistent Latin romanization, for example "Reminder to call Moran tomorrow about the tza'atzuim". It never writes Hebrew script, because the base is the English-only whisper-small.en, which has no Hebrew tokens to drift into.
• Speaker: fine-tuned on one speaker (Daniel Rosehill). It is a personal model and may help other speakers less.
• Format: ggml-whisper-hebrish-small-en-acft-q8_0.bin is a whisper.cpp GGML Q8_0 file (264 MB). That is the same format, size and header as FUTO's own "English-244" model (small_en_acft_q8_0.bin). The repository root also has the same weights in Transformers format (model.safetensors, fp32).
• ACFT: FUTO runs Whisper with a dynamic audio context. The encoder only processes as much of the 30-second window as the clip is long, which makes short dictations fast. Stock and normally fine-tuned Whisper models break under this setting: they repeat themselves or never stop (see the table). FUTO's ACFT procedure fixes this, and it was applied here after the fine-tune.

Website: hebrish.app: get the Hebrish keyboard for Android and the desktop apps, see the results, and find every model, dataset and demo in one place. Android updates through F-Droid: add the Hebrish F-Droid repository (https://fdroid.hebrish.app/repo; steps).

Install on your phone (FUTO Keyboard)

1. On the phone, download ggml-whisper-hebrish-small-en-acft-q8_0.bin (264 MB).
2. Import it in either of these ways:
   • tap the downloaded file and choose FUTO Keyboard under "Open with", or
   • open FUTO Keyboard Settings → Languages & Models → Import resource file and pick the .bin.
3. The importer shows "Import - Voice Input Model" and asks you to choose a language. Pick English (the language you dictate in). The model replaces English's voice input model. The app does not ask whether the model is English-only or multilingual; whisper.cpp works that out from the file.
4. Use voice input as usual. The language's entry under Languages & Models now shows the voice model as (Imported).
5. To undo, open the same entry, tap the voice input model, and choose Revert to Default.

Optional, not tested with this model: Personal Dictionary for Voice (in voice input settings) sends your personal dictionary words to Whisper as a (Glossary: ...) prompt. Adding the Hebrew words you use most may help further.

These steps come from FUTO Keyboard's source (ImportResourceActivity.kt, commit 70a5d390, 2026-09-23), not from a screen recording. The importer identifies the file as a voice model from its first four bytes (whisper.cpp's ggml magic) and then asks for a language. The file also went through FUTO's own bundled whisper.cpp, built on a desktop, with the keyboard's exact inference settings. That check produced the "FUTO runtime" rows below.

Results

The test sets are the same as Whisper Hebrish v2's, so the numbers are directly comparable:

• test_cs: 155 English sentences with Hebrew words. Its Hebrew terms are disjoint from training and it was never trained on.
• held-term recall: the share of those test-only Hebrew terms that come out in the canonical spelling.
• own EN: 33 of the speaker's plain English sentences.
• Libri: 200 LibriSpeech test-clean clips.

WER is normalised (case and punctuation removed); lower is better. "FUTO ctx" is FUTO's runtime context, max(160, min(1500, ceil(samples/320) + 32)) encoder positions. "Full" is the normal 30-second window.

Model	Context / decoding	test_cs WER	held-term recall	own EN WER	Libri WER
openai/whisper-small.en (stock)	full, greedy	0.117	0.26	0.044	0.024
openai/whisper-small.en (stock)	FUTO ctx, greedy	0.941	0.23	0.978	0.241
FUTO acft-whisper-small.en ("English-244", FUTO's default top English model)	FUTO ctx, greedy	0.118	0.24	0.046	0.025
FUTO small_en_acft_q8_0.bin	FUTO runtime, q8_0, beam 5	0.118	0.26	0.041	–
Whisper Hebrish small.en, before ACFT	full, greedy	0.062	0.52	0.032	0.023
Whisper Hebrish small.en, before ACFT	FUTO ctx, greedy	1.228	0.52	0.666	0.251
Whisper Hebrish Small ACFT	full, greedy	0.063	0.51	0.036	0.027
Whisper Hebrish Small ACFT	FUTO ctx, greedy	0.062	0.50	0.032	0.023
Whisper Hebrish Small ACFT	FUTO ctx, beam 5	0.056	0.52	0.029	0.021
Whisper Hebrish Small ACFT (this .bin)	FUTO runtime, q8_0, beam 5	0.058	0.50	0.028	–
Whisper Hebrish v2 (large-v3-turbo, 809 M)	full, greedy	0.052	0.48	0.029	0.022

On the phone, this model halves FUTO's stock English-244 error rate on code-switched speech (0.118 → 0.058). It gets about twice as many unseen Hebrew terms right (0.26 → 0.50), and plain English improves too (0.041 → 0.028). It comes within about 0.006 WER of the large v2 model at a quarter of the parameters. ACFT costs nothing at full context, and under FUTO's context it brings the fine-tune from 1.23 WER (unusable) to 0.062.

Known weakness: dictations longer than 30 seconds. The test_long set (stitched 30 to 60 s clips, run through the Transformers chunked pipeline) scores 0.21 to 0.27 WER, against 0.09 for v2. The small model handles a 30-second boundary worse. Typical keyboard dictation is shorter than that.

Per-utterance hypotheses for every row are in evals/. The scoring code is in the training repository.

Training

1. Fine-tune. Full fine-tune of openai/whisper-small.en on 1,929 clips (215 minutes) of the speaker's read speech:
   • 70% code-switching sentences, 20% his own plain English, and 10% public English replay to avoid forgetting.
   • The target text is cased and punctuated, with Hebrew in canonical romanization.
   • 10 epochs, lr 1.25e-5 (linear, 10% warmup), effective batch 32, bf16 autocast.
   • The best checkpoint was chosen on validation WER (epoch 8, 0.033).
   • The data, mix and loop are identical to Whisper Hebrish v2; only the base and the learning rate differ.
2. ACFT. FUTO's procedure (futo-org/whisper-acft finetune.ipynb at commit f0700d0), with FUTO's settings unchanged:
   • Adam, lr 1e-6, batch 1, 8 epochs.
   • Encoder context round(50 × seconds) ± jitter.
   • MSE between the decoder hidden states at reduced context and those of a frozen copy at full context, teacher-forced on the reference text.
   • It ran over the same 1,929 training clips, with the fine-tuned model as the reference.
3. Conversion. whisper.cpp v1.9.5 (d1be6fd): models/convert-h5-to-ggml.py, then whisper-quantize … q8_0. On 12 test clips, whisper.cpp greedy output matched PyTorch after normalisation 12 of 12 times.
   • sha256 40dfdc7da4a57c1d27bd394e192fd1ddb142cb0b66d32e2b7956086e54fe09a5.

training/ holds the run configs, the ACFT log (acft-result.json) and the conversion check (ggml-verify.json).

Use outside FUTO

# whisper.cpp, with the same dynamic context FUTO uses (here for a 6 s clip: 6*50+32 = 332)
whisper-cli -m ggml-whisper-hebrish-small-en-acft-q8_0.bin -f clip.wav -bs 5 -ac 332

from transformers import pipeline
asr = pipeline("automatic-speech-recognition", model="Hebrish/Whisper-Hebrish-Small-ACFT")
print(asr("clip.wav")["text"])   # English-only model: do not pass language/task

License

MIT. The base model openai/whisper-small.en is MIT. The ACFT method is from FUTO's whisper-acft repository (MIT).