Add ACFT short-window variants (5s/10s/30s, int8 drq) converted from futo-org/acft-whisper-tiny
This PR adds ACFT short-window variants of whisper-tiny: fixed 5 s / 10 s / 30 s window TFLite graphs converted from the audio-context fine-tuned checkpoint futo-org/acft-whisper-tiny (Apache-2.0), using the same encode/decode-split graph interface as the existing exports in this repo.
Why short windows
Whisper always encodes a 30 s window; for sub-5 s voice commands most encoder work is spent on padding. The encoder conv stack is length-agnostic and the encoder positional table is a fixed sinusoid, so slicing the first N rows of embed_positions yields a shorter window — exactly what whisper.cpp's audio_ctx parameter does at runtime. Stock Whisper decoders destabilize when the window shrinks (repetition loops: stock base at a 5 s window collapsed to CER 10.4 on a sentence clip in our evaluation); the futo-org ACFT (audio-context fine-tuning) checkpoints are fine-tuned via self-distillation (L2 on hidden states) to tolerate exactly this. Method and training: futo-org/whisper-acft — full credit to FUTO for the checkpoints; this PR only converts them (no retraining).
Files
Conversion: transformers TFWhisperForConditionalGeneration → two-signature graph (same pipeline as the 30 s exports in this repo), converter-time dynamic-range quantization (int8 weights, fp32 activations, tf.lite.Optimize.DEFAULT).
| File | Window | Encode input |
|---|---|---|
acft_whisper_tiny_5s_drq.tflite |
5 s | [1, 80, 500] |
acft_whisper_tiny_10s_drq.tflite |
10 s | [1, 80, 1000] |
acft_whisper_tiny_30s_drq.tflite |
30 s | [1, 80, 3000] |
Tokenizer/vocab unchanged (51865) — the stock whisper-tiny tokenizer.json works as-is.
REQUIRED integration note
These are fixed short-window graphs. The encode signature input is [1, 80, frames] with frames = 500 / 1000 / 3000. The runtime must size the mel window from the encode signature's input shape (compute/pad the log-mel to exactly that frame count) instead of assuming 3000 frames. The decode signature input order of these exports is (mask, audio, tokens) — opposite of the stock export in this repo — so bind decode tensors by shape/name rather than position. Reference implementation: the JNI bridge in LiteRT-LM-Unity, which auto-detects 500/1000/3000-frame windows from the signature on one code path.
Validation
Desktop probe of these files (LiteRT interpreter, XNNPACK CPU, greedy decode; CER vs punctuation-normalized references, whitespace removed):
| Window | Clip | Transcript | CER | Encode s | Decode s (steps) |
|---|---|---|---|---|---|
| 5s | 2025년 3월 5일 전술평가 결과 보고 (3.98 s, ko) | 2015년 3월호일 전술 평가 결과 보고 | 0.118 | 0.015 | 0.36 (12) |
| 5s | The current weather in Seoul is cloudy (2.9 s, en) | The current weather in Seoul is cloudy. | 0.000 | 0.013 | 0.26 (8) |
| 5s | 소리 키워줘 (1.32 s, ko command) | 소리 키워줘 | 0.000 | 0.014 | 0.14 (4) |
| 5s | 음량 증가 (1.15 s, ko command) | 능량 증가 | 0.250 | 0.015 | 0.23 (6) |
| 10s | 2025년 3월 5일 전술평가 결과 보고 | 2015년 3월호일 전술 평가 결과보고 | 0.118 | 0.046 | 0.43 (13) |
| 10s | The current weather in Seoul is cloudy | The current weather and soil is cloudy. | 0.125 | 0.034 | 0.28 (8) |
| 10s | 소리 키워줘 | 소리 키워줘 | 0.000 | 0.040 | 0.15 (4) |
| 10s | 음량 증가 | 능냥 증가 | 0.500 | 0.040 | 0.24 (6) |
| 30s | 2025년 3월 5일 전술평가 결과 보고 | 2015년 3월호일 전술 평가 결과보고 | 0.118 | 0.214 | 0.67 (13) |
| 30s | The current weather in Seoul is cloudy | The current weather and soul is cloudy. | 0.094 | 0.210 | 0.45 (8) |
| 30s | 소리 키워줘 | 소리 키워줘 | 0.000 | 0.210 | 0.25 (4) |
| 30s | 음량 증가 | 는 양증가 | 0.500 | 0.206 | 0.31 (5) |
The short windows track the checkpoint's own 30 s-window behavior (same clips right/wrong at every window; the residual errors are tiny's model-capacity errors, present at 30 s too) at ~14x encoder speedup for the 5 s window (0.013–0.015 s vs 0.206–0.214 s per encode). In the same evaluation on whisper-base, the ACFT 5 s f32 graph reproduced the stock 30 s transcripts clip-for-clip at ~8x encoder speedup, ~20x with drq weights; short-window graphs are also device-validated on Android arm64 (Snapdragon 865, 0.5–0.8 s end-to-end per command clip for tiny/base 5 s).
Caveats
- The 5 s/10 s graphs hard-truncate longer audio — pick the window that matches your audio length; keep a 30 s graph for dictation.
- tiny retains its stock model-capacity limits on short noisy/quiet clips; base is the recommended minimum for non-English voice commands.
License: Apache-2.0 (this repo's license; the source ACFT checkpoint is also Apache-2.0).
Converted by the LiteRT-LM-Unity project (https://github.com/Leuconoe/LiteRT-LM-Unity).
Withdrawing this PR — we consolidated all six ACFT short-window conversions (tiny/base/small × multilingual/.en, 5s/10s/30s each) into a single dedicated repo instead of adding files piecemeal to the per-model repos: litert-community/whisper-acft.
The same files (identical content) plus the full integration notes and per-model validation tables live there. Sorry for the noise, and thanks for hosting the stock exports these graphs interface-match!