| --- |
| pipeline_tag: automatic-speech-recognition |
| license: apache-2.0 |
| base_model: openai/whisper-tiny |
| library_name: zeromodels |
| tags: |
| - keras |
| - zeromodels |
| - whisper |
| - automatic-speech-recognition |
| - audio |
| - multilingual |
| - arxiv:2212.04356 |
| - pytorch |
| - jax |
| - tf |
| --- |
| |
| ## ***See [our collection](https://huggingface.co/collections/zeromodels/whisper-6a8eaf34918224be11047f86) for all versions of Whisper.*** |
|
|
| # Run Whisper with Keras 3: JAX, PyTorch, or TensorFlow |
|
|
| [](https://github.com/IMvision12/ZeroModels) [](https://imvision12.github.io/ZeroModels/whisper/) [](https://huggingface.co/collections/zeromodels/whisper-6a8eaf34918224be11047f86) |
|
|
| # zeromodels/whisper_tiny |
| |
| Paper: [Robust Speech Recognition via Large-Scale Weak Supervision (arXiv:2212.04356)](https://arxiv.org/abs/2212.04356) · [HF Papers](https://huggingface.co/papers/2212.04356) |
| |
| Whisper is a **multilingual** encoder-decoder ASR model trained on large-scale weak supervision. Use `task="transcribe"` to keep the source language or `task="translate"` to render English. Pass `language=None` to let the model detect the spoken language. Output is cased and punctuated. |
| |
| For more details on the model, please go to the upstream [model card](https://huggingface.co/openai/whisper-tiny). |
| |
| Pure-**Keras 3** conversion of [`openai/whisper-tiny`](https://huggingface.co/openai/whisper-tiny) for [zeromodels](https://github.com/IMvision12/ZeroModels). One implementation runs unmodified on **TensorFlow / Torch / JAX**. |
| |
| This is an **ASR** checkpoint (`WhisperConditionalGenerate`, 39M). |
| |
| ## ✨ Quick start |
| |
| ```python |
| import os |
| os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow" |
|
|
| import soundfile as sf |
| from zeromodels.models.whisper import ( |
| WhisperProcessor, |
| WhisperConditionalGenerate, |
| ) |
| |
| model = WhisperConditionalGenerate.from_weights("zeromodels/whisper_tiny") |
| processor = WhisperProcessor.from_weights("zeromodels/whisper_tiny") |
|
|
| audio, sr = sf.read("your_audio.wav", dtype="float32") # 16 kHz mono |
| # task="transcribe" keeps the source language; "translate" -> English. |
| text = model.generate(audio, processor, language="en", task="transcribe") |
| print(repr(text[0])) |
| ``` |
| |
| Load any Whisper variant the same way with `from_weights("zeromodels/<variant>")`: |
|
|
| | Variant | Hub | Notes | |
| |---|---|---| |
| | `whisper_tiny` | [`zeromodels/whisper_tiny`](https://huggingface.co/zeromodels/whisper_tiny) | 39M | |
| | `whisper_base` | [`zeromodels/whisper_base`](https://huggingface.co/zeromodels/whisper_base) | 74M | |
| | `whisper_small` | [`zeromodels/whisper_small`](https://huggingface.co/zeromodels/whisper_small) | 244M | |
| | `whisper_medium` | [`zeromodels/whisper_medium`](https://huggingface.co/zeromodels/whisper_medium) | 769M | |
| | `whisper_large` | [`zeromodels/whisper_large`](https://huggingface.co/zeromodels/whisper_large) | 1.55B | |
| | `whisper_large_v2` | [`zeromodels/whisper_large_v2`](https://huggingface.co/zeromodels/whisper_large_v2) | 1.55B | |
| | `whisper_large_v3` | [`zeromodels/whisper_large_v3`](https://huggingface.co/zeromodels/whisper_large_v3) | 128 mel bins | |
| | `whisper_large_v3_turbo` | [`zeromodels/whisper_large_v3_turbo`](https://huggingface.co/zeromodels/whisper_large_v3_turbo) | 4 decoder layers | |
|
|
| ## Tips |
|
|
| - Set `KERAS_BACKEND` **before** importing Keras / zeromodels. |
| - Prefer `WhisperProcessor.from_weights(...)` so mel bins match the variant (v3 uses 128). |
| - Clips are padded to a 30 s window; chunk longer audio yourself. |
| - See [Whisper docs](https://imvision12.github.io/ZeroModels/whisper/) and [Loading Weights](https://imvision12.github.io/ZeroModels/loading_weights/). |
| - Community / upstream safetensors still work via the `hf:` prefix, e.g. `WhisperConditionalGenerate.from_weights("hf:openai/whisper-tiny")`. |
|
|
| ## Special Thanks |
|
|
| A huge thank you to the OpenAI Whisper authors for creating and releasing these models. |
|
|
| License: Apache 2.0. |
|
|
