File size: 4,058 Bytes
e1da05f 4f55449 4c48135 e1da05f 4c48135 e1da05f 4f55449 e1da05f 4f55449 e1da05f 24233ab 4f55449 24233ab 4f55449 4c48135 4f55449 4c48135 e1da05f a266927 4f55449 e1da05f 4f55449 4c48135 4f55449 a266927 4f55449 4c48135 4f55449 e1da05f 4f55449 4c48135 4f55449 4c48135 4f55449 4c48135 4f55449 4c48135 a266927 4f55449 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 | ---
pipeline_tag: automatic-speech-recognition
license: apache-2.0
base_model: openai/whisper-tiny
library_name: zeromodels
tags:
- keras
- zeromodels
- whisper
- automatic-speech-recognition
- audio
- multilingual
- arxiv:2212.04356
- pytorch
- jax
- tf
---
## ***See [our collection](https://huggingface.co/collections/zeromodels/whisper-6a8eaf34918224be11047f86) for all versions of Whisper.***
# Run Whisper with Keras 3: JAX, PyTorch, or TensorFlow
[](https://github.com/IMvision12/ZeroModels) [](https://imvision12.github.io/ZeroModels/whisper/) [](https://huggingface.co/collections/zeromodels/whisper-6a8eaf34918224be11047f86)
# zeromodels/whisper_tiny
Paper: [Robust Speech Recognition via Large-Scale Weak Supervision (arXiv:2212.04356)](https://arxiv.org/abs/2212.04356) · [HF Papers](https://huggingface.co/papers/2212.04356)
Whisper is a **multilingual** encoder-decoder ASR model trained on large-scale weak supervision. Use `task="transcribe"` to keep the source language or `task="translate"` to render English. Pass `language=None` to let the model detect the spoken language. Output is cased and punctuated.
For more details on the model, please go to the upstream [model card](https://huggingface.co/openai/whisper-tiny).
Pure-**Keras 3** conversion of [`openai/whisper-tiny`](https://huggingface.co/openai/whisper-tiny) for [zeromodels](https://github.com/IMvision12/ZeroModels). One implementation runs unmodified on **TensorFlow / Torch / JAX**.
This is an **ASR** checkpoint (`WhisperConditionalGenerate`, 39M).
## ✨ Quick start
```python
import os
os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow"
import soundfile as sf
from zeromodels.models.whisper import (
WhisperProcessor,
WhisperConditionalGenerate,
)
model = WhisperConditionalGenerate.from_weights("zeromodels/whisper_tiny")
processor = WhisperProcessor.from_weights("zeromodels/whisper_tiny")
audio, sr = sf.read("your_audio.wav", dtype="float32") # 16 kHz mono
# task="transcribe" keeps the source language; "translate" -> English.
text = model.generate(audio, processor, language="en", task="transcribe")
print(repr(text[0]))
```
Load any Whisper variant the same way with `from_weights("zeromodels/<variant>")`:
| Variant | Hub | Notes |
|---|---|---|
| `whisper_tiny` | [`zeromodels/whisper_tiny`](https://huggingface.co/zeromodels/whisper_tiny) | 39M |
| `whisper_base` | [`zeromodels/whisper_base`](https://huggingface.co/zeromodels/whisper_base) | 74M |
| `whisper_small` | [`zeromodels/whisper_small`](https://huggingface.co/zeromodels/whisper_small) | 244M |
| `whisper_medium` | [`zeromodels/whisper_medium`](https://huggingface.co/zeromodels/whisper_medium) | 769M |
| `whisper_large` | [`zeromodels/whisper_large`](https://huggingface.co/zeromodels/whisper_large) | 1.55B |
| `whisper_large_v2` | [`zeromodels/whisper_large_v2`](https://huggingface.co/zeromodels/whisper_large_v2) | 1.55B |
| `whisper_large_v3` | [`zeromodels/whisper_large_v3`](https://huggingface.co/zeromodels/whisper_large_v3) | 128 mel bins |
| `whisper_large_v3_turbo` | [`zeromodels/whisper_large_v3_turbo`](https://huggingface.co/zeromodels/whisper_large_v3_turbo) | 4 decoder layers |
## Tips
- Set `KERAS_BACKEND` **before** importing Keras / zeromodels.
- Prefer `WhisperProcessor.from_weights(...)` so mel bins match the variant (v3 uses 128).
- Clips are padded to a 30 s window; chunk longer audio yourself.
- See [Whisper docs](https://imvision12.github.io/ZeroModels/whisper/) and [Loading Weights](https://imvision12.github.io/ZeroModels/loading_weights/).
- Community / upstream safetensors still work via the `hf:` prefix, e.g. `WhisperConditionalGenerate.from_weights("hf:openai/whisper-tiny")`.
## Special Thanks
A huge thank you to the OpenAI Whisper authors for creating and releasing these models.
License: Apache 2.0.
|