See our collection for all versions of Whisper.

Run Whisper with Keras 3: JAX, PyTorch, or TensorFlow

kerasformers/whisper_small

Paper: Robust Speech Recognition via Large-Scale Weak Supervision (arXiv:2212.04356) · HF Papers

Whisper is a multilingual encoder-decoder ASR model trained on large-scale weak supervision. Use task="transcribe" to keep the source language or task="translate" to render English. Pass language=None to let the model detect the spoken language. Output is cased and punctuated.

For more details on the model, please go to the upstream model card.

Pure-Keras 3 conversion of openai/whisper-small for kerasformers. One implementation runs unmodified on TensorFlow / Torch / JAX.

This is an ASR checkpoint (WhisperSpeechToText, 244M).

✨ Quick start

import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

import soundfile as sf
from kerasformers.models.whisper import (
    WhisperProcessor,
    WhisperSpeechToText,
)

model = WhisperSpeechToText.from_weights("kerasformers/whisper_small")
processor = WhisperProcessor.from_weights("kerasformers/whisper_small")

audio, sr = sf.read("your_audio.wav", dtype="float32")  # 16 kHz mono
# task="transcribe" keeps the source language; "translate" -> English.
text = model.generate(audio, processor, language="en", task="transcribe")
print(repr(text[0]))

Load any Whisper variant the same way with from_weights("kerasformers/<variant>"):

Variant	Hub	Notes
`whisper_tiny`	`kerasformers/whisper_tiny`	39M
`whisper_base`	`kerasformers/whisper_base`	74M
`whisper_small`	`kerasformers/whisper_small`	244M
`whisper_medium`	`kerasformers/whisper_medium`	769M
`whisper_large`	`kerasformers/whisper_large`	1.55B
`whisper_large_v2`	`kerasformers/whisper_large_v2`	1.55B
`whisper_large_v3`	`kerasformers/whisper_large_v3`	128 mel bins
`whisper_large_v3_turbo`	`kerasformers/whisper_large_v3_turbo`	4 decoder layers

Tips

Set KERAS_BACKEND before importing Keras / kerasformers.
Prefer WhisperProcessor.from_weights(...) so mel bins match the variant (v3 uses 128).
Clips are padded to a 30 s window; chunk longer audio yourself.
See Whisper docs and Loading Weights.
Community / upstream safetensors still work via the hf: prefix, e.g. WhisperSpeechToText.from_weights("hf:openai/whisper-small").