tarekziade's picture
tarekziade HF Staff
MIC WhisperSpeech-integration (files_sha256 52dbb64a41ed)
77fb2ff verified
|
Raw History Blame Contribute Delete
3.98 kB
# Whisper: assisted speech recognition with Transformers
This checkpoint-only contract compares greedy transcription with an encoder-decoder assistant on frozen speech. It complements the tone-based [Whisper contract](https://huggingface.co/hf-internal-testing/Whisper-integration). Neither slim weight kind exposes the historical beginning-token suppression bug on this sample, so fixture-only runs declare this path unsupported.
The sample is the original LibriSpeech FLAC `1272-128104-0000`, copied from a pinned dataset revision. Its [provenance](../../audio/1272-128104-0000.json) records the source, attribution, CC-BY-4.0 license, and checksum. Run the blocks from this page's directory. The contract checks the audio bytes before inference; no dataset download or random input generation is needed.
## Transcribe speech greedily
```py runnable:speech
import hashlib
import json
from pathlib import Path
import soundfile as sf
import torch
from transformers import WhisperForConditionalGeneration, WhisperProcessor, set_seed
import os # doc-builder: hide
model_id = "openai/whisper-tiny"
model_id = os.environ.get("MIC_MODEL", model_id) # doc-builder: hide
variant = json.loads(os.environ.get("MIC_MODEL_KWARGS", "{}")) # doc-builder: hide
device = "cuda" if torch.cuda.is_available() else "cpu"
set_seed(0)
audio_dir = Path("../../audio")
provenance = json.loads((audio_dir / "1272-128104-0000.json").read_text())
speech_path = audio_dir / "1272-128104-0000.flac"
speech_sha256 = hashlib.sha256(speech_path.read_bytes()).hexdigest()
assert speech_sha256 == provenance["sha256"], "speech sample differs from its pinned checksum"
speech, sampling_rate = sf.read(speech_path, dtype="float32")
processor = WhisperProcessor.from_pretrained(model_id)
model = WhisperForConditionalGeneration.from_pretrained(model_id).to(device).eval()
if variant: # doc-builder: hide
model = WhisperForConditionalGeneration.from_pretrained(model_id, **variant).to(device).eval()
inputs = processor(speech, sampling_rate=sampling_rate, return_tensors="pt").to(device, model.dtype)
generation = {"language": "en", "task": "transcribe", "max_new_tokens": 64, "do_sample": False}
with torch.no_grad():
greedy = model.generate(inputs.input_features, **generation)
greedy_text = processor.batch_decode(greedy, skip_special_tokens=True)
print(greedy_text)
```
## Use an encoder-decoder assistant
An assistant drafts tokens that the main model verifies. Here a separate copy of the same tiny checkpoint supplies its own encoder; this checks the encoder-decoder assistant path, rather than measuring a speedup. Its transcript must equal greedy decoding.
```py runnable:speech:2
assistant = WhisperForConditionalGeneration.from_pretrained(model_id).to(device).eval()
if variant: # doc-builder: hide
assistant = WhisperForConditionalGeneration.from_pretrained(model_id, **variant).to(device).eval()
with torch.no_grad():
assisted = model.generate(inputs.input_features, **generation, assistant_model=assistant)
assisted_text = processor.batch_decode(assisted, skip_special_tokens=True)
print(assisted_text)
if os.environ.get("MIC_RUNNER"): # doc-builder: hide
result = {
"runtime": {
"device": model.device.type,
"dtype": str(model.dtype).removeprefix("torch."),
"attn_implementation": model.config._attn_implementation,
},
"speech": {"id": provenance["id"], "sha256": speech_sha256, "sampling_rate": sampling_rate},
"greedy_tokens": greedy.tolist(),
"greedy_text": greedy_text,
"assisted_tokens": assisted.tolist(),
"assisted_text": assisted_text,
"assisted_matches_greedy": torch.equal(greedy, assisted),
}
print(json.dumps(result))
```
This inference-only probe complements the original contract's training checks. Results apply to the pinned checkpoint, speech sample, framework build, and runtime; agreement is not a speech-recognition accuracy measurement.