README.md

---
license: mit
language: en
library_name: transformers
pipeline_tag: automatic-speech-recognition
base_model: openai/whisper-small
tags:
- asr
- speech-recognition
- whisper
- audio
---

# talk

A `whisper-small` encoder-decoder, fine-tuned on short passages of live microphone audio, for English speech recognition at 16 kHz mono. The weights are float32 and occupy roughly 967 MB. The fine-tuning was slight: across 37 sampled encoder and decoder tensors the relative L2 distance from `openai/whisper-small` has a median of 0.0013 and a maximum of 0.024, with none exceeding 5%. That is a light touch consistent with a small amount of in-domain audio, and it is why this is best treated as `whisper-small` with a nudge rather than as a model of its own. The audio it was tuned on is not published, so the corpus itself remains a matter of record rather than something you can check here.

Two corrections to the previous version of this card, both of which pointed at the wrong model. The metadata claimed a Wav2Vec2 lineage, naming `facebook/wav2vec2-base-960h` as a base model and `facebook/multilingual_librispeech` as the training set. Neither is true of the published checkpoint: `config.json` records `_name_or_path: openai/whisper-small` with `WhisperForConditionalGeneration`, and the documented training signal was live recordings, not a fixed corpus. The earlier card also described `main.py`, `train.py`, `test_transcription.py` and a `requirements.txt`; none of those files have ever been in this repository, so the instructions it gave could not be followed.

The tokenizer and the feature extractor are now present in the repository, taken from [`openai/whisper-small`](https://huggingface.co/openai/whisper-small), which supplies the same 51865-token vocabulary and 80-bin mel filterbank the checkpoint expects, since that is what it was fine-tuned from. An earlier version of this repository shipped neither, and `AutoProcessor.from_pretrained` failed outright while `AutoTokenizer` returned a one-token vocabulary. Both are fixed; `WhisperForConditionalGeneration`, `AutoProcessor` and the `automatic-speech-recognition` pipeline now all load from this identifier alone. The repository's own `generation_config.json` was kept as fine-tuned rather than replaced with the upstream one. A duplicate `pytorch_model.bin` and a 761-byte `training_log.pt` left over from debugging were removed.

## Usage

```python
import torch
from transformers import WhisperForConditionalGeneration, AutoProcessor, pipeline

model = WhisperForConditionalGeneration.from_pretrained("harpertoken/talk")
processor = AutoProcessor.from_pretrained("harpertoken/talk")

transcriber = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    torch_dtype=torch.float32,
)

# Supply 16 kHz mono float32 samples as a numpy array. The pipeline rejects a
# placeholder such as (0, 16000): it raises TypeError on a tuple. Load real audio
# with soundfile or librosa, for example:
#     audio, sample_rate = sf.read("clip.wav", dtype="float32")
#     assert sample_rate == 16000
audio = load_audio()          # your loader, returning np.ndarray of shape (n,)
print(transcriber(audio, return_timestamps=True))
```

## Limitations

The fine-tuning corpus is a small quantity of live microphone audio, from one room, one machine and likely one speaker, so this should be read as an adaptation of `whisper-small` rather than an independent model. I have no transcription accuracy measurement for it: the card this replaces listed a character error rate with no evaluation behind it, and that has been removed. Expect degradation on accents, noisy rooms, far-field audio and any language other than English. If you need a dependable transcriber, start from `openai/whisper-small`, whose published WER figures you can actually check.

## Attribution

Whisper follows Radford et al., *Robust Speech Recognition via Large-Scale Weak Supervision* (2022).