---
license: mit
library_name: transformers
tags:
  - prosody
  - speech-recognition
  - custom_code
pipeline_tag: automatic-speech-recognition
---

# Prosody2text

Two signals — **interpolated f0 and intensity at 200 Hz** — into a mel
spectrogram, and the mel into words. No phonetic content at any point: the
model sees only how the pitch moved and how loud it was.

That is the claim this repo exists to let you check.

Companion to the paper. The interactive version, where you *draw* a contour
instead of speaking one, is at
[Daporte/Prosody2text](https://huggingface.co/spaces/Daporte/Prosody2text).

## Use it

```python
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "Daporte/prosody2text-weights", trust_remote_code=True
)

result = model.transcribe("utterance.wav")
print(result.text)              # the chosen hypothesis
print(result.candidates)        # the whole beam -- ten of them
print(result.mel.shape)         # the predicted mel
print(result.features.f0_hz)    # what the conformer was actually given
print(result.fallback)          # True if this is the no-speech boilerplate
```

`trust_remote_code=True` is needed because the architecture is not one of
Transformers' own: a conformer and a fine-tuned Whisper held alongside each
other, each loaded from its own checkpoint. The code it runs is the modeling
files in this repo.

Already have the contours? Hand them over directly — physical units, Hz and
dB on the 200 Hz grid:

```python
result = model(f0_hz=f0, intensity_db=intensity_db)
```

The normalisation to the model's own scale happens inside, so it cannot be
applied twice — which produces a mel that looks plausible and is wrong.

`AutoModel.from_pretrained(..., whisper=False)` loads the conformer alone: a
45 MB download rather than 3.1 GB, and enough to look at the predicted mel.

## Evaluate on a dataset

`evaluate_dataset.py` runs the pipeline over any HuggingFace speech dataset
and scores it against the reference transcripts. A microphone tells you the
pipeline runs; a dataset tells you whether it is *right*.

```sh
pip install transformers torch numpy scipy safetensors \
            amfm-decompy==1.0.12.2 praat-parselmouth \
            datasets jiwer soundfile

hf download Daporte/prosody2text-weights evaluate_dataset.py --local-dir .

python evaluate_dataset.py \
    --dataset espnet/yodas-granary \
    --data_files data/en001/asr_only/00000000.parquet \
    --split train --count 3000 --fp16 \
    --save ours_en001.jsonl
```

With `uv` instead of `pip`, in an empty directory:

```sh
uv init --bare                 # uv add needs a pyproject.toml to add to
uv add transformers torch numpy scipy safetensors \
       amfm-decompy==1.0.12.2 praat-parselmouth \
       datasets jiwer soundfile
uv run hf download Daporte/prosody2text-weights evaluate_dataset.py \
    --local-dir .
uv run python evaluate_dataset.py --dataset espnet/yodas-granary \
    --data_files data/en001/asr_only/00000000.parquet \
    --split train --count 3000 --fp16 --save ours_en001.jsonl
```

`uv run python`, not `python`: the bare command is the system interpreter,
which does not have any of the above installed.

Streaming by default, so this does not download the whole split. Each line
of the output is one utterance:

```json
{"prediction": "...", "candidates": ["...", "..."], "reference": "...", "id": "0"}
```

Those field names match the published inference runs', so comparing against
them is a join on `id` rather than a translation.

Anything with an audio column and a transcript works. The transcript column
is found from `text`, `sentence`, `transcription`, `transcript` or
`normalized_text`; pass `--text_column` otherwise. `--help` lists the rest.

### On a GPU

Whisper at beam 10 is about **1 s per utterance on an RTX 3090** against
**~20 s on CPU**, so a few thousand utterances is the difference between an
hour and a day. `--fp16` halves the VRAM and is a further 1.8× faster; it
transcribed every template identically to fp32.

If it reports `whisper: cpu` on a machine that has a GPU, the usual cause is
that `libcuda.so.1` sits somewhere the dynamic loader does not search —
`nvidia-smi` works, but torch reports no driver at all. The model detects
this and prints the fix. On most distributions none of this arises.

### What the numbers mean

WER is reported **over the corpus**, not averaged per utterance: it is total
errors over total reference words, so averaging would weight a three-word
reference the same as a thirty-word one and give a different, wrong number.
The normalisation — punctuation removed, case folded — matches the published
runs, because case and punctuation alone would otherwise count as errors and
the figure would measure typography rather than the model.

Each line also prints the substitution/deletion/insertion split, which is
worth reading: on this pipeline a hypothesis is usually the right *length*
with the wrong words, and that is a different failure from dropping or
inventing half the utterance.

## What is here

| | | |
|---|---|---|
| `conformer/` | 45 MB | f0 + intensity → mel |
| `whisper/` | 3.1 GB | Whisper-medium fine-tuned on prosody-predicted mels |
| `tokenizer/` | 5 MB | the stock whisper-medium tokenizer, vendored |
| `config.json` | | the mel contract, and the `auto_map` |
| `modeling_prosody.py`, `configuration_prosody.py` | | the architecture |
| `features.py`, `speech.py`, `conformer.py`, `conformer_model.py`, `whisper_decoder.py` | | the pipeline |
| `evaluate_dataset.py` | | dataset evaluation |

Only inference files. A training checkpoint also carries optimizer state —
6 GB of Adam moments for this model — which nothing here reads.

## Notes

**The feature extraction is not interchangeable with sensible-looking
alternatives.** The yaapt settings, the half-frame zero-padding, the
six-frame trim and the two clamps all move the contour, and the conformer
only ever saw one combination of them. `speech.py` carries that combination
and is checked bit-identical against the reference implementation.

**The mel contract lives in `config.json`.** `mel_bins` and `length_ratio`
must match what the conformer was trained with; a wrong `length_ratio`
mis-times the mel against the audio without failing, which is the worst way
for it to be wrong.

**Recordings are trimmed; dataset clips usually need not be.** A live
microphone hands over the room before and after the words, and yaapt finds a
pitch in noise, so an untrimmed recording puts several hundred milliseconds
of nothing into the mel. The trim cuts anything more than 18 dB under the
loudest frame. `--no_trim`, or `trim=False`, turns it off — which is the
closer match to the published runs.

**The conformer is the trained module, not an export.** `generate()` is
non-autoregressive and sizes its output from the real input length, so a
phrase is predicted at its own duration. An earlier version of this repo
shipped an ONNX export instead, whose input width is baked in at 2000
frames — every phrase padded to ten seconds, with the decoder attending over
the padding. Measured against the training pipeline's own predictions that
is 0.9357 correlation against 0.9985 here, which is two orders of magnitude
more than the f0 extraction differs by. The architecture in
`conformer_model.py` is extracted verbatim from the training code and checked
bit-identical to it on CPU.

**This is not trainable as it stands.** It subclasses `PreTrainedModel` so
that `from_pretrained`, the `AutoModel` API and download metrics work, but
`forward` runs the two models in sequence under `no_grad` rather than
building one differentiable graph, and the weights load from two checkpoints
rather than one state dict. It exists to run the published pipeline.

## Licence

MIT for the code. The weights derive from
[openai/whisper-medium](https://huggingface.co/openai/whisper-medium) and
carry its terms.