---
license: apache-2.0
base_model: openai/whisper-large-v3
library_name: transformers
pipeline_tag: automatic-speech-recognition
language:
- en
tags:
- textpro-slm
- prosody
- speech
- audio
- original-checkpoint
---
# WhisperPro

**Speech → transcript + compact prosody.**

WhisperPro is the unified speech encoder behind TextPro-SLM. It extracts synchronized transcript tokens and continuous prosody features, preserving acoustic information that a transcript alone can miss.

[Paper](https://openreview.net/forum?id=Jsjmo8eX29) · [GitHub project](https://github.com/dreamtheater123/TextPro-SLM) · [Paired checkpoint](https://huggingface.co/qqjz/TextPro-SLM-7B-Global)

## Release and pairing

This repository contains the **original WhisperPro checkpoint (step 9092)** and is paired with [TextPro-SLM-7B-Global](https://huggingface.co/qqjz/TextPro-SLM-7B-Global).

The original interface supports audio clips of at most **30 seconds**. Use the separate 30Plus release for the segmentation-based long-audio extension.

| Component | Specification |
| --- | --- |
| Backbone | OpenAI Whisper large-v3 |
| Prosody features | Decoder layer 16, 1280 dimensions |
| Compact representation | Mean pooling over valid generated-token positions |
| Weights | Inference model in safetensors; training-only Mel reconstructor excluded |

## Method

The accompanying paper studies how reducing the speech–text mismatch from the input side can preserve semantic capabilities while learning paralinguistic understanding, using roughly **1,000 hours of LLM training audio** for the paper models. This model card documents the released artifact; it does not introduce new benchmark measurements.

## Download

```bash
pip install "huggingface-hub>=0.36,<1"
hf download qqjz/WhisperPro --local-dir WhisperPro
```

This Hub repository contains **checkpoint artifacts and model documentation only**. Inference code, dependencies, and usage examples are being prepared separately in the [GitHub project](https://github.com/dreamtheater123/TextPro-SLM); they are not included in this checkpoint release. Prosody-aware inference requires that implementation rather than loading the text backbone alone.

## Files and export

```text
model.safetensors        WhisperPro inference weights
config.json              Whisper architecture configuration
codec_config.json        Prosody-layer selection
generation_config.json   ASR generation configuration
*token* / vocab / merges  Processor and tokenizer artifacts
SHA256SUMS.json          Artifact sizes and SHA-256 hashes
```

The processor matches the one used by the paired inference pipeline. All retained weight tensors are unchanged; excluding the training-only Mel reconstructor does not change transcript/prosody extraction.

## Model family

| Encoder | Paired 7B model | Audio handling |
| --- | --- | --- |
| [WhisperPro](https://huggingface.co/qqjz/WhisperPro) | [TextPro-SLM-7B-Global](https://huggingface.co/qqjz/TextPro-SLM-7B-Global) | Original, at most 30 seconds |
| [WhisperPro-30Plus](https://huggingface.co/qqjz/WhisperPro-30Plus) | [TextPro-SLM-7B-30Plus](https://huggingface.co/qqjz/TextPro-SLM-7B-30Plus) | Segmentation-based long-audio extension |

## Limitations

Transcription errors and shifts in language or recording conditions can affect the extracted features. Compact mean pooling compresses time-varying prosody. The 30Plus processing design is subject to segmentation effects and the LLM's finite transcript context; no unlimited-duration or long-audio accuracy claim is implied.

## License and acknowledgements

Apache-2.0; see `LICENSE` and `NOTICE`. This checkpoint is derived from [OpenAI Whisper large-v3](https://huggingface.co/openai/whisper-large-v3).

## Citation

```bibtex
@inproceedings{cui2026textproslm,
  title={Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM},
  author={Cui, Wenqian and Li, Xiao-Hui and Tan, Daxin and Zheng, Qiyong and King, Irwin},
  booktitle={Advances in Neural Information Processing Systems},
  year={2026},
  url={https://openreview.net/forum?id=Jsjmo8eX29}
}
```