| --- |
| base_model: |
| - Qwen/Qwen2.5-7B-Instruct |
| - qqjz/WhisperPro-30Plus |
| language: |
| - en |
| library_name: pytorch |
| license: apache-2.0 |
| pipeline_tag: audio-text-to-text |
| tags: |
| - textpro-slm |
| - prosody |
| - speech |
| - audio |
| - long-audio |
| --- |
| |
| # TextPro-SLM-7B-30Plus |
|
|
| **Your speech LLM can be a prosody-aware text LLM.** |
|
|
| TextPro-SLM combines transcript tokens with a compact prosody embedding. Transcript tokens carry linguistic content, while prosody carries complementary acoustic information. This input-side factorization brings spoken input closer to the native input format of a text LLM. |
|
|
| [Paper](https://openreview.net/forum?id=Jsjmo8eX29) · [Paper on HF](https://huggingface.co/papers/2605.05927) · [GitHub project](https://github.com/dreamtheater123/TextPro-SLM) · [Paired checkpoint](https://huggingface.co/qqjz/WhisperPro-30Plus) |
|
|
| ## Release and pairing |
|
|
| This repository contains the **30Plus extension** and is paired with [WhisperPro-30Plus](https://huggingface.co/qqjz/WhisperPro-30Plus). |
|
|
| The paired inference design handles recordings longer than 30 seconds through VAD segmentation, previous-transcript context, transcript concatenation, and equal-weight averaging of segment prosody vectors. Each Whisper segment remains at most 30 seconds; this is not a native extension of Whisper's context window. This release is separate from the original checkpoints, and original-paper scores are not measurements of the 30Plus extension. |
|
|
| | Component | Specification | |
| | --- | --- | |
| | Text backbone | Qwen2.5-7B-Instruct, fully fine-tuned | |
| | Prosody projector | 1280 → 3584 → 3584, GELU, BF16 | |
| | Fusion | One global prosody embedding prepended to transcript tokens | |
| | Audio preprocessing | Mono, 16 kHz | |
| | LLM context | 32,768 tokens | |
| | Output | Text response | |
|
|
| ## Method |
|
|
| The accompanying paper studies how reducing the speech–text mismatch from the input side can preserve semantic capabilities while learning paralinguistic understanding, using roughly **1,000 hours of LLM training audio** for the paper models. This model card documents the released artifact; it does not introduce new benchmark measurements. |
|
|
| ## Download |
|
|
| ```bash |
| pip install "huggingface-hub>=0.36,<1" |
| hf download qqjz/TextPro-SLM-7B-30Plus --local-dir TextPro-SLM-7B-30Plus |
| ``` |
|
|
| This Hub repository contains **checkpoint artifacts and model documentation only**. Inference code, dependencies, and usage examples are being prepared separately in the [GitHub project](https://github.com/dreamtheater123/TextPro-SLM); they are not included in this checkpoint release. Prosody-aware inference requires that implementation rather than loading the text backbone alone. |
|
|
| ## Files and export |
|
|
| ```text |
| llm/ Fine-tuned LLM weights, configuration, and tokenizer |
| projector.safetensors Compact trained prosody projector |
| textpro_config.json Encoder pairing, fusion, and prosody-token settings |
| SHA256SUMS.json Artifact sizes and SHA-256 hashes |
| ``` |
|
|
| The supplied tokenizer preserves the trained `<|repo_name|>` prosody marker (ID 151663). The configuration pins the paired encoder revision. The projector is stored as compact safetensors rather than a serialized training storage buffer. |
|
|
| ## Model family |
|
|
| | Encoder | Paired 7B model | Audio handling | |
| | --- | --- | --- | |
| | [WhisperPro](https://huggingface.co/qqjz/WhisperPro) | [TextPro-SLM-7B-Global](https://huggingface.co/qqjz/TextPro-SLM-7B-Global) | Original, at most 30 seconds | |
| | [WhisperPro-30Plus](https://huggingface.co/qqjz/WhisperPro-30Plus) | [TextPro-SLM-7B-30Plus](https://huggingface.co/qqjz/TextPro-SLM-7B-30Plus) | Segmentation-based long-audio extension | |
|
|
| ## Limitations |
|
|
| Responses may contain transcription errors or hallucinations. Performance can vary with language, recording conditions, speaker characteristics, and domain. Global pooling compresses time-varying prosody. The model returns text; it does not synthesize speech. The 30Plus processing design is subject to segmentation effects and the LLM's finite transcript context; no unlimited-duration or long-audio accuracy claim is implied. |
|
|
| ## License and acknowledgements |
|
|
| Apache-2.0; see `LICENSE` and `NOTICE`. The LLM is derived from [Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct). The paired WhisperPro encoder is derived from OpenAI Whisper large-v3. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{cui2026textproslm, |
| title={Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM}, |
| author={Cui, Wenqian and Li, Xiao-Hui and Tan, Daxin and Zheng, Qiyong and King, Irwin}, |
| booktitle={Advances in Neural Information Processing Systems}, |
| year={2026}, |
| url={https://openreview.net/forum?id=Jsjmo8eX29} |
| } |
| ``` |
