Any-to-Any
Transformers
Safetensors
English
step_audio_2
text-generation
costa
speech-to-speech
spoken-dialogue
speech-token-compression
custom_code
Instructions to use D-YZ/CoSTA-EN with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use D-YZ/CoSTA-EN with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("D-YZ/CoSTA-EN", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
|
Download README.md from D-YZ/CoSTA-EN: direct link, hf CLI and curl.
- Browser
- Download file 4.76 kB
-
https://huggingface.co/D-YZ/CoSTA-EN/resolve/main/README.md
- Command line
-
hf download hf://D-YZ/CoSTA-EN/README.md
-
curl -L -o README.md https://huggingface.co/D-YZ/CoSTA-EN/resolve/main/README.md
4.76 kB
| license: apache-2.0 | |
| language: | |
| - en | |
| base_model: stepfun-ai/Step-Audio-2-mini | |
| base_model_relation: finetune | |
| library_name: transformers | |
| pipeline_tag: any-to-any | |
| tags: | |
| - costa | |
| - speech-to-speech | |
| - spoken-dialogue | |
| - speech-token-compression | |
| - custom_code | |
| - arxiv:2610.21143 | |
| # CoSTA-EN | |
| **CoSTA-EN** is the English dialogue-adapted checkpoint from | |
| [CoSTA: Compressed Speech Token Adaptation for Efficient Spoken Dialogue Models](https://arxiv.org/abs/2610.21143). | |
| It is based on [Step-Audio-2-mini](https://huggingface.co/stepfun-ai/Step-Audio-2-mini). | |
| [Paper](https://arxiv.org/abs/2610.21143) · [Code and full instructions](https://github.com/yingzhuodeng/CoSTA) | |
| ## Model | |
| CoSTA compresses the speech-token sequences generated by the language model using | |
| a learned BPE vocabulary. Generated tokens are expanded losslessly to the original | |
| speech-token space before waveform synthesis. The original speech tokenizer and | |
| waveform decoder are retained. | |
| This is the final checkpoint after vocabulary extension, pronunciation adaptation, | |
| and dialogue adaptation. It contains BF16 weights with 238,352 text-and-speech token | |
| IDs, including 80,000 added speech tokens. The matching speech BPE vocabulary has | |
| 86,656 entries: 6,656 original units and 80,000 merged units. | |
| ## Files | |
| - `model-*.safetensors` and `model.safetensors.index.json`: complete model weights. | |
| - `config.json`, `configuration_step_audio_2.py`, and `modeling_step_audio_2.py`: | |
| model configuration and custom Transformers implementation. | |
| - `tokenizer.json` and companion tokenizer files: the LLM tokenizer. | |
| - `speech_bpe_tokenizer.json`: the matching speech BPE tokenizer. This is distinct | |
| from the LLM `tokenizer.json`. | |
| - `speech_bpe_vocab_int.jsonl`: BPE-token expansions to original speech-token IDs. | |
| - `audio_bpeid_to_llmid.json`: mapping from speech BPE IDs to global LLM token IDs. | |
| ## Download and inference | |
| Use the inference code and dependency versions in the | |
| [CoSTA repository](https://github.com/yingzhuodeng/CoSTA#-environment-setup). | |
| The supported workflow uses Transformers 4.49.0 and the custom model implementation; | |
| a generic Transformers `pipeline` is not the speech-dialogue inference entry point. | |
| After cloning the repository and installing its dependencies, run from its root: | |
| ```bash | |
| export COSTA_LANGUAGE=en | |
| source scripts/env.sh | |
| hf download D-YZ/CoSTA-EN --local-dir "checkpoints/CoSTA-EN" | |
| hf download stepfun-ai/Step-Audio-2-mini \ | |
| --include "token2wav/*" --local-dir "$BASE_MODEL_PATH" | |
| export MODEL_PATH="$PWD/checkpoints/CoSTA-EN" | |
| export MODEL_NAME=CoSTA-EN | |
| export BPE_TOKENIZER_PATH="$MODEL_PATH/speech_bpe_tokenizer.json" | |
| export BPE_MAPPING_PATH="$MODEL_PATH/audio_bpeid_to_llmid.json" | |
| export DATASET_NAME=AlpacaEval | |
| export JSONL_PATH="$DATA_ROOT/eval/$DATASET_NAME/test_norm.jsonl" | |
| CUDA_VISIBLE_DEVICES=0 bash inference/infer_costa.sh | |
| ``` | |
| Prepare the evaluation JSONL and referenced audio files as described in the | |
| [data and inference instructions](https://github.com/yingzhuodeng/CoSTA#-model-inference). | |
| Prepared training and evaluation data are released separately and are not included | |
| in this model repository. The external `token2wav` decoder is obtained from the base | |
| model as shown above. For the bilingual checkpoint, select the appropriate input | |
| manifest for the language being evaluated while retaining this checkpoint's own BPE files. | |
| ## Training and evaluation | |
| The adaptation sequence is mean-initialized vocabulary extension, pronunciation | |
| adaptation with an embedding/LM-head stage followed by a full-LLM stage, and | |
| speech-to-speech dialogue adaptation. Pronunciation data is based on Emilia with | |
| Seed-VC voice conversion; dialogue preparation follows OpenS2S with CosyVoice 2 | |
| assistant-speech synthesis. The paper reports evaluation on URO-Bench. | |
| See the [paper](https://arxiv.org/abs/2610.21143) for the experimental settings, | |
| metrics, results, and limitations. In particular, much of the adaptation data is | |
| synthetic, and evaluation focuses on general speech dialogue. Model outputs can | |
| contain factual errors and depend on the input language and acoustic conditions. | |
| ## License and attribution | |
| This checkpoint is released under Apache-2.0. See `LICENSE` and `NOTICE`. | |
| The custom model implementation derives from Step-Audio-2-mini and contains | |
| CoSTA changes for training loss, embedding resizing, and inference timing. | |
| ## Citation | |
| ```bibtex | |
| @misc{deng2026costa, | |
| title={{CoSTA}: Compressed Speech Token Adaptation for Efficient Spoken Dialogue Models}, | |
| author={Yingzhuo Deng and Chong Li and Bingxin Liu and Yi Shu and Tianyu Peng and Can Xie and Siyao Song and Shuo Ren and Jiajun Zhang}, | |
| year={2026}, | |
| eprint={2610.21143}, | |
| archivePrefix={arXiv}, | |
| primaryClass={cs.CL}, | |
| url={https://arxiv.org/abs/2610.21143} | |
| } | |
| ``` | |