CoSTA-EN / README.md
D-YZ's picture
Add files using upload-large-folder tool
a9747b8 verified
|
Raw History Blame Contribute Delete
4.76 kB
---
license: apache-2.0
language:
- en
base_model: stepfun-ai/Step-Audio-2-mini
base_model_relation: finetune
library_name: transformers
pipeline_tag: any-to-any
tags:
- costa
- speech-to-speech
- spoken-dialogue
- speech-token-compression
- custom_code
- arxiv:2610.21143
---
# CoSTA-EN
**CoSTA-EN** is the English dialogue-adapted checkpoint from
[CoSTA: Compressed Speech Token Adaptation for Efficient Spoken Dialogue Models](https://arxiv.org/abs/2610.21143).
It is based on [Step-Audio-2-mini](https://huggingface.co/stepfun-ai/Step-Audio-2-mini).
[Paper](https://arxiv.org/abs/2610.21143) · [Code and full instructions](https://github.com/yingzhuodeng/CoSTA)
## Model
CoSTA compresses the speech-token sequences generated by the language model using
a learned BPE vocabulary. Generated tokens are expanded losslessly to the original
speech-token space before waveform synthesis. The original speech tokenizer and
waveform decoder are retained.
This is the final checkpoint after vocabulary extension, pronunciation adaptation,
and dialogue adaptation. It contains BF16 weights with 238,352 text-and-speech token
IDs, including 80,000 added speech tokens. The matching speech BPE vocabulary has
86,656 entries: 6,656 original units and 80,000 merged units.
## Files
- `model-*.safetensors` and `model.safetensors.index.json`: complete model weights.
- `config.json`, `configuration_step_audio_2.py`, and `modeling_step_audio_2.py`:
model configuration and custom Transformers implementation.
- `tokenizer.json` and companion tokenizer files: the LLM tokenizer.
- `speech_bpe_tokenizer.json`: the matching speech BPE tokenizer. This is distinct
from the LLM `tokenizer.json`.
- `speech_bpe_vocab_int.jsonl`: BPE-token expansions to original speech-token IDs.
- `audio_bpeid_to_llmid.json`: mapping from speech BPE IDs to global LLM token IDs.
## Download and inference
Use the inference code and dependency versions in the
[CoSTA repository](https://github.com/yingzhuodeng/CoSTA#-environment-setup).
The supported workflow uses Transformers 4.49.0 and the custom model implementation;
a generic Transformers `pipeline` is not the speech-dialogue inference entry point.
After cloning the repository and installing its dependencies, run from its root:
```bash
export COSTA_LANGUAGE=en
source scripts/env.sh
hf download D-YZ/CoSTA-EN --local-dir "checkpoints/CoSTA-EN"
hf download stepfun-ai/Step-Audio-2-mini \
--include "token2wav/*" --local-dir "$BASE_MODEL_PATH"
export MODEL_PATH="$PWD/checkpoints/CoSTA-EN"
export MODEL_NAME=CoSTA-EN
export BPE_TOKENIZER_PATH="$MODEL_PATH/speech_bpe_tokenizer.json"
export BPE_MAPPING_PATH="$MODEL_PATH/audio_bpeid_to_llmid.json"
export DATASET_NAME=AlpacaEval
export JSONL_PATH="$DATA_ROOT/eval/$DATASET_NAME/test_norm.jsonl"
CUDA_VISIBLE_DEVICES=0 bash inference/infer_costa.sh
```
Prepare the evaluation JSONL and referenced audio files as described in the
[data and inference instructions](https://github.com/yingzhuodeng/CoSTA#-model-inference).
Prepared training and evaluation data are released separately and are not included
in this model repository. The external `token2wav` decoder is obtained from the base
model as shown above. For the bilingual checkpoint, select the appropriate input
manifest for the language being evaluated while retaining this checkpoint's own BPE files.
## Training and evaluation
The adaptation sequence is mean-initialized vocabulary extension, pronunciation
adaptation with an embedding/LM-head stage followed by a full-LLM stage, and
speech-to-speech dialogue adaptation. Pronunciation data is based on Emilia with
Seed-VC voice conversion; dialogue preparation follows OpenS2S with CosyVoice 2
assistant-speech synthesis. The paper reports evaluation on URO-Bench.
See the [paper](https://arxiv.org/abs/2610.21143) for the experimental settings,
metrics, results, and limitations. In particular, much of the adaptation data is
synthetic, and evaluation focuses on general speech dialogue. Model outputs can
contain factual errors and depend on the input language and acoustic conditions.
## License and attribution
This checkpoint is released under Apache-2.0. See `LICENSE` and `NOTICE`.
The custom model implementation derives from Step-Audio-2-mini and contains
CoSTA changes for training loss, embedding resizing, and inference timing.
## Citation
```bibtex
@misc{deng2026costa,
title={{CoSTA}: Compressed Speech Token Adaptation for Efficient Spoken Dialogue Models},
author={Yingzhuo Deng and Chong Li and Bingxin Liu and Yi Shu and Tianyu Peng and Can Xie and Siyao Song and Shuo Ren and Jiajun Zhang},
year={2026},
eprint={2610.21143},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2610.21143}
}
```