Any-to-Any
ESPnet
PyTorch
English
audio
multimodal
speech-language-model
audio-understanding
audio-generation
audio-to-text
text-to-audio
rich-caption
speech
music
environmental-sound
Instructions to use espnet/bagpiper with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- ESPnet
How to use espnet/bagpiper with ESPnet:
unknown model type (must be text-to-speech or automatic-speech-recognition)
- Notebooks
- Google Colab
- Kaggle
Add standard ESPnet configuration for download counting
Browse filesPublish the existing ESPnet configuration as config.yaml so normal repository downloads include a Hugging Face download-counting query file. Document the configuration filename and preserve release integrity checks.
- config.yaml +57 -0
config.yaml
ADDED
|
@@ -0,0 +1,57 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
job_type: speechlm
|
| 2 |
+
|
| 3 |
+
# Reconstructed config that reproduces the model behind the public
|
| 4 |
+
# stage-2 pretrained BASE checkpoint (exp/opuslm_v2_stage2_pretrain_base,
|
| 5 |
+
# step 260000). The original launch script referenced this filename but it
|
| 6 |
+
# was never committed; recreated here as the base-tokenizer analog of
|
| 7 |
+
# conf/train_stage2_qwen3.yaml (Qwen3-8B-Base backbone/tokenizer).
|
| 8 |
+
# Vocab check: 256 special + 151936 (Qwen3-8B-Base) + 8200 (Xcodec 8x1025)
|
| 9 |
+
# = 160392 = embed_tokens rows in the checkpoint.
|
| 10 |
+
|
| 11 |
+
multimodal_io:
|
| 12 |
+
text:
|
| 13 |
+
tokenizer_name: Qwen/Qwen3-8B-Base
|
| 14 |
+
discrete_audio:
|
| 15 |
+
codec_choice: Xcodec
|
| 16 |
+
codec_hf_model_tag: hf-audio/xcodec-hubert-general
|
| 17 |
+
ssl_choice: null
|
| 18 |
+
ssl_hf_model_tag: null
|
| 19 |
+
delay_interleave: true
|
| 20 |
+
stream_weights: [0.125, 0.125, 0.125, 0.125, 0.125, 0.125, 0.125, 0.125]
|
| 21 |
+
continuous_audio:
|
| 22 |
+
encoder_choice: huggingface
|
| 23 |
+
encoder_hf_model_tag: Qwen/Qwen3-Omni-30B-A3B-Instruct
|
| 24 |
+
attn_implementation: flash_attention_3
|
| 25 |
+
dtype: bfloat16
|
| 26 |
+
|
| 27 |
+
model:
|
| 28 |
+
model_choice: parallel
|
| 29 |
+
model_hf_tag: Qwen/Qwen3-8B-Base
|
| 30 |
+
model_conf:
|
| 31 |
+
attn_implementation: flash_attention_3
|
| 32 |
+
dtype: bfloat16
|
| 33 |
+
# NOTE: compile_transformer_body / freeze_text_embeddings appear in the
|
| 34 |
+
# original stage configs but are dead keys in the current espnet2/speechlm
|
| 35 |
+
# code (referenced nowhere). transformers 5.x silently drops them; 4.57.x
|
| 36 |
+
# forwards them into the HF constructor and crashes. Omitted here.
|
| 37 |
+
activation_checkpointing: true
|
| 38 |
+
|
| 39 |
+
preprocessor:
|
| 40 |
+
audio_input: continuous_audio
|
| 41 |
+
audio_output: discrete_audio
|
| 42 |
+
loss_region: assistant
|
| 43 |
+
audio_cfg: 0.05
|
| 44 |
+
|
| 45 |
+
data_loading:
|
| 46 |
+
batchfy_method: pack
|
| 47 |
+
batch_size: 15000
|
| 48 |
+
save_loader_state: false
|
| 49 |
+
seed: 43
|
| 50 |
+
num_workers: 6
|
| 51 |
+
|
| 52 |
+
trainer:
|
| 53 |
+
deepspeed_config: conf/deepspeed_stage2.json
|
| 54 |
+
freeze_param: [multimodal_io_dict.discrete_audio, multimodal_io_dict.continuous_audio]
|
| 55 |
+
max_step: 700000
|
| 56 |
+
save_interval: 5000
|
| 57 |
+
log_interval: 1
|