File size: 1,907 Bytes
b6d2bc8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 | ---
pretty_name: EchoChat
language:
- en
tags:
- audio
- speech
- conversational
library_name: transformers
license: other
license_name: echochat-component-licenses
license_link: LICENSE
---
# EchoChat
EchoChat is an audio-conditioned conversational model with text and speech-token
outputs. This repository provides the language model, tokenizer, and audio
encoder weights.
## Files and loading
- Root: language model weights, tokenizer, configuration and custom model code.
- `audio_encoder/`: complete audio encoder weights, configuration and its custom implementation.
- `LICENSE`, `licenses/`, `THIRD_PARTY_NOTICES.md`: applicable licenses and notices.
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "ddwang2000/EchoChat"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
repo, trust_remote_code=True,
torch_dtype=torch.bfloat16,
attn_implementation="flash_attention_2",
).to("cuda").eval()
```
The custom audio encoder class is `WhisperEncoder` in
`audio_encoder/modeling_audio_encoder.py`. Instantiate it with the supplied
configuration and strictly load `audio_encoder/model.safetensors`. Its custom
attention implementation requires a compatible CUDA/FlashAttention runtime.
This is a component checkpoint, not a standard Transformers text-generation
pipeline. End-to-end audio use requires compatible feature extraction, prompting,
and text/speech-token generation logic. Waveform synthesis additionally requires
a separately licensed compatible decoder, which is not included here.
## Dependencies
The component configuration targets Transformers 4.51.3. Runtime dependencies
include PyTorch, Transformers, safetensors, tiktoken, einops, and a compatible
FlashAttention build for CUDA execution. Inspect the custom code before enabling
`trust_remote_code=True`.
|