---
license: cc-by-nc-sa-4.0
library_name: generic
tags:
- audio
- speech
- multilingual
- automatic-speech-recognition
- safetensors
- bfloat16
- e-branchformer
- acoustic-encoder
datasets:
- espnet/yodas
- facebook/voxpopuli
- facebook/multilingual_librispeech
- google/fleurs
- openslr/librispeech_asr
- speechcolab/gigaspeech
- cheulyop/ksponspeech
- espnet/mms_ulab_v2
---

# XEUS: Cross-lingual E-Branchformer Universal Speech Representation (SafeTensors BF16)

This repository provides the **SafeTensors (bfloat16)** conversion and standalone pure-PyTorch implementation of **XEUS**, originally developed by Carnegie Mellon University's [WAVLab](https://www.wavlab.org/).

- **Original Hub Repository:** [espnet/xeus](https://huggingface.co/espnet/xeus)
- **Paper:** [XEUS: A Cross-lingual Encoder for Universal Speech](https://wanchichen.github.io/pdf/xeus.pdf)
- **Architecture:** E-Branchformer (19 layers, 577M parameters)
- **Pretraining scale:** 1,000,000+ hours across 4,000+ languages
- **Input:** 16 kHz raw audio waveform `(batch_size, time_samples)`
- **Output:** Frame representations `(batch_size, time_frames, 1024)` at 50 Hz frame rate (stride 320, 20ms per frame)
- **Precision:** `bfloat16` (1.07 GB vs 2.15 GB in float32, 50% storage & memory savings)
- **Weight format:** `model.safetensors` (fast, zero arbitrary code execution, memory-mapped)

---

## Architecture Specifications

| Parameter | Value |
| :--- | :--- |
| Pretrained Objective | HuBERT-style masked prediction + acoustic noise & reverberation augmentation |
| Input Format | Raw mono waveform at 16,000 Hz |
| Frontend | 7-layer Conv1d extractor (strides: 5, 2, 2, 2, 2, 2, 2; total stride: 320) |
| Feature Dimension ($d_{model}$) | 1024 |
| Intermediate FFN Dimension | 4096 |
| Number of E-Branchformer Layers | 19 |
| Attention Heads | 8 |
| CGMLP / CSGU Kernel Size | 31 (depthwise) |
| Output Frame Rate | 50.0 Hz |
| Default Precision | `torch.bfloat16` |

---

## Standalone Quickstart (No ESPnet installation required)

This repository includes a standalone [`modeling_xeus.py`](modeling_xeus.py) implementation that runs on pure PyTorch without requiring ESPnet or custom C++ dependencies.

### Installation

```bash
pip install torch safetensors huggingface_hub
```

### Inference Example

```python
import torch
from huggingface_hub import hf_hub_download

# Download modeling code and load weights
# Or copy modeling_xeus.py directly to your codebase
from modeling_xeus import XeusTrunk

# Automatically downloads bfloat16 model.safetensors from this repository
model = XeusTrunk.from_pretrained("giangndm/xeus", dtype=torch.bfloat16)
model.eval()

# Input: 16 kHz waveform (batch_size, num_samples)
# Example: 1.0 second of dummy audio
audio = torch.randn(1, 16000)

with torch.no_grad():
    # Shape: [1, 49, 1024] (50 Hz frame rate)
    features = model(audio)

print("Acoustic representation shape:", features.shape, "dtype:", features.dtype)
```

---

## Weight Verification & Conversion Details

The weights in `model.safetensors` were converted directly from `model/xeus_checkpoint_new.pth` (commit `02acdd3ac9445561eeb7e9e7f096e4482a32b3a7`) of `espnet/xeus` to `bfloat16` SafeTensors format.
- Tensors: All 763 tensors converted to contiguous bfloat16 format.
- Size: ~1.07 GB (compact footprint for efficient training and inference).

---

## License

This model and its weights are distributed under the **Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC-BY-NC-SA 4.0)** license, matching the original license established by the authors at CMU WAVLab / ESPnet. See [`LICENSE`](LICENSE) for complete legal terms.

---

## Citation

```bibtex
@article{chen2024xeus,
  title={XEUS: A Cross-lingual Encoder for Universal Speech},
  author={Chen, Wan-Chi and Choi, Kwanghee and Kim, David and Watanabe, Shinji},
  journal={arXiv preprint arXiv:2406.11832},
  year={2024}
}
```