| --- |
| license: cc-by-nc-sa-4.0 |
| library_name: generic |
| tags: |
| - audio |
| - speech |
| - multilingual |
| - automatic-speech-recognition |
| - safetensors |
| - bfloat16 |
| - e-branchformer |
| - acoustic-encoder |
| datasets: |
| - espnet/yodas |
| - facebook/voxpopuli |
| - facebook/multilingual_librispeech |
| - google/fleurs |
| - openslr/librispeech_asr |
| - speechcolab/gigaspeech |
| - cheulyop/ksponspeech |
| - espnet/mms_ulab_v2 |
| --- |
| |
| # XEUS: Cross-lingual E-Branchformer Universal Speech Representation (SafeTensors BF16) |
|
|
| This repository provides the **SafeTensors (bfloat16)** conversion and standalone pure-PyTorch implementation of **XEUS**, originally developed by Carnegie Mellon University's [WAVLab](https://www.wavlab.org/). |
|
|
| - **Original Hub Repository:** [espnet/xeus](https://huggingface.co/espnet/xeus) |
| - **Paper:** [XEUS: A Cross-lingual Encoder for Universal Speech](https://wanchichen.github.io/pdf/xeus.pdf) |
| - **Architecture:** E-Branchformer (19 layers, 577M parameters) |
| - **Pretraining scale:** 1,000,000+ hours across 4,000+ languages |
| - **Input:** 16 kHz raw audio waveform `(batch_size, time_samples)` |
| - **Output:** Frame representations `(batch_size, time_frames, 1024)` at 50 Hz frame rate (stride 320, 20ms per frame) |
| - **Precision:** `bfloat16` (1.07 GB vs 2.15 GB in float32, 50% storage & memory savings) |
| - **Weight format:** `model.safetensors` (fast, zero arbitrary code execution, memory-mapped) |
|
|
| --- |
|
|
| ## Architecture Specifications |
|
|
| | Parameter | Value | |
| | :--- | :--- | |
| | Pretrained Objective | HuBERT-style masked prediction + acoustic noise & reverberation augmentation | |
| | Input Format | Raw mono waveform at 16,000 Hz | |
| | Frontend | 7-layer Conv1d extractor (strides: 5, 2, 2, 2, 2, 2, 2; total stride: 320) | |
| | Feature Dimension ($d_{model}$) | 1024 | |
| | Intermediate FFN Dimension | 4096 | |
| | Number of E-Branchformer Layers | 19 | |
| | Attention Heads | 8 | |
| | CGMLP / CSGU Kernel Size | 31 (depthwise) | |
| | Output Frame Rate | 50.0 Hz | |
| | Default Precision | `torch.bfloat16` | |
| |
| --- |
| |
| ## Standalone Quickstart (No ESPnet installation required) |
| |
| This repository includes a standalone [`modeling_xeus.py`](modeling_xeus.py) implementation that runs on pure PyTorch without requiring ESPnet or custom C++ dependencies. |
| |
| ### Installation |
| |
| ```bash |
| pip install torch safetensors huggingface_hub |
| ``` |
| |
| ### Inference Example |
| |
| ```python |
| import torch |
| from huggingface_hub import hf_hub_download |
| |
| # Download modeling code and load weights |
| # Or copy modeling_xeus.py directly to your codebase |
| from modeling_xeus import XeusTrunk |
| |
| # Automatically downloads bfloat16 model.safetensors from this repository |
| model = XeusTrunk.from_pretrained("giangndm/xeus", dtype=torch.bfloat16) |
| model.eval() |
|
|
| # Input: 16 kHz waveform (batch_size, num_samples) |
| # Example: 1.0 second of dummy audio |
| audio = torch.randn(1, 16000) |
|
|
| with torch.no_grad(): |
| # Shape: [1, 49, 1024] (50 Hz frame rate) |
| features = model(audio) |
| |
| print("Acoustic representation shape:", features.shape, "dtype:", features.dtype) |
| ``` |
| |
| --- |
| |
| ## Weight Verification & Conversion Details |
| |
| The weights in `model.safetensors` were converted directly from `model/xeus_checkpoint_new.pth` (commit `02acdd3ac9445561eeb7e9e7f096e4482a32b3a7`) of `espnet/xeus` to `bfloat16` SafeTensors format. |
| - Tensors: All 763 tensors converted to contiguous bfloat16 format. |
| - Size: ~1.07 GB (compact footprint for efficient training and inference). |
| |
| --- |
| |
| ## License |
| |
| This model and its weights are distributed under the **Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC-BY-NC-SA 4.0)** license, matching the original license established by the authors at CMU WAVLab / ESPnet. See [`LICENSE`](LICENSE) for complete legal terms. |
| |
| --- |
| |
| ## Citation |
| |
| ```bibtex |
| @article{chen2024xeus, |
| title={XEUS: A Cross-lingual Encoder for Universal Speech}, |
| author={Chen, Wan-Chi and Choi, Kwanghee and Kim, David and Watanabe, Shinji}, |
| journal={arXiv preprint arXiv:2406.11832}, |
| year={2024} |
| } |
| ``` |
| |
