| --- |
| license: apache-2.0 |
| library_name: pytorch |
| tags: |
| - forced-alignment |
| - word-alignment |
| - speech |
| - audio |
| - gguf |
| --- |
| |
| # FuseAlign multilingual (CJK fine-tune, step 2,500) |
|
|
| Word-level forced aligner: given a recording and its transcript, it predicts each word's start and end on a 1 ms |
| grid plus a presence probability. A reproduction of FuseAlign ([Vaidya et al., 2026](https://arxiv.org/abs/2609.33650)) |
| by Zyphra: the multilingual model (100,000 steps), fine-tuned 2,500 steps on CJK junction-corrected labels. |
|
|
| Code, usage, long-form alignment and the C++/ggml CLI: |
| [github.com/Zyphra/fusealign-multilingual](https://github.com/Zyphra/fusealign-multilingual). |
|
|
| ## Files |
|
|
| | File | Size | Use | |
| | --- | ---: | --- | |
| | `fusealign-multilingual-cjkft-step2500.safetensors` | 1.34 GB | PyTorch package (`torch.hub`, `fusealign_hub`): FuseAlign fp32, Dia text encoder bf16 | |
| | `fusealign-multilingual-cjkft-step2500-f32.gguf` | 1.35 GB | ggml port: FuseAlign F32, Dia BF16, Silero VAD F32 | |
| | `fusealign-multilingual-cjkft-step2500-q8_0.gguf` | 0.51 GB | ggml port: transformer matmuls Q8_0, other Dia tensors BF16, the rest F32 | |
| |
| ## Usage |
| |
| ```python |
| import soundfile as sf |
| import torch |
| |
| aligner = torch.hub.load("Zyphra/fusealign-multilingual", "fusealign", trust_repo=True, device="cuda") |
| wav, sr = sf.read("speech.wav", dtype="float32") |
| words = aligner.align(torch.from_numpy(wav), sr, "the transcript of the recording") |
| # [WordAlignment(word='the', start=..., end=..., presence=..., placed=True), ...] |
| ``` |
| |
| ## Evaluation |
| |
| % of word boundaries off by more than 50 ms (the paper's asymmetric error), against |
| [Qwen3-ForcedAligner-0.6B](https://huggingface.co/Qwen/Qwen3-ForcedAligner-0.6B) (qwen-asr 0.0.6, bf16, whole clips) |
| on the same clips, transcripts and references: |
| |
| | Corpus | FuseAlign | Qwen3-FA | |
| | --- | ---: | ---: | |
| | TIMIT | 16.55 | 20.17 | |
| | Buckeye | 24.58 | 24.60 | |
| | AMI-silver | 49.38 | 58.22 | |
| | TIMIT-long | 17.25 | 20.04 | |
| | BZNSYP Mandarin (gold) | 15.94 | 28.04 | |
| | FLEURS Mandarin | 17.95 | 11.06 | |
| | FLEURS Japanese | 14.02 | 8.38 | |
| | FLEURS Korean | 8.39 | 11.53 | |
| | FLEURS Spanish | 19.66 | 11.71 | |
| | FLEURS Portuguese | 9.30 | 10.24 | |
| | FLEURS French | 13.15 | 12.35 | |
| | FLEURS German | 13.29 | 13.91 | |
| | FLEURS Russian | 12.74 | 11.18 | |
| |
| FLEURS references are Montreal Forced Aligner output, not hand labels; BZNSYP's are hand-labelled. |
| |
| ## License |
| |
| Apache-2.0. The checkpoint and GGUFs include the Dia text encoder weights from |
| [nari-labs/Dia-1.6B-0626](https://huggingface.co/nari-labs/Dia-1.6B-0626) (Apache-2.0); the GGUFs include |
| [Silero VAD](https://github.com/snakers4/silero-vad) weights (MIT). |
| |
| ## Citation |
| |
| ```bibtex |
| @article{vaidya2026fusealign, |
| title = {FuseAlign: Forced Alignment in the Wild}, |
| author = {Vaidya, Mithilesh and Bailey, Stephen and Badam, Sumukh and Bendel, Matthew and He, Xingzhe}, |
| journal = {arXiv preprint arXiv:2609.33650}, |
| year = {2026} |
| } |
| ``` |
| |
