README.md

---
license: cc-by-nc-sa-4.0
language:
  - zh
  - en
tags:
  - forced-alignment
  - singing-voice
  - phoneme-alignment
  - breath-detection
  - hubert
library_name: singalign
pipeline_tag: audio-to-audio
---

# SingAlign — released checkpoints

Weights for [**SingAlign**](https://github.com/pymaster17/SingAlign), a unified
forced-alignment toolkit for singing voice. The code lives in the GitHub
repository; this repository holds only the checkpoints it downloads.

Files here are named exactly as `configs/models/<name>.yaml` declares them, so
the toolkit's resolver (`modules/common/api/checkpoints.py`) fetches them by
name and verifies the pinned SHA256 before it puts anything on disk.

| File | Family | Size | SHA256 |
|---|---|---:|---|
| `hfa-mandarin-fullweak-v1.ckpt` | HubertFA | 54.2 MB | `d38f7d5839e2cd93e8dec5b77b6bd3ed28aa1318d5ed069c2cd81c899dd3c057` |
| `hfa-english-fullweak-v2.ckpt` | HubertFA | 54.1 MB | `878de0899cb188c05537934a6f55bfd7aaabcbde1c411aba30d1e6f50a995712` |
| `hfa-english-fullweak-v1.ckpt` | HubertFA (superseded by v2) | 54.1 MB | `c1b2baa939a87b3ed832c97e18eba9fcb27214b39f50ff346c062601bd11c513` |
| `nll-hubert-ls960-v1.pt` | breath head (NLL) for the English aligner | 9.9 MB | `03587e7a476aea7f137904e6034420d0a4b255d12949e85ec16b20077ae29ce8` |

## `hfa-mandarin-fullweak-v1`

A HubertFA alignment head over a **frozen `chinese-hubert-base` front-end**,
trained in-repo with `train_hfa.py`. It is SingAlign's default model.

- **Recipe**: `configs/hfa/train_fullweak_v1.yaml`, data
  `configs/hfa/binarize_fullweak.yaml`.
- **Training data**: 51.4 h full-label (GTSinger, M4Singer, Opencpop) +
  7627 h weak-label Mandarin singing (lyrics only, no phoneme durations).
- **Step 15000**, selected by CBER on a held-out validation set — *not* the
  final step-60000 weights, whose validation curve had already degraded.
- **Phoneme inventory**: 117 phones, `sofa-expanded-v1` profile (identity
  against the toolkit's canonical Mandarin inventory).
- **G2P**: G2pW — no pronunciation dictionary to configure.

### Results

Scored on `cloudtest-verified-v1`: 8 recently released original songs
(90 clips, 1971 phones, 535.6 s) with hand-verified `word` + `ph` tiers, none
of whose song IDs or "artist – title" pairs appear in any training manifest.
Every system was fed the **same gold phoneme sequence** with G2P and breath
detection off, so this measures acoustic alignment only.

| System | VER20 ↓ | VER50 ↓ | mIoU ↑ | CBER ↓ |
|---|---:|---:|---:|---:|
| **`hfa-mandarin-fullweak-v1`** | 0.2265 | **0.0571** | **0.8030** | **0.1737** |
| HubertFA v0.0.7 (upstream ONNX) | **0.2075** | 0.0786 | 0.7734 | 0.2223 |
| SOFA ConvNeXt (in-repo) | 0.2519 | 0.0835 | 0.7747 | 0.2159 |
| SOFA `pretrained_mandarin_singing` | 0.2575 | 0.0946 | 0.7501 | 0.2901 |
| STARS (Chinese) | 0.3379 | 0.1674 | 0.6072 | 0.6547 |

VER20/VER50 are vlabeler edit ratios at 20 ms / 50 ms boundary tolerance and
include `SP`; mIoU and CBER exclude it. Point estimates, no confidence
intervals.

**Read this as an error profile, not a ranking.** This model leads on VER50,
mIoU and CBER but trails upstream v0.0.7 by ~9% on VER20: it makes *fewer large
errors and more small ones*. If your criterion is a 20 ms tolerance, the
upstream model is the better pick.

### Runtime dependency

The torch path does **not** carry its own SSL front-end. The checkpoint records
`hubert_config.model_path`, and those encoder weights
([`TencentGameMate/chinese-hubert-base`](https://huggingface.co/TencentGameMate/chinese-hubert-base),
~380 MB) must be present. `python scripts/download_assets.py` in the toolkit
fetches this checkpoint, the encoder and the G2pW model together.

### Usage

```bash
git clone https://github.com/pymaster17/SingAlign && cd SingAlign
uv sync --extra pitch && source .venv/bin/activate
python scripts/download_assets.py           # this checkpoint + encoder + G2pW
python infer_one.py -a audio.wav -t "中文歌词" --out_formats textgrid,json
```

```python
from modules.api import HubertFATorchAligner

aligner = HubertFATorchAligner()             # resolves to this checkpoint
results = aligner.align([{"id": "x", "audio_path": "a.wav", "text": "中文歌词"}])
```

## `hfa-english-fullweak-v2`

The English model: the same alignment head over a **frozen
`facebook/hubert-base-ls960` front-end**, trained with the recipe of
`hfa-english-fullweak-v1` below on data whose only change is *where the clips
start*.

- **Recipe**: `configs/hfa/train_en_fullweak_v6.yaml` (identical to v1's
  except the run name), data `configs/hfa/binarize_en_fullweak_v6.yaml`.
- **Training data**: 21.4 h full-label across 13 singers (as v1; Project-AIdol
  and NUS-48E re-cut so that a silence goes whole to one neighbouring segment
  instead of being split) + 1845 h weak-label English singing selected by
  `singmos >= 3.5` only, so the corpus's silence-cut `sub` segments (37 %) are
  in the pool alongside the lyric-timed `short` ones.
- **Step 56000**, selected by CBER on held-out singers.
- **Phoneme inventory / G2P**: as v1 (`english-arpabet-v1`, G2pEn); the
  vocabulary file is byte-identical, so the two are drop-in for each other.

### Why v2 exists

v1 puts a spurious sub-millisecond `SP` at the head of about a third of clips
that start on the voice. The cause was traced to its weak-label pool: a
`wer` filter silently restricted it to lyric-timed segments, three quarters of
which start mid-voice, and CTC training on those teaches the shared phone
channels to read onset frames as "not yet a phone". Mixing in silence-cut
segments removes it: **32 % → 5.3 %** of voice-initial GTSinger-en clips
(`bench/tools/probe_leading_sp.py`; `docs/experiments/leading_sp_probe.md`).

### Results

Same four held-out NUS-48E singers as v1 (201 clips, 0.53 h):

| System | CBER ↓ | VER20 ↓ | VER50 ↓ | mIoU ↑ |
|---|---:|---:|---:|---:|
| **`hfa-english-fullweak-v2`** | 0.1951 | 0.1327 | 0.0634 | 0.7823 |
| `hfa-english-fullweak-v1` | 0.1975 | 0.1318 | 0.0616 | 0.7821 |
| Same recipe, full label only (21.4 h) | **0.1940** | **0.1300** | 0.0617 | **0.7833** |

A tie with v1 on alignment quality; the caveat below about the weak label
buying nothing measurable on clean studio audio still applies.

### Batched inference **is** bit-reproducible

Unlike v1, this checkpoint **passes** the toolkit's batch-invariance gate:
`bench/tools/validate_hfa_batch.py` on the same 48 items, batch 8, measures a
maximum boundary shift of **0.002 ms** against a 1.0 ms tolerance.

### Runtime dependency and usage

As v1: [`facebook/hubert-base-ls960`](https://huggingface.co/facebook/hubert-base-ls960)
must be present, and the model is named explicitly:

```python
from modules.api import HubertFATorchAligner

aligner = HubertFATorchAligner(ckpt="hfa-english-fullweak-v2", g2p="G2pEn")
results = aligner.align([{"id": "x", "audio_path": "a.wav", "text": "english lyrics"}])
```

## `hfa-english-fullweak-v1` (superseded)

Kept because the `CrawlSinger-en` corpus's `align_conf` column was written by
it; use `hfa-english-fullweak-v2` for new work. The same alignment head over a **frozen `facebook/hubert-base-ls960`
front-end** — same HuBERT-base recipe as the Mandarin model, so only the
encoder path and the phoneme inventory differ.

- **Recipe**: `configs/hfa/train_en_fullweak.yaml`, data
  `configs/hfa/binarize_en_fullweak.yaml`.
- **Training data**: 21.4 h full-label across **13 singers** (GTSinger-en,
  ACV-001, Project-AIdol, 8 of NUS-48E's 12 singers) + 1845 h weak-label
  English singing (lyrics only, no phoneme durations).
- **Step 54000**, selected by CBER on held-out singers — *not* the final
  step-60000 weights.
- **Phoneme inventory**: the 39 CMUdict ARPAbet phones (stress stripped) plus
  `SP`, `english-arpabet-v1` profile.
- **G2P**: G2pEn (CMUdict + a seq2seq for OOV) — no dictionary to configure.

### Results

Scored on **four held-out NUS-48E singers** (201 clips, 0.53 h) — two from each
mode of the corpus's F0 distribution, none of whom appears in training. The
same suite selected the checkpoint, so read these as in-suite numbers.

| System | CBER ↓ | VER20 ↓ | VER50 ↓ | mIoU ↑ |
|---|---:|---:|---:|---:|
| `hfa-english-fullweak-v1` | 0.1975 | 0.1318 | **0.0616** | 0.7821 |
| Same recipe, full label only (21.4 h) | **0.1940** | **0.1300** | 0.0617 | **0.7833** |

**The weak label bought nothing measurable here.** The two models tie, and the
gap is smaller than this run's own validation-point scatter. It is released as
the default English model on the grounds that the Mandarin line measured weak
label buying robustness on *rough* recordings — but English has no rough-domain
held-out set to check that on, because the validation set is studio singing
while the weak label is crawled audio. If your material is clean, expect
nothing from the extra 1845 h. The full account is in
`docs/experiments/hfa_en_full_weak_v1.md` in the toolkit.

### Batched inference is not bit-reproducible

This checkpoint **does not pass** the toolkit's batch-invariance gate:
`bench/tools/validate_hfa_batch.py` measures a **10.5 ms** maximum boundary
shift between batch-1 and batch-8 against a 1.0 ms tolerance. Scale: one
boundary of one clip in 48, between content phones (`SH | IY`) rather than on
an `SP` edge, so an adapter does not drop it.

Batching changes the floating-point reduction order in the linear layers, and
weak-label training leaves the logits flat enough at an acoustically ambiguous
boundary that the last-bit difference moves the Viterbi path. A deterministic
tie-break cannot fix it — the inputs genuinely differ. Set `max_batch_size=1`
if you need reproducibility. For comparison, the full-label-only model measures
0.003 ms on the same items.

### Runtime dependency

[`facebook/hubert-base-ls960`](https://huggingface.co/facebook/hubert-base-ls960),
**not** the Mandarin encoder — the two are not interchangeable.

### Usage

```python
from modules.api import HubertFATorchAligner

aligner = HubertFATorchAligner(ckpt="hfa-english-fullweak-v1", g2p="G2pEn")
results = aligner.align([{"id": "x", "audio_path": "a.wav", "text": "english lyrics"}])
```

## `nll-hubert-ls960-v1`

A **breath (non-lexical) head**, not an aligner: a 2.45 M-parameter CVNT that
labels each 10 ms frame `None` / `AP` from the HuBERT units the aligner has
already computed, and whose breaths are spliced into the finished alignment as
`AP` without touching any other boundary. It rides on
`hfa-english-fullweak-v2` (or v1) as `ap_detector="nll"` and is selected automatically
for any HubertFA aligner whose front-end is `hubert-base-ls960`.

- **Why it exists**: a breath head is bound to the SSL front-end it was
  trained on. The head shipped inside upstream HubertFA's ONNX bundle was
  trained on `chinese-hubert-base` units; run over `hubert-base-ls960` units
  (same 768 width, unrelated latent space) it recalls **0.001** of the breaths
  in held-out English singing -- nothing. The toolkit now refuses that pairing.
- **Recipe**: `configs/nll/train_v2.yaml`, data `configs/nll/binarize_v2.yaml`;
  class-weighted CE + focal + dice, 15000 steps, **step 7500** selected on
  frame-F1 over the annotation-consistent held-out groups.
- **Training data (59.3 h)**: GTSinger singers whose breath annotation agrees
  with the upstream head's convention (ZH ×2, EN-Alto-2, EN-Tenor-1, IT ×3,
  ES-Soprano-1; 32.7 h) + M4Singer (26.7 h). Five GTSinger languages label the
  whole inter-phrase gap as breath; a head trained on all of GTSinger learns
  "gap ⇒ breath" and its precision collapses on any other corpus.
- **Binary**: `None` / `AP`. No tail-breath (`EP`) class.

### Results

Frame-F1 at threshold 0.5, scored only inside the gold non-content regions
(where a breath could be), on songs held out by title:

| Held-out set | This head | Upstream head on its own front-end |
|---|---:|---:|
| GTSinger Chinese (2 singers, 1.7 h) | 0.885 / 0.872 | 0.844 / 0.790 |
| GTSinger English, consistent singers (1.3 h) | 0.782 / 0.750 | 0.746 / 0.712 |
| M4Singer held-out songs (3.0 h) | **0.893** | 0.826 |
| Opencpop, all 3756 clips (5.2 h, never trained on) | 0.815 | **0.931** |

The Opencpop gap is a known limitation: this head still over-fires on
Opencpop's silent gaps (precision 0.69 at recall 0.99). Full account, including
the per-singer annotation probe:
`docs/experiments/nll_hubert_ls960_v1.md` in the toolkit.

### Usage

```python
from modules.api import HubertFATorchAligner

aligner = HubertFATorchAligner(ckpt="hfa-english-fullweak-v2", g2p="G2pEn",
                               ap_detector="nll")   # resolves to this head
```

Pairing it with a `chinese-hubert-base` aligner raises; the Mandarin line keeps
the upstream head.

## Training data, attribution and terms

All checkpoints are derived from corpora with their own terms. Attribution is
required by several of them; the ShareAlike terms are why this repository is
CC-BY-NC-SA-4.0 rather than CC-BY-NC-4.0.

| Corpus | Used by | Terms |
|---|---|---|
| [GTSinger](https://github.com/GTSinger/GTSinger) | all three | CC BY-NC-SA 4.0 — attribution, non-commercial, **ShareAlike** |
| [M4Singer](https://github.com/M4Singer/M4Singer) | Mandarin, breath head | CC BY-NC-SA 4.0; cite Zhang et al., *M4Singer*, NeurIPS 2022 |
| Opencpop | Mandarin | research corpus; cite its paper |
| [Project-AIdol](https://github.com/lottev1991) | English | CC BY-SA 4.0. Created by **Lotte V (@lottev1991)**. The dataset asks that it not be used with voice changers (RVC and similar), and that models featuring the voice not be publicly released without prior permission — this is an aligner, which predicts phone boundaries and cannot reproduce a voice. |
| ACV-001 | English | supplied as supplementary data; check the source for its current terms |
| NUS-48E | English | shared for **research purposes only**; cite Duan, Fang, Li, Sim and Wang, *The NUS Sung and Spoken Lyrics Corpus*, APSIPA ASC 2013 |
| CrawlSinger (zh / en) | both | scraped singing; the underlying recordings are third-party copyrighted works |

No audio from any of these corpora is redistributed here — only trained
weights.

## Caveats

- **`confidence` does not transfer across models.** SOFA scores land in
  ~[0.79, 1] and HubertFA in ~[0.67, 0.95], and the score is a top-2 pairwise
  margin. Any threshold inherited from another model has to be re-calibrated.
- **One language per checkpoint.** Each model's inventory and frozen encoder
  are language-specific; there is no multilingual checkpoint. The breath head
  is language-independent but **front-end-specific**.
- The weak-label half of both models' training data is scraped singing whose
  underlying recordings are third-party copyrighted works. The weights are
  released for **non-commercial research** for that reason, under
  **CC-BY-NC-SA-4.0** — ShareAlike because GTSinger, in both models' full-label
  data, carries it. The toolkit's source code is MIT and licensed separately.

## Citation

Upstream work this builds on: [SOFA](https://github.com/qiuqiao/SOFA) and
[HubertFA](https://github.com/wolfgitpr/HubertFA).