--- license: cc-by-nc-sa-4.0 language: - zh - en tags: - forced-alignment - singing-voice - phoneme-alignment - breath-detection - hubert library_name: singalign pipeline_tag: audio-to-audio --- # SingAlign — released checkpoints Weights for [**SingAlign**](https://github.com/pymaster17/SingAlign), a unified forced-alignment toolkit for singing voice. The code lives in the GitHub repository; this repository holds only the checkpoints it downloads. Files here are named exactly as `configs/models/.yaml` declares them, so the toolkit's resolver (`modules/common/api/checkpoints.py`) fetches them by name and verifies the pinned SHA256 before it puts anything on disk. | File | Family | Size | SHA256 | |---|---|---:|---| | `hfa-mandarin-fullweak-v1.ckpt` | HubertFA | 54.2 MB | `d38f7d5839e2cd93e8dec5b77b6bd3ed28aa1318d5ed069c2cd81c899dd3c057` | | `hfa-english-fullweak-v2.ckpt` | HubertFA | 54.1 MB | `878de0899cb188c05537934a6f55bfd7aaabcbde1c411aba30d1e6f50a995712` | | `hfa-english-fullweak-v1.ckpt` | HubertFA (superseded by v2) | 54.1 MB | `c1b2baa939a87b3ed832c97e18eba9fcb27214b39f50ff346c062601bd11c513` | | `nll-hubert-ls960-v1.pt` | breath head (NLL) for the English aligner | 9.9 MB | `03587e7a476aea7f137904e6034420d0a4b255d12949e85ec16b20077ae29ce8` | ## `hfa-mandarin-fullweak-v1` A HubertFA alignment head over a **frozen `chinese-hubert-base` front-end**, trained in-repo with `train_hfa.py`. It is SingAlign's default model. - **Recipe**: `configs/hfa/train_fullweak_v1.yaml`, data `configs/hfa/binarize_fullweak.yaml`. - **Training data**: 51.4 h full-label (GTSinger, M4Singer, Opencpop) + 7627 h weak-label Mandarin singing (lyrics only, no phoneme durations). - **Step 15000**, selected by CBER on a held-out validation set — *not* the final step-60000 weights, whose validation curve had already degraded. - **Phoneme inventory**: 117 phones, `sofa-expanded-v1` profile (identity against the toolkit's canonical Mandarin inventory). - **G2P**: G2pW — no pronunciation dictionary to configure. ### Results Scored on `cloudtest-verified-v1`: 8 recently released original songs (90 clips, 1971 phones, 535.6 s) with hand-verified `word` + `ph` tiers, none of whose song IDs or "artist – title" pairs appear in any training manifest. Every system was fed the **same gold phoneme sequence** with G2P and breath detection off, so this measures acoustic alignment only. | System | VER20 ↓ | VER50 ↓ | mIoU ↑ | CBER ↓ | |---|---:|---:|---:|---:| | **`hfa-mandarin-fullweak-v1`** | 0.2265 | **0.0571** | **0.8030** | **0.1737** | | HubertFA v0.0.7 (upstream ONNX) | **0.2075** | 0.0786 | 0.7734 | 0.2223 | | SOFA ConvNeXt (in-repo) | 0.2519 | 0.0835 | 0.7747 | 0.2159 | | SOFA `pretrained_mandarin_singing` | 0.2575 | 0.0946 | 0.7501 | 0.2901 | | STARS (Chinese) | 0.3379 | 0.1674 | 0.6072 | 0.6547 | VER20/VER50 are vlabeler edit ratios at 20 ms / 50 ms boundary tolerance and include `SP`; mIoU and CBER exclude it. Point estimates, no confidence intervals. **Read this as an error profile, not a ranking.** This model leads on VER50, mIoU and CBER but trails upstream v0.0.7 by ~9% on VER20: it makes *fewer large errors and more small ones*. If your criterion is a 20 ms tolerance, the upstream model is the better pick. ### Runtime dependency The torch path does **not** carry its own SSL front-end. The checkpoint records `hubert_config.model_path`, and those encoder weights ([`TencentGameMate/chinese-hubert-base`](https://huggingface.co/TencentGameMate/chinese-hubert-base), ~380 MB) must be present. `python scripts/download_assets.py` in the toolkit fetches this checkpoint, the encoder and the G2pW model together. ### Usage ```bash git clone https://github.com/pymaster17/SingAlign && cd SingAlign uv sync --extra pitch && source .venv/bin/activate python scripts/download_assets.py # this checkpoint + encoder + G2pW python infer_one.py -a audio.wav -t "中文歌词" --out_formats textgrid,json ``` ```python from modules.api import HubertFATorchAligner aligner = HubertFATorchAligner() # resolves to this checkpoint results = aligner.align([{"id": "x", "audio_path": "a.wav", "text": "中文歌词"}]) ``` ## `hfa-english-fullweak-v2` The English model: the same alignment head over a **frozen `facebook/hubert-base-ls960` front-end**, trained with the recipe of `hfa-english-fullweak-v1` below on data whose only change is *where the clips start*. - **Recipe**: `configs/hfa/train_en_fullweak_v6.yaml` (identical to v1's except the run name), data `configs/hfa/binarize_en_fullweak_v6.yaml`. - **Training data**: 21.4 h full-label across 13 singers (as v1; Project-AIdol and NUS-48E re-cut so that a silence goes whole to one neighbouring segment instead of being split) + 1845 h weak-label English singing selected by `singmos >= 3.5` only, so the corpus's silence-cut `sub` segments (37 %) are in the pool alongside the lyric-timed `short` ones. - **Step 56000**, selected by CBER on held-out singers. - **Phoneme inventory / G2P**: as v1 (`english-arpabet-v1`, G2pEn); the vocabulary file is byte-identical, so the two are drop-in for each other. ### Why v2 exists v1 puts a spurious sub-millisecond `SP` at the head of about a third of clips that start on the voice. The cause was traced to its weak-label pool: a `wer` filter silently restricted it to lyric-timed segments, three quarters of which start mid-voice, and CTC training on those teaches the shared phone channels to read onset frames as "not yet a phone". Mixing in silence-cut segments removes it: **32 % → 5.3 %** of voice-initial GTSinger-en clips (`bench/tools/probe_leading_sp.py`; `docs/experiments/leading_sp_probe.md`). ### Results Same four held-out NUS-48E singers as v1 (201 clips, 0.53 h): | System | CBER ↓ | VER20 ↓ | VER50 ↓ | mIoU ↑ | |---|---:|---:|---:|---:| | **`hfa-english-fullweak-v2`** | 0.1951 | 0.1327 | 0.0634 | 0.7823 | | `hfa-english-fullweak-v1` | 0.1975 | 0.1318 | 0.0616 | 0.7821 | | Same recipe, full label only (21.4 h) | **0.1940** | **0.1300** | 0.0617 | **0.7833** | A tie with v1 on alignment quality; the caveat below about the weak label buying nothing measurable on clean studio audio still applies. ### Batched inference **is** bit-reproducible Unlike v1, this checkpoint **passes** the toolkit's batch-invariance gate: `bench/tools/validate_hfa_batch.py` on the same 48 items, batch 8, measures a maximum boundary shift of **0.002 ms** against a 1.0 ms tolerance. ### Runtime dependency and usage As v1: [`facebook/hubert-base-ls960`](https://huggingface.co/facebook/hubert-base-ls960) must be present, and the model is named explicitly: ```python from modules.api import HubertFATorchAligner aligner = HubertFATorchAligner(ckpt="hfa-english-fullweak-v2", g2p="G2pEn") results = aligner.align([{"id": "x", "audio_path": "a.wav", "text": "english lyrics"}]) ``` ## `hfa-english-fullweak-v1` (superseded) Kept because the `CrawlSinger-en` corpus's `align_conf` column was written by it; use `hfa-english-fullweak-v2` for new work. The same alignment head over a **frozen `facebook/hubert-base-ls960` front-end** — same HuBERT-base recipe as the Mandarin model, so only the encoder path and the phoneme inventory differ. - **Recipe**: `configs/hfa/train_en_fullweak.yaml`, data `configs/hfa/binarize_en_fullweak.yaml`. - **Training data**: 21.4 h full-label across **13 singers** (GTSinger-en, ACV-001, Project-AIdol, 8 of NUS-48E's 12 singers) + 1845 h weak-label English singing (lyrics only, no phoneme durations). - **Step 54000**, selected by CBER on held-out singers — *not* the final step-60000 weights. - **Phoneme inventory**: the 39 CMUdict ARPAbet phones (stress stripped) plus `SP`, `english-arpabet-v1` profile. - **G2P**: G2pEn (CMUdict + a seq2seq for OOV) — no dictionary to configure. ### Results Scored on **four held-out NUS-48E singers** (201 clips, 0.53 h) — two from each mode of the corpus's F0 distribution, none of whom appears in training. The same suite selected the checkpoint, so read these as in-suite numbers. | System | CBER ↓ | VER20 ↓ | VER50 ↓ | mIoU ↑ | |---|---:|---:|---:|---:| | `hfa-english-fullweak-v1` | 0.1975 | 0.1318 | **0.0616** | 0.7821 | | Same recipe, full label only (21.4 h) | **0.1940** | **0.1300** | 0.0617 | **0.7833** | **The weak label bought nothing measurable here.** The two models tie, and the gap is smaller than this run's own validation-point scatter. It is released as the default English model on the grounds that the Mandarin line measured weak label buying robustness on *rough* recordings — but English has no rough-domain held-out set to check that on, because the validation set is studio singing while the weak label is crawled audio. If your material is clean, expect nothing from the extra 1845 h. The full account is in `docs/experiments/hfa_en_full_weak_v1.md` in the toolkit. ### Batched inference is not bit-reproducible This checkpoint **does not pass** the toolkit's batch-invariance gate: `bench/tools/validate_hfa_batch.py` measures a **10.5 ms** maximum boundary shift between batch-1 and batch-8 against a 1.0 ms tolerance. Scale: one boundary of one clip in 48, between content phones (`SH | IY`) rather than on an `SP` edge, so an adapter does not drop it. Batching changes the floating-point reduction order in the linear layers, and weak-label training leaves the logits flat enough at an acoustically ambiguous boundary that the last-bit difference moves the Viterbi path. A deterministic tie-break cannot fix it — the inputs genuinely differ. Set `max_batch_size=1` if you need reproducibility. For comparison, the full-label-only model measures 0.003 ms on the same items. ### Runtime dependency [`facebook/hubert-base-ls960`](https://huggingface.co/facebook/hubert-base-ls960), **not** the Mandarin encoder — the two are not interchangeable. ### Usage ```python from modules.api import HubertFATorchAligner aligner = HubertFATorchAligner(ckpt="hfa-english-fullweak-v1", g2p="G2pEn") results = aligner.align([{"id": "x", "audio_path": "a.wav", "text": "english lyrics"}]) ``` ## `nll-hubert-ls960-v1` A **breath (non-lexical) head**, not an aligner: a 2.45 M-parameter CVNT that labels each 10 ms frame `None` / `AP` from the HuBERT units the aligner has already computed, and whose breaths are spliced into the finished alignment as `AP` without touching any other boundary. It rides on `hfa-english-fullweak-v2` (or v1) as `ap_detector="nll"` and is selected automatically for any HubertFA aligner whose front-end is `hubert-base-ls960`. - **Why it exists**: a breath head is bound to the SSL front-end it was trained on. The head shipped inside upstream HubertFA's ONNX bundle was trained on `chinese-hubert-base` units; run over `hubert-base-ls960` units (same 768 width, unrelated latent space) it recalls **0.001** of the breaths in held-out English singing -- nothing. The toolkit now refuses that pairing. - **Recipe**: `configs/nll/train_v2.yaml`, data `configs/nll/binarize_v2.yaml`; class-weighted CE + focal + dice, 15000 steps, **step 7500** selected on frame-F1 over the annotation-consistent held-out groups. - **Training data (59.3 h)**: GTSinger singers whose breath annotation agrees with the upstream head's convention (ZH ×2, EN-Alto-2, EN-Tenor-1, IT ×3, ES-Soprano-1; 32.7 h) + M4Singer (26.7 h). Five GTSinger languages label the whole inter-phrase gap as breath; a head trained on all of GTSinger learns "gap ⇒ breath" and its precision collapses on any other corpus. - **Binary**: `None` / `AP`. No tail-breath (`EP`) class. ### Results Frame-F1 at threshold 0.5, scored only inside the gold non-content regions (where a breath could be), on songs held out by title: | Held-out set | This head | Upstream head on its own front-end | |---|---:|---:| | GTSinger Chinese (2 singers, 1.7 h) | 0.885 / 0.872 | 0.844 / 0.790 | | GTSinger English, consistent singers (1.3 h) | 0.782 / 0.750 | 0.746 / 0.712 | | M4Singer held-out songs (3.0 h) | **0.893** | 0.826 | | Opencpop, all 3756 clips (5.2 h, never trained on) | 0.815 | **0.931** | The Opencpop gap is a known limitation: this head still over-fires on Opencpop's silent gaps (precision 0.69 at recall 0.99). Full account, including the per-singer annotation probe: `docs/experiments/nll_hubert_ls960_v1.md` in the toolkit. ### Usage ```python from modules.api import HubertFATorchAligner aligner = HubertFATorchAligner(ckpt="hfa-english-fullweak-v2", g2p="G2pEn", ap_detector="nll") # resolves to this head ``` Pairing it with a `chinese-hubert-base` aligner raises; the Mandarin line keeps the upstream head. ## Training data, attribution and terms All checkpoints are derived from corpora with their own terms. Attribution is required by several of them; the ShareAlike terms are why this repository is CC-BY-NC-SA-4.0 rather than CC-BY-NC-4.0. | Corpus | Used by | Terms | |---|---|---| | [GTSinger](https://github.com/GTSinger/GTSinger) | all three | CC BY-NC-SA 4.0 — attribution, non-commercial, **ShareAlike** | | [M4Singer](https://github.com/M4Singer/M4Singer) | Mandarin, breath head | CC BY-NC-SA 4.0; cite Zhang et al., *M4Singer*, NeurIPS 2022 | | Opencpop | Mandarin | research corpus; cite its paper | | [Project-AIdol](https://github.com/lottev1991) | English | CC BY-SA 4.0. Created by **Lotte V (@lottev1991)**. The dataset asks that it not be used with voice changers (RVC and similar), and that models featuring the voice not be publicly released without prior permission — this is an aligner, which predicts phone boundaries and cannot reproduce a voice. | | ACV-001 | English | supplied as supplementary data; check the source for its current terms | | NUS-48E | English | shared for **research purposes only**; cite Duan, Fang, Li, Sim and Wang, *The NUS Sung and Spoken Lyrics Corpus*, APSIPA ASC 2013 | | CrawlSinger (zh / en) | both | scraped singing; the underlying recordings are third-party copyrighted works | No audio from any of these corpora is redistributed here — only trained weights. ## Caveats - **`confidence` does not transfer across models.** SOFA scores land in ~[0.79, 1] and HubertFA in ~[0.67, 0.95], and the score is a top-2 pairwise margin. Any threshold inherited from another model has to be re-calibrated. - **One language per checkpoint.** Each model's inventory and frozen encoder are language-specific; there is no multilingual checkpoint. The breath head is language-independent but **front-end-specific**. - The weak-label half of both models' training data is scraped singing whose underlying recordings are third-party copyrighted works. The weights are released for **non-commercial research** for that reason, under **CC-BY-NC-SA-4.0** — ShareAlike because GTSinger, in both models' full-label data, carries it. The toolkit's source code is MIT and licensed separately. ## Citation Upstream work this builds on: [SOFA](https://github.com/qiuqiao/SOFA) and [HubertFA](https://github.com/wolfgitpr/HubertFA).