Model card: add paper benchmark results and paper link
Browse files
README.md
CHANGED
|
@@ -26,3 +26,70 @@ Alignment via Soft Dynamic Programming"* ([arXiv:2606.25460](https://arxiv.org/a
|
|
| 26 |
Each checkpoint is a PyTorch state dict with the model `hparams` and the dill-serialized
|
| 27 |
peak-detection parameters. Load them with the FALCON code (`predict.py` / `app.py`), which
|
| 28 |
auto-downloads from this repo when `HF_MODEL_REPO` is set.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
Each checkpoint is a PyTorch state dict with the model `hparams` and the dill-serialized
|
| 27 |
peak-detection parameters. Load them with the FALCON code (`predict.py` / `app.py`), which
|
| 28 |
auto-downloads from this repo when `HF_MODEL_REPO` is set.
|
| 29 |
+
|
| 30 |
+
---
|
| 31 |
+
|
| 32 |
+
<details>
|
| 33 |
+
<summary><b>Benchmark results</b> (from the paper — click to expand)</summary>
|
| 34 |
+
|
| 35 |
+
<br>
|
| 36 |
+
|
| 37 |
+
Accuracy = % of reference boundaries matched within the ms tolerance. **Specialist** = trained on the target English corpus; **joint** = `falcon_joint_multilingual.pt`; multilingual rows are zero-shot (no target-language training data). Bold = best in column-block.
|
| 38 |
+
|
| 39 |
+
**Phone-level — English**
|
| 40 |
+
|
| 41 |
+
| Dataset | Model | t≤10 | t≤25 | t≤50 | t≤100 |
|
| 42 |
+
|---|---|---|---|---|---|
|
| 43 |
+
| TIMIT | MFA | **38.6** | 72.3 | 81.1 | 84.6 |
|
| 44 |
+
| TIMIT | FALCON specialist | 37.66 | **83.88** | **94.85** | **98.62** |
|
| 45 |
+
| TIMIT | FALCON joint | 34.70 | 82.62 | 94.91 | 98.60 |
|
| 46 |
+
| Buckeye | MFA | **35.3** | 60.6 | 68.9 | 72.7 |
|
| 47 |
+
| Buckeye | FALCON specialist | 29.69 | **69.93** | **90.07** | **97.40** |
|
| 48 |
+
| Buckeye | FALCON joint | 28.87 | 69.40 | 89.53 | 97.13 |
|
| 49 |
+
|
| 50 |
+
**Phoneme-level — zero-shot multilingual**
|
| 51 |
+
|
| 52 |
+
| Test set | Model | ≤10 | ≤15 | ≤20 | ≤25 | ≤50 | ≤100 |
|
| 53 |
+
|---|---|---|---|---|---|---|---|
|
| 54 |
+
| Dutch — IFA | **FALCON joint** | **26.85** | **36.16** | **44.56** | **51.17** | **69.94** | **84.11** |
|
| 55 |
+
| Dutch — IFA | FALCON specialist | 26.86 | 35.79 | 43.85 | 50.34 | 68.68 | 83.22 |
|
| 56 |
+
| Dutch — IFA | MFA | 11.01 | 14.70 | 19.05 | 21.80 | 33.90 | 51.02 |
|
| 57 |
+
| German — PHONDAT | **FALCON joint** | **25.63** | **34.12** | **41.87** | **49.07** | **70.04** | **84.58** |
|
| 58 |
+
| German — PHONDAT | FALCON specialist | 25.08 | 33.37 | 40.76 | 47.43 | 68.27 | 82.44 |
|
| 59 |
+
| German — PHONDAT | MFA | 20.60 | 31.75 | 37.17 | 45.83 | 66.78 | 79.19 |
|
| 60 |
+
| Hebrew | **FALCON joint** | **21.98** | **30.10** | **36.91** | **42.78** | **63.07** | **80.41** |
|
| 61 |
+
| Hebrew | FALCON specialist | 21.03 | 27.78 | 34.30 | 39.79 | 59.38 | 77.76 |
|
| 62 |
+
|
| 63 |
+
**Word-level — English**
|
| 64 |
+
|
| 65 |
+
| Dataset | Model | t≤10 | t≤25 | t≤50 | t≤100 |
|
| 66 |
+
|---|---|---|---|---|---|
|
| 67 |
+
| TIMIT | FALCON spec (MFA-G2P) | 49.22 | **81.79** | **93.04** | 98.37 |
|
| 68 |
+
| TIMIT | FALCON joint (MFA-G2P) | **49.50** | 80.60 | 92.86 | **98.46** |
|
| 69 |
+
| TIMIT | MFA | 41.60 | 72.80 | 89.40 | 97.40 |
|
| 70 |
+
| TIMIT | MMS | 18.60 | 43.50 | 75.70 | 94.70 |
|
| 71 |
+
| TIMIT | WhisperX | 22.40 | 52.70 | 82.40 | 94.20 |
|
| 72 |
+
| TIMIT | Nvidia-Canary-1b | 9.23 | 23.11 | 44.23 | 72.81 |
|
| 73 |
+
| Buckeye | FALCON spec (MFA-G2P) | 50.06 | 77.85 | **91.51** | **96.63** |
|
| 74 |
+
| Buckeye | FALCON joint (MFA-G2P) | **50.42** | **77.98** | 91.01 | 96.55 |
|
| 75 |
+
| Buckeye | MFA | 39.80 | 69.90 | 84.90 | 91.80 |
|
| 76 |
+
| Buckeye | MMS | 25.00 | 52.70 | 75.00 | 87.90 |
|
| 77 |
+
| Buckeye | WhisperX | 18.80 | 43.10 | 67.40 | 77.40 |
|
| 78 |
+
| Buckeye | Nvidia-Canary-1b | 8.06 | 18.83 | 36.31 | 63.29 |
|
| 79 |
+
|
| 80 |
+
**Word-level — zero-shot multilingual**
|
| 81 |
+
|
| 82 |
+
| Dataset | Model | t≤10 | t≤25 | t≤50 | t≤100 |
|
| 83 |
+
|---|---|---|---|---|---|
|
| 84 |
+
| German — PHONDAT | FALCON (MFA-G2P) | **44.20** | **68.48** | **86.12** | **95.11** |
|
| 85 |
+
| German — PHONDAT | MFA | 29.9 | 65.4 | 82.1 | 94.3 |
|
| 86 |
+
| German — PHONDAT | MMS | 21.8 | 44.3 | 74.9 | 91.8 |
|
| 87 |
+
| Dutch — IFA | FALCON (MFA-G2P) | **26.38** | **45.15** | 61.16 | 76.49 |
|
| 88 |
+
| Dutch — IFA | MFA | 4.7 | 7.3 | 11.6 | 19.0 |
|
| 89 |
+
| Dutch — IFA | MMS | 16.0 | 37.9 | **62.9** | **76.6** |
|
| 90 |
+
| Hebrew | FALCON | **31.91** | **56.72** | 75.18 | 87.89 |
|
| 91 |
+
| Hebrew | MMS | 14.3 | 41.3 | **76.5** | **94.7** |
|
| 92 |
+
|
| 93 |
+
</details>
|
| 94 |
+
|
| 95 |
+
<sub><b>Paper:</b> <i>Fully Differentiable Neural Forced Alignment via Soft Dynamic Programming</i> — <a href="https://arxiv.org/abs/2606.25460">arXiv:2606.25460</a></sub>
|