MLSpeech commited on
Commit
b1b29a1
·
verified ·
1 Parent(s): 2737a77

Model card: add paper benchmark results and paper link

Browse files
Files changed (1) hide show
  1. README.md +67 -0
README.md CHANGED
@@ -26,3 +26,70 @@ Alignment via Soft Dynamic Programming"* ([arXiv:2606.25460](https://arxiv.org/a
26
  Each checkpoint is a PyTorch state dict with the model `hparams` and the dill-serialized
27
  peak-detection parameters. Load them with the FALCON code (`predict.py` / `app.py`), which
28
  auto-downloads from this repo when `HF_MODEL_REPO` is set.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
26
  Each checkpoint is a PyTorch state dict with the model `hparams` and the dill-serialized
27
  peak-detection parameters. Load them with the FALCON code (`predict.py` / `app.py`), which
28
  auto-downloads from this repo when `HF_MODEL_REPO` is set.
29
+
30
+ ---
31
+
32
+ <details>
33
+ <summary><b>Benchmark results</b> (from the paper — click to expand)</summary>
34
+
35
+ <br>
36
+
37
+ Accuracy = % of reference boundaries matched within the ms tolerance. **Specialist** = trained on the target English corpus; **joint** = `falcon_joint_multilingual.pt`; multilingual rows are zero-shot (no target-language training data). Bold = best in column-block.
38
+
39
+ **Phone-level — English**
40
+
41
+ | Dataset | Model | t≤10 | t≤25 | t≤50 | t≤100 |
42
+ |---|---|---|---|---|---|
43
+ | TIMIT | MFA | **38.6** | 72.3 | 81.1 | 84.6 |
44
+ | TIMIT | FALCON specialist | 37.66 | **83.88** | **94.85** | **98.62** |
45
+ | TIMIT | FALCON joint | 34.70 | 82.62 | 94.91 | 98.60 |
46
+ | Buckeye | MFA | **35.3** | 60.6 | 68.9 | 72.7 |
47
+ | Buckeye | FALCON specialist | 29.69 | **69.93** | **90.07** | **97.40** |
48
+ | Buckeye | FALCON joint | 28.87 | 69.40 | 89.53 | 97.13 |
49
+
50
+ **Phoneme-level — zero-shot multilingual**
51
+
52
+ | Test set | Model | ≤10 | ≤15 | ≤20 | ≤25 | ≤50 | ≤100 |
53
+ |---|---|---|---|---|---|---|---|
54
+ | Dutch — IFA | **FALCON joint** | **26.85** | **36.16** | **44.56** | **51.17** | **69.94** | **84.11** |
55
+ | Dutch — IFA | FALCON specialist | 26.86 | 35.79 | 43.85 | 50.34 | 68.68 | 83.22 |
56
+ | Dutch — IFA | MFA | 11.01 | 14.70 | 19.05 | 21.80 | 33.90 | 51.02 |
57
+ | German — PHONDAT | **FALCON joint** | **25.63** | **34.12** | **41.87** | **49.07** | **70.04** | **84.58** |
58
+ | German — PHONDAT | FALCON specialist | 25.08 | 33.37 | 40.76 | 47.43 | 68.27 | 82.44 |
59
+ | German — PHONDAT | MFA | 20.60 | 31.75 | 37.17 | 45.83 | 66.78 | 79.19 |
60
+ | Hebrew | **FALCON joint** | **21.98** | **30.10** | **36.91** | **42.78** | **63.07** | **80.41** |
61
+ | Hebrew | FALCON specialist | 21.03 | 27.78 | 34.30 | 39.79 | 59.38 | 77.76 |
62
+
63
+ **Word-level — English**
64
+
65
+ | Dataset | Model | t≤10 | t≤25 | t≤50 | t≤100 |
66
+ |---|---|---|---|---|---|
67
+ | TIMIT | FALCON spec (MFA-G2P) | 49.22 | **81.79** | **93.04** | 98.37 |
68
+ | TIMIT | FALCON joint (MFA-G2P) | **49.50** | 80.60 | 92.86 | **98.46** |
69
+ | TIMIT | MFA | 41.60 | 72.80 | 89.40 | 97.40 |
70
+ | TIMIT | MMS | 18.60 | 43.50 | 75.70 | 94.70 |
71
+ | TIMIT | WhisperX | 22.40 | 52.70 | 82.40 | 94.20 |
72
+ | TIMIT | Nvidia-Canary-1b | 9.23 | 23.11 | 44.23 | 72.81 |
73
+ | Buckeye | FALCON spec (MFA-G2P) | 50.06 | 77.85 | **91.51** | **96.63** |
74
+ | Buckeye | FALCON joint (MFA-G2P) | **50.42** | **77.98** | 91.01 | 96.55 |
75
+ | Buckeye | MFA | 39.80 | 69.90 | 84.90 | 91.80 |
76
+ | Buckeye | MMS | 25.00 | 52.70 | 75.00 | 87.90 |
77
+ | Buckeye | WhisperX | 18.80 | 43.10 | 67.40 | 77.40 |
78
+ | Buckeye | Nvidia-Canary-1b | 8.06 | 18.83 | 36.31 | 63.29 |
79
+
80
+ **Word-level — zero-shot multilingual**
81
+
82
+ | Dataset | Model | t≤10 | t≤25 | t≤50 | t≤100 |
83
+ |---|---|---|---|---|---|
84
+ | German — PHONDAT | FALCON (MFA-G2P) | **44.20** | **68.48** | **86.12** | **95.11** |
85
+ | German — PHONDAT | MFA | 29.9 | 65.4 | 82.1 | 94.3 |
86
+ | German — PHONDAT | MMS | 21.8 | 44.3 | 74.9 | 91.8 |
87
+ | Dutch — IFA | FALCON (MFA-G2P) | **26.38** | **45.15** | 61.16 | 76.49 |
88
+ | Dutch — IFA | MFA | 4.7 | 7.3 | 11.6 | 19.0 |
89
+ | Dutch — IFA | MMS | 16.0 | 37.9 | **62.9** | **76.6** |
90
+ | Hebrew | FALCON | **31.91** | **56.72** | 75.18 | 87.89 |
91
+ | Hebrew | MMS | 14.3 | 41.3 | **76.5** | **94.7** |
92
+
93
+ </details>
94
+
95
+ <sub><b>Paper:</b> <i>Fully Differentiable Neural Forced Alignment via Soft Dynamic Programming</i> — <a href="https://arxiv.org/abs/2606.25460">arXiv:2606.25460</a></sub>