File size: 3,417 Bytes
b1e227a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 | ---
license: agpl-3.0
library_name: pytorch
language:
- ar
- ckb
- fa
- sd
- ur
datasets:
- google/fleurs
tags:
- forced-alignment
- phoneme-recognition
- phoneme-segmentation
- speech
- multilingual
---
# mbfa-abjad (p4mbfa · abjad)
A context-free CNN phoneme encoder and forced aligner (p4mbfa, the successor of CUPE / p3cupe),
for the **abjad** language group of [standard_g2p](https://github.com/tabahi/CharsiuG2P). Each 5 ms frame is classified
from at most 120 ms of audio (38.9 ms receptive field), so the model cannot learn any language's
phonotactics. Phone boundaries come from a segmental Viterbi decoder over the known phone
sequence, refined to sub-frame precision at the crossing of neighbouring posteriors.
| head | labels | size |
|---|---|---|
| `ph` | local tokens of `abjad` (incl. `<blank> SIL noise <unk>`) | 90 |
| `phg` | gold phoneme groups, shared by every language group | 15 |
| `tone` | tone values, shared by every language group; present but untrained (no language in this group has a tone layer) | 22 |
Languages (the FLEURS languages it was trained on): ar, ckb, fa, sd, ur. standard_g2p maps every member of the group
onto the same tokens, so other members may align too, untested.
## Training
**Data:** FLEURS abjad, all 38.2 h, training noise_level 0.02.
**Checkpoint:** experiment `mp01a`, epoch 5, selected on FLEURS val_loss (best of 20 epochs; final epoch 2.988).
Trunk and phg_head from latin ma02a (final epoch), ph_head and tone_head trained fresh.
| metric | value |
|---|---|
| `val_loss` | 2.8549 |
| `val_frame_acc` | 0.4126 |
| `val_frame_acc_groups` | 0.5149 |
`val_*` are on held-out FLEURS clips but are scored against the model's own alignments (there are no boundary labels outside English), so they measure self-consistency, not boundary accuracy.
Labels are standard_g2p dictionary pronunciations (gold inventory `9438371ed6dd`),
not phonetic transcriptions of what was said.
## Usage
The code is in [https://github.com/tabahi/bfa_models](https://github.com/tabahi/bfa_models) (`p4mbfa/`); clone it and run from its root.
```python
from p4mbfa.inference import MbfaAligner
aligner = MbfaAligner.from_pretrained("Tabahi/mbfa-abjad")
wav = aligner.load_audio("clip.wav") # mono, 16000 Hz
phones = ["SIL", "h", "ɛ", "l", "o", "SIL"] # this group's tokens (config.json labels.tokens)
for seg in aligner.align(wav, phones):
print(seg["token"], seg["start_ms"], seg["end_ms"])
```
`aligner.encode(wav)` returns the per-frame log posteriors of all three heads.
Text -> tokens is standard_g2p's job (`goldG2P.phonemize_sentence` then
`lang_group_inventory.to_local`); `config.json` lists the token strings. `from_pretrained`
downloads only `config.json` and `model.safetensors`.
## Fine-tuning
`ckpt/abjad_fleurs38h_mp01a_e5_val_loss=2.855.ckpt` is the training checkpoint (PyTorch Lightning, weights only, a pickle: load
it only if you trust this repo). Download it, then point a p4mbfa yaml at it to continue
training, or to train a new language group on its trunk and shared `phg` head:
```bash
hf download Tabahi/mbfa-abjad "ckpt/abjad_fleurs38h_mp01a_e5_val_loss=2.855.ckpt" --local-dir tmp/hf/mbfa-abjad
```
```yaml
ckpt_path: "tmp/hf/mbfa-abjad/ckpt/abjad_fleurs38h_mp01a_e5_val_loss=2.855.ckpt"
reset_fine_heads: true # new lang_group: rebuild ph_head / tone_head; false = same group
```
License: AGPL-3.0.
|