tujia-ctc v0.3
Syllable-level speech recognition for Tujia (Northern dialect) with tone marks. Takes
16 kHz mono audio and returns a sequence of toned syllable labels with time spans, e.g.
la35, tsΚ°i53, ΙiΙ35.
- Architecture:
wav2vec2-xlsr-53-espeak-cv-ft(fine-tuned, encoder from runv03f)- LayerNorm + 2Γ BiLSTM(256) + Linear(512 β 1199). A 4-class tone auxiliary CTC head is used during training only.
- Labels: 1,198 toned syllables + blank; tones are
21 / 35 / 53 / 55 - Trainable parameters: 4,822,196 (head only, 19 MB); the fine-tuned encoder is distributed separately
- Training data: third-party Tujia recordings, 5,243 clips (audio and transcripts are not distributed)
- Best epoch: 8 (validation SER 0.5126 on a 141-clip split)
- Encoder: fine-tuned from
facebook/wav2vec2-xlsr-53-espeak-cv-ft(Apache-2.0)
What changed from v0.2
| v0.2 | v0.3 | |
|---|---|---|
| Labels | 708 toned syllables | 1,198 toned syllables |
| Held-out SER (353 clips) | 0.8627 | 0.7330 |
| Substitution rate | 0.5656 | 0.3057 |
| Deletion rate | 0.1941 | 0.3901 |
The inventory now covers every toned syllable the corpus supports instead of the 707 that survived the earlier merging, and the model substitutes far less. The price is that it deletes more: it emits fewer syllables than the reference on faint or fast speech. v0.2 leaned the other way (over-segmentation), so the two are not a straight upgrade for every use β if you need every syllable covered, run v0.2 alongside and compare.
The v0.3 encoder cannot be replaced by either the original XLSR-53 weights or the
v0.2 encoder: all three carry different features. Use TujiaNLP/tujia-xlsr53-v03f,
fetched by fetch_encoder.sh.
Package contents
tujia-ctc-0.3/
βββ model/
β βββ tujia-ctc-v0.3.pt head weights (includes vocab, head args, tones)
β βββ vocab.txt 1199 labels, one per line (blank is index 0)
β βββ model_card.json architecture, metrics, license
βββ infer.py standalone inference script
βββ fetch_encoder.sh downloads the fine-tuned encoder (~1.26 GB)
βββ requirements.txt
βββ SHA256SUMS
βββ MANIFEST.json
βββ NOTICE
βββ LICENSE MIT (code only)
Quick start
pip install -r requirements.txt
./fetch_encoder.sh ./models # Hugging Face by default
# mainland mirror: TUJIA_ENCODER_SOURCE=modelscope ./fetch_encoder.sh ./models
TUJIA_XLSR=./models/tujia-xlsr53-v03f python infer.py your_audio.wav
The encoder is hosted on Hugging Face (TujiaNLP/tujia-xlsr53-v03f) and mirrored on
ModelScope under the same id. It is not bundled here because of its size.
Mainland China. Hugging Face is normally unreachable without a proxy. Two options:
TUJIA_ENCODER_SOURCE=modelscope ./fetch_encoder.sh ./models # ModelScope mirror
HF_ENDPOINT=https://hf-mirror.com ./fetch_encoder.sh ./models # HF mirror
Output (one syllable per line):
0.00 0.42 xei35 0.912
0.66 1.04 tΚ°a21 0.884
1.04 1.42 Κ¨Κ°i55 0.803
--json out.json writes structured output; --start / --duration decode a single span;
--format ipa|ye changes the printed form.
Manual encoder setup (no fetch_encoder.sh)
--encoder and TUJIA_XLSR accept any directory containing a complete encoder
(config.json plus weights). models/tujia-xlsr53-v03f is only the default that
fetch_encoder.sh writes to, so an existing copy can be pointed at directly without
renaming it.
fetch_encoder.sh is a POSIX shell script and does not run on Windows. On Windows, copy
the encoder folder yourself and set the variable:
xcopy /E /I D:\somewhere\encoder_best models\tujia-xlsr53-v03f
set TUJIA_XLSR=models\tujia-xlsr53-v03f
python infer.py your_audio.wav --device cpu
PowerShell: $env:TUJIA_XLSR="models\tujia-xlsr53-v03f".
Input formats
infer.py reads audio with soundfile, which handles wav, flac and ogg. Container files
such as mp4 or m4a must be demuxed first:
ffmpeg -i video.mp4 -vn -ac 1 -ar 16000 audio.wav
Metrics
Scored as syllable error rate (SER): token sequences are aligned with NeedlemanβWunsch
and the edit distance is divided by the reference length. Tone error is computed the same
way over the tone subsequence only. Greedy CTC, no lexicon and no n-gram rescoring.
SER is an error rate β lower is better, and 1 β SER approximates syllable accuracy
(0.7330 = about a quarter of the syllables right).
The 141-clip validation split was used for early stopping, so 0.5126 is not an independent number. The held-out comparison below uses the 353 clips that appear in neither v0.3's training nor its validation split (0.40 h, 8 speakers, 300 of them one male teacher). v0.2's 708-label predictions are folded to the 0.6.3 inventory before scoring.
| Metric | v0.2 | v0.3 |
|---|---|---|
| SER | 0.8627 | 0.7330 |
| SER, no tone | 0.7034 | 0.6457 |
| Tone SER | 0.6228 | 0.5832 |
| Substitution rate | 0.5656 | 0.3057 |
| Deletion rate | 0.1941 | 0.3901 |
| Insertion rate | 0.1030 | 0.0372 |
The 856-clip test split used for v0.2 cannot be reused here: 851 of its clips are in v0.3's training set.
Known limitations
- Under-generation. Deletion rate 0.39 against 0.19 for v0.2. Quiet, fast or
far-field speech loses syllables instead of producing wrong ones; check durations
against the audio before trusting a short transcript.
tujia-asr1.0.1 exposes--blank-bias/TUJIA_BLANK_BIASto push back: 1.0 moves ~10 points of deletion into substitution and restores output length from 0.62 to 0.75 of the reference, at a 0.6-point SER cost. The demo Space runs at 1.0. - In-domain data only. Few speakers and one collection source. Other dialect points are likely to degrade.
- Tone errors are the dominant residual. About one tone in three is wrong on the
held-out set even though the label carries the tone; the confusion is spread across
21 / 35 / 53 / 55. - Evaluation is teacher-heavy. 300 of the 353 held-out clips come from one male teacher, so 0.7330 is closer to an in-domain number than a cross-domain one.
Data and license
- Code: MIT (see LICENSE).
- Model weights: research use only. No rights to the underlying recordings or transcripts are granted.
- The training corpus is not distributed with this package.
- Upstream encoder
facebook/wav2vec2-xlsr-53-espeak-cv-ftis Apache-2.0 and must retain its attribution. The fine-tuned encoder weights are research-use only. - Parent project: https://github.com/ibaibing/TujiaNLP (field collection, annotation standards, orthography conversion; corpus terms are recorded there)