tujia-ctc v0.3

Syllable-level speech recognition for Tujia (Northern dialect) with tone marks. Takes 16 kHz mono audio and returns a sequence of toned syllable labels with time spans, e.g. la35, tsΚ°i53, Ι•iΙ›35.

  • Architecture: wav2vec2-xlsr-53-espeak-cv-ft (fine-tuned, encoder from run v03f)
    • LayerNorm + 2Γ— BiLSTM(256) + Linear(512 β†’ 1199). A 4-class tone auxiliary CTC head is used during training only.
  • Labels: 1,198 toned syllables + blank; tones are 21 / 35 / 53 / 55
  • Trainable parameters: 4,822,196 (head only, 19 MB); the fine-tuned encoder is distributed separately
  • Training data: third-party Tujia recordings, 5,243 clips (audio and transcripts are not distributed)
  • Best epoch: 8 (validation SER 0.5126 on a 141-clip split)
  • Encoder: fine-tuned from facebook/wav2vec2-xlsr-53-espeak-cv-ft (Apache-2.0)

What changed from v0.2

v0.2 v0.3
Labels 708 toned syllables 1,198 toned syllables
Held-out SER (353 clips) 0.8627 0.7330
Substitution rate 0.5656 0.3057
Deletion rate 0.1941 0.3901

The inventory now covers every toned syllable the corpus supports instead of the 707 that survived the earlier merging, and the model substitutes far less. The price is that it deletes more: it emits fewer syllables than the reference on faint or fast speech. v0.2 leaned the other way (over-segmentation), so the two are not a straight upgrade for every use β€” if you need every syllable covered, run v0.2 alongside and compare.

The v0.3 encoder cannot be replaced by either the original XLSR-53 weights or the v0.2 encoder: all three carry different features. Use TujiaNLP/tujia-xlsr53-v03f, fetched by fetch_encoder.sh.

Package contents

tujia-ctc-0.3/
β”œβ”€β”€ model/
β”‚   β”œβ”€β”€ tujia-ctc-v0.3.pt    head weights (includes vocab, head args, tones)
β”‚   β”œβ”€β”€ vocab.txt            1199 labels, one per line (blank is index 0)
β”‚   └── model_card.json      architecture, metrics, license
β”œβ”€β”€ infer.py                 standalone inference script
β”œβ”€β”€ fetch_encoder.sh         downloads the fine-tuned encoder (~1.26 GB)
β”œβ”€β”€ requirements.txt
β”œβ”€β”€ SHA256SUMS
β”œβ”€β”€ MANIFEST.json
β”œβ”€β”€ NOTICE
└── LICENSE                  MIT (code only)

Quick start

pip install -r requirements.txt
./fetch_encoder.sh ./models            # Hugging Face by default
# mainland mirror: TUJIA_ENCODER_SOURCE=modelscope ./fetch_encoder.sh ./models
TUJIA_XLSR=./models/tujia-xlsr53-v03f python infer.py your_audio.wav

The encoder is hosted on Hugging Face (TujiaNLP/tujia-xlsr53-v03f) and mirrored on ModelScope under the same id. It is not bundled here because of its size.

Mainland China. Hugging Face is normally unreachable without a proxy. Two options:

TUJIA_ENCODER_SOURCE=modelscope ./fetch_encoder.sh ./models   # ModelScope mirror
HF_ENDPOINT=https://hf-mirror.com ./fetch_encoder.sh ./models # HF mirror

Output (one syllable per line):

    0.00     0.42  xei35       0.912
    0.66     1.04  tΚ°a21       0.884
    1.04     1.42  Κ¨Κ°i55       0.803

--json out.json writes structured output; --start / --duration decode a single span; --format ipa|ye changes the printed form.

Manual encoder setup (no fetch_encoder.sh)

--encoder and TUJIA_XLSR accept any directory containing a complete encoder (config.json plus weights). models/tujia-xlsr53-v03f is only the default that fetch_encoder.sh writes to, so an existing copy can be pointed at directly without renaming it.

fetch_encoder.sh is a POSIX shell script and does not run on Windows. On Windows, copy the encoder folder yourself and set the variable:

xcopy /E /I D:\somewhere\encoder_best models\tujia-xlsr53-v03f
set TUJIA_XLSR=models\tujia-xlsr53-v03f
python infer.py your_audio.wav --device cpu

PowerShell: $env:TUJIA_XLSR="models\tujia-xlsr53-v03f".

Input formats

infer.py reads audio with soundfile, which handles wav, flac and ogg. Container files such as mp4 or m4a must be demuxed first:

ffmpeg -i video.mp4 -vn -ac 1 -ar 16000 audio.wav

Metrics

Scored as syllable error rate (SER): token sequences are aligned with Needleman–Wunsch and the edit distance is divided by the reference length. Tone error is computed the same way over the tone subsequence only. Greedy CTC, no lexicon and no n-gram rescoring. SER is an error rate β€” lower is better, and 1 βˆ’ SER approximates syllable accuracy (0.7330 = about a quarter of the syllables right).

The 141-clip validation split was used for early stopping, so 0.5126 is not an independent number. The held-out comparison below uses the 353 clips that appear in neither v0.3's training nor its validation split (0.40 h, 8 speakers, 300 of them one male teacher). v0.2's 708-label predictions are folded to the 0.6.3 inventory before scoring.

Metric v0.2 v0.3
SER 0.8627 0.7330
SER, no tone 0.7034 0.6457
Tone SER 0.6228 0.5832
Substitution rate 0.5656 0.3057
Deletion rate 0.1941 0.3901
Insertion rate 0.1030 0.0372

The 856-clip test split used for v0.2 cannot be reused here: 851 of its clips are in v0.3's training set.

Known limitations

  1. Under-generation. Deletion rate 0.39 against 0.19 for v0.2. Quiet, fast or far-field speech loses syllables instead of producing wrong ones; check durations against the audio before trusting a short transcript. tujia-asr 1.0.1 exposes --blank-bias / TUJIA_BLANK_BIAS to push back: 1.0 moves ~10 points of deletion into substitution and restores output length from 0.62 to 0.75 of the reference, at a 0.6-point SER cost. The demo Space runs at 1.0.
  2. In-domain data only. Few speakers and one collection source. Other dialect points are likely to degrade.
  3. Tone errors are the dominant residual. About one tone in three is wrong on the held-out set even though the label carries the tone; the confusion is spread across 21 / 35 / 53 / 55.
  4. Evaluation is teacher-heavy. 300 of the 353 held-out clips come from one male teacher, so 0.7330 is closer to an in-domain number than a cross-domain one.

Data and license

  • Code: MIT (see LICENSE).
  • Model weights: research use only. No rights to the underlying recordings or transcripts are granted.
  • The training corpus is not distributed with this package.
  • Upstream encoder facebook/wav2vec2-xlsr-53-espeak-cv-ft is Apache-2.0 and must retain its attribution. The fine-tuned encoder weights are research-use only.
  • Parent project: https://github.com/ibaibing/TujiaNLP (field collection, annotation standards, orthography conversion; corpus terms are recorded there)
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support