Nepali-Swar-Experimental

A 16.1 M parameter semantic language model for Nepali and English speech. Swar (स्वर) is Nepali for voice, sound, or musical note.

Experimental. A research checkpoint, not a production system.

What this is, and what it is not

This is one component of a TTS system, not a complete one. It converts text into CosyVoice3 speech tokens. It does not produce audio by itself.

text  ->  [ Nepali-Swar, 16.1 M ]  ->  speech tokens  ->  [ CosyVoice3 flow + HiFT ]  ->  audio
                  this repo                                       frozen, not ours

The decoder is the frozen flow-matching model and HiFT vocoder from Fun-CosyVoice3-0.5B. This model replaces only the 0.5 B Qwen LM in that stack — which is the point: 16.1 M parameters in place of 500 M, a 31× reduction in the component that carries the language.

Results

	dev loss	dev accuracy
chance (6,569 classes)	8.790	0.015%
Nepali-Swar (16.1 M)	4.1980	18.23%
on call-centre register	2.9938	31.53%
Fun-CosyVoice3-0.5B reference	3.765	17.6%

Accuracy is roughly 1,200× chance. That is less impressive than it sounds: next-speech-token prediction has many valid answers for the same text — small differences in pitch or timing are different token ids but equally correct speech — so ~18% sits near the intrinsic ceiling rather than measuring quality. Loss is the number with real headroom.

The shipped checkpoint is tuned for call-centre register, which is why its score on that slice (2.9938 / 31.53%) is far stronger than its general score.

Files

file
swar.pt	the model, 64 MB. Carries lm_config, char_vocab and control_buckets inside it.
char_vocab.json	154-character vocabulary — part of the model
control_buckets.json	rate/pause bucket edges — part of the model
sample.wav	5.9 s sample decoded through CosyVoice3's vocoder

⚠️ char_vocab.json and control_buckets.json are not metadata. Character ids are positions in that file, and control ids are defined by those bucket edges. Running these weights against different ones is silent corruption — no error, just wrong sounds. They are embedded in the .pt for that reason.

Usage

import torch
from nvoice.lm import build
from nvoice.vocab import CharVocab, SOS, TASK, ctrl_prefix

ck = torch.load("swar.pt", map_location="cpu")
model = build("M16", **{k: v for k, v in ck["lm_config"].items() if k != "head_dim"})
model.load_state_dict(ck["model"]); model.to("cuda").eval()
vocab = CharVocab(ck["char_vocab"]["chars"])          # travels inside the checkpoint

ids = vocab.encode("नमस्कार, हजुरलाई कसरी सहयोग गर्न सक्छु?")
prefix = [SOS] + ctrl_prefix("callcenter", None, None, None) + ids + [TASK]
tokens = model.generate(prefix, max_new=760, min_new=12, device="cuda")
# tokens -> audio via CosyVoice3's flow + HiFT

Control tokens

Four optional dials precede the text: style (neutral / callcenter), rate, pause, pitch — five buckets each, or masked. pitch is reserved and non-functional. Rate control is weak.

Limitations

• Not a standalone TTS. Requires CosyVoice3's flow + vocoder (~1.4 GB).
• Experimental quality. No MOS, no native-listener evaluation, no CER gate.
• Timbre comes from the reference clip, not the model — this is a voice-cloning stack, and the model carries language, not identity.
• Long paragraphs are best synthesised one sentence at a time; an autoregressive model of this size can drift or stall on very long inputs.

License

Apache-2.0. The decoder it depends on (Fun-CosyVoice3-0.5B) carries its own terms.