Some questions about the model

#1
by Thanushttk - opened

It's great to finally see an Indic speech to speech model! As someone working in the same space, I have a few questions.

Could you confirm if the model uses the following stack?

  • Speech Input: Qwen3-ASR 1.7B (~4.2 GB)
  • Response Generation: Gemma 4 E4B, 4-bit quantized, with a separate LoRA adapter (~5 GB)
  • Speech Output: Qwen3-TTS 12Hz 0.6B, with two fixed voice vectors (~2.6 GB)
  • Turn Detection: wav2vec2 XLS-R 300M with a policy head
  • Conversation Memory: XLM-R Large

Also, could you publish the serving and inference code, tokenizers, configuration files, and selected_packages.json referenced in the manifest?

A few more questions:

  1. Which evaluations have you conducted (WER, MOS, turn-detection accuracy, and latency)? Which test datasets did you use, and what were the results?

  2. Is the system fully full-duplex? Can users interrupt it mid-response? Does it support backchanneling and listening while speaking?

  3. Does any component process non-verbal audio cues such as emotion, laughter, hesitation, and breathing, or is everything converted into text?

  4. Can the TTS generate paralinguistic expressions such as laughter, sighs, and different emotions? If so, how are these controlled?

  5. How is voice cloning implemented, given that the speaker encoder is excluded? Where do the two fixed voice vectors come from?

Would love to know more about the architecture and implementation. Great work!

Vector Notion org

Hi @Thanushttk , thanks for the detailed questions. We have now updated the model card with the architecture, streaming behavior, same-dataset ASR comparisons, evaluation limitations, voice-vector provenance and the files included/excluded from this release. The answers below are for your reference; the updated card brings the details together for other readers too.

  1. Architecture: your component list matches the published v1 weights bundle: fine-tuned Qwen3-ASR 1.7B, quantized Gemma 4 E4B with a separate LoRA adapter, Qwen3-TTS 0.6B, wav2vec2 XLS-R 300M with acoustic/policy heads, and XLM-R Large with memory adapters/state heads. This is an integrated modular pipeline. The newer hosted stack uses a later ASR checkpoint, Sarvam 105B and VoxCPM2; those changes are separate from the v1 download.

  2. Evaluations: the card now includes WER/CER for all five ASR models on the same 916 Indic recordings, plus the separate 84-recording English comparison. It explicitly distinguishes those results from v1's 22.66% WER on a different 560-clip development set. It also records TTS intelligibility results and the detector's 85% clip-end turn-completion accuracy, with dataset and protocol limitations. We do not have a formal human-rated MOS study or comprehensive twelve-language mouth-to-ear p50/p95 benchmark to report.

  3. Streaming and full-duplex: the serving runtime incrementally processes incoming speech, streams reply text into TTS and sends audio chunks before the full reply finishes. It supports receiving input during output and barge-in cancellation. Those behaviors require controller and playback integration beyond the weights. Overlapping backchannels and robust full-duplex behavior across languages/devices have not been comprehensively validated. Other models also support streaming; it is not exclusive to Swarit.

  4. Non-verbal cues and expression: wav2vec processes audio directly for activity, language, turn completion and policy. Response generation primarily uses transcripts and conversation state. Reliable recognition of laughter, breathing, hesitation and vocal emotion is unverified. V1 has no validated laughter/sigh/emotion control interface. The newer hosted VoxCPM2 system provides reference-based natural, happy and empathetic delivery, with manual or text-based automatic selection; this is separate from acoustic emotion recognition and is not included in v1.

  5. Voice cloning: the supplied vectors were extracted offline using the original speaker encoder from selected IISc SYSPIN/LIMMITS24 recordings. Serving reuses them across output languages without keeping the encoder resident. Creating a new vector requires the matching encoder and preprocessing, which are excluded. The manifest and license notice contain attribution and processing provenance.

  6. Code and reproducibility: this is currently a weights-only upload. Serving code, tokenizers and runtime configurations are excluded, and selected_packages.json refers to an internal source path. The weights alone do not reproduce the complete runtime. We have not announced a publication date for the supporting files.

Thanks again for raising these questions and helping us make the release documentation clearer.

Sign up or log in to comment