Nampfiev1995/tasu-cps-projector

🎙️ TASU Speech-LLM: User & Inference Documentation

This repository provides TASU (Token-Aligned Speech Understanding) — a modular framework for zero-shot speech reasoning and Voice Assistant capabilities using Gipformer CTC + Qwen 3.5-9B.

---

🌟 1. Checkpoint & Hugging Face Model

• Hugging Face Repository: Nampfiev1995/tasu-cps-projector
• Primary Checkpoint: step_500.pt (Trained on full Stage 2 Speech-Reasoning & Voice Assistant QA datasets)

---

🎵 2. Audio Data Input Requirements

Parameter	Value / Requirement
Sample Rate	16,000 Hz (16 kHz)
Channels	1 (Mono)
Supported File Formats	.wav, .mp3, .flac, .m4a, .webm, .ogg
Acoustic Feature Extraction	Kaldi 80-bin Mel Fbank (num_mel_bins=80, frame_shift=10ms, frame_length=25ms)

---

💬 3. Prompt Formats & Chat Templates

A. Voice Assistant Mode (Direct Conversational Answer)

• System Prompt:

  You are a helpful voice assistant. Listen to the user's speech and answer their request accurately.
  

• Assistant Prefix (No Thinking, Lowest Latency):

  <|im_start|>assistant\n<think>\n\n</think>\n\n
  

B. Thinking & Reasoning Mode

• System Prompt:

  You are a helpful voice assistant. Listen to the user's speech and answer their request accurately.
  

• Assistant Prefix (Generates <think>...</think> internal reasoning chain):

  <|im_start|>assistant\n
  

C. Pure ASR Transcription Mode (Zero Extra Words)

• System Prompt:

  You transcribe Vietnamese and Vietnamese-English smart-home speech exactly. Return only the transcript.
  

---

🚀 4. Commands for Running Inference

Option A: Command Line Single Audio File Prediction (predict_single_audio.py)

Run inference on any single .wav or .mp3 file. If using Hugging Face repo ID, the checkpoint is automatically downloaded:

export PATH="/Utilisateurs/tnguye28/tools/ffmpeg:/home/tools:$PATH"
export LD_LIBRARY_PATH="/Utilisateurs/tnguye28/.conda/envs/auden_cuda124/lib:/Utilisateurs/tnguye28/Speech-LLMs/Auden/.venv/lib/python3.11/site-packages/nvidia/nvtx/lib:${LD_LIBRARY_PATH:-}"
export CUDA_VISIBLE_DEVICES=0

/Utilisateurs/tnguye28/.conda/envs/auden_cuda124/bin/python predict_single_audio.py \
  --config configs/speech_llm/tasu_baseline_b_zeroshot_eval_pw733_test_128.yaml \
  --checkpoint Nampfiev1995/tasu-cps-projector \
  --audio /Utilisateurs/tnguye28/Speech-LLMs-reasoning/audio-test/may_gio.wav

Option B: Interactive Gradio Web Demo App (app_tasu.py)

Launch an interactive web dashboard with microphone recording, file upload, prompt customization, and public HTTPS links (--share):

export PATH="/Utilisateurs/tnguye28/tools/ffmpeg:/home/tools:$PATH"
export LD_LIBRARY_PATH="/Utilisateurs/tnguye28/.conda/envs/auden_cuda124/lib:/Utilisateurs/tnguye28/Speech-LLMs/Auden/.venv/lib/python3.11/site-packages/nvidia/nvtx/lib:${LD_LIBRARY_PATH:-}"
export CUDA_VISIBLE_DEVICES=0

/Utilisateurs/tnguye28/.conda/envs/auden_cuda124/bin/python app_tasu.py \
  --config configs/speech_llm/tasu_baseline_b_zeroshot_eval_pw733_test_128.yaml \
  --checkpoint Nampfiev1995/tasu-cps-projector \
  --port 7860 \
  --share

---

🔌 6. How to Integrate Any Custom Audio Encoder with TASU

TASU's architecture is 100% modular. You can swap Gipformer with Zipformer, Wav2Vec2, Conformer, HuBERT, or Whisper.

A. Mathematical Contract

To work with TASU, your audio encoder must meet only one requirement:

$$\text{Encoder}({\mathbf{x}}_{\text{audio}})\stackrel{\text{CTC Head}}{\to }\mathbf{Z}\in {\mathbb{R}}^{B\times T\times V}\backslash text\{Encoder\}(\backslash mathbf\{x\}\_\{\backslash text\{audio\}\})\; \backslash xrightarrow\{\backslash text\{CTC\; Head\}\}\; \backslash mathbf\{Z\}\; \backslash in\; \backslash mathbb\{R\}^\{B\; \backslash times\; T\; \backslash times\; V\}$$

• $B$ = Batch size
• $T$ = Number of time frames (e.g. $T = 50 \times \text{seconds}$)
• $V$ = Vocabulary size (e.g., $V = 10,008$ for Qwen 10k subwords or $V = 521$ for characters)

Universal Vocabulary Rule: ANY CTC model can be integrated with TASU as long as its CTC vocabulary $V$ maps into the target LLM's token embedding space.

• Subword Tokens (e.g., 10k Qwen subwords): Mapped via ctc_local_to_qwen.json.
• Character / Byte Tokens (e.g., 521 characters): Mapped directly to Qwen's byte/character embedding rows.
• Phoneme / Acoustic Units: Mapped via an acoustic-to-LLM index dictionary.

The QwenCtcEmbeddingBridge computes $\mathbf{S}{\text{speech}} = \mathbf{P} \cdot \mathbf{E}{\text{LLM}} \in \mathbb{R}^{T \times D_{\text{LLM}}}$, converting any CTC frame probability distribution $\mathbf{P}$ into dense, continuous LLM speech embeddings.

B. Step-by-Step Developer Implementation Guide

1. Define your Audio Encoder Module (src/models/my_custom_encoder.py)

import torch
import torch.nn as nn

class MyCustomSpeechEncoder(nn.Module):
    def __init__(self, vocab_size: int, hidden_dim: int = 768):
        super().__init__()
        self.encoder = MyAudioBackbone(...)  # e.g., Whisper, Zipformer, Conformer
        self.ctc_head = nn.Linear(hidden_dim, vocab_size)

    def forward(self, features: torch.Tensor, feature_lens: torch.Tensor):
        # 1. Forward through audio backbone -> (B, T, D_enc)
        enc_out, out_lens = self.encoder(features, feature_lens)
        # 2. Project to vocabulary logits -> (B, T, V)
        logits = self.ctc_head(enc_out)
        return logits, out_lens

2. Register in train_ctc_head.py (build_model)

Add your backend to the build_model factory function:

elif backend == "my_custom_encoder":
    from src.models.my_custom_encoder import MyCustomSpeechEncoder
    return MyCustomSpeechEncoder(vocab_size=vocab_size, hidden_dim=768)

3. Connect to TASU Bridge & Projector

Once your custom encoder produces logits and out_lens, pass them directly to the Bridge and Projector:

# 1. Forward through your custom encoder
logits, out_lens = custom_encoder(features, feature_lens)

# 2. Bridge converts CTC frame posteriors to LLM embedding space
speech_repr, speech_lens = bridge(
    logits,
    out_lens,
    blank_threshold=0.9,
    max_frames=192,
    top_k=10,
)

# 3. Project to LLM Context
projected_speech = projector(speech_repr)

# 4. Pass to LLM (Qwen 3.5-9B, Qwen 3-32B, Llama 3, etc.)
outputs = qwen_llm(inputs_embeds=projected_speech, ...)

---

C. Example Config YAML for Custom Encoder

speech:
  backend: my_custom_encoder
  model_name: path/or/name/of/encoder

ctc:
  config: configs/ctc/qwen10k_vocab_b128_bucket_probe.yaml
  checkpoint: path/to/custom_ctc_checkpoint.pt

qwen:
  model_name: Qwen/Qwen3.5-9B