My Tiny Audio (Qwen3.5-0.8B + Granite Speech 5.0)

A lightweight, hackable, end-to-end Speech-to-Text Large Language Model.


๐Ÿ“– Model Overview

my-tiny-audio is a lightweight multimodal speech recognition model built with:

  1. Audio Encoder (Frozen): ibm-granite/granite-speech-5.0-470m-turboctc-nc (470M Conformer-based feature extractor).
  2. Audio Projector (Trained): 2-layer MLP projector (1024 โ†’ 1024 GELU โ†’ 1024) with frame pooling stride 1.
  3. Text Decoder (Fine-tuned & Merged): Qwen/Qwen3.5-0.8B with LoRA adapter ($r=16, \alpha=32$) mathematically fused into the base weights via merge_and_unload().
  4. Quantized GGUF Exports: Both full precision (BF16) and 8-bit quantized (Q8_0) GGUF files included for native llama.cpp and Ollama deployment.

๐Ÿ“Š Training Specifications

  • Hardware: Single consumer GPU (NVIDIA GeForce RTX 4060 8GB VRAM).
  • Dataset: LibriSpeech ASR clean/validation (dev-clean, 5.4 hours audio).
  • Batching: Per-device batch size 2, gradient accumulation steps 2 (effective batch size 4).
  • Optimization: AdamW ($\text{LR}=2\times 10^{-4}$ with warmup and cosine decay).
  • Convergence:
    • Initial Loss: 4.238 (Step 1)
    • Final Loss: 0.1069 (Step 800)
    • Status: Successfully converged with stable gradients.

๐Ÿ“ฆ Repository Structure

aaron-hu26/my-tiny-audio/
โ”œโ”€โ”€ model.safetensors          # 1.51 GB (Unified weights: trained Projector + merged Qwen3.5-0.8B)
โ”œโ”€โ”€ config.json                # Root multimodal ASRConfig with Hugging Face auto_map
โ”œโ”€โ”€ preprocessor_config.json   # Granite Speech 5.0 feature extraction config
โ”œโ”€โ”€ tokenizer.json             # Aligned 248k vocab tokenizer
โ”œโ”€โ”€ tokenizer_config.json      # Tokenizer configuration
โ”œโ”€โ”€ chat_template.jinja        # Chat template
โ”œโ”€โ”€ asr_modeling.py            # Custom multimodal ASR architecture
โ”œโ”€โ”€ asr_pipeline.py            # Custom ASR pipeline implementation
โ”œโ”€โ”€ projectors.py              # MLP audio projector definition
โ”œโ”€โ”€ requirements.txt           # Inference dependencies
โ”œโ”€โ”€ qwen_decoder_q8_0.gguf     # 775 MB (8-bit quantized GGUF for Ollama/llama.cpp)
โ”œโ”€โ”€ qwen_decoder.gguf          # 1.51 GB (BF16 lossless GGUF)
โ”œโ”€โ”€ Modelfile                  # Ollama model definition file
โ”œโ”€โ”€ app.py                     # Single-file Gradio web demo with microphone & metrics
โ””โ”€โ”€ decoder/                   # Standalone merged Qwen3.5-0.8B text decoder

๐Ÿš€ Quickstart

1. Hugging Face Pipeline (Standard & Recommended)

You can load and run the model directly from the Hugging Face Hub using the standard transformers pipeline:

from transformers import pipeline

pipe = pipeline(
    "automatic-speech-recognition",
    model="aaron-hu26/my-tiny-audio",
    trust_remote_code=True,
    device="cuda",  # ๆˆ– "cpu"
)

print(pipe("/tmp/test_audio.wav")["text"])

2. Standalone Web UI (Microphone & Audio Upload)

You can launch the complete Web UI with a single command:

git clone https://huggingface.co/aaron-hu26/my-tiny-audio
cd my-tiny-audio
pip install -r requirements.txt gradio
python app.py

Open http://localhost:7860 to record from your microphone or upload any WAV/MP3/FLAC audio with live RTF and throughput metrics.

3. Run GGUF with Ollama

The repository bundles an 8-bit quantized GGUF model (qwen_decoder_q8_0.gguf) and a ready-to-use Modelfile:

# Register the model in Ollama
ollama create my-tiny-audio -f Modelfile

# Run interactive conversation
ollama run my-tiny-audio

๐Ÿ“„ License & Attribution

Downloads last month
-
Safetensors
Model size
0.8B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support