Instructions to use aaron-hu26/my-tiny-audio with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use aaron-hu26/my-tiny-audio with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf aaron-hu26/my-tiny-audio:Q8_0 # Run inference directly in the terminal: llama cli -hf aaron-hu26/my-tiny-audio:Q8_0
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf aaron-hu26/my-tiny-audio:Q8_0 # Run inference directly in the terminal: llama cli -hf aaron-hu26/my-tiny-audio:Q8_0
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf aaron-hu26/my-tiny-audio:Q8_0 # Run inference directly in the terminal: ./build/bin/llama cli -hf aaron-hu26/my-tiny-audio:Q8_0
Use Docker
docker model run hf.co/aaron-hu26/my-tiny-audio:Q8_0
- Llama new
- LM Studio
- Jan
- Ollama
How to use aaron-hu26/my-tiny-audio with Ollama:
ollama run hf.co/aaron-hu26/my-tiny-audio:Q8_0
- Unsloth Desktop
- Pi
How to use aaron-hu26/my-tiny-audio with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf aaron-hu26/my-tiny-audio:Q8_0
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "aaron-hu26/my-tiny-audio:Q8_0" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use aaron-hu26/my-tiny-audio with Docker Model Runner:
docker model run hf.co/aaron-hu26/my-tiny-audio:Q8_0
- Lemonade
How to use aaron-hu26/my-tiny-audio with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull aaron-hu26/my-tiny-audio:Q8_0
Run and chat with the model
lemonade run user.my-tiny-audio-Q8_0
List all available models
lemonade list
- Hermes Agent
How to use aaron-hu26/my-tiny-audio with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf aaron-hu26/my-tiny-audio:Q8_0
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default aaron-hu26/my-tiny-audio:Q8_0
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use aaron-hu26/my-tiny-audio with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf aaron-hu26/my-tiny-audio:Q8_0
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "aaron-hu26/my-tiny-audio:Q8_0" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
My Tiny Audio (Qwen3.5-0.8B + Granite Speech 5.0)
A lightweight, hackable, end-to-end Speech-to-Text Large Language Model.
๐ Model Overview
my-tiny-audio is a lightweight multimodal speech recognition model built with:
- Audio Encoder (Frozen):
ibm-granite/granite-speech-5.0-470m-turboctc-nc(470M Conformer-based feature extractor). - Audio Projector (Trained): 2-layer MLP projector (
1024โ1024GELU โ1024) with frame pooling stride 1. - Text Decoder (Fine-tuned & Merged):
Qwen/Qwen3.5-0.8Bwith LoRA adapter ($r=16, \alpha=32$) mathematically fused into the base weights viamerge_and_unload(). - Quantized GGUF Exports: Both full precision (BF16) and 8-bit quantized (
Q8_0) GGUF files included for nativellama.cppandOllamadeployment.
๐ Training Specifications
- Hardware: Single consumer GPU (NVIDIA GeForce RTX 4060 8GB VRAM).
- Dataset: LibriSpeech ASR
clean/validation(dev-clean, 5.4 hours audio). - Batching: Per-device batch size 2, gradient accumulation steps 2 (effective batch size 4).
- Optimization: AdamW ($\text{LR}=2\times 10^{-4}$ with warmup and cosine decay).
- Convergence:
- Initial Loss: 4.238 (Step 1)
- Final Loss: 0.1069 (Step 800)
- Status: Successfully converged with stable gradients.
๐ฆ Repository Structure
aaron-hu26/my-tiny-audio/
โโโ model.safetensors # 1.51 GB (Unified weights: trained Projector + merged Qwen3.5-0.8B)
โโโ config.json # Root multimodal ASRConfig with Hugging Face auto_map
โโโ preprocessor_config.json # Granite Speech 5.0 feature extraction config
โโโ tokenizer.json # Aligned 248k vocab tokenizer
โโโ tokenizer_config.json # Tokenizer configuration
โโโ chat_template.jinja # Chat template
โโโ asr_modeling.py # Custom multimodal ASR architecture
โโโ asr_pipeline.py # Custom ASR pipeline implementation
โโโ projectors.py # MLP audio projector definition
โโโ requirements.txt # Inference dependencies
โโโ qwen_decoder_q8_0.gguf # 775 MB (8-bit quantized GGUF for Ollama/llama.cpp)
โโโ qwen_decoder.gguf # 1.51 GB (BF16 lossless GGUF)
โโโ Modelfile # Ollama model definition file
โโโ app.py # Single-file Gradio web demo with microphone & metrics
โโโ decoder/ # Standalone merged Qwen3.5-0.8B text decoder
๐ Quickstart
1. Hugging Face Pipeline (Standard & Recommended)
You can load and run the model directly from the Hugging Face Hub using the standard transformers pipeline:
from transformers import pipeline
pipe = pipeline(
"automatic-speech-recognition",
model="aaron-hu26/my-tiny-audio",
trust_remote_code=True,
device="cuda", # ๆ "cpu"
)
print(pipe("/tmp/test_audio.wav")["text"])
2. Standalone Web UI (Microphone & Audio Upload)
You can launch the complete Web UI with a single command:
git clone https://huggingface.co/aaron-hu26/my-tiny-audio
cd my-tiny-audio
pip install -r requirements.txt gradio
python app.py
Open http://localhost:7860 to record from your microphone or upload any WAV/MP3/FLAC audio with live RTF and throughput metrics.
3. Run GGUF with Ollama
The repository bundles an 8-bit quantized GGUF model (qwen_decoder_q8_0.gguf) and a ready-to-use Modelfile:
# Register the model in Ollama
ollama create my-tiny-audio -f Modelfile
# Run interactive conversation
ollama run my-tiny-audio
๐ License & Attribution
- Built using the tiny-audio architecture.
- Base models: Granite Speech 5.0 by IBM and Qwen3.5-0.8B by Alibaba Cloud Qwen Team.
- License: MIT License.
- Downloads last month
- -