VibeVoice-ASR-HF-GGUF

VibeVoice-ASR (Transformers-compatible version, microsoft/VibeVoice-ASR-HF) is Microsoft Research's MIT-licensed unified speech-to-text model. It processes up to 60 minutes of continuous audio in a single pass within a 64K-token context, so speaker tracking and semantic coherence hold across the whole recording. It jointly performs transcription, speaker diarization, and timestamping, and outputs JSON-like structured text giving who (Speaker), when (Start/End), and what (Content). It handles over 50 languages without an explicit language setting, including code-switching, and accepts optional context or hotwords such as names and technical terms to improve accuracy on domain-specific content. To handle long audio, it chunks the input into 60-second segments and caches convolution states between them. The chunk size can be reduced through tokenizer_chunk_size if memory is tight. It is available in Transformers v5.3.0 and later through VibeVoiceAsrForConditionalGeneration. It supports batch inference, chat-template inputs, training with the model's built-in loss, torch.compile, and pipeline usage. On the Open ASR Leaderboard it averages 7.77% WER (for example 2.20% on LibriSpeech test-clean and 17.20% on AMI) with an RTFx of 51.8. The technical report also gives diarization (DER) and speaker-attributed WER (cpWER, tcpWER) results.

Model Files

File Name Quant Type File Size File Link Description
VibeVoice-ASR-HF.BF16.gguf BF16 15.2 GB Link Full BF16 weights. Highest quality, largest file size.
VibeVoice-ASR-HF.Q3_K_L.gguf Q3_K_L 4.09 GB Link Lower quality but usable, good for low RAM availability.
VibeVoice-ASR-HF.Q3_K_M.gguf Q3_K_M 3.81 GB Link Low quality, smallest file size.
VibeVoice-ASR-HF.Q4_K_M.gguf Q4_K_M 4.68 GB Link Good quality, default size for most use cases, recommended.
VibeVoice-ASR-HF.Q4_K_S.gguf Q4_K_S 4.46 GB Link Slightly lower quality with more space savings, recommended.
VibeVoice-ASR-HF.Q5_K_M.gguf Q5_K_M 5.44 GB Link High quality, recommended.
VibeVoice-ASR-HF.Q5_K_S.gguf Q5_K_S 5.32 GB Link High quality with slight space savings, recommended.
VibeVoice-ASR-HF.Q6_K.gguf Q6_K 6.25 GB Link Very high quality, near perfect, recommended.
VibeVoice-ASR-HF.Q8_0.gguf Q8_0 8.1 GB Link Extremely high quality, generally unneeded but max available quant.
VibeVoice-ASR-HF.mmproj-bf16.gguf mmproj-bf16 1.43 GB Link Multimodal projection file in BF16 format. Used for audio/ASR capabilities.

Install dependencies and clone llama.cpp

!pip install transformers huggingface-hub

!git clone https://github.com/ggml-org/llama.cpp.git
%cd /data/llama.cpp

Download GGUF model files

Download the quantized Q5_K_M model and multimodal projector into ./model/VibeVoice-ASR-HF-GGUF:

from huggingface_hub import snapshot_download

# Target local directory matching your path structure
local_dir = "./model/VibeVoice-ASR-HF-GGUF"

snapshot_download(
    repo_id="prithivMLmods/VibeVoice-ASR-HF-GGUF",
    local_dir=local_dir,
    allow_patterns=[
        "VibeVoice-ASR-HF.Q5_K_M.gguf",
        "VibeVoice-ASR-HF.mmproj-bf16.gguf",
    ],
    resume_download=True,
)

Build llama-mtmd-cli

Build the multimodal CLI with the CUDA backend enabled:

!cmake --build . --config Release -j$(nproc) --target llama-mtmd-cli

Download sample audio

import os

os.makedirs("/data/llama.cpp/model/VibeVoice-ASR-HF-GGUF", exist_ok=True)
!wget -O /data/llama.cpp/model/VibeVoice-ASR-HF-GGUF/sample.wav \
"https://cdn-uploads.huggingface.co/production/uploads/65bb837dbfb878f46c77de4c/gObkjtDcqnbGl2wu_p3UP.wav"

Audio transcription (Plain Text Output)

Run inference using llama-mtmd-cli with dynamic duration injection and Python regex post-processing to output only clean plain text:

import subprocess
import wave
import re

audio_path = "/data/llama.cpp/model/VibeVoice-ASR-HF-GGUF/sample.wav"

# 1. Get exact audio duration
with wave.open(audio_path, "rb") as wf:
    duration = wf.getnframes() / float(wf.getframerate())

# 2. Keep 'Start time, End time, Content' prompt anchor to prevent infinite repetition loops
prompt = (
    f"This is a {duration:.2f} seconds audio, "
    "please transcribe it with these keys: Start time, End time, Content"
)

cmd = [
    "/data/llama.cpp/build/bin/llama-mtmd-cli",
    "-m", "/data/llama.cpp/model/VibeVoice-ASR-HF-GGUF/VibeVoice-ASR-HF.Q5_K_M.gguf",
    "--mmproj", "/data/llama.cpp/model/VibeVoice-ASR-HF-GGUF/VibeVoice-ASR-HF.mmproj-bf16.gguf",
    "--audio", audio_path,
    "-p", prompt,
    "-ngl", "99",
    "-c", "8192",
    "-n", "512",
    "--temp", "0.0"
]

# 3. Capture output and extract only plain text content
res = subprocess.run(cmd, capture_output=True, text=True)

segments = re.findall(r'\{"Start":.*?"Content":\s*"(.*?)"\}', res.stdout)
plain_text = " ".join(segments)

print(plain_text)

Output

The stale smell of old beer lingers. It takes heat to bring out the odor. A cold dip restores health and zest. A salt pickle tastes fine with ham. Tacos al pastor are my favorite. A zestful food is the hot cross bun.

llama.cpp

LLM inference in C/C++ β€” https://github.com/ggml-org/llama.cpp

Quantized using llama.cpp by prithivMLmods.

Downloads last month
281
GGUF
Model size
8B params
Architecture
qwen2
Hardware compatibility
Log In to add your hardware

3-bit

4-bit

5-bit

6-bit

8-bit

16-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for prithivMLmods/VibeVoice-ASR-HF-GGUF

Quantized
(7)
this model

Collections including prithivMLmods/VibeVoice-ASR-HF-GGUF