MOSS-Transcribe-Diarize-GGUF/README.md

---
license: apache-2.0
base_model: OpenMOSS-Team/MOSS-Transcribe-Diarize
tags:
- gguf
- audio.cpp
- asr
- diarization
---

# MOSS-Transcribe-Diarize GGUF

BF16, Q8_0, and Q4_K weights for native [audio.cpp](https://github.com/0xShug0/audio.cpp)
transcription, speaker diarization, and segment timestamps.

Upstream: [OpenMOSS-Team/MOSS-Transcribe-Diarize](https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize),
pinned to revision `704aa4a9c304e8520be88901e0d1960158ef5b15`.
The model is licensed under Apache-2.0; see `LICENSE`.

`moss-transcribe-diarize-bf16.gguf` contains all 683 original tensors (908.5M
parameters) and the tokenizer/configuration sidecars. Every BF16 tensor was
verified byte-for-byte against the original safetensors checkpoint. Whisper
tensor names are mapped to the existing framework loader layout; tensor values
and shapes are unchanged. The GGUF is slightly larger because it also embeds
tokenizer/configuration files and metadata.

No separate Whisper, Qwen, speaker model, VAD, or MOSS audio codec download is
needed. Non-16-kHz audio requires the SOXR runtime library. Every variant is
generated directly from the pinned safetensors checkpoint, never from another
GGUF. Quantized variants retain unsupported tensors in floating-point precision.

| File | Approximate size | Primary-fixture quality |
| --- | ---: | --- |
| `moss-transcribe-diarize-bf16.gguf` | 1.833 GB | Original weights; reference choice |
| `moss-transcribe-diarize-q8_0.gguf` | 1.132 GB | Passes the same strict gate as BF16 |
| `moss-transcribe-diarize-q4_k.gguf` | 0.758 GB | Not parity-safe: segment/timestamp drift |

Q8 preserves words and speakers on the 28-second and 327.6-second fixtures,
with at most 30 ms timestamp difference from Python FP32. Q4_K preserves their
words and speakers but changes segment boundaries, with up to 680 ms drift on
matched segments. Do not use Q4_K when faithful timing/segmentation is required.
These fixtures do not establish accuracy on arbitrary recordings.

```bash
audiocpp_cli --task asr --family moss_transcribe_diarize \
  --model moss-transcribe-diarize-bf16.gguf --backend cuda --threads 8 \
  --audio recording.wav --log --log-file transcription.log
```

The native implementation uses F32 activations and an F16 decode KV cache.
`--instruct "..."` (or `--request-option instruct=...`) replaces the default transcription instruction.
Add `--mode streaming` to receive transcript deltas during decoding. Complete
audio is required first; this is output streaming, not live microphone input.
The embedded spec supports both modes. All requests use
128-token chunked prefill directly into the GPU-resident FP16 decode
cache. It removes duplicate prefill KV storage and the full square prompt
mask, without offloading KV to CPU or truncating context. Cache memory and
the block mask still grow with context capacity; VRAM is not constant.

Tiny timestamp differences are possible versus Python FP32. Python BF16 may
also choose a different word or segment break due to numerical sensitivity.
RTX 5090, CUDA Debug, eight threads, default 5120-token limit; warm medians of
three sequential requests after one cold request, excluding model load/upload:

| Precision | 28 s inference / RTF | 327.6 s inference / RTF |
| --- | ---: | ---: |
| BF16 | 0.626 s / 0.0224 | 8.846 s / 0.0270 |
| Q8_0 | 0.588 s / 0.0210 | 8.402 s / 0.0256 |
| Q4_K | 0.590 s / 0.0211 | 8.318 s / 0.0254 |

Q4_0 was also measured: 0.897 s and 8.216 s. Q4_K was retained for its better
short-input latency and unchanged words; Q4_0 changed one word on the long
fixture and split the short transcript into 14 segments instead of six.
This is end-to-end latency, not equal-token kernel throughput.

These regenerated local packages have not been uploaded or published as an
audio.cpp release.