Config is in Mistral's layout, so mlx-audio can't load it
#2
by flo5k5 - opened
Hi, and thanks for the conversions!
This repo's config.json uses Mistral's original layout (dim, n_layers, multimodal.whisper_model_args, …) with nomodel_type, unlike mlx-community/Voxtral-Mini-4B-Realtime-2602-fp16 and -4bit, which use mlx-audio's layout
(model_type: voxtral_realtime, decoder, encoder_args). As a result mlx-audio's load_model can't read it, and
oMLX discovers it as an LLM and fails with KeyError: 'model_type'.
Re-converting from the fp16 repo with mlx-audio works:
python -m mlx_audio.convert --hf-path mlx-community/Voxtral-Mini-4B-Realtime-2602-fp16 \
--mlx-path Voxtral-Mini-4B-Realtime-2602-6bit -q --q-bits 6
I published that conversion, with streaming latency and French WER measurements, at
flo5k5/Voxtral-Mini-4B-Realtime-2602-MLX-6bit,
in case it helps. The 6-bit version keeps up with live speech on an M5 Pro, where fp16 falls behind.