Velvet2-3B-MLX-8bit

๐Ÿ‡ฎ๐Ÿ‡น Conversione non ufficiale in formato MLX (quantizzazione a 8 bit) di AlmawaveLabs/Velvet2-3B, per l'uso su Mac con Apple Silicon tramite mlx-lm o LM Studio.

๐Ÿ‡ฌ๐Ÿ‡ง Unofficial 8-bit MLX conversion of AlmawaveLabs/Velvet2-3B, for Apple Silicon Macs via mlx-lm or LM Studio.

This repository is not affiliated with or endorsed by Almawave. All credit for the model goes to the original authors. For the model description, training details, evaluations and intended use, please refer to the original model card.

Conversion details

Original model AlmawaveLabs/Velvet2-3B (BF16 safetensors)
Original revision f36ebe942894130d86199d3abe70d0f7a7a4d446
Format MLX
Quantization 8-bit (mlx-lm default group size)
Tool mlx-lm
Converted on Apple MacBook (M4)

Command used:

mlx_lm.convert \
  --hf-path AlmawaveLabs/Velvet2-3B \
  --mlx-path Velvet2-3B-MLX-8bit \
  -q --q-bits 8

No fine-tuning or other changes were made to the weights beyond quantization. Tokenizer and chat template are copied unchanged from the original repository.

Usage

LM Studio

Search for rbizzoni/Velvet2-3B-MLX-8bit in LM Studio's model search and download it (requires the MLX runtime on Apple Silicon).

mlx-lm

pip install -U mlx-lm
mlx_lm.generate \
  --model rbizzoni/Velvet2-3B-MLX-8bit \
  --prompt "Spiegami in tre frasi cos'รจ la Costituzione italiana."
from mlx_lm import load, generate

model, tokenizer = load("rbizzoni/Velvet2-3B-MLX-8bit")

messages = [
    {"role": "system", "content": "You are Velvet2-3B, an LLM created by Almawave Labs."},
    {"role": "user", "content": "Qual รจ la capitale d'Italia?"},
]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)

print(generate(model, tokenizer, prompt=prompt, max_tokens=256, verbose=True))

Notes

  • Tokenizer warning: when loading, transformers may print a warning about an "incorrect regex pattern" and suggest fix_mistral_regex=True. This check targets an issue in some Mistral tokenizers. Velvet2 uses its own tokenizer (from Velvet-14B), so the warning is likely a false positive. The tokenizer was not modified in this conversion.
  • Tool calling: the original model provides a dedicated tool-calling template and parser for vLLM (tool_chat_template.jinja, tool_parser_plugin.py). These are not used by mlx-lm or LM Studio, so tool calling may not work as documented in the original card.
  • Quantization can slightly reduce output quality compared to the original BF16 weights.

License and usage restrictions

This conversion is distributed under the same license as the original model (Apache 2.0). The original model card also lists usage restrictions and governance terms, including limitations on commercial and institutional use and on processing sensitive healthcare, financial or government data without appropriate safeguards. These terms apply to this conversion as well. Please read them on the original model page before use.

Credits

Downloads last month
43
Safetensors
Model size
4B params
Tensor type
U32
ยท
BF16
ยท
MLX
Hardware compatibility
Log In to add your hardware

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for rbizzoni/Velvet2-3B-MLX-8bit

Quantized
(1)
this model