nativ-community/flint-5bit

Flint 5BIT

Converted from RecursiveSI/flint at revision 2a247be0c0aec895ccb0ac7ca92229ec6df37288 with MLX-VLM main, version 0.7.10, and MLX 0.32.4.

5BIT weight quantization using affine with group size 64. All 47 MoE routers use 8-bit affine quantization with group size 64. Normalization parameters, attention sinks and other non-quantized parameters remain BF16.

The upstream tokenizer, chat template, generation defaults and Apache-2.0 license are preserved byte for byte. Thinking remains controlled by the original template. No thinking-disable override is included. See the source model card for architecture, intended use and limitations. These converted weights have not been evaluated on the source model's published benchmark suite.

Usage

pip install -U git+https://github.com/Blaizzy/mlx-vlm.git@6d9ba328abbd6129c7461eda86280aa9fbfd9986
mlx_vlm.generate --model nativ-community/flint-5bit --prompt "Write a Python function that merges overlapping intervals." --max-tokens 1024

The source defaults are temperature 1.0, top-p 0.95 and top-k 20. Reasoning consumes the output budget; use a larger budget for complex tasks. The source configuration supports a combined prompt/output window of 262,144 tokens; this conversion's smoke tests do not validate long-context quality.

Reproduce conversion

pip install mlx==0.32.4
mlx_vlm.convert --hf-path RecursiveSI/flint --revision 2a247be0c0aec895ccb0ac7ca92229ec6df37288 --mlx-path flint-5bit --dtype bfloat16 -q --q-bits 5 --q-group-size 64 --q-mode affine

Preserve the canonical upstream sidecars after conversion:

python - <<'PY'
from pathlib import Path
from huggingface_hub import hf_hub_download
import shutil

destination = Path("flint-5bit")
for name in ("tokenizer.json", "tokenizer_config.json", "chat_template.jinja", "generation_config.json", "LICENSE"):
    target = destination / name
    if target.exists():
        target.chmod(target.stat().st_mode | 0o200)
    source = hf_hub_download("RecursiveSI/flint", name, revision="2a247be0c0aec895ccb0ac7ca92229ec6df37288")
    shutil.copy2(source, target)
PY

Use the pinned MLX-VLM commit and MLX version above. The conversion preserves model-specific router precision. Options, per-module overrides and all shard SHA-256 hashes are recorded in conversion.json. The original tokenizer, template, generation config and license are copied unchanged after conversion. SHA256SUMS covers the published artifacts.

Validation

The saved checkpoint passed strict reloading, finite prefill/decode logits, cache-position checks, arithmetic, short retrieval and simple Python-function smoke checks on Apple M5 Max with MLX 0.32.4. All three prompts used the native template, greedy decoding and a 512-token output budget. See validation.json for exact prompts, outputs and checks. Generated code was parsed as an AST, not executed.

These are smoke checks, not a quality benchmark or proof of equivalence to BF16 or Transformers. Quantization can change answers, and floating-point 4-bit formats may differ in quality from affine quantization.