bison-1 MLX 4-bit

4-bit MLX build of Baseten's bison-1 decision model, measured on my 64GB M5 Pro Mac mini. Official weights are by baselabs. This repository is a code port of their System One decision path to Apple Silicon, plus the 4-bit weights. The port was built with a coding agent.

This is not a chat model. bison-1 answers by scoring options in one prefill. It takes a state and typed questions (noul, choice, or score) and returns probabilities. There is no generated text.

License follows the official checkpoint: MIT and Apache 2.0. See LICENSE and NOTICE.

Source: baselabs/bison-1 (31B dense, built on Gemma 4).

What's in this repo

  • MLX affine 4-bit weights, group size 64, following mlx-lm's Gemma 4 defaults (norms stay unquantized, embeddings and Linear weights are quantized). The 4-bit files are 16.08 GB on disk, down from 60.88 GB for the official BF16 snapshot.
  • engine/, a small System One decision engine: qev-labels-v2 prompts, slot_lm readout (256 slots), and Baseten's attention overrides (block-bidirectional mask on full-attention layers, causal sliding-window layers).
  • Tokenizer, chat template, and readout.json from the official repo.

Precision is in the repo name on purpose: this is 4-bit MLX, not BF16.

Numbers on my 64GB M5 Pro Mac mini

Another workload kept running during these measurements. These are not isolated-idle figures, and I am not reporting a system memory peak as the model's memory. The measured results:

  • Public JevBench, 231 items: 221/231 (easy 48/48, original 72/72, hard 101/111). Median 0.824 s per item.
  • Latency at 513 tokens, 1 warmup + 20 timed, median: 1.544 s per decision (p95 1.591 s, 0.648 decisions/s).
  • Hosted-API agreement on a fixed 30-item slice: 29/30 decisions matched Baseten's hosted System One API. Mean absolute probability difference 0.0176, max 0.248. The one miss was hard-opus-b-multi_hop-05: local predicted 2, hosted predicted 3, and the JevBench gold label was 2. Local matched gold; it still counts as a miss against hosted. This is hosted-API agreement, not local BF16 OneJev parity.
  • The same 231 items on Baseten's hosted API scored 218/231 (easy 48/48, original 72/72, hard 98/111). That is a different runtime, not a like-for-like comparison.

Raw JSON, charts, and setup notes: pratikgx/bison-1-m5-pro-mac-mini-bench. Write-up: local-ai-field-notes.

Runtime was mlx 0.32.4 and mlx-lm 0.32.2.

Use it

Install mlx-lm 0.32.2 or newer, clone or snapshot this repo, and add engine/ to PYTHONPATH.

from engine.engine_mlx import DecisionEngine

engine = DecisionEngine("/path/to/bison-1-MLX-4bit")
out = engine.decide(
    state="The customer asked where their order is. The tracking page shows it shipped yesterday.",
    questions={
        "decision": {
            "type": "choice",
            "instructions": "What does the customer want?",
            "criteria": {
                "track_order": "They want shipment status",
                "cancel_order": "They want to cancel",
                "billing_question": "They have a billing question",
            },
        }
    },
)
print(out["answers"]["decision"])

Do not use a generic chat generate loop. The model is meant to be scored at the slot letters in one prefill, with thinking off.

How the 4-bit files were built

The official BF16 safetensors were converted tensor-by-tensor with mx.quantize (affine, group 64), matching mlx-lm's Gemma 4 rules: norms stay unquantized, and embeddings and Linear weights are quantized when the last dimension is divisible by 64. The converter also lifts per_layer_config into global_head_dim=512 and num_global_key_value_heads=4, which is what mlx-lm reads for the full-attention layers.

Credits

  • Model: Baseten / baselabs / baselabs/bison-1
  • Prompt and readout conventions follow OneJev/qev (qev-labels-v2) plus Baseten's published attention and slot_lm overrides
  • This MLX port and 4-bit conversion were built with a coding agent on my Mac mini
Downloads last month
-
Safetensors
Model size
31B params
Tensor type
U32
Β·
BF16
Β·
MLX
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for pratikgx/bison-1-MLX-4bit

Finetuned
baselabs/bison-1
Quantized
(1)
this model