Instructions to use pratikgx/bison-1-MLX-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use pratikgx/bison-1-MLX-4bit with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download pratikgx/bison-1-MLX-4bit --local-dir bison-1-MLX-4bit
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
bison-1 MLX 4-bit
4-bit MLX build of Baseten's bison-1 decision model, measured on my 64GB M5 Pro Mac mini. Official weights are by baselabs. This repository is a code port of their System One decision path to Apple Silicon, plus the 4-bit weights. The port was built with a coding agent.
This is not a chat model. bison-1 answers by scoring options in one prefill. It takes a state and typed questions (noul, choice, or score) and returns probabilities. There is no generated text.
License follows the official checkpoint: MIT and Apache 2.0. See LICENSE and NOTICE.
Source: baselabs/bison-1 (31B dense, built on Gemma 4).
What's in this repo
- MLX affine 4-bit weights, group size 64, following mlx-lm's Gemma 4 defaults (norms stay unquantized, embeddings and Linear weights are quantized). The 4-bit files are 16.08 GB on disk, down from 60.88 GB for the official BF16 snapshot.
engine/, a small System One decision engine: qev-labels-v2 prompts, slot_lm readout (256 slots), and Baseten's attention overrides (block-bidirectional mask on full-attention layers, causal sliding-window layers).- Tokenizer, chat template, and
readout.jsonfrom the official repo.
Precision is in the repo name on purpose: this is 4-bit MLX, not BF16.
Numbers on my 64GB M5 Pro Mac mini
Another workload kept running during these measurements. These are not isolated-idle figures, and I am not reporting a system memory peak as the model's memory. The measured results:
- Public JevBench, 231 items: 221/231 (easy 48/48, original 72/72, hard 101/111). Median 0.824 s per item.
- Latency at 513 tokens, 1 warmup + 20 timed, median: 1.544 s per decision (p95 1.591 s, 0.648 decisions/s).
- Hosted-API agreement on a fixed 30-item slice: 29/30 decisions matched Baseten's hosted System One API. Mean absolute probability difference 0.0176, max 0.248. The one miss was
hard-opus-b-multi_hop-05: local predicted 2, hosted predicted 3, and the JevBench gold label was 2. Local matched gold; it still counts as a miss against hosted. This is hosted-API agreement, not local BF16 OneJev parity. - The same 231 items on Baseten's hosted API scored 218/231 (easy 48/48, original 72/72, hard 98/111). That is a different runtime, not a like-for-like comparison.
Raw JSON, charts, and setup notes: pratikgx/bison-1-m5-pro-mac-mini-bench. Write-up: local-ai-field-notes.
Runtime was mlx 0.32.4 and mlx-lm 0.32.2.
Use it
Install mlx-lm 0.32.2 or newer, clone or snapshot this repo, and add engine/ to PYTHONPATH.
from engine.engine_mlx import DecisionEngine
engine = DecisionEngine("/path/to/bison-1-MLX-4bit")
out = engine.decide(
state="The customer asked where their order is. The tracking page shows it shipped yesterday.",
questions={
"decision": {
"type": "choice",
"instructions": "What does the customer want?",
"criteria": {
"track_order": "They want shipment status",
"cancel_order": "They want to cancel",
"billing_question": "They have a billing question",
},
}
},
)
print(out["answers"]["decision"])
Do not use a generic chat generate loop. The model is meant to be scored at the slot letters in one prefill, with thinking off.
How the 4-bit files were built
The official BF16 safetensors were converted tensor-by-tensor with mx.quantize (affine, group 64), matching mlx-lm's Gemma 4 rules: norms stay unquantized, and embeddings and Linear weights are quantized when the last dimension is divisible by 64. The converter also lifts per_layer_config into global_head_dim=512 and num_global_key_value_heads=4, which is what mlx-lm reads for the full-attention layers.
Credits
- Model: Baseten / baselabs / baselabs/bison-1
- Prompt and readout conventions follow OneJev/qev (qev-labels-v2) plus Baseten's published attention and slot_lm overrides
- This MLX port and 4-bit conversion were built with a coding agent on my Mac mini
- Downloads last month
- -
4-bit