Inkling 2 Mini — MLX 8-bit

Independent affine MLX conversion of the official Thinking Machines checkpoint, revision 6e7c8248f75265bdd4fbab5273304243b5d8c2cb. The source is the authors' trained NVFP4/MXFP8 release, not an original BF16 checkpoint. This conversion starts from that source, not from our 5-bit quant. Eight-bit storage reduces additional affine reconstruction error; it cannot recover precision absent from the source. No improvement in benchmark quality over 5-bit is claimed.

Format

  • Language linear/expert matrices, embeddings and output head: 8-bit affine, group size 32, min/max round-to-nearest.
  • Vision tower, expert routing and loop/message gates retain their source floating-point precision.
  • Images are included. All 24 stored layers and 48 layer applications per token are retained; the middle layers still run three times.
  • Weight-only conversion; no activation-calibrated GPTQ/AWQ claim.
  • Target plus vision weights: 30.04 GB / 27.98 GiB.
  • Approximately 26.4B logical parameters, per upstream. Packed-storage counters may report another number.
  • The auxiliary mtp.safetensors is not bundled. Drafting is not enabled or qualified by this upload.

Run on Apple Silicon

Use a separate environment and a suitable weights/cache directory. On machines using Time Machine, exclude replaceable bulk downloads before downloading them.

python3 -m venv .venv-inkling
.venv-inkling/bin/python -m pip install 'mlx==0.32.4' 'transformers==5.23.0' pillow \
  'git+https://github.com/ml-explore/mlx-lm.git@0c4c99192230f1943dd468d297f0b8a64ae7b7c2'
MLX_ENABLE_TF32=0 .venv-inkling/bin/python -m mlx_lm.generate \
  --model araysuter/Inkling-2-Mini-MLX-8bit --prompt "Write a Python function to reverse a list." --max-tokens 256 --temp 0

The pinned MLX-LM port is required; a stock version without Inkling support cannot load this architecture. Memory use grows with context, concurrency, images and runtime workspace. The upload alone does not qualify two 128K windows on a 64 GiB machine. The measured 5-bit comparison and two-window capacity qualification are below.

For an authenticated OpenAI-compatible API, use the TensorFold fork adapter guide with this checkpoint as the model path. Its tested maximum context is 131,072 tokens total input plus output; other machines need their own RAM qualification.

Validation

Text arithmetic, Python syntax and image color/shape smoke checks passed. The tiny public-text probe has delta NLL 0.008163 and perplexity ratio 1.008196 against the official release. These are not model quality benchmark scores. See VALIDATION.json and QUANTIZATION_MANIFEST.json for results and the conversion recipe. Scripts contain no weights or credentials.

Model license: Apache-2.0; source model terms apply. This is an independent conversion, not an official Thinking Machines release.

Measured 5-bit comparison on M5 Max

Prompt processing and generation speed comparison

M5 Max, 64 GiB, MLX 0.32.4, pinned MLX-LM above; fresh KV, greedy decoding, 256 output tokens, one attempt per cell. Two-user rates are per-user medians, not combined throughput. All two-user outputs matched their own quant's solo reference hashes. Different quants are not expected to generate identical tokens. These synthetic speed probes are not accuracy benchmarks.

Prompt tokens Users 5-bit prompt tok/s 8-bit prompt tok/s 5-bit generation tok/s 8-bit generation tok/s
4,000 1 1742 1694 43.0 40.4
8,000 1 1699 1651 42.3 39.7
32,000 1 1410 1390 38.8 36.6
4,000 2 906 882 21.2 19.8
8,000 2 863 843 20.9 19.6
32,000 2 709 696 19.1 18.0

Eight-bit generation was about 6% slower, and prompt processing about 1–3% slower, than the 5-bit conversion. MLX target allocations immediately after loading were 19.21 GiB for 5-bit and 27.98 GiB for 8-bit, before context/workspace and other process memory.

Two full 128K windows

Two concurrent requests each completed with 131,064 prompt tokens and eight output tokens (131,072 total per request). This is a capacity test, not a long-context generation-speed measurement.

  • Peak measured backend process-group physical footprint: 38.60 GiB.
  • Minimum available host RAM: 11.06 GiB.
  • Additional swap: 0 MiB; memory pressure remained normal.
  • The initial conservative 5.5 GiB-per-window admission estimate refused the second request. With measured BF16 cache growth and a 4.5 GiB-per-window estimate, both completed. The 8 GiB host reserve and stop condition remained unchanged. No KV precision was reduced.

The locally qualified gateway model profile uses memory_required_gib: 31, memory_per_request_gib: 4.5, request_reserve_gib: 8, max_concurrent: 2, and context_window: 131072. Use these with the adapter guide linked above; do not copy the 5-bit profile's 22 GiB model estimate for this checkpoint. These are measurements on one host, not a promise about other machines or background workloads. Admission still checks live RAM.

The API was switched to this 8-bit checkpoint after public image, text, EOS, authentication and two-user hash checks passed. Drafting remains disabled. Eight-bit storage is not full precision and cannot recover precision absent from the authors' original NVFP4/MXFP8 training/release weights.

Raw timings, hashes and memory receipts · Chart redraw script

Downloads last month
-
Safetensors
Model size
26B params
Tensor type
U32
·
BF16
·
F32
·
MLX
Hardware compatibility
Log In to add your hardware

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for araysuter/Inkling-2-Mini-MLX-8bit

Quantized
(40)
this model

Collection including araysuter/Inkling-2-Mini-MLX-8bit