Instructions to use araysuter/Inkling-2-Mini-MLX-8bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use araysuter/Inkling-2-Mini-MLX-8bit with MLX:
# Make sure mlx-vlm is installed # pip install --upgrade mlx-vlm from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config # Load the model model, processor = load("araysuter/Inkling-2-Mini-MLX-8bit") config = load_config("araysuter/Inkling-2-Mini-MLX-8bit") # Prepare input image = ["http://images.cocodataset.org/val2017/000000039769.jpg"] prompt = "Describe this image." # Apply chat template formatted_prompt = apply_chat_template( processor, config, prompt, num_images=1 ) # Generate output output = generate(model, processor, formatted_prompt, image) print(output) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use araysuter/Inkling-2-Mini-MLX-8bit with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "araysuter/Inkling-2-Mini-MLX-8bit"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "araysuter/Inkling-2-Mini-MLX-8bit" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent
How to use araysuter/Inkling-2-Mini-MLX-8bit with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "araysuter/Inkling-2-Mini-MLX-8bit"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default araysuter/Inkling-2-Mini-MLX-8bit
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use araysuter/Inkling-2-Mini-MLX-8bit with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "araysuter/Inkling-2-Mini-MLX-8bit"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "araysuter/Inkling-2-Mini-MLX-8bit" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Inkling 2 Mini — MLX 8-bit
Independent affine MLX conversion of the official
Thinking Machines checkpoint,
revision 6e7c8248f75265bdd4fbab5273304243b5d8c2cb.
The source is the authors' trained NVFP4/MXFP8 release, not an original BF16 checkpoint.
This conversion starts from that source, not from our 5-bit quant. Eight-bit storage
reduces additional affine reconstruction error; it cannot recover precision absent
from the source. No improvement in benchmark quality over 5-bit is claimed.
Format
- Language linear/expert matrices, embeddings and output head: 8-bit affine, group size 32, min/max round-to-nearest.
- Vision tower, expert routing and loop/message gates retain their source floating-point precision.
- Images are included. All 24 stored layers and 48 layer applications per token are retained; the middle layers still run three times.
- Weight-only conversion; no activation-calibrated GPTQ/AWQ claim.
- Target plus vision weights: 30.04 GB / 27.98 GiB.
- Approximately 26.4B logical parameters, per upstream. Packed-storage counters may report another number.
- The auxiliary
mtp.safetensorsis not bundled. Drafting is not enabled or qualified by this upload.
Run on Apple Silicon
Use a separate environment and a suitable weights/cache directory. On machines using Time Machine, exclude replaceable bulk downloads before downloading them.
python3 -m venv .venv-inkling
.venv-inkling/bin/python -m pip install 'mlx==0.32.4' 'transformers==5.23.0' pillow \
'git+https://github.com/ml-explore/mlx-lm.git@0c4c99192230f1943dd468d297f0b8a64ae7b7c2'
MLX_ENABLE_TF32=0 .venv-inkling/bin/python -m mlx_lm.generate \
--model araysuter/Inkling-2-Mini-MLX-8bit --prompt "Write a Python function to reverse a list." --max-tokens 256 --temp 0
The pinned MLX-LM port is required; a stock version without Inkling support cannot load this architecture. Memory use grows with context, concurrency, images and runtime workspace. The upload alone does not qualify two 128K windows on a 64 GiB machine. The measured 5-bit comparison and two-window capacity qualification are below.
For an authenticated OpenAI-compatible API, use the TensorFold fork adapter guide with this checkpoint as the model path. Its tested maximum context is 131,072 tokens total input plus output; other machines need their own RAM qualification.
Validation
Text arithmetic, Python syntax and image color/shape smoke checks passed.
The tiny public-text probe has delta NLL 0.008163 and perplexity ratio
1.008196 against the official release. These are not model quality
benchmark scores. See VALIDATION.json and QUANTIZATION_MANIFEST.json for
results and the conversion recipe. Scripts contain no weights or credentials.
Model license: Apache-2.0; source model terms apply. This is an independent conversion, not an official Thinking Machines release.
Measured 5-bit comparison on M5 Max
M5 Max, 64 GiB, MLX 0.32.4, pinned MLX-LM above; fresh KV, greedy decoding, 256 output tokens, one attempt per cell. Two-user rates are per-user medians, not combined throughput. All two-user outputs matched their own quant's solo reference hashes. Different quants are not expected to generate identical tokens. These synthetic speed probes are not accuracy benchmarks.
| Prompt tokens | Users | 5-bit prompt tok/s | 8-bit prompt tok/s | 5-bit generation tok/s | 8-bit generation tok/s |
|---|---|---|---|---|---|
| 4,000 | 1 | 1742 | 1694 | 43.0 | 40.4 |
| 8,000 | 1 | 1699 | 1651 | 42.3 | 39.7 |
| 32,000 | 1 | 1410 | 1390 | 38.8 | 36.6 |
| 4,000 | 2 | 906 | 882 | 21.2 | 19.8 |
| 8,000 | 2 | 863 | 843 | 20.9 | 19.6 |
| 32,000 | 2 | 709 | 696 | 19.1 | 18.0 |
Eight-bit generation was about 6% slower, and prompt processing about 1–3% slower, than the 5-bit conversion. MLX target allocations immediately after loading were 19.21 GiB for 5-bit and 27.98 GiB for 8-bit, before context/workspace and other process memory.
Two full 128K windows
Two concurrent requests each completed with 131,064 prompt tokens and eight output tokens (131,072 total per request). This is a capacity test, not a long-context generation-speed measurement.
- Peak measured backend process-group physical footprint: 38.60 GiB.
- Minimum available host RAM: 11.06 GiB.
- Additional swap: 0 MiB; memory pressure remained normal.
- The initial conservative 5.5 GiB-per-window admission estimate refused the second request. With measured BF16 cache growth and a 4.5 GiB-per-window estimate, both completed. The 8 GiB host reserve and stop condition remained unchanged. No KV precision was reduced.
The locally qualified gateway model profile uses memory_required_gib: 31,
memory_per_request_gib: 4.5, request_reserve_gib: 8, max_concurrent: 2,
and context_window: 131072. Use these with the adapter guide linked above;
do not copy the 5-bit profile's 22 GiB model estimate for this checkpoint.
These are measurements on one host, not a promise about other machines or
background workloads. Admission still checks live RAM.
The API was switched to this 8-bit checkpoint after public image, text, EOS, authentication and two-user hash checks passed. Drafting remains disabled. Eight-bit storage is not full precision and cannot recover precision absent from the authors' original NVFP4/MXFP8 training/release weights.
Raw timings, hashes and memory receipts · Chart redraw script
- Downloads last month
- -
8-bit
Model tree for araysuter/Inkling-2-Mini-MLX-8bit
Base model
thinkingmachines/Inkling-2-Mini