---
title: README
emoji: 🐢
colorFrom: green
colorTo: gray
sdk: static
pinned: false
license: mit
---

<p align="center">
  <img src="https://cdn-avatars.huggingface.co/v1/production/uploads/6693b1fe8437b4d339a5e4e9/LW7MVkHebrqJZyPD8ea55.png" width="96" alt="Agnostic">
</p>

# Agnostic

**Quantized weights, built for the metal.**

**Website: [lisa-ml.com](https://lisa-ml.com/)**

Agnostic publishes **MLX-format, affine-quantized builds** of recent language
and multimodal models, shaped for fast local inference on Apple Silicon and
packed for [`lisa`](https://lisa-ml.com/) — a pure-Rust Metal inference engine
with no Python, no MLX runtime, and no tensor framework at load time. Built from
scratch on the Metal C API, with an OpenAI-compatible server.

Every checkpoint here is a **faithful re-pack of the public original**, not a
fine-tune: same weights, same tokenizer, same chat template, reduced precision.
Quantization is deliberately boring — one `bf16` scale/bias pair per group,
packed `u32` payloads, group 32/64 as required by the kernels that consume them.

## Models

| Model | Base | Bits | Group | Notes |
|---|---|---|---|---|
| [Qwen3.8-Flash-Next-4bit](https://huggingface.co/agnosticeng/Qwen3.8-Flash-Next-4bit) | [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) | 4 | 32 | `qwen4_exp` text tower + MTP head + merged n-gram table |
| [Qwen3.8-27B-4bit](https://huggingface.co/agnosticeng/Qwen3.8-27B-4bit) | [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) | 4 | 64 | vision tower included, MTP head bundled |
| [Qwen3.8-27B-2bit](https://huggingface.co/agnosticeng/Qwen3.8-27B-2bit) | [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) | 2 | 64 | text-only, MTP head bundled |
| [MiMo-V2.6-Distill-Qwen-9B-4bit](https://huggingface.co/agnosticeng/MiMo-V2.6-Distill-Qwen-9B-4bit) | [XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B) | 4 | 64 | vision tower included, image-text-to-text |

## What's in the box

- **Affine quantization** (`mx.quantize`): packed `u32` weights plus one
  `bf16` scale and bias per group along the input dimension.
- **Group size is fixed by the checkpoint** and must be honored by the loader.
  Group 32 is required by lisa's fused Flash-Next decode kernels; other builds
  use group 64 unless noted.
- **Only linear layers are quantized.** Embeddings, norms, convolutions and the
  vision patch embedder keep original precision.
- **MTP heads ride along** where the base model has one, for lossless
  speculative decoding: `mtp/` holds the multi-token-prediction sidecar.
- **Transformers are reworked** to the engine's expected layout at publish time
  (prefix renames, MoE `switch_mlp` packing, conv1d transposes, RMSNorm fold) —
  loading is the contract test, not a separate transform pass.

## Usage

With **lisa**, `--model` takes a repo id directly:

```bash
lisa serve --model agnosticeng/Qwen3.8-Flash-Next-4bit --addr 127.0.0.1:8000
```

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Hello"}],"max_tokens":64}'
```

With **MLX / mlx-vlm** these are ordinary quantized checkpoints:

```python
from mlx_vlm import load, generate
model, processor = load("agnosticeng/MiMo-V2.6-Distill-Qwen-9B-4bit")
```

Greedy decoding is bit-exact against the reference engine's goldens
(**256/256**, serial and MTP depths 1–6 on the public long-copy gate).

## Conversion

Built from the public originals by streaming the checkpoint shard by shard
(download → quantize → delete), so no host ever holds the full-precision model
twice. Scripts and the exact transforms are documented on each model card.

## Links

- Website — [lisa-ml.com](https://lisa-ml.com/)
- Engine — [lisa](https://lisa-ml.com/)

## Licensing

Weights carry the **base model's license** — see each card and its `LICENSE`
file. We claim no rights over the upstream models; please credit and comply with
the original terms.