README.md

---
base_model: mistralai/Leanstral-2603
library_name: mlx
tags:
  - rotorquant
  - kv-cache-quantization
  - mlx
  - 2-bit
  - weight-quantization
  - leanstral
  - lean4
  - formal-proofs
  - theorem-proving
  - quantized
  - apple-silicon
  - mistral
  - moe
license: apache-2.0
pipeline_tag: text-generation
---

> [!TIP]
> **KV-cache quantization without any fork (recommended, 2026):** upstream
> llama.cpp/Ollama now cover this natively — use `-ctk q8_0 -ctv q8_0`
> (~half KV memory, negligible quality loss: perplexity +0.002–0.05) or
> `-ctk q4_0 -ctv q4_0` (~quarter memory, ≈7.6% perplexity increase). In
> Ollama: `OLLAMA_KV_CACHE_TYPE=q8_0` with `OLLAMA_FLASH_ATTENTION=1`. Keep
> K and V types symmetric to stay on the fast fused Flash-Attention path.
> Since April 2026, mainline llama.cpp also applies Hadamard rotation to
> KV activations ([PR #21038](https://github.com/ggml-org/llama.cpp/pull/21038)),
> which greatly improves low-bit KV quality (opt-out:
> `LLAMA_ATTN_ROT_DISABLE=1`).
>
> The RotorQuant/TurboQuant fork flow below is **experimental/legacy**: the
> TurboQuant llama.cpp PR was closed without merging (June 2026) and the fork
> is unmaintained relative to mainline. It is NOT required to use this model.
<!-- kv-upstream-note -->

# Leanstral-RotorQuant-MLX-2bit

**2-bit MLX weight-quantized [Leanstral-2603](https://huggingface.co/mistralai/Leanstral-2603) with [RotorQuant](https://github.com/scrya-com/rotorquant) KV-cache quantization for high-throughput Lean 4 formal proof generation on Apple Silicon.**

Leanstral is the first open-source AI agent purpose-built for Lean 4 formal proofs -- generating both executable code and machine-checkable mathematical proofs. This variant combines **dual compression**: 2-bit MLX weight quantization for aggressive model size reduction plus the legacy RotorQuant KV-cache fork (superseded by upstream llama.cpp KV options), delivering.

## Overview

This repository provides an aggressively compressed configuration with RotorQuant's superior throughput: MLX 2-bit weight quantization minimizes the static memory footprint, while RotorQuant's rotation-aware KV-cache compression delivers KV-cache handling.

| Spec | Value |
|------|-------|
| Base model | [mistralai/Leanstral-2603](https://huggingface.co/mistralai/Leanstral-2603) |
| Architecture | Mistral MoE (~119B parameters, 7 consolidated shards) |
| Weight quantization | 2-bit (MLX) |
| KV-cache quantization | RotorQuant |
| Weight memory | ~30 GB |
| Prefill speedup | 5.3x vs TurboQuant |
| Decode speedup | 28% vs TurboQuant |
| Runtime | MLX (Apple Silicon) |
| License | Apache 2.0 |
| Use case | Lean 4 formal verification, theorem proving, mathematical proofs |

## Quickstart

```python
from mlx_lm import load, generate

model, tokenizer = load("majentik/Leanstral-RotorQuant-MLX-2bit")

prompt = "Prove that for all natural numbers n, n + 0 = n in Lean 4:"
response = generate(
    model,
    tokenizer,
    prompt=prompt,
    max_tokens=512,
)
print(response)
```

## About the RotorQuant / TurboQuant labels

RotorQuant and TurboQuant are this project's **release labels**, not distinct
quantization algorithms — for any given tier, both brand repos carry
byte-identical weights produced with the standard MLX / llama.cpp quantizers.
No brand-specific speedup is claimed or measured. The KV-cache fork these
labels originally referred to is legacy; for KV-cache memory savings use the
upstream options described above (`-ctk/-ctv q8_0`, `OLLAMA_KV_CACHE_TYPE`).

## Memory Estimates

| Component | Estimate |
|-----------|----------|
| Model weights (2-bit) | ~30 GB |
| KV-cache | Reduced via RotorQuant |
| Recommended hardware | MacBook Pro M2/M3/M4 Max (64 GB+) or Mac Studio |

## Lean 4 Use Case

Leanstral excels at:
- **Formal verification** -- generating machine-checkable proofs of mathematical theorems
- **Theorem proving** -- interactive and automated proof search in Lean 4
- **Code generation** -- writing verified Lean 4 programs with correctness guarantees
- **Proof repair** -- fixing incomplete or broken proof scripts

## See Also

- [mistralai/Leanstral-2603](https://huggingface.co/mistralai/Leanstral-2603) -- Base model
- [majentik/Leanstral-TurboQuant-MLX-2bit](https://huggingface.co/majentik/Leanstral-TurboQuant-MLX-2bit) -- MLX 2-bit + TurboQuant
- [RotorQuant repository](https://github.com/scrya-com/rotorquant)

## Quant trade-off (MLX lane)

| Bits | Approx size | Use case | Recommendation |
|---|---|---|---|
| **2-bit** | ~31 GB | Aggressive quantization | **Very low-RAM Macs** |
| 3-bit | ~43 GB | Lossy but small | Low-RAM Macs |
| 4-bit | ~50 GB | Balanced default | Recommended for most Macs |
| 5-bit | ~60 GB | Higher fidelity | Quality-sensitive |
| 6-bit | ~71 GB | Approaching FP16 quality | High-fidelity |
| 8-bit | ~90 GB | Near-lossless reference | Fidelity-critical work |

(Current variant — **2bit** — is bolded.)

## Variants in this family

(Showing 8 sibling variants under `majentik/leanstral-*`. The current variant — `RotorQuant-MLX-2bit` — is **bolded**.)

| Variant | Runtime | Approx size | Use case |
|---|---|---|---|
| **RotorQuant-MLX-2bit** | mlx-lm | card-only | Apple Silicon, smallest |
| [RotorQuant-MLX-8bit](https://huggingface.co/majentik/leanstral-rotorquant-mlx-8bit) | mlx-lm | card-only | Apple Silicon reference |
| [TurboQuant-MLX-2bit](https://huggingface.co/majentik/leanstral-turboquant-mlx-2bit) | mlx-lm | card-only | Apple Silicon, smallest |
| [TurboQuant-MLX-8bit](https://huggingface.co/majentik/leanstral-turboquant-mlx-8bit) | mlx-lm | card-only | Apple Silicon reference |