README.md

---
license: apache-2.0
library_name: mlx
pipeline_tag: image-text-to-text
base_model: HemmingwayAI/Astrea-R8-Chat-9B
base_model_relation: quantized
language:
- en
tags:
- mlx
- optiq
- quantized
- mixed-precision
- 4bit
- 8bit
- qwen3.5
- chat
- creative-writing
- vision-language-model
- image-text-to-text
- apple-silicon
---

# mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit

> **Built with [mlx-optiq](https://mlx-optiq.com)**, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. [Try the Lab](https://mlx-optiq.com/docs/lab/) · [All OptiQ quants](https://mlx-optiq.com/models) · [Docs](https://mlx-optiq.com/docs/)

**Altworld's creative-writing model, quantized for a Mac.** A mixed-precision MLX quant of [HemmingwayAI/Astrea-R8-Chat-9B](https://huggingface.co/HemmingwayAI/Astrea-R8-Chat-9B), a Qwen3.5-9B fine-tune built for chat and fiction. 8.0 GB on disk, down from 18.8 GB bf16. See the [original card](https://huggingface.co/HemmingwayAI/Astrea-R8-Chat-9B) for what the model is trained to do and how Altworld evaluates it.

OptiQ quantizes the language tower to mixed 4/8-bit and keeps the vision tower at bf16 in a sidecar, so one checkpoint does both text and images.

## What it is

| Property | Value |
|---|---|
| Base | HemmingwayAI/Astrea-R8-Chat-9B (Qwen3.5-9B fine-tune) |
| Language tower | per-layer 4/8-bit: 116 layers at 4-bit, 134 at 8-bit (250 total) |
| Vision tower | bf16, kept in `optiq_vision.safetensors` (333 tensors) |
| On disk | 7.10 GB language + 0.91 GB vision |
| Group size | 64 |
| Context | 262,144 tokens |

Following `llama.cpp`'s naming for mixed quants, the "4bit" label denotes the predominant precision, not the weighted average.

## How the bit-widths were chosen

The per-layer allocation was not measured on this model. It was transferred from [mlx-community/Qwen3.5-9B-OptiQ-4bit](https://huggingface.co/mlx-community/Qwen3.5-9B-OptiQ-4bit), whose allocation came from a KL-divergence sensitivity sweep over a [six-domain calibration mix](https://mlx-optiq.com/blog/calibration-mix). The two language towers share an architecture exactly, and all **250 tensors matched with none unmatched**.

## Checked after conversion

Run through this quant and its parent, Qwen3.5-9B-OptiQ-4bit, on Apple Silicon with greedy decoding and thinking off:

| Prompt | Astrea | Parent |
|---|---|---|
| Capital of France, one word | Paris | Paris |
| 120 km in 1.5 h, speed in km/h | 80 | 80 |
| `is_palindrome(s)` ignoring case and non-letters | correct function | correct function |
| What makes a good short-story opening, two sentences | coherent answer in its own wording | coherent answer |

Image input: shown a red rectangle, a blue circle and the number 42, it answered "The image contains a red square, a blue circle, and the number 42 written in black."

## Serve it with n-gram drafting

Measured on a 24 GB M4, batch 1, greedy, median of three runs each:

| Workload | Plain | `--ngram-draft 16` |
|---|---|---|
| Edit (rename a parameter across a file and return it) | 16.6 tok/s | **44.9 tok/s (2.7x)** |
| Code generation | 16.7 tok/s | 18.0 tok/s (1.08x) |
| Prose from a 1,024-token prompt | 16.6 tok/s | 16.2 tok/s (0.98x) |

```bash
pip install mlx-optiq
optiq serve --model mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit --ngram-draft 16
```

This checkpoint has no MTP head, so `--mtp` does not apply.

## Use it

```bash
# chat, images, and every app that talks to optiq serve
optiq serve --model mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit --ngram-draft 16
optiq lab --model mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit
```

Text only, with stock mlx-lm:

```python
from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit")
messages = [{"role": "user", "content": "Open a short story set on a night ferry."}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
print(generate(model, tokenizer, prompt=prompt, max_tokens=400))
```