| --- |
| license: apache-2.0 |
| library_name: mlx |
| pipeline_tag: image-text-to-text |
| base_model: HemmingwayAI/Astrea-R8-Chat-9B |
| base_model_relation: quantized |
| language: |
| - en |
| tags: |
| - mlx |
| - optiq |
| - quantized |
| - mixed-precision |
| - 4bit |
| - 8bit |
| - qwen3.5 |
| - chat |
| - creative-writing |
| - vision-language-model |
| - image-text-to-text |
| - apple-silicon |
| --- |
| |
| # mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit |
|
|
| > **Built with [mlx-optiq](https://mlx-optiq.com)**, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. [Try the Lab](https://mlx-optiq.com/docs/lab/) · [All OptiQ quants](https://mlx-optiq.com/models) · [Docs](https://mlx-optiq.com/docs/) |
|
|
| **Altworld's creative-writing model, quantized for a Mac.** A mixed-precision MLX quant of [HemmingwayAI/Astrea-R8-Chat-9B](https://huggingface.co/HemmingwayAI/Astrea-R8-Chat-9B), a Qwen3.5-9B fine-tune built for chat and fiction. 8.0 GB on disk, down from 18.8 GB bf16. See the [original card](https://huggingface.co/HemmingwayAI/Astrea-R8-Chat-9B) for what the model is trained to do and how Altworld evaluates it. |
|
|
| OptiQ quantizes the language tower to mixed 4/8-bit and keeps the vision tower at bf16 in a sidecar, so one checkpoint does both text and images. |
|
|
| ## What it is |
|
|
| | Property | Value | |
| |---|---| |
| | Base | HemmingwayAI/Astrea-R8-Chat-9B (Qwen3.5-9B fine-tune) | |
| | Language tower | per-layer 4/8-bit: 116 layers at 4-bit, 134 at 8-bit (250 total) | |
| | Vision tower | bf16, kept in `optiq_vision.safetensors` (333 tensors) | |
| | On disk | 7.10 GB language + 0.91 GB vision | |
| | Group size | 64 | |
| | Context | 262,144 tokens | |
|
|
| Following `llama.cpp`'s naming for mixed quants, the "4bit" label denotes the predominant precision, not the weighted average. |
|
|
| ## How the bit-widths were chosen |
|
|
| The per-layer allocation was not measured on this model. It was transferred from [mlx-community/Qwen3.5-9B-OptiQ-4bit](https://huggingface.co/mlx-community/Qwen3.5-9B-OptiQ-4bit), whose allocation came from a KL-divergence sensitivity sweep over a [six-domain calibration mix](https://mlx-optiq.com/blog/calibration-mix). The two language towers share an architecture exactly, and all **250 tensors matched with none unmatched**. |
|
|
| ## Checked after conversion |
|
|
| Run through this quant and its parent, Qwen3.5-9B-OptiQ-4bit, on Apple Silicon with greedy decoding and thinking off: |
|
|
| | Prompt | Astrea | Parent | |
| |---|---|---| |
| | Capital of France, one word | Paris | Paris | |
| | 120 km in 1.5 h, speed in km/h | 80 | 80 | |
| | `is_palindrome(s)` ignoring case and non-letters | correct function | correct function | |
| | What makes a good short-story opening, two sentences | coherent answer in its own wording | coherent answer | |
|
|
| Image input: shown a red rectangle, a blue circle and the number 42, it answered "The image contains a red square, a blue circle, and the number 42 written in black." |
|
|
| ## Serve it with n-gram drafting |
|
|
| Measured on a 24 GB M4, batch 1, greedy, median of three runs each: |
|
|
| | Workload | Plain | `--ngram-draft 16` | |
| |---|---|---| |
| | Edit (rename a parameter across a file and return it) | 16.6 tok/s | **44.9 tok/s (2.7x)** | |
| | Code generation | 16.7 tok/s | 18.0 tok/s (1.08x) | |
| | Prose from a 1,024-token prompt | 16.6 tok/s | 16.2 tok/s (0.98x) | |
|
|
| ```bash |
| pip install mlx-optiq |
| optiq serve --model mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit --ngram-draft 16 |
| ``` |
|
|
| This checkpoint has no MTP head, so `--mtp` does not apply. |
|
|
| ## Use it |
|
|
| ```bash |
| # chat, images, and every app that talks to optiq serve |
| optiq serve --model mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit --ngram-draft 16 |
| optiq lab --model mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit |
| ``` |
|
|
| Text only, with stock mlx-lm: |
|
|
| ```python |
| from mlx_lm import load, generate |
| |
| model, tokenizer = load("mlx-community/Astrea-R8-Chat-9B-OptiQ-4bit") |
| messages = [{"role": "user", "content": "Open a short story set on a night ferry."}] |
| prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False) |
| print(generate(model, tokenizer, prompt=prompt, max_tokens=400)) |
| ``` |
|
|
