Qwythos-9B-Claude-Mythos-5-1M-MLX-bf16-mtp

This repository contains a bfloat16 MLX conversion of empero-ai/Qwythos-9B-Claude-Mythos-5-1M with an integrated Multi-Token Prediction (MTP) head, for Apple Silicon inference with oMLX (the community fork that supports MTP draft-mode decoding).

No additional fine-tuning was performed for this repository. The weights were converted from the upstream checkpoint to MLX-compatible safetensors while preserving the upstream Apache-2.0 license and model behavior.

v3 Update Notice

This MLX conversion has been refreshed with the upstream v3 files. If you downloaded this model before the v3 refresh, please redownload or update this repository.

v3 is a hotfix for the embedded chat template. The updated files:

• update the embedded chat template for preserved reasoning and adaptive thinking;
• fix looping during long generation traces;
• fix agentic use in harnesses such as OpenCode, Abacus, Hermes, and Claude Code.

Users with older local copies should update this MLX model before using it with oMLX.

Compatibility Notice

oMLX is required to load this model. Standard mlx-vlm, mlx-lm, and LM Studio do NOT support MTP and will fail with ValueError: Received 15 parameters not in model: language_model.mtp.*. See the MTP section below for details.

Model Summary

• Format: MLX safetensors
• Precision: bfloat16
• Parameters: about 9B
• Context length: 1,048,576 tokens in the model config
• Architecture: Qwen3.5-style hybrid attention text model
• Primary use: local text generation and reasoning on Apple Silicon
• Upstream model: empero-ai/Qwythos-9B-Claude-Mythos-5-1M

Qwythos-9B is a reasoning-focused model derived from Qwen3.5-9B and post-trained by Empero AI on Claude Mythos and Claude Fable reasoning traces. The upstream model card describes strong emphasis on long-context reasoning, native function calling, tool-augmented workflows, cybersecurity, biomedical reasoning, math, and agentic tasks.

Installation

pip install -U git+https://github.com/jundot/omlx.git

Recommended Sampling

The upstream model is a reasoning model. It is usually best to allow enough generation budget for a reasoning trace and final answer.

Suggested defaults:

generation_kwargs = {
    "temperature": 0.6,
    "top_p": 0.95,
    "top_k": 20,
    "repetition_penalty": 1.05,
    "max_tokens": 4096,
}

Increase max_tokens for difficult reasoning, tool-use, code, or long-context tasks.

MTP (Multi-Token Prediction) — oMLX Edition

What is MTP?

Multi-Token Prediction (MTP) adds a shallow auxiliary head on top of the base model to predict the next 1–3 tokens in parallel. During decoding this enables speculative draft-mode inference — the MTP head proposes candidate tokens and the main trunk verifies them, yielding a 1.5×–2× speedup on hardware that supports it.

The upstream empero-ai/Qwythos-9B-Claude-Mythos-5-1M safetensors release ships without an MTP head. The MTP head is only present in the companion GGUF (Qwythos-9B-Claude-Mythos-5-1M-MTP-BF16.gguf). This *-mtp/ variant restores the MTP head for the MLX ecosystem.

Why oMLX and not standard mlx-vlm?

Standard mlx-vlm and mlx-lm do not include MTP layer definitions for Qwen3.5-family models. Loading this model with either of them produces:

ValueError: Received 15 parameters not in model:
language_model.mtp.fc.weight, language_model.mtp.layers.0.*, ...

oMLX (jundot/omlx) is a community fork of the MLX ecosystem that adds monkey patches for MTP support. It dynamically registers the language_model.mtp.* layers at load time and implements speculative draft-mode decoding in its inference engine. Without oMLX, the MTP weights are present but unusable.

LM Studio ships its own bundled mlx-vlm backend and also lacks MTP support — this model will not work in LM Studio.

How to Use with oMLX

# Install oMLX (community fork with MTP support)
pip install -U git+https://github.com/jundot/omlx.git

# Start an OpenAI-compatible server with MTP draft-mode
omlx serve --model /path/to/Qwythos-9B-Claude-Mythos-5-1M-MLX-bf16-mtp

With the oMLX server, draft-mode speculation is enabled automatically via the mtp_num_hidden_layers: 1 config. You can also invoke it programmatically:

from omlx import load, generate

model_id = "/path/to/Qwythos-9B-Claude-Mythos-5-1M-MLX-bf16-mtp"

model, tokenizer = load(model_id)

# MTP draft-mode: the model uses its auxiliary head to propose
# speculative tokens, speeding up autoregressive decoding.
result = generate(
    model=model,
    tokenizer=tokenizer,
    prompt="Explain speculative decoding with MTP.",
    max_tokens=2048,
    temperature=0.6,
    # draft_mode=True enables the MTP head automatically
    # when mtp_num_hidden_layers > 0 in config.json
)

print(result)

MTP Extraction & Merge Details

This *-mtp/ directory was produced by the following pipeline:

1. Extract MTP weights from GGUF — The original MTP head was extracted from empero-ai/Qwythos-9B-Claude-Mythos-5-1M-MTP-BF16.gguf (block blk.32 nextn tensors) using llama.cpp's GGUF parser.

2. Remap to MLX key names — The extracted PyTorch/LlamaCPP keys were converted to MLX safetensors naming convention:

   language_model.mtp.*          ← MTP auxiliary head weights
   language_model.ln_head.*      ← final layernorm (included)
   

3. Merge into safetensors shards — The MTP weights were merged into the last safetensors shard (model-00004-of-00004.safetensors, ~3.1 GB), bringing the total shard count to 4 and adding 15 MTP-specific keys (layer norm + projection weight/bias for depth-1 next-token prediction).

4. Config update — config.json was patched with:

   "text_config": {
       "mtp_num_hidden_layers": 1,
       "mtp_use_dedicated_embeddings": false
   }
   

Result: 4 safetensors shards (5.0G + 4.9G + 5.0G + 3.1G = ~18 GB total), with the MTP head fully integrated and recognized by oMLX draft-mode decoding.

Verification

The MTP head was validated against the GGUF trunk at depths 1–3, confirming weight parity with the original Qwythos-9B-Claude-Mythos-5-1M-MTP-BF16.gguf.

Files

File	Size	Description
config.json	3.1 KB	Updated with mtp_num_hidden_layers: 1
model-00001-of-00004.safetensors	5.0 GB	Base model shard 1/4
model-00002-of-00004.safetensors	4.9 GB	Base model shard 2/4
model-00003-of-00004.safetensors	5.0 GB	Base model shard 3/4
model-00004-of-00004.safetensors	3.1 GB	Base model shard 4/4 + MTP head
tokenizer.json	—	Qwen3.5 tokenizer
tokenizer_config.json	—	Chat template (chatml)

Function Calling

The upstream Qwythos model is designed to follow Qwen3.5-style tool-calling templates. Use the tokenizer chat template with a tools argument when your runtime supports it, then parse emitted tool call blocks in your application.

Exact tool execution, validation, and safety policy should be handled outside the model.

Limitations and Safety

• This is a format conversion, not a new training run. See the upstream model card for training data, benchmark details, and original limitations.
• The model can produce incorrect details, especially for identifiers, citations, medical facts, security facts, and fast-changing topics. Use retrieval, tools, or human review when exactness matters.
• The upstream model is intentionally less refusal-oriented than many assistant models. Add application-level safety controls for public or end-user-facing deployments.
• Do not rely on this model as a sole source for medical, legal, financial, or security-critical decisions.

License

This MLX conversion is released under the same license as the upstream model: Apache-2.0.

Acknowledgements

• Original model: empero-ai/Qwythos-9B-Claude-Mythos-5-1M
• Original developers: Empero AI
• Base family: Qwen3.5
• MTP support: oMLX (community fork with MTP monkey patches)
• MLX ecosystem: Apple MLX