README.md

---
license: mit
base_model:
  - peculiar-ragdoll/Cyber-Tiel-Coder-35B-A3B-MLX-oQ6e-MTP
base_model_relation: quantized
pipeline_tag: image-text-to-text
library_name: mlx
language:
  - en
  - zh
tags:
  - mlx
  - omlx
  - apple-silicon
  - m1
  - oq
  - qwen35moe
  - moe
  - mtp
  - speculative-decoding
  - fixed-mtp
  - fp16
  - agentic-coding
  - abliterated
  - uncensored
  - vision
---

# CyberTiel-Coder-35B-MLX (FP16 tensors · fixed MTP head · tuned for oMLX on Apple Silicon)

**English** · [中文](#中文说明)

A 35B-A3B MoE coding/agent model (6-bit oQ6e, vision-capable, abliterated) in the exact form we run every day on a Mac Studio M1 Ultra 64 GB with [oMLX](https://github.com/jundot/omlx) 0.7.0.
Compared with the upstream pack it has two local changes, both measured:

1. **Non-quantized BF16 tensors cast to FP16** (M1/M2 have no native bfloat16). Prefill **+28–50 %**, decode at 16K context **+22 %**, quality unchanged (206/260 = 206/260).
2. **Rebuilt MTP head** ("fixedmtp": 42-key fused layout) so that oMLX's Lightning MTP speculative decoding runs with an **adaptive depth** → **94–116 tok/s** chat decode.

Everything below is measured on one machine; numbers are not a promise for yours. Full method and raw data: [tuning notes dataset](https://huggingface.co/datasets/YCF-AI/omlx-m1-ultra-tuning-notes) · [GitHub](https://github.com/6667084/omlx-m1-ultra-tuning-notes).

> ⚠️ **Abliterated (refusal-removed) model.** Meant for emergency / red-team / research work. Run it inside an OS-level sandbox with restricted network and execution rights, and guard against prompt injection. You are responsible for how you use it.

![local stack overview](images/fig11_local_stack_overview.png)

*Updated 2026-10-10: the retrieval pair of this stack is now [BGE-M3 8-bit](https://huggingface.co/YCF-AI/bge-m3-8bit-MLX) + [bge-reranker-v2-m3 (FP16 embedding tables)](https://huggingface.co/YCF-AI/bge-reranker-v2-m3-fp16emb); the previous embedding model, Qwen3-Embedding-0.6B, was retired. 更新于 2026-10-10：本机栈的检索组合现为上述 BGE-M3 8-bit + bge-reranker-v2-m3，旧嵌入模型 Qwen3-Embedding-0.6B 已下线。*

## At a glance

| Item | Value |
|---|---|
| Architecture | Qwen3.5-MoE style, 35B total / ~3B active (256 experts, 8 active), vision tower included |
| Quantization | oQ6e mixed (≈6-bit) + non-quantized tensors in **FP16** (this repo) |
| Size on disk | ≈ 28 GB (6 shards + MTP head) |
| Context | 131072 configured (native 262144); 16K / 32K / 64K / 128K measured below |
| Max output | 32768 (configured; context and output share one budget) |
| Speed (M1 Ultra, chat) | **107–116 tok/s** decode; prefill 4K ≈ 1890, 16K ≈ 1640 tok/s |
| Thinking | On by default; `reasoning_effort` none/low/medium/xhigh works |
| Vision | Yes (emergency use: OCR digits sometimes wrong, counting weak) |
| Tool calling | Yes |
| License | MIT (upstream card; see `LICENSE`, `README-upstream.md`) |

**Lineage:** `ornith-ai/Ornith-1.5-35B-A3B` → `huihui-ai/Huihui-Ornith-…-abliterated` (refusal removal) → `peculiar-ragdoll/Cyber-Tiel-Coder-35B-A3B-MLX-oQ6e-MTP` (rev `a443d4e`) → our fixedmtp head → FP16 tensor cast (this repo). The previous BF16 build of this repo stays reachable as revision `32e067dd15c07fbfcf841d53efd011f90c6527b7` (recommended for Macs with native bf16, M3 and later — untested by us).

## Measured results

### 1. BF16 → FP16 on M1 (the biggest free win)

![fp16 cast](images/fig3_fp16_cast_prefill_decode.png)

| Metric (ABBA, same session) | BF16 (as published) | FP16 (this repo) |
|---|---|---|
| Prefill 1K / 4K / 16K (tok/s) | 1085 / 1406 / 1096 | **1393 / 1891 / 1644** |
| First token at 16K prompt | 14.9 s | **10.0 s** |
| Decode at 16K context | 96 tok/s | **117 tok/s** |
| Chat decode (3 prompts) | 102.5 / 108.7 | **109.1 / 112.3** |
| 260-question quality set (T=0) | 206 / 260 | 206 / 260 |

What was cast: 1569 tensors (scales, biases, norms, embeddings, MTP head, vision tower; 2.94 GB). Max |x| = 25.4 (far inside the FP16 range), 26 103 tiny values underflowed to zero. The quantized U32 data and `model.safetensors.index.json` are unchanged. Script: [`recipe/to_fp16_v2.py`](recipe/to_fp16_v2.py) (aborts if any value would overflow).
A pure short-decode test showed *no* gain and misled us once — always measure prefill at 1K/4K/16K and 16K decode.

### 2. MTP depth (Lightning MTP, chat decode tok/s, BF16 baseline sweep)

![mtp depth](images/fig1_cybertiel_mtp_depth.png)

Leave `mtp_fixed_depth` and `mtp_adaptive_max_depth` **empty** (0.7.0 model default = adaptive): 94.2 tok/s vs 89.8 for fixed d2, 84.7 for d3, 59.4 with MTP off. Note: oMLX 0.7.0 silently ignores the old `mtp_num_draft_tokens` field.

### 3. Long context (hot cache off)

| Context | Prefill tok/s | Decode tok/s |
|---|---:|---:|
| 16K | 1620 | ~100 |
| 32K | 1320 | 86–89 |
| 64K | 957 | 71 |
| 128K | 614 (first token 3.6 min) | 60 |

Peak memory with this model alone at 128K: 36.8 GB (fits the default 48 GB Metal limit on a 64 GB Mac).

### 4. Code quality depends on the thinking mode you use

![thinking flip](images/fig8_code_candidates_thinking_flip.png)

We rejected a newer upstream requant (09-29) and a rival code model after testing in the **thinking-on production mode** (LiveCodeBench-60 subset 38 vs 40; 324 code questions 247 vs 242, p = 0.30). Benchmarking with thinking off would have picked the wrong model.

### 5. What we tried and rejected for this model

TurboQuant KV (decode −20 %), ANE prefill (slower), DFlash / SpecPrefill (no gain), `model_type_override=llm` to save memory (**decode −23 %, image requests silently hallucinate — never do this**), hot cache on 64 GB (starves long-context admission).

## Recommended oMLX settings

Per-model entry: [`omlx/model_settings.entry.json`](omlx/model_settings.entry.json) (key = model directory name).

```json
{
  "max_context_window": 131072, "max_tokens": 32768,
  "temperature": 0.6, "top_p": 0.95, "top_k": 20, "repetition_penalty": 1.0,
  "enable_thinking": true, "thinking_budget_enabled": false,
  "mtp_enabled": true, "vlm_mtp_enabled": false,
  "ttl_seconds": 1800, "model_alias": "CyberTiel-35B"
}
```

Global (see the dataset repo's `config/` for the full redacted files): hot cache **off** (`hot_cache_max_size: "0"`), SSD cache on, `max_concurrent_requests: 8`, `memory_guard_tier: balanced` with `soft_threshold` left at 0.85 (0.70 legacy values make models evict each other), TurboQuant/DFlash/SpecPrefill/ANE prefill all off.

## Thinking control

| Goal | Request field |
|---|---|
| Thinking off | `"reasoning_effort": "none"` (or `chat_template_kwargs: {"enable_thinking": false}`) |
| Effort | `"reasoning_effort": "low" \| "medium" \| "xhigh"` (`high`/`max` = xhigh); default medium |
| Where the reasoning goes | separate `reasoning_content` field |

## Quick start

```bash
# 1) download (mirror example for mainland China)
HF_ENDPOINT=https://hf-mirror.com hf download YCF-AI/CyberTiel-Coder-35B-MLX \
  --local-dir ~/models/CyberTiel-Coder-35B-MLX
# 2) point oMLX model_dirs at ~/models, merge omlx/model_settings.entry.json into ~/.omlx/model_settings.json, restart oMLX
# 3) chat
curl http://127.0.0.1:8000/v1/chat/completions -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"model":"CyberTiel-35B","messages":[{"role":"user","content":"Write a Python LRU cache."}],"reasoning_effort":"low"}'
```

The MTP head (`model-mtp.safetensors`) is used by oMLX's Lightning MTP; we have **not** tested other runtimes with it. Verify downloaded files with the SHA-256 values in the HF file listing.

## Moving to another machine — checklist

1. Install the same oMLX version (0.7.0 or newer), put the model folder under a `model_dirs` path.
2. Merge the settings entry above; keep MTP depth fields empty.
3. Disable hot cache, keep concurrency 8, keep the memory guard at `balanced`.
4. Measure *your* machine: 3-prompt chat decode, prefill 1K/4K/16K, 16K decode, then (only if you changed something) a fixed quality set with McNemar. Our protocol is in the dataset repo.
5. If the chip has native bf16 (M3+), compare the BF16 revision with this one before choosing.

## Known limits

- Vision is an emergency capability: digit OCR errors, weak counting. Use a dedicated VLM for precision ([MiniCPM-V-4.6 build](https://huggingface.co/YCF-AI/MiniCPM-V-4.6-Huihui-abliterated-8bit-MLX)).
- Abliteration removes refusals, not hallucinations.
- All speed numbers: Mac Studio M1 Ultra 64-core GPU, 64 GB, macOS 15.8, oMLX 0.7.0 (build 2987), MLX 0.32.2, mlx-lm 0.31.4.dev, mlx-vlm 0.7.1.

---

<a id="中文说明"></a>

# 中文说明

**[English](#cybertiel-coder-35b-mlx-fp16-tensors--fixed-mtp-head--tuned-for-omlx-on-apple-silicon)** · 中文

35B-A3B MoE 代码/智能体模型（6-bit oQ6e、带视觉、去审查），就是我们每天在 Mac Studio M1 Ultra 64 GB 上用 [oMLX](https://github.com/jundot/omlx) 0.7.0 实际运行的版本。相对上游打包有两处本地改动，均有实测：

1. **非量化 BF16 张量转 FP16**（M1/M2 没有原生 bf16）：prefill **+28–50 %**、16K 上下文解码 **+22 %**，质量不变（260 题 206 = 206）。
2. **重建 MTP 头**（fixedmtp，42 键融合布局），配合 oMLX Lightning MTP **自适应深度**，对话解码 **94–116 tok/s**。

所有数字来自同一台机器，不保证在你的机器上一致。完整方法与原始数据见[调优笔记数据集](https://huggingface.co/datasets/YCF-AI/omlx-m1-ultra-tuning-notes)、[GitHub](https://github.com/6667084/omlx-m1-ultra-tuning-notes)。

> ⚠️ **去审查（abliterated）模型**，仅用于应急测试 / 红队 / 研究。请放进 OS 级沙箱，限制网络与执行权限，并防范提示注入；使用责任自负。

## 概览

| 项目 | 值 |
|---|---|
| 架构 | Qwen3.5-MoE 系，总参数 35B / 激活约 3B（256 专家激活 8），含视觉塔 |
| 量化 | oQ6e 混合（约 6-bit）+ 非量化张量 **FP16**（本仓库） |
| 体积 | 约 28 GB（6 个分片 + MTP 头） |
| 上下文 | 配置 131072（原生 262144）；16K/32K/64K/128K 实测见下 |
| 最大输出 | 32768（上下文与输出共用一个预算） |
| 速度（M1 Ultra） | 解码 **107–116 tok/s**；prefill 4K ≈ 1890、16K ≈ 1640 tok/s |
| 思考模式 | 默认开；`reasoning_effort` 支持 none/low/medium/xhigh |
| 视觉 | 有（应急：OCR 数字偶错、计数弱） |
| 工具调用 | 有 |
| 许可 | MIT（上游卡片；见 `LICENSE`、`README-upstream.md`） |

**血统：** `ornith-ai/Ornith-1.5-35B-A3B` → `huihui-ai/Huihui-Ornith-…-abliterated` (refusal removal) → `peculiar-ragdoll/Cyber-Tiel-Coder-35B-A3B-MLX-oQ6e-MTP`（rev `a443d4e`）→ 自建 fixedmtp 头 → FP16 张量转换（本仓库）。本仓库此前的 BF16 版仍可用 revision `32e067dd15c07fbfcf841d53efd011f90c6527b7` 取回（原生支持 bf16 的 M3 及以后机型可优先对比，我们未测）。

## 实测结果

**1. BF16 → FP16（M1 上最大的免费收益）**——图见上（英文部分 `fp16 cast`）。同时段 ABBA：prefill 1K/4K/16K 1085/1406/1096 → **1393/1891/1644** tok/s；16K 首字 14.9 → **10.0 s**；16K 上下文解码 96 → **117**；对话解码 102.5/108.7 → **109.1/112.3**；260 题 206 = 206。
转换了 1569 个张量（scales、biases、norm、embedding、MTP 头、视觉塔；2.94 GB），最大绝对值 25.4，26103 个极小值下溢为 0；量化 U32 数据与 index 不变。脚本 [`recipe/to_fp16_v2.py`](recipe/to_fp16_v2.py)（任何值溢出即中止）。**只测短上下文纯解码会得出"无收益"的错误结论**，必须测 1K/4K/16K prefill 与 16K 解码。

**2. MTP 深度**——`mtp_fixed_depth` 与 `mtp_adaptive_max_depth` **留空**（0.7.0 默认=自适应）最优：94.2 tok/s，固定 d2 89.8、d3 84.7、关 MTP 59.4。注意 0.7.0 已**静默忽略**旧字段 `mtp_num_draft_tokens`。

**3. 长上下文（hot cache 关闭）**：16K 1620/~100、32K 1320/86–89、64K 957/71、128K 614（首字 3.6 分钟）/60（prefill/解码 tok/s）。单模型 128K 峰值内存 36.8 GB，64 GB Mac 默认 48 GB Metal 上限内可用。

**4. 代码质量取决于你实际使用的思考模式**：我们用**开思考的生产口径**否决了上游 09-29 重量化版与另一款代码模型（LCB60 38 vs 40；324 题 247 vs 242，p=0.30）；关思考评测会选错模型。

**5. 试过并否决**：TurboQuant KV（解码 −20 %）、ANE prefill（更慢）、DFlash/SpecPrefill（无收益）、`model_type_override=llm` 省内存（**解码 −23 %、图片请求静默幻觉，切勿使用**）、64 GB 上开 hot cache（挤占长上下文准入）。

## 推荐 oMLX 设置

单模型条目见 [`omlx/model_settings.entry.json`](omlx/model_settings.entry.json)（键名=模型目录名），参数见上方英文部分 JSON。全局：hot cache **关**（`hot_cache_max_size: "0"`）、SSD 缓存开、`max_concurrent_requests: 8`、`memory_guard_tier: balanced` 且 `soft_threshold` 保持 0.85（遗留的 0.70 会让模型互相驱逐）、TurboQuant/DFlash/SpecPrefill/ANE prefill 全关。完整脱敏配置见数据集仓库 `config/`。

## 思考控制

| 目标 | 请求字段 |
|---|---|
| 关闭思考 | `"reasoning_effort": "none"`（或 `chat_template_kwargs: {"enable_thinking": false}`） |
| 力度 | `"reasoning_effort": "low" \| "medium" \| "xhigh"`（`high`/`max` 等同 xhigh），默认 medium |
| 推理内容 | 走独立的 `reasoning_content` 字段 |

## 快速开始

```bash
# 下载（国内用镜像、直连不走代理）
HF_ENDPOINT=https://hf-mirror.com hf download YCF-AI/CyberTiel-Coder-35B-MLX --local-dir ~/models/CyberTiel-Coder-35B-MLX
# 把 ~/models 加入 oMLX 的 model_dirs，合并 omlx/model_settings.entry.json 到 ~/.omlx/model_settings.json，重启 oMLX
```

MTP 头 `model-mtp.safetensors` 由 oMLX Lightning MTP 使用，其他运行时我们**未测**。下载后请用 HF 文件列表中的 SHA-256 校验。

## 换机器部署清单

1. 安装同版本 oMLX（0.7.0 或更新），把模型目录放进 `model_dirs`。
2. 合并上面的设置条目，MTP 深度字段留空。
3. 关 hot cache、并发 8、内存守卫 `balanced`。
4. 在**你的机器**上实测：3 提示词对话解码、1K/4K/16K prefill、16K 解码；若改了东西，再用固定题集 + McNemar 检验。协议见数据集仓库。
5. 芯片支持原生 bf16（M3+）时，先对比 BF16 版本再选。

## 已知限制

- 视觉仅应急：数字 OCR 会错、计数弱；精确任务请用专用 VLM（[MiniCPM-V-4.6 版](https://huggingface.co/YCF-AI/MiniCPM-V-4.6-Huihui-abliterated-8bit-MLX)）。
- 去审查只去掉拒答，不去掉幻觉。
- 全部速度数据：Mac Studio M1 Ultra 64 核 GPU、64 GB、macOS 15.8、oMLX 0.7.0（build 2987）、MLX 0.32.2、mlx-lm 0.31.4.dev、mlx-vlm 0.7.1。