Instructions to use infosave/flint-cmf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- cortiq
How to use infosave/flint-cmf with cortiq:
# one Rust binary, no additional dependencies cargo install cortiq-cli # or a prebuilt binary from github.com/infosave2007/cmf/releases hf download infosave/flint-cmf --include "*.cmf" --local-dir . ls *.cmf # some repos ship more than one quantization
cortiq run FILE.cmf --prompt "What is the capital of France?"
cortiq serve FILE.cmf --port 8080 # OpenAI-compatible server
- Notebooks
- Google Colab
- Kaggle
docs: document v0.8.30 native Metal Flint path
Browse files
README.md
CHANGED
|
@@ -148,16 +148,42 @@ and route locality all matter.
|
|
| 148 |
|
| 149 |
### Apple silicon / Metal status
|
| 150 |
|
| 151 |
-
The
|
| 152 |
-
|
| 153 |
-
|
| 154 |
-
|
| 155 |
-
|
| 156 |
-
|
| 157 |
-
|
| 158 |
-
|
| 159 |
-
|
| 160 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 161 |
|
| 162 |
## Integrity and legacy files
|
| 163 |
|
|
@@ -289,15 +315,41 @@ CMF_GPU=wgpu CMF_GPU_WGPU_GRAPH=1 CMF_MULTISTEP=4 \
|
|
| 289 |
на каждом prompt. Скорость зависит от GPU, драйвера, свободной VRAM, контекста,
|
| 290 |
длины вывода и локальности маршрутов.
|
| 291 |
|
| 292 |
-
|
| 293 |
-
|
| 294 |
-
|
| 295 |
-
|
| 296 |
-
|
| 297 |
-
|
| 298 |
-
|
| 299 |
-
|
| 300 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 301 |
|
| 302 |
### Целостность и старые файлы
|
| 303 |
|
|
|
|
| 148 |
|
| 149 |
### Apple silicon / Metal status
|
| 150 |
|
| 151 |
+
The released q2tp layout has a native-Metal **direct selected-expert** path.
|
| 152 |
+
The host keeps the source-exact route selection, then native Metal executes
|
| 153 |
+
the six selected q2tp gate/up pairs, q4tp down projections, and the detached
|
| 154 |
+
q4tp shared expert through bounded mmap-backed views. This avoids allocating a
|
| 155 |
+
complete routed-expert bank on unified memory. The complete Flint profile
|
| 156 |
+
(including learned attention sinks and the no-RoPE full-attention layers) is
|
| 157 |
+
still executed; this is a hybrid route, not a fully resident token graph.
|
| 158 |
+
|
| 159 |
+
The direct path checks layout, admission, command completion, and readback;
|
| 160 |
+
any refusal falls back to the CPU path rather than silently changing the
|
| 161 |
+
result. It retains one checked submit/wait boundary for each of the 47 MoE
|
| 162 |
+
layers, so it is not appropriate to infer a general Metal tok/s promise from
|
| 163 |
+
the Vulkan table. `CMF_METAL_FLINT_PAGE_GRAPH=1` remains experimental and
|
| 164 |
+
default-off because it can increase unified-memory pressure. Leave the direct
|
| 165 |
+
path selected by default:
|
| 166 |
+
|
| 167 |
+
```bash
|
| 168 |
+
# Native Metal on Apple silicon; do not set CMF_SDOT for a quality run.
|
| 169 |
+
CMF_GPU=1 CMF_GPU_PROBE=0 CMF_GPU_BLOCK=0 \
|
| 170 |
+
CMF_METAL_FLINT_VIEW_CACHE_MB=0 \
|
| 171 |
+
cortiq run flint-q2tp.cmf --prompt "Explain a bounded LRU cache." \
|
| 172 |
+
--greedy --max-tokens 128
|
| 173 |
+
```
|
| 174 |
+
|
| 175 |
+
#### Exact activation policy
|
| 176 |
+
|
| 177 |
+
When `CMF_SDOT` is **unset**, Flint automatically uses float activations for
|
| 178 |
+
its host-side quantized routes, including persistent row-pool workers.
|
| 179 |
+
`CMF_SDOT=0` explicitly requests that same strict contract. `CMF_SDOT=1` is
|
| 180 |
+
an experimental speed override, not a quality or compatibility mode, and must
|
| 181 |
+
not be used for quality claims. On the native-Metal greedy diagnostic (`Hi`,
|
| 182 |
+
24 generated tokens), the default and `CMF_SDOT=0` runs produced identical
|
| 183 |
+
dumped logits at every step (24/24; maximum absolute difference 0); the
|
| 184 |
+
explicit `=1` run diverged after the first generated token. This verifies the
|
| 185 |
+
runtime policy on that diagnostic, not equivalence to an upstream quality
|
| 186 |
+
evaluation.
|
| 187 |
|
| 188 |
## Integrity and legacy files
|
| 189 |
|
|
|
|
| 315 |
на каждом prompt. Скорость зависит от GPU, драйвера, свободной VRAM, контекста,
|
| 316 |
длины вывода и локальности маршрутов.
|
| 317 |
|
| 318 |
+
Для опубликованного q2tp Native Metal использует **прямой путь выбранных
|
| 319 |
+
экспертов**. Host сохраняет source-exact выбор маршрута, а Metal выполняет
|
| 320 |
+
шесть выбранных q2tp gate/up-пар, q4tp down-проекции и отдельный q4tp shared
|
| 321 |
+
expert через ограниченные mmap-backed views. Поэтому на unified memory не
|
| 322 |
+
выделяется полный банк routed-experts. Полный профиль Flint (включая learned
|
| 323 |
+
attention sinks и full-attention слои без RoPE) исполняется, но это гибридный
|
| 324 |
+
путь, а не полностью resident token graph.
|
| 325 |
+
|
| 326 |
+
Путь проверяет layout, admission, completion command buffer и readback; при
|
| 327 |
+
отказе он возвращается к CPU, а не меняет результат молча. На каждом из 47
|
| 328 |
+
MoE-слоёв остаётся проверяемая граница submit/wait, поэтому из Vulkan-таблицы
|
| 329 |
+
нельзя выводить общую скорость Metal. `CMF_METAL_FLINT_PAGE_GRAPH=1` остаётся
|
| 330 |
+
экспериментальным и выключен по умолчанию: он может повышать давление на
|
| 331 |
+
unified memory. Оставьте direct path по умолчанию:
|
| 332 |
+
|
| 333 |
+
```bash
|
| 334 |
+
# Native Metal на Apple silicon; для quality-run не задавайте CMF_SDOT.
|
| 335 |
+
CMF_GPU=1 CMF_GPU_PROBE=0 CMF_GPU_BLOCK=0 \
|
| 336 |
+
CMF_METAL_FLINT_VIEW_CACHE_MB=0 \
|
| 337 |
+
cortiq run flint-q2tp.cmf --prompt "Объясни bounded LRU cache." \
|
| 338 |
+
--greedy --max-tokens 128
|
| 339 |
+
```
|
| 340 |
+
|
| 341 |
+
#### Политика точных активаций
|
| 342 |
+
|
| 343 |
+
Если `CMF_SDOT` **не задана**, Flint автоматически использует float
|
| 344 |
+
activations в host-side quantized routes, включая persistent row-pool workers.
|
| 345 |
+
`CMF_SDOT=0` явно выбирает тот же строгий контракт. `CMF_SDOT=1` —
|
| 346 |
+
экспериментальный speed override, а не режим качества или совместимости; его
|
| 347 |
+
нельзя использовать для заявлений о качестве. В native-Metal greedy
|
| 348 |
+
диагностике (`Hi`, 24 сгенерированных токена) default и `CMF_SDOT=0`
|
| 349 |
+
совпали во всех dumped logits (24/24; максимальная абсолютная разница 0), а
|
| 350 |
+
явный `=1` разошёлся после первого сгенерированного токена. Это проверка
|
| 351 |
+
runtime policy на данном тесте, а не доказательство эквивалентности upstream
|
| 352 |
+
оценке качества.
|
| 353 |
|
| 354 |
### Целостность и старые файлы
|
| 355 |
|