Add v4 transfer matrices (ridge only, no intercept) for Qwen3.8-Flash-Next-FP8 and Qwen3.6-35B-A3B: final calibration mix incl. thinking-off code set
2e9c72a verified |
Download README.md from mkleinegger/apex: direct link, hf CLI and curl.
- Browser
- Download file 6.58 kB
-
https://huggingface.co/mkleinegger/apex/resolve/main/README.md
- Command line
-
hf download hf://mkleinegger/apex/README.md
-
curl -L -o README.md https://huggingface.co/mkleinegger/apex/resolve/main/README.md
6.58 kB
| tags: | |
| - mixture-of-experts | |
| - expert-pruning | |
| # APEX-MoE calibration priors | |
| Calibration files for APEX-MoE, prompt-based expert pruning for mixture-of-experts language models: | |
| at the end of prefill, each request keeps a fixed share of the routed experts in every MoE layer, | |
| and decode routes among those only. These files hold the per-expert statistics the selection score | |
| needs, fitted once per model on a calibration set. | |
| | folder | model | MoE layers x experts | | |
| |---|---|---| | |
| | `qwen3.6-35b-a3b` | [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) | 40 x 256 | | |
| | `qwen3.8-flash-next-fp8` | [Qwen/Qwen3.8-Flash-Next-FP8](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8) | 48 x 512 | | |
| | `deepseek-v4.1-flash` | [deepseek-ai/DeepSeek-V4.1-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash) | 40 x 384 | | |
| Each folder holds, for MoE layer `l` and routed expert `e`: | |
| - `gain.npy` `[L, E]`: the expert's gain, its router-weighted mean output norm on the calibration | |
| set, `sum g * ||f(x)|| / sum g`; | |
| - `prior.npy` `[L, E]`: the expert's share of the summed REAP saliency `g * ||f(x)||` on the | |
| calibration set (a static prior); | |
| - `transfer.npz`: the transfer prior, a per-layer ridge regression of a request's decode profile | |
| on its prompt's routing: `matrix` `[L, E, E]`, `offset` `[L, E]`, `lam` (ridge strength), `power` | |
| (2: fitted on the square root of the decode share); | |
| - `meta.json`: model revision, calibration runs, fit settings. | |
| ## Selection score | |
| Per request and layer, with `m` the prompt's summed router weight per expert, `c` its route count | |
| per expert, `n` its token count and `share(v) = v / sum(v)`: | |
| ``` | |
| prior = share(max(0, share(c) @ matrix[l] + offset[l]) ** power) | |
| score = n * share(gain[l] ** 2 * m) + 256 * prior | |
| ``` | |
| The `round(keep * E)` highest-scoring experts are kept (ties: lowest expert id); decode masks the | |
| other router logits before the top-k. Settings used with these files: gain power 2, prior strength | |
| 256, keep 25% and 50%. | |
| ## Calibration | |
| 1024 prompts (`calmix1024`, seed 0): 256 each from Open-Platypus, OpenThoughts-114k, | |
| evol-codealpaca-v1 and FineWeb-Edu passages, decontaminated against the evaluated benchmarks. Each | |
| model answered every prompt itself with thinking on; a teacher-forced replay of prompt and answer | |
| recorded the routed experts' router weights and output norms. The Qwen3.6 files were fitted on the | |
| BF16 checkpoint. | |
| ## v3 priors (`v3/`) | |
| Refit priors for Qwen3.6-35B-A3B (`v3/qwen3.6-35b-a3b`) and Qwen3.8-Flash-Next-FP8 | |
| (`v3/qwen3.8-flash-next-fp8`) on a larger calibration set, used **ridge only**: the transfer | |
| prediction alone ranks the experts (prior strength 1e9; the prompt term only breaks ties). Besides | |
| `gain.npy`, `prior.npy` and `transfer.npz` (as above) each folder holds: | |
| - `transfer_window.npz`: the window ridge for dynamic re-selection during decode, a per-layer ridge | |
| regression of the square root of the next 256 decode tokens' saliency share on the last window's | |
| unmasked route-count share and the prompt's: `matrix` `[L, 2E, E]`, `offset` `[L, E]`, `window` | |
| (256), `lam`, `power` (2), `pairs` (calibration window pairs); | |
| - `meta.json`: model revision, calibration runs and excluded domains, fit settings (including | |
| `ridge_intercept`: Qwen3.6 without, `offset` 0; Qwen3.8 with). | |
| Static selection, per request and layer (`c`: the prompt's route counts per expert): | |
| ``` | |
| keep top round(keep * E) of share(c) @ matrix[l] + offset[l] | |
| ``` | |
| Dynamic selection: every 256 decode tokens the set is chosen again as the top of | |
| `[share(w) | share(c)] @ window.matrix[l] + window.offset[l]`, `w` the unmasked top-k route counts | |
| of the last 256 decode tokens, plus 0.5 standard deviations of that prediction for the experts kept | |
| now. | |
| Calibration (2,304 requests, each answered by the model itself): `calmix1024` as above; `calmcq512`, | |
| 256 MMLU dev and 256 ARC-Challenge questions, answered once with thinking off (16k) and once with | |
| thinking on; `calv3-1024` without its Tulu-3 rows, 256 each from the MATH train split, the MBPP train | |
| split and GPQA-style questions, three shards with thinking on and one off. The Qwen3.6 files were | |
| fitted on the BF16 checkpoint. | |
| ## v4 priors (`v4/`) | |
| The final calibration mix, fitted for Qwen3.8-Flash-Next-FP8 (`v4/qwen3.8-flash-next-fp8`) and | |
| Qwen3.6-35B-A3B (`v4/qwen3.6-35b-a3b`, fitted on the BF16 checkpoint). Ridge only: each folder | |
| holds `transfer.npz` (`matrix` `[L, E, E]`, `offset` `[L, E]` all zero, `lam`, `power` 2) and | |
| `meta.json` (model revision, fit settings, calibration sets and runs). There is no `gain.npy` or | |
| `prior.npy`: the selection uses T alone. | |
| Fit, per MoE layer `l`: `sqrt(s) ~ r @ matrix[l]` by ridge regression without intercept over the | |
| calibration requests (`lambda = 1`, scaled by `tr(R R^T) / requests`), with `r` the prompt's | |
| route-count share per expert and `s` the response's REAP saliency share (router weight times expert | |
| output norm), measured by teacher-forcing the model's own response. | |
| Selection, per request and layer (`c`: the prompt's route counts per expert): | |
| ``` | |
| keep top round(keep * E) of share(c) @ matrix[l] | |
| ``` | |
| Calibration (3,584 requests, each answered by the model itself at temperature 1.0): | |
| | set | requests | data | | |
| |---|---|---| | |
| | calmix1024 | 1,024 | Open-Platypus, OpenThoughts-114k, evol-codealpaca-v1, FineWeb-Edu | | |
| | callong-ot384 | 384 | OpenThoughts-114k problems with the longest reference answers | | |
| | callong-otcode384 | 384 | OpenThoughts-114k code problems with the longest reference answers | | |
| | calagent-swe256 | 256 | SWE-smith trajectories cut before an agent turn, SWE-agent's tools | | |
| | calagent-xlam256 | 256 | xLAM function-calling queries with their tools | | |
| | calown-wp256 | 256 | WritingPrompts curated | | |
| | calown-personas256 | 256 | tulu-3-sft-personas-math | | |
| | calown-cfcots384 | 384 | codeforces-cots (Python) problems with the longest R1 traces | | |
| | calotcode-nothink384 | 384 | further OpenThoughts-114k code problems, answered with thinking off | | |
| All sets are answered with thinking on except `calotcode-nothink384`. No prompt contains a question | |
| of MATH-500, GPQA-Diamond, MMLU-Pro, MBPP+ or HumanEval+ (substring or shared word 13-gram); the | |
| thinking-off code set is also checked against LiveCodeBench v6 (2025-02 to 2025-05). No data is | |
| drawn from LiveCodeBench, LeetCode or AtCoder. | |
| ## Use | |
| With [apex-engine](https://github.com/IST-DASLab/apex-engine) (FreeToken with APEX): | |
| ```bash | |
| ft serve --model Qwen/Qwen3.6-35B-A3B --moe-strategy offload \ | |
| --apex-dir <this repo>/qwen3.6-35b-a3b --apex-keep 0.25 | |
| ``` | |