apex / README.md
mkleinegger's picture
Add v4 transfer matrices (ridge only, no intercept) for Qwen3.8-Flash-Next-FP8 and Qwen3.6-35B-A3B: final calibration mix incl. thinking-off code set
2e9c72a verified
|
Raw History Blame Contribute Delete
6.58 kB
---
tags:
- mixture-of-experts
- expert-pruning
---
# APEX-MoE calibration priors
Calibration files for APEX-MoE, prompt-based expert pruning for mixture-of-experts language models:
at the end of prefill, each request keeps a fixed share of the routed experts in every MoE layer,
and decode routes among those only. These files hold the per-expert statistics the selection score
needs, fitted once per model on a calibration set.
| folder | model | MoE layers x experts |
|---|---|---|
| `qwen3.6-35b-a3b` | [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) | 40 x 256 |
| `qwen3.8-flash-next-fp8` | [Qwen/Qwen3.8-Flash-Next-FP8](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8) | 48 x 512 |
| `deepseek-v4.1-flash` | [deepseek-ai/DeepSeek-V4.1-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash) | 40 x 384 |
Each folder holds, for MoE layer `l` and routed expert `e`:
- `gain.npy` `[L, E]`: the expert's gain, its router-weighted mean output norm on the calibration
set, `sum g * ||f(x)|| / sum g`;
- `prior.npy` `[L, E]`: the expert's share of the summed REAP saliency `g * ||f(x)||` on the
calibration set (a static prior);
- `transfer.npz`: the transfer prior, a per-layer ridge regression of a request's decode profile
on its prompt's routing: `matrix` `[L, E, E]`, `offset` `[L, E]`, `lam` (ridge strength), `power`
(2: fitted on the square root of the decode share);
- `meta.json`: model revision, calibration runs, fit settings.
## Selection score
Per request and layer, with `m` the prompt's summed router weight per expert, `c` its route count
per expert, `n` its token count and `share(v) = v / sum(v)`:
```
prior = share(max(0, share(c) @ matrix[l] + offset[l]) ** power)
score = n * share(gain[l] ** 2 * m) + 256 * prior
```
The `round(keep * E)` highest-scoring experts are kept (ties: lowest expert id); decode masks the
other router logits before the top-k. Settings used with these files: gain power 2, prior strength
256, keep 25% and 50%.
## Calibration
1024 prompts (`calmix1024`, seed 0): 256 each from Open-Platypus, OpenThoughts-114k,
evol-codealpaca-v1 and FineWeb-Edu passages, decontaminated against the evaluated benchmarks. Each
model answered every prompt itself with thinking on; a teacher-forced replay of prompt and answer
recorded the routed experts' router weights and output norms. The Qwen3.6 files were fitted on the
BF16 checkpoint.
## v3 priors (`v3/`)
Refit priors for Qwen3.6-35B-A3B (`v3/qwen3.6-35b-a3b`) and Qwen3.8-Flash-Next-FP8
(`v3/qwen3.8-flash-next-fp8`) on a larger calibration set, used **ridge only**: the transfer
prediction alone ranks the experts (prior strength 1e9; the prompt term only breaks ties). Besides
`gain.npy`, `prior.npy` and `transfer.npz` (as above) each folder holds:
- `transfer_window.npz`: the window ridge for dynamic re-selection during decode, a per-layer ridge
regression of the square root of the next 256 decode tokens' saliency share on the last window's
unmasked route-count share and the prompt's: `matrix` `[L, 2E, E]`, `offset` `[L, E]`, `window`
(256), `lam`, `power` (2), `pairs` (calibration window pairs);
- `meta.json`: model revision, calibration runs and excluded domains, fit settings (including
`ridge_intercept`: Qwen3.6 without, `offset` 0; Qwen3.8 with).
Static selection, per request and layer (`c`: the prompt's route counts per expert):
```
keep top round(keep * E) of share(c) @ matrix[l] + offset[l]
```
Dynamic selection: every 256 decode tokens the set is chosen again as the top of
`[share(w) | share(c)] @ window.matrix[l] + window.offset[l]`, `w` the unmasked top-k route counts
of the last 256 decode tokens, plus 0.5 standard deviations of that prediction for the experts kept
now.
Calibration (2,304 requests, each answered by the model itself): `calmix1024` as above; `calmcq512`,
256 MMLU dev and 256 ARC-Challenge questions, answered once with thinking off (16k) and once with
thinking on; `calv3-1024` without its Tulu-3 rows, 256 each from the MATH train split, the MBPP train
split and GPQA-style questions, three shards with thinking on and one off. The Qwen3.6 files were
fitted on the BF16 checkpoint.
## v4 priors (`v4/`)
The final calibration mix, fitted for Qwen3.8-Flash-Next-FP8 (`v4/qwen3.8-flash-next-fp8`) and
Qwen3.6-35B-A3B (`v4/qwen3.6-35b-a3b`, fitted on the BF16 checkpoint). Ridge only: each folder
holds `transfer.npz` (`matrix` `[L, E, E]`, `offset` `[L, E]` all zero, `lam`, `power` 2) and
`meta.json` (model revision, fit settings, calibration sets and runs). There is no `gain.npy` or
`prior.npy`: the selection uses T alone.
Fit, per MoE layer `l`: `sqrt(s) ~ r @ matrix[l]` by ridge regression without intercept over the
calibration requests (`lambda = 1`, scaled by `tr(R R^T) / requests`), with `r` the prompt's
route-count share per expert and `s` the response's REAP saliency share (router weight times expert
output norm), measured by teacher-forcing the model's own response.
Selection, per request and layer (`c`: the prompt's route counts per expert):
```
keep top round(keep * E) of share(c) @ matrix[l]
```
Calibration (3,584 requests, each answered by the model itself at temperature 1.0):
| set | requests | data |
|---|---|---|
| calmix1024 | 1,024 | Open-Platypus, OpenThoughts-114k, evol-codealpaca-v1, FineWeb-Edu |
| callong-ot384 | 384 | OpenThoughts-114k problems with the longest reference answers |
| callong-otcode384 | 384 | OpenThoughts-114k code problems with the longest reference answers |
| calagent-swe256 | 256 | SWE-smith trajectories cut before an agent turn, SWE-agent's tools |
| calagent-xlam256 | 256 | xLAM function-calling queries with their tools |
| calown-wp256 | 256 | WritingPrompts curated |
| calown-personas256 | 256 | tulu-3-sft-personas-math |
| calown-cfcots384 | 384 | codeforces-cots (Python) problems with the longest R1 traces |
| calotcode-nothink384 | 384 | further OpenThoughts-114k code problems, answered with thinking off |
All sets are answered with thinking on except `calotcode-nothink384`. No prompt contains a question
of MATH-500, GPQA-Diamond, MMLU-Pro, MBPP+ or HumanEval+ (substring or shared word 13-gram); the
thinking-off code set is also checked against LiveCodeBench v6 (2025-02 to 2025-05). No data is
drawn from LiveCodeBench, LeetCode or AtCoder.
## Use
With [apex-engine](https://github.com/IST-DASLab/apex-engine) (FreeToken with APEX):
```bash
ft serve --model Qwen/Qwen3.6-35B-A3B --moe-strategy offload \
--apex-dir <this repo>/qwen3.6-35b-a3b --apex-keep 0.25
```