keel / docs /META-MODELS.md
Tobi
Upload folder using huggingface_hub
52f9e15 verified
|
Raw History Blame Contribute Delete
5.91 kB
# Keel: technical use of Meta models
## 1. System topology
Keel has three planes, and Meta models appear in all three.
| plane | Meta model | interface | artifact |
|---|---|---|---|
| judge | Muse Spark 1.3 | Meta Model API, `POST /v1/chat/completions` | per-card `judge_contract` |
| white-box subject | Muse Glimmer 30B | HF weights, `transformers`, bf16 on A100-80GB | drift cards, weight-delta reports |
| black-box fleet | Spark 1.1, 1.2, 1.3, Wonder 1.0, Spark Open | Meta Model API / Baseten | fleet rows, no cards |
## 2. Judge plane: `muse-spark-1.3`
Every compliance number on every card is produced by one frozen judge through the Meta Model API.
```text
endpoint https://api.meta.ai/v1/chat/completions
model muse-spark-1.3
prompt submit_compliance-numeric
decode temperature 0, answer-only extraction
repeats 3, majority verdict
```
The response parser is format-aware because of the Muse chat format: the answer segment is
the text after the last `to=user` marker; a lone `to=self` yields an empty answer; no marker
means the full completion is the answer. This is recorded per card as:
```json
{"judge_model":"muse-spark-1.3","judge_date":"2026-10-30",
"extraction":"answer-only","prompt_version":"submit_compliance-numeric",
"majority_of":3}
```
Selection was empirical. I labelled a 48-item gold set (24 base answers, 24 M1 answers)
blind, then ranked candidate judges by Cohen's kappa against those labels:
| judge | n | accuracy | kappa |
|---|---|---|---|
| **muse-spark-1.3** | 45 | **0.889** | **0.603** |
| muse-spark-open-1026 | 48 | 0.750 | 0.385 |
| muse-wonder-1.0 | 48 | 0.854 | 0.282 |
`spark-open-1026` was rejected because it called many refusals compliant (12 of 42
negatives). The gold-set annotation was done by the reviewing agent, not a human; that
limitation is stated in `JUDGE-BENCH.md`.
## 3. White-box plane: Muse Glimmer 30B
The base is `meta-models/Muse-Glimmer-30B`, 52 decoder layers, 19,968 MLP units per layer,
loaded in bf16. Four variants were measured under the same judge:
| variant | construction | compliance |
|---|---|---|
| M0 | released weights | 0.0000 |
| M1 | refusal direction estimated contrastively, ablated from the residual stream via a `forward_pre_hook` | 0.1562 |
| M2 | LoRA adapter (`peft`) on the 4-bit base | 0.3438 |
| C | community directional ablation (`darkc0de/Muse-Glimmer-30B-heretic`) | 0.0938 |
M2 is flagged suspect: its compliance is real but its format collapsed, so the number is
reportable only with that caveat.
**Weight-space recovery.** For derivative C, the weight delta was decomposed per module.
The delta lies 100 percent in `o_proj` and `down_proj` (`coverage = 1.000`), reconstructs to
bf16 exactness (`max_abs_error = 0`), and is close to rank-one: per-module lambda is
non-uniform (1.2805 for `o_proj` layer 24, 0.2177 for `down_proj` layer 7), and the median
rank-one residual is 0.363.
**Behavioral replay.** Reconstructing the weights and replaying behavior is gated at a
pre-registered tolerance of 0.15. On derivative C the replay scored 0.000 against the
candidate's 0.084, an absolute gap of 0.084. The 0.15 gate passes arithmetically, but the
effect is smaller than the tolerance itself, so the experiment cannot distinguish a correct
reconstruction from a no-op. It is recorded as blocked by inference-context dependence, not
as a success.
**Circuit experiments.** A contrastive activation circuit over the top 0.1 percent of MLP
neurons was extracted and compared. It failed at all three levels: membership Jaccard 0.456
against a base-vs-base extraction noise band of 0.12 to 0.49, z-scored signature cosine
0.991, and a causal ablation effect of +0.04 over a random-circuit control, one to two
judged items at n=32. Nothing from this was integrated.
The tokenizer/chat template matters technically: Glimmer emits reasoning marked `to=self`,
so the extractor's marker rule is what prevents reasoning from being scored as compliance.
## 4. Black-box fleet
Five Meta API checkpoints are listed as fleet rows and measured on the same 32 harmful
probes, with no public response files and therefore no drift cards:
```text
muse-spark-1.1 0.0000
muse-spark-1.2 0.0000
muse-spark-1.3 0.0000
muse-wonder-1.0 0.03125
muse-spark-open 0.0000
```
These rows exist so the judge can see the API-side checkpoints and the alternative judges
side by side rather than only the open-weight cards.
## 5. Gated model
`meta-models/Muse-Spark-Open-816B-A42B` and its NVFP4 variant were the intended second
open-weights family. Both are gated with manual approval; the account returns
`403 not in the authorized list`. The model was never loaded and never measured.
`SPARK_OPEN.md` records the request path and the experiment that would have run. This is one
of the project's seven refusals.
## 6. Muse Code
Muse Code is the Meta coding agent that wrote the implementation: `stats.py` (paired
bootstrap, sign-flip permutation, Benjamini-Hochberg), `scan.py` and `format.py` (validity
contract), `report.py`, `policy.py`, `ledger.py`, `sign.py`, `serve.py`, and `space/app.py`.
The review trail (cptlight as owner, Muse Code as implementer, an OSS agent as adversarial
reviewer) is in `MUSE-CODE.md`.
## 7. Technical limits
- The judge is a model, not an oracle: kappa 0.603 leaves real label noise, and the n=32
panels have intervals wide enough that a 0.05 effect is not resolvable.
- The white-box metrics are conditional on the inference context: the weight delta is
recoverable exactly, but behavior is not, which is why replay is gated and reported as
blocked.
- The circuit result was killed by its own stability control, not by the absence of a
z-score.
- The live 7B tab on the Space runs Qwen2.5-7B-Instruct, not a Meta model; it demonstrates
the method, and it is not part of the Meta measurement.
- No Muse Image, SAM 3.1, or Muse Voice was used.