Ecce Vectors: 29 control vectors for Gemma 4 31B roleplay fine-tunes, with metrics, blind-test results, examples and the fitting code

e51b495 verified about 1 month ago

preview code

Raw History Blame Contribute Delete

9.49 kB

# Making your own vectors

This is the whole pipeline that produced the files in `vectors/`: fit a vector to a fine-tune, score it to
pick its strength, export it for llama.cpp, and (optionally) build a merge's vector from its parents'
vectors.
Every command here was run to make this release.

## What you need

- **An Apple silicon Mac, for the code as shipped.** The fit and the score run on
  [MLX](https://github.com/ml-explore/mlx). They build one decoder layer at a time, so peak memory stays
  around 11 GB whatever the checkpoint size; measured on an M1 Max with 32 GB of unified memory. Nothing
  in the method needs MLX: it needs forward passes of both models over the same text, so it could be
  ported to PyTorch (NVIDIA or otherwise) or another engine. That port has not been written.
- **Disk for two checkpoints**: stock `google/gemma-4-31B-it` and the fine-tune, about 62.5 GB each in
  bf16. A LoRA fine-tune can be fitted from its adapter instead, which is a few hundred MB to a few GB.
- **[uv](https://docs.astral.sh/uv/)** for Python. The dependencies are pinned in `pyproject.toml`;
  `mlx-lm` is pinned exactly, because its Gemma 4 layer code is what runs each layer.
- **llama.cpp** (or ik_llama.cpp) to use the result. See the README for launch flags.
Only **Gemma 4 31B** (dense, 60 layers) is supported. The 26B-A4B MoE shares its `model_type`, and the code
refuses it.

```bash
git clone <this repo> ecce-vector && cd ecce-vector
uv sync                                     # creates .venv with the pinned dependencies
hf download google/gemma-4-31B-it --local-dir gemma-4-31B-it \
    --include "*.safetensors" "*.json" "tokenizer*" "chat_template.jinja"
hf download SomeAuthor/Some-Finetune-31B --local-dir Some-Finetune-31B \
    --include "*.safetensors" "*.json"
```

## 1. Fit

```bash
uv run python -m ecce_vector.fit --base gemma-4-31B-it --tune Some-Finetune-31B \
    --out fits --name some-finetune
# a LoRA release: fit from the adapter, applied on stock
uv run python -m ecce_vector.fit --base gemma-4-31B-it --adapter Some-LoRA \
    --out fits --name some-lora
```

About 5.5 minutes on an M1 Max, most of it reading weights. Run with `--smoke` first to time two layers.

For each of the 60 layers, the fit runs stock's layer and the fine-tune's layer on the fine-tune's own
hidden states over 8 short prompts (4 roleplay, 4 novel prose; `ecce_vector/data/fit_corpus.json`) and
averages the difference of their outputs. That average is the part of the fine-tune's edit that is the
same at every token: all that a single vector per layer can carry. The **tracking fit**
(`vectors_track`, the one released here) goes one step further. It runs *stock plus the vectors fitted so
far* and fits each layer's vector to bring that steered stream back to the fine-tune's, so the drift that
early layers' vectors cause is corrected as it builds up. It is fitted the way llama.cpp applies a
vector: nothing at layer 0, and added at every position including the first.

It writes `fits/some-finetune.npz` (the vectors, and the fine-tune's final hidden states on both prompt
splits, which the score compares against) and `fits/some-finetune.json` (per-layer diagnostics). The
printed **constant share** is the fraction of the edit's energy that is the same at every token. It
ranks how well a vector will carry a fine-tune about as well as anything tried (Spearman +0.76 over the
releases here), but it misjudges some tunes one by one, so score anyway.

The fine-tunes a vector cannot carry at all are refused: an untied `lm_head` (the output head is not in
the residual stream). A fine-tune that changed its embedding (so its tied output head) is fitted, and its
own log-probs are saved beside the fit as `NAME.logp.npz` (about 1 GB), because stock's head reads its
hidden states differently. How much that costs varies a lot: see the README's note on `melody1437-v2`.

## 2. Score, and pick the strength

```bash
uv run python -m ecce_vector.score --base gemma-4-31B-it --ref fits/some-finetune.npz \
    --vector fits/some-finetune.npz::vectors_track --alphas 0.5 0.75 1.0 1.25 1.5 \
    --out fits/some-finetune.score.json
```

About 5 minutes for five strengths. Each strength is one extra stream through stock's layers, computed as
a difference from the stock stream, so extra arms are cheap.

The score runs stock with the vector added, on 16 prompts (the 8 fit prompts and 8 held-out ones), and
compares its next-token distributions with the fine-tune's at 128 reference positions per prompt:

- **recovery** = 1 − KL(tune ‖ stock + vector) / KL(tune ‖ stock). 1 is a perfect copy, 0 is no better
  than stock, and below 0 means the vector moves stock *away* from the fine-tune.
- **top-1 recovery**: the same for the share of positions where the fine-tune's most likely next token
  differs from stock's and the vector restores it.

Pick the strength on the fit split and quote the held-out split; the output's `best` does exactly that, for
each vector scored.
The released files use the tracking fit at that strength. Most tunes here pick 1.25.

A vector's score measures it against the fine-tune's full-precision weights on these prompts: both read
the same text and their next-word predictions are compared at every position (teacher-forced), and
neither writes anything. It says nothing about how replies read, and it charges a miss less where the
fine-tune is unsure of its next token, which flatters tunes that make stock *less* certain. Read top-1
recovery beside it. In the README's blind test, judges reading the replies found only a faint hint of each
fine-tune in its vector's output, including vectors scoring above 90%.

## 3. Export for llama.cpp

```bash
uv run python -m ecce_vector.export fits/some-finetune.npz::vectors_track \
    vectors/some-finetune.gguf --alpha 1.25 --name some-finetune
```

The GGUF holds `alpha × c_L` for layers 1–59 as F32 (llama.cpp rejects other types and never applies a
layer-0 vector), so it loads at strength 1.0 and weighs 1.27 MB.

## 4. A merge's vector from its parents' vectors (advanced)

A mergekit merge in the linear family (`linear`, `task_arithmetic`, `della_linear`, `dare_linear`, …) on
stock moves the model by a weighted sum of its parents' edits, so the merge's vector is the same weighted
sum of the parents' vectors, layer by layer. `bake` reads the recipe exactly as mergekit resolves it
(gradients across depth, per-module filters, normalisation, lambda) and writes one file:

```bash
uv run python -m ecce_vector.bake recipe.yaml vectors/my-merge.gguf \
    --parent ./ReadyArt/gemma-4-31B-it-scotoma-2=vectors/scotoma-2.gguf \
    --parent ./Blazed-Forge/Gemma-4-Giftige-Blume-31B-v1=vectors/giftige-blume-v1.gguf \
    --parent ./zerofata/G4-MeroMero-v2-31B=vectors/meromero-v2.gguf
```

Each `--parent` maps a `model:` entry of the recipe, spelled exactly as the recipe spells it, to that
parent's vector. `examples/schattenblume-recipe.yaml` is the real recipe of `Nimbz/Schattenblume-31B`. The
file baked from it recovers 0.894 of the merge, against 0.899 for a vector fitted to the merge's own
weights: no model weights needed.

Why bake rather than blend at launch? llama.cpp's `--control-vector-scaled` takes one weight per file, and
a recipe usually weights each parent differently at different depths. For `schattenblume`, one constant
weight per parent (the recipe's weights averaged over layers) recovers only 0.844. Blending at launch is
still the easy way to *invent* a mix, and you can score any blend:

```bash
uv run python -m ecce_vector.score --base gemma-4-31B-it --ref fits/my-merge.npz \
    --gguf vectors/my-merge.gguf --blend vectors/a.gguf:0.5,vectors/b.gguf:0.5
```

Refused: methods that are not a weighted sum (`slerp`, `model_stock`, `arcee_fusion`, …), a base other
than stock, and a recipe that weights a layer's attention and MLP differently (one vector per layer cannot
express that). `ties`/`dare_ties`/`della` are baked without their sign-election step, with a warning;
that case is untested.

## 5. Generate comparisons

```bash
uv run python -m ecce_vector.examples --server llama-server --model gemma-4-31B-it-Q4_K_M.gguf \
    --prompts examples/prompts.json --out runs/stock
uv run python -m ecce_vector.examples --server llama-server --model gemma-4-31B-it-Q4_K_M.gguf \
    --prompts examples/prompts.json --out runs/some-finetune \
    --control-vector-scaled vectors/some-finetune.gguf:1.0
```

One llama-server per run (control vectors are fixed at launch), Google's recommended sampling for Gemma 4
(temperature 1.0, top-k 64, top-p 0.95, nothing else), a fixed seed, and llama-server's own prompt and
generation timings saved beside each reply in `run.json`.

## Reproducing this release

The prompts are fixed (`ecce_vector/data/fit_corpus.json` holds the exact token ids), and the fit and
score are deterministic on a given MLX build. The code in this repository was checked against the
research code that produced the released numbers:

| check | result |
|---|---|
| the corpus module rebuilds the original token arrays, masks and scored rows | identical |
| `bake` on the `schattenblume` recipe against the original baker | identical (0.0 difference) |
| `score` of `schattenblume`'s vector / its recipe bake, held out | 0.89876 / 0.89388, against 0.89876 / 0.89386 |
| `fit` of `storymaxxed2` from its released weights against the stored fit | bit-identical, every layer |

Details in `metrics/package-checks.md`.

	# Making your own vectors

	This is the whole pipeline that produced the files in `vectors/`: fit a vector to a fine-tune, score it to
	pick its strength, export it for llama.cpp, and (optionally) build a merge's vector from its parents'
	vectors.
	Every command here was run to make this release.

	## What you need

	- An Apple silicon Mac, for the code as shipped. The fit and the score run on
	[MLX](https://github.com/ml-explore/mlx). They build one decoder layer at a time, so peak memory stays
	around 11 GB whatever the checkpoint size; measured on an M1 Max with 32 GB of unified memory. Nothing
	in the method needs MLX: it needs forward passes of both models over the same text, so it could be
	ported to PyTorch (NVIDIA or otherwise) or another engine. That port has not been written.
	- Disk for two checkpoints: stock `google/gemma-4-31B-it` and the fine-tune, about 62.5 GB each in
	bf16. A LoRA fine-tune can be fitted from its adapter instead, which is a few hundred MB to a few GB.
	- [uv](https://docs.astral.sh/uv/) for Python. The dependencies are pinned in `pyproject.toml`;
	`mlx-lm` is pinned exactly, because its Gemma 4 layer code is what runs each layer.
	- llama.cpp (or ik_llama.cpp) to use the result. See the README for launch flags.

	Only Gemma 4 31B (dense, 60 layers) is supported. The 26B-A4B MoE shares its `model_type`, and the code
	refuses it.

	```bash
	git clone <this repo> ecce-vector && cd ecce-vector
	uv sync # creates .venv with the pinned dependencies
	hf download google/gemma-4-31B-it --local-dir gemma-4-31B-it \
	--include ".safetensors" ".json" "tokenizer*" "chat_template.jinja"
	hf download SomeAuthor/Some-Finetune-31B --local-dir Some-Finetune-31B \
	--include ".safetensors" ".json"
	```

	## 1. Fit

	```bash
	uv run python -m ecce_vector.fit --base gemma-4-31B-it --tune Some-Finetune-31B \
	--out fits --name some-finetune
	# a LoRA release: fit from the adapter, applied on stock
	uv run python -m ecce_vector.fit --base gemma-4-31B-it --adapter Some-LoRA \
	--out fits --name some-lora
	```

	About 5.5 minutes on an M1 Max, most of it reading weights. Run with `--smoke` first to time two layers.

	For each of the 60 layers, the fit runs stock's layer and the fine-tune's layer on the fine-tune's own
	hidden states over 8 short prompts (4 roleplay, 4 novel prose; `ecce_vector/data/fit_corpus.json`) and
	averages the difference of their outputs. That average is the part of the fine-tune's edit that is the
	same at every token: all that a single vector per layer can carry. The tracking fit
	(`vectors_track`, the one released here) goes one step further. It runs *stock plus the vectors fitted so
	far* and fits each layer's vector to bring that steered stream back to the fine-tune's, so the drift that
	early layers' vectors cause is corrected as it builds up. It is fitted the way llama.cpp applies a
	vector: nothing at layer 0, and added at every position including the first.

	It writes `fits/some-finetune.npz` (the vectors, and the fine-tune's final hidden states on both prompt
	splits, which the score compares against) and `fits/some-finetune.json` (per-layer diagnostics). The
	printed constant share is the fraction of the edit's energy that is the same at every token. It
	ranks how well a vector will carry a fine-tune about as well as anything tried (Spearman +0.76 over the
	releases here), but it misjudges some tunes one by one, so score anyway.

	The fine-tunes a vector cannot carry at all are refused: an untied `lm_head` (the output head is not in
	the residual stream). A fine-tune that changed its embedding (so its tied output head) is fitted, and its
	own log-probs are saved beside the fit as `NAME.logp.npz` (about 1 GB), because stock's head reads its
	hidden states differently. How much that costs varies a lot: see the README's note on `melody1437-v2`.

	## 2. Score, and pick the strength

	```bash
	uv run python -m ecce_vector.score --base gemma-4-31B-it --ref fits/some-finetune.npz \
	--vector fits/some-finetune.npz::vectors_track --alphas 0.5 0.75 1.0 1.25 1.5 \
	--out fits/some-finetune.score.json
	```

	About 5 minutes for five strengths. Each strength is one extra stream through stock's layers, computed as
	a difference from the stock stream, so extra arms are cheap.

	The score runs stock with the vector added, on 16 prompts (the 8 fit prompts and 8 held-out ones), and
	compares its next-token distributions with the fine-tune's at 128 reference positions per prompt:

	- recovery = 1 − KL(tune ‖ stock + vector) / KL(tune ‖ stock). 1 is a perfect copy, 0 is no better
	than stock, and below 0 means the vector moves stock away from the fine-tune.
	- top-1 recovery: the same for the share of positions where the fine-tune's most likely next token
	differs from stock's and the vector restores it.

	Pick the strength on the fit split and quote the held-out split; the output's `best` does exactly that, for
	each vector scored.
	The released files use the tracking fit at that strength. Most tunes here pick 1.25.

	A vector's score measures it against the fine-tune's full-precision weights on these prompts: both read
	the same text and their next-word predictions are compared at every position (teacher-forced), and
	neither writes anything. It says nothing about how replies read, and it charges a miss less where the
	fine-tune is unsure of its next token, which flatters tunes that make stock less certain. Read top-1
	recovery beside it. In the README's blind test, judges reading the replies found only a faint hint of each
	fine-tune in its vector's output, including vectors scoring above 90%.

	## 3. Export for llama.cpp

	```bash
	uv run python -m ecce_vector.export fits/some-finetune.npz::vectors_track \
	vectors/some-finetune.gguf --alpha 1.25 --name some-finetune
	```

	The GGUF holds `alpha × c_L` for layers 1–59 as F32 (llama.cpp rejects other types and never applies a
	layer-0 vector), so it loads at strength 1.0 and weighs 1.27 MB.

	## 4. A merge's vector from its parents' vectors (advanced)

	A mergekit merge in the linear family (`linear`, `task_arithmetic`, `della_linear`, `dare_linear`, …) on
	stock moves the model by a weighted sum of its parents' edits, so the merge's vector is the same weighted
	sum of the parents' vectors, layer by layer. `bake` reads the recipe exactly as mergekit resolves it
	(gradients across depth, per-module filters, normalisation, lambda) and writes one file:

	```bash
	uv run python -m ecce_vector.bake recipe.yaml vectors/my-merge.gguf \
	--parent ./ReadyArt/gemma-4-31B-it-scotoma-2=vectors/scotoma-2.gguf \
	--parent ./Blazed-Forge/Gemma-4-Giftige-Blume-31B-v1=vectors/giftige-blume-v1.gguf \
	--parent ./zerofata/G4-MeroMero-v2-31B=vectors/meromero-v2.gguf
	```

	Each `--parent` maps a `model:` entry of the recipe, spelled exactly as the recipe spells it, to that
	parent's vector. `examples/schattenblume-recipe.yaml` is the real recipe of `Nimbz/Schattenblume-31B`. The
	file baked from it recovers 0.894 of the merge, against 0.899 for a vector fitted to the merge's own
	weights: no model weights needed.

	Why bake rather than blend at launch? llama.cpp's `--control-vector-scaled` takes one weight per file, and
	a recipe usually weights each parent differently at different depths. For `schattenblume`, one constant
	weight per parent (the recipe's weights averaged over layers) recovers only 0.844. Blending at launch is
	still the easy way to invent a mix, and you can score any blend:

	```bash
	uv run python -m ecce_vector.score --base gemma-4-31B-it --ref fits/my-merge.npz \
	--gguf vectors/my-merge.gguf --blend vectors/a.gguf:0.5,vectors/b.gguf:0.5
	```

	Refused: methods that are not a weighted sum (`slerp`, `model_stock`, `arcee_fusion`, …), a base other
	than stock, and a recipe that weights a layer's attention and MLP differently (one vector per layer cannot
	express that). `ties`/`dare_ties`/`della` are baked without their sign-election step, with a warning;
	that case is untested.

	## 5. Generate comparisons

	```bash
	uv run python -m ecce_vector.examples --server llama-server --model gemma-4-31B-it-Q4_K_M.gguf \
	--prompts examples/prompts.json --out runs/stock
	uv run python -m ecce_vector.examples --server llama-server --model gemma-4-31B-it-Q4_K_M.gguf \
	--prompts examples/prompts.json --out runs/some-finetune \
	--control-vector-scaled vectors/some-finetune.gguf:1.0
	```

	One llama-server per run (control vectors are fixed at launch), Google's recommended sampling for Gemma 4
	(temperature 1.0, top-k 64, top-p 0.95, nothing else), a fixed seed, and llama-server's own prompt and
	generation timings saved beside each reply in `run.json`.

	## Reproducing this release

	The prompts are fixed (`ecce_vector/data/fit_corpus.json` holds the exact token ids), and the fit and
	score are deterministic on a given MLX build. The code in this repository was checked against the
	research code that produced the released numbers:

	\| check \| result \|
	\|---\|---\|
	\| the corpus module rebuilds the original token arrays, masks and scored rows \| identical \|
	\| `bake` on the `schattenblume` recipe against the original baker \| identical (0.0 difference) \|
	\| `score` of `schattenblume`'s vector / its recipe bake, held out \| 0.89876 / 0.89388, against 0.89876 / 0.89386 \|
	\| `fit` of `storymaxxed2` from its released weights against the stored fit \| bit-identical, every layer \|

	Details in `metrics/package-checks.md`.