marcodene/onward-probes-apertus-8b-instruct

ONWARD degeneration probes for Apertus-8B-Instruct

Linear probes that read the residual stream of swiss-ai/Apertus-8B-Instruct-2509 while it generates and score every generated token for repetitive degeneration, a loop that keeps repeating until the token limit is reached. Each probe comes with thresholds that make it raise an alarm on at most 1%, 5% or 10% of healthy responses. The probes are evaluated in the paper Degeneration Detection Is Not Early Warning: False-Alarm Evaluation with ONWARD (TODO: paper link).

Probes

Folder	Probe	Layer	Trained	Released seed
frozen-L15	frozen	15	the probe only, base model frozen	43
adapted-L15	adapted	15	LoRA adapters (rank 16) on blocks 0–15 and the probe	43
adapted-L30	adapted	30	LoRA adapters (rank 16) on blocks 0–30 and the probe	42

Layer ℓ is the output of decoder block ℓ, with blocks numbered from 0 (model.model.layers[ℓ], hidden_states[ℓ + 1] in Transformers). A probe applies a layer norm and a linear map to that hidden state and returns one logit per token. A frozen probe reads the activations the base model computes during generation. An adapted probe reads them after its LoRA adapters, which must be active when it scores. The released seed of each probe is the one of three with the highest warning coverage on validation at the 1% budget.

Each folder holds probe.safetensors (layer norm and linear weights), config.json (base model, pinned revision b946d40447b2, layer), thresholds.json (thresholds per budget, calibrated on validation), seeds.json (metrics of all three seeds) and, for adapted probes, the PEFT adapter files.

Usage

The thresholds hold for responses generated and scored as in the paper: a single user turn formatted with the model's chat template without the beginning-of-sequence token and tokenized without special tokens, sampling with temperature 0.7 and top-p 0.9, and at most 4,096 new tokens.

import json, torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from transformers import AutoModelForCausalLM, AutoTokenizer

repo, probe = "marcodene/onward-probes-apertus-8b-instruct", "frozen-L15"
path = snapshot_download(repo, allow_patterns=[f"{probe}/*"]) + f"/{probe}"
config = json.load(open(f"{path}/config.json"))
tau = {t["target_negative_fpr"]: t["tau"] for t in json.load(open(f"{path}/thresholds.json"))["thresholds"]}[0.05]

tokenizer = AutoTokenizer.from_pretrained(config["base_model"], revision=config["base_model_revision"])
model = AutoModelForCausalLM.from_pretrained(config["base_model"], revision=config["base_model_revision"],
                                             dtype=torch.bfloat16, device_map="cuda")
if config["adapted"]:
    from peft import PeftModel
    model = PeftModel.from_pretrained(model, path)

weights = load_file(f"{path}/probe.safetensors", device="cuda")
norm = torch.nn.LayerNorm(config["hidden_size"], device="cuda")
norm.load_state_dict({"weight": weights["norm.weight"], "bias": weights["norm.bias"]})
head = torch.nn.Linear(config["hidden_size"], 1, device="cuda")
head.load_state_dict({"weight": weights["linear.weight"], "bias": weights["linear.bias"]})

def prompt_ids(prompt):
    text = tokenizer.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False,
                                         add_generation_prompt=True)
    return tokenizer(text.replace(tokenizer.bos_token, ""), add_special_tokens=False)["input_ids"]

@torch.no_grad()
def score(prompt, generated_ids):
    ids = prompt_ids(prompt)
    out = model(torch.tensor([ids + list(generated_ids)], device="cuda"), output_hidden_states=True)
    states = out.hidden_states[config["hidden_states_index"]][0, len(ids):]
    return torch.sigmoid(head(norm(states.float()))).squeeze(-1)   # one score per generated token

scores = score(prompt, generated_ids)
first_alarm = (scores >= tau).nonzero()   # the first row is the first alarm, if any

A frozen probe can also run during generation by reading the same hidden state at each new token, at no extra cost. An adapted probe changes the model's outputs if its adapters stay active during generation, so it needs a separate scoring pass over the generated tokens.

Thresholds

An alarm is raised at the first token whose score reaches τ. τ is the lowest threshold at which at most the given share of healthy validation responses raise an alarm.

Folder	τ at 1%	τ at 5%	τ at 10%
frozen-L15	0.9931	0.9660	0.9314
adapted-L15	0.9972	0.9577	0.7673
adapted-L30	0.9978	0.8768	0.4903

Results

In-domain test split at the 5% budget, with onsets annotated by an LLM judge. False alarms: share of healthy test responses that raise an alarm. Post-onset coverage: share of tokens from the onset onward that are flagged. Warn₂₅₆: share of the 256 tokens before the onset that are flagged. Early-warning rate: share of alarmed degenerate responses whose first alarm comes before the onset. Median lead: first alarm minus onset, negative when the alarm comes first. All metrics at 1%, 5% and 10% are in each folder's seeds.json.

Folder	Seed	False alarms	Post-onset coverage	Warn₂₅₆	Early-warning rate	Median lead (tokens)
frozen-L15	42	5.3%	0.860	0.238	0.658	-181
frozen-L15	43 (released)	6.0%	0.850	0.231	0.667	-197
frozen-L15	44	5.5%	0.863	0.243	0.658	-191
adapted-L15	42	4.2%	0.968	0.553	0.718	-253
adapted-L15	43 (released)	5.0%	0.964	0.570	0.744	-247
adapted-L15	44	5.3%	0.967	0.588	0.735	-274
adapted-L30	42 (released)	4.6%	0.981	0.618	0.752	-284
adapted-L30	43	4.7%	0.979	0.623	0.718	-243
adapted-L30	44	4.4%	0.958	0.581	0.701	-280

Data

Prompts come from seven sources covering mathematics, competitive programming, instruction following and medical reasoning. Each prompt was answered 10 times. Splits are assigned per prompt. A response that emits the end-of-sequence token is treated as healthy. A response that reaches the token limit is labelled by an LLM judge, which reads the prompt and the response and quotes where the loop begins; that position is the onset.

Limitations

• The thresholds are calibrated for the prompt format and sampling settings above. Other settings change the score distribution, and the false-alarm rate is then no longer controlled.
• Long healthy responses are more likely to raise a false alarm than short ones.
• A response that loops and then emits the end-of-sequence token counts as healthy, so an alarm on it counts as a false alarm.
• The released seed is the best of three on validation, which does not make it the best on test. The results table lists all three seeds so the spread between them can be read.

Citation

@inproceedings{TODO,
  title  = {Degeneration Detection Is Not Early Warning: False-Alarm Evaluation with ONWARD},
  author = {De Negri, Marco and Sartori, Luca and Baggi, Lorenzo and Reihs, Moritz and Durech, Eduard Frank and El-Assady, Mennatallah and Pyatkin, Valentina and Hedström, Anna},
  year   = {2026},
}