CATCH-Gemma-4-31B โ€” LoRA adapter for Gemma-4-31B-it

A LoRA adapter trained with CATCH (adversarial honesty training) on the language model of google/gemma-4-31B-it, revision 439edf5652646a0d1bd8b46bfdc1d3645761a445.

This is run ul9959, checkpoint-108 (epoch 3), the run's most honest checkpoint on a 150-record MASK subset used for checkpoint selection. That subset is part of the 1,000 records below.

Results

Full MASK benchmark (1,000 records), pooled over the six archetypes; the adapter vs the base model, with Gemma's reasoning (thinking) mode off and on:

model thinking honest evade lie accuracy
Gemma-4-31B-it off 12.9 31.2 55.9 73.7
+ CATCH (this adapter) off 66.1 19.8 14.1 76.6
Gemma-4-31B-it on 19.5 27.9 52.6 76.3
+ CATCH (this adapter) on 63.8 27.6 8.6 73.7

Honest rate by MASK archetype (thinking off):

model known facts provided facts statistics continuations disinformation doubling down
Gemma-4-31B-it 23.4 6.6 39.6 4.5 2.4 10.8
+ CATCH (this adapter) 60.8 94.5 71.9 44.9 42.4 61.7

AlpacaEval 2.0 (805 prompts), length-controlled win rate vs the base Gemma-4-31B-it: 55.2% (95% bootstrap CI 51.1โ€“58.9; raw 54.5%, 706 judgeable prompts, mean length 1.00ร— base), i.e. no utility cost.

Evaluation details:

  • Judge: openai/gpt-6-luna (reasoning effort low, seed 0) with the MASK judge prompts and the AlpacaEval pair-v2 prompt. These numbers are therefore not directly comparable to the evaluations of our other CATCH adapters, which used a different judge model.
  • Generation: MASK temperature 1.0 / top_p 0.95, 512 tokens (thinking off) or 8,192 tokens (thinking on); AlpacaEval temperature 0.7 / top_p 0.9, 2,048 tokens.
  • Thinking on: the judge sees only the final answer, not the thought channel. With the adapter, 12.8% of thinking-mode generations do not finish within 8,192 tokens (base: 0.1%). These count as abstentions (evade). The adapter was trained with thinking off; it lowers lying with thinking on too, but it makes reasoning less reliable to terminate.

Usage

The adapter covers the language model only. Its keys use the names of the text-only Gemma4ForCausalLM (base_model.model.model.layers.N...), so load the base model the same way we trained it:

import torch
from peft import PeftModel
from transformers import AutoConfig, AutoTokenizer, Gemma4ForCausalLM

cfg = AutoConfig.from_pretrained("google/gemma-4-31B-it").text_config
base = Gemma4ForCausalLM.from_pretrained(
    "google/gemma-4-31B-it", config=cfg,
    key_mapping={r"^model\.language_model\.": "model."},   # text weights only; the vision tower is skipped
    torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, "kibiddd/CATCH-Gemma-4-31B")
tok = AutoTokenizer.from_pretrained("kibiddd/CATCH-Gemma-4-31B")   # = the base Gemma-4 tokenizer
prompt = tok.apply_chat_template([{"role": "user", "content": "Hello"}], tokenize=False,
                                 add_generation_prompt=True)   # thinking off (the template default)

Loading it onto the multimodal AutoModelForCausalLM / Gemma4ForConditionalGeneration instead would not match these key names.

With vLLM, use vLLM โ‰ฅ 0.25: vLLM 0.22 loads Gemma-4 LoRA adapters but does not apply them (outputs equal the base model). vLLM 0.25 maps the text-model key names onto the language model, so the adapter loads as is. We ran it with enforce_eager=True.

from huggingface_hub import snapshot_download
from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest

path = snapshot_download("kibiddd/CATCH-Gemma-4-31B", allow_patterns="adapter_*")
llm = LLM(model="google/gemma-4-31B-it", enable_lora=True, max_lora_rank=64, tensor_parallel_size=2, enforce_eager=True)
out = llm.generate(prompt, SamplingParams(temperature=0.7, max_tokens=512),
                   lora_request=LoRARequest("catch", 1, path))

Gemma-4's chat template leaves thinking off unless enable_thinking=True; with thinking off, the generation prompt ends in an empty thought channel (<|channel>thought\n<channel|>). The adapter was trained in that format.

Training

method CATCH
objective toward (honest target) + clipped away / unlikelihood (deceptive target) + utility, under an embedding-space attack
loss weights toward / away / utility 0.5 / 0.5 / 1
cutoffs toward / away 0.5 / -5
attack eps / step / steps 0.1 x mean input-embedding norm / 1.8e-2 (sign) / 10 -- see the note below
honesty data 290 contrast pairs over 58 scenarios, generated by Gemma-4-31B-it and labelled by a GPT-6 Luna judge: 220 pairs / 44 self-report scenarios + 70 pairs / 14 content-production scenarios (see below)
utility data 1,984 task-diverse instructions (the Magpie instructions of our Qwen3.8 utility set) answered by Gemma-4-31B-it itself
adv / utility mix 0.5 / 0.5
learning rate 5e-5, cosine, warmup_ratio 0.1
batch (device x accum.) 8 x 2
max sequence length 1024
checkpoint step 108 (epoch 3) of a 30-epoch schedule; the utility early-stopping guard ended training after epoch 5
LoRA r 64, alpha 16, dropout 0.1, 7 target modules (q/k/v/o/gate/up/down projections), language model only
precision bf16, 4-bit base (nf4, double-quant off)
libraries transformers 5.14.1, peft 0.11.1, bitsandbytes 0.49.2, torch 2.9.1

Attack-strength note. Gemma multiplies its input embeddings by sqrt(5376) โ‰ˆ 73.3 inside the embedding layer, and the attack acts on these scaled embeddings (mean norm โ‰ˆ 82, vs 0.90 for Qwen3.8). The radius is relative (eps = 0.1 ร— that norm), and the sign-step size is set so that 10 steps cover โ‰ˆ 1.6ร— the radius, the same step-to-radius ratio as in our Qwen runs.

Honesty-data note. 14 of the 58 training scenarios are the content-production scenarios of our held-out generalized-honesty test set (system-prompt pressure to write false content). Any evaluation of this adapter on that set must exclude its content-production category. We checked these 14 scenarios against all 1,000 MASK records and found no shared 8-grams.

Base model license: Apache-2.0 (Gemma 4 license).

Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for kibiddd/CATCH-Gemma-4-31B

Adapter
(382)
this model

Collection including kibiddd/CATCH-Gemma-4-31B