LatentRisk-Probe

Measure how strongly a frozen language model's hidden activations align with harmful-request labels. This repository contains two small sparse linear readouts, one for Gemma 3 1B and one for Qwen3 0.6B, plus the extraction code and benchmark controls. Input a prompt; receive a benchmark risk score and a thresholded label without generating a response.

Use the probes to investigate over-refusal, representation differences and failure clusters in a safety research dashboard. The Gemma probe reaches 80.75% balanced accuracy on XSTest, while both probes falsely flag roughly 43–46% of a difficult benign suite. They are diagnostic instruments for studying behavior, not reliable production filters or detectors of a person's intent.

Application example

Audit benign prompts that look risky. Compare a request about recognizing a fraudulent message with one asking how to deceive a victim. Inspect the scores alongside the original text and known benchmark labels, then review high-scoring benign cases to understand over-refusal. The two prompts in the code example are illustrative; a score is an uncalibrated model readout, not evidence about the user's intentions.

Inference

Download probe.py, authenticate for the Gemma backbone if needed, then:

from probe import LatentRiskProbe

probe = LatentRiskProbe(backbone="gemma")  # Or backbone="qwen"; pin revision for frozen use.
scores = probe.score_texts([
    "How can I recognize a fraudulent message?",
    "How can I deceive someone into sending me money?",
])
print(scores)

Output is an uncalibrated benchmark risk score and a label at 0.5. It is not a probability of real intent. The base language model must be downloaded separately; the probe files are about 9 KB each. GPU reproduction: PyTorch2.6.0, Transformers4.57.1, NumPy2.1.3, scikit-learn1.6.1, Safetensors0.5.3, bfloat16/SDPA. CPU float32 is a different activation configuration from the reported GPU run.

Cross-dataset results

XSTest: 450 prompts, including 250 safe and 200 unsafe requests. Threshold 0.5, selected without using XSTest.

Readout	Balanced accuracy	AUROC	Benign false-positive rate	Harmful false-negative rate
Gemma sparse probe	80.75%	0.8978	12.0%	26.5%
Qwen sparse probe	74.25%	0.8292	26.0%	25.5%
Character TF-IDF	59.85%	0.6260	32.8%	47.5%
Word TF-IDF	57.00%	0.5979	20.0%	66.0%
Length-only logistic	52.80%	0.5582	86.4%	8.0%

The fixed hidden-state-index-7 controls score 72.8% (Gemma) and 60.5% (Qwen) balanced accuracy. Shuffled-label controls score 48.45% and 47.9%; Gaussian-feature controls score 52.6% and 48.3%. The selected hidden states and regularization were chosen on validation only.

Failure regions and transfer

Suite	Gemma	Qwen
OR-Bench category holdout, balanced accuracy (222)	77.48%	85.59%
Difficult benign OR-Bench, false-positive rate (188)	45.74%	42.55%
SimpleSafetyTests, false-negative rate (100 unsafe)	20.0%	26.0%
Authored Portuguese stress test, balanced accuracy (24)	75.0%	100.0%

The Portuguese result is only 12 authored pairs, with no independent human review. It does not demonstrate broad multilingual generalization. Single-class suites have no AUROC or balanced-accuracy claim. Benchmark labels can be disputable, especially for hard benign/dual-use prompts.

Training and selection

Frozen OR-Bench benign/toxic prompts, length-matched within category: 816 balanced training examples, 122 validation examples from the held-out unethical category, and 222 held-out examples from privacy/illegal. Exact normalized-prompt deduplication spans all partitions. Near-duplicates may remain.

Independent tests: XSTest by Röttger et al., SimpleSafetyTests by Vidgen et al., 188 unused difficult benign OR-Bench prompts, and 24 authored Portuguese requests. Upstream CC-BY attribution, immutable revisions, hashes, split labels and preparation code are recorded in data/manifest.json and data/prepare_latent_probe.py. These benchmarks may overlap the backbones' pretraining; a fresh sealed evaluation remains outstanding.

Features are the final prompt-token hidden state at the assistant-generation boundary, with explicit cumulative attention-mask position IDs, left padding/truncation and a 512-token limit. No responses were generated. L1 logistic regression uses a training-fitted StandardScaler; hidden-state index and C∈{0.01,0.1,1} maximize validation AUROC. Text baselines fit only training text and select C on the same validation set.

Gemma: pinned base dcc83ea841ab6100d6b47a070329e1ba4cf78752, hidden-state index 9 (after transformer block 8), C=1, 138 nonzero features. Qwen: base c1899de289a04d12100db370d81485cdf75e47ca, hidden-state index 17 (after block 16), C=0.1, 62 nonzero features. Index 0 is the embedding state.

Serialized raw-feature readouts fold the training scaler into a double-precision weight/bias. An independent audit reconstructs every accuracy/Brier/log-loss from captured activations; tiny changes in training-set AUROC from numerically tied probabilities are bounded explicitly. No test threshold or layer tuning was applied after opening the independent suites.

Limits and licenses

This study has one split/seed, existing public benchmarks and a small authored language suite. It supplies no causal steering validation, covert-intent detector, response-level moderator, robustness guarantee or production refusal policy. Both missed harms and over-refusal remain substantial.

Gemma-associated components are subject to the included Gemma Terms and Section3.2 restrictions, including the Prohibited Use Policy. Qwen-associated readouts use Apache2.0. Benchmark text/labels retain their CC-BY4.0 terms. See LICENSE.md, GEMMA_TERMS.txt, GEMMA_PROHIBITED_USE_POLICY.txt, APACHE_LICENSE.txt and NOTICE. No backbone weights are redistributed.