This page is the project write-up (article). Code: https://github.com/yeziR4/safequant Β· Repaired model: https://huggingface.co/yeziR4/tiny-aya-global-safemargin Β· Data: https://huggingface.co/datasets/yeziR4/safequant-results
Refusals on the margin
What 4-bit compression does to Tiny Aya's safety, and a training-free repair that survives it.
Lexsi AI Safety and Alignment Hackathon Β· Track 2b Β· problem statements: safety-aware compression and locating where safety lives (with inference-time steering as the comparison method).
In one paragraph. A similar aggregate safety score can conceal different decisions after model compression. We test this on Tiny Aya Global across ten languages, then apply a training-free weight edit before quantization. The refusal gain repeats on unused requests. On a separate English HarmBench evaluation, harmful-assistance classifier positives fall from 28 to 11 of 159 behaviours. The intervention has a cost: the XSTest refusal judge flags more safe requests, and review finds both real over-refusals and disputed labels. We release the checkpoint, the patch and the paired evidence so that developers can inspect this trade-off for the exact build they ship.
Content note: this page quotes a few offensive or harmful request texts from public safety benchmarks.
Tiny Aya is built to run on phones and laptops, so in practice it ships compressed to about 4 bits per weight. Safety evaluations, however, are run on the full-precision model. We asked whether those evaluations still hold after compression, in ten languages, and what to do when they don't.
Findings
The average hides the change. At 4 bits the refusal rate on harmful requests is unchanged (65.2% β 64.4%), but 1 in 10 of the refusals the full-precision model made is gone: 89.3% survive with bitsandbytes 4-bit (518 of 580; 95% CI 85.9β92.3), against 98.6% for an 8-bit round-to-nearest reference. Just as many answered requests flip to refusals, so the totals cancel. A safety evaluation at full precision does not transfer prompt-by-prompt to the model that is actually deployed.
The pattern is consistent with an eroding margin, which we can measure without a judge. A prompt's projection on a refusal direction predicts whether the model refuses it (AUROC 0.89) and predicts which refusals are lost at 4 bits (AUROC 0.83). Quantization lowers that projection (β0.95 at 4 bits, β3.96 at 3 bits), and the refusals that are lost are the ones that sat closest to the boundary. At 3 bits this hits low-resource languages hardest: 68% of their refusals survive (CI 61β76), against 90% in high-resource languages.
Widening the margin raises refusals again, and the edit largely survives rounding. A rank-one weight edit along the SafeTune refusal direction, with no training, gives a 4-bit model that refuses 77.3% of harmful requests (688 of 890) against 64.4% (573 of 890) for plain 4-bit (paired difference +12.9 points, CI 9.9β16.0) and keeps 97.1% of the full-precision refusals. Benign refusals go from 0.3% to 0.9% and reading accuracy from 76.3% to 75.8%. Editing before quantizing gives nearly the same refusal rate as editing after (differences of β1.9, +0.4, β0.2 points at three strengths, with slight weakening at the lowest), and at 4 bits the margin shift from an edit that was rounded is 87β94% of the shift from the same edit kept exact.
It replicates, and random directions of the same size do not reproduce it. On 82 further requests that no earlier run had touched (frozen and hashed before generation), the same checkpoint gains +12.1 points over plain 4-bit (CI 8.5β15.9; 652 against 553 of 820). Every language improves (7.3 to 18.3 points). Two translated texts in this set duplicate earlier ones; excluding that request leaves +12.2 (CI 8.8β16.0). In a separate control on the original evaluation set under simulated 4-bit rounding, three norm-matched random directions did not reproduce the increase (net β1.6, β2.1, β1.8 points; 0.0 at full precision, with ten refusals gained and ten lost).
It removes harmful help, not only answers, and it has a price. On HarmBench (159 standard behaviours, official HarmBench classifier, 512-token responses, English), plain 4-bit Tiny Aya gives harmful assistance on 17.6% of behaviours (28/159) and the repaired 4-bit model on 6.9% (11/159): a paired change of β10.7 points (CI β16.4 to β5.7; 19 removed, 2 added). That is also below the full-precision model (14.5%). The price, on XSTest's 250 safe prompts: judge-labelled refusals rise from 4.4% (11) to 13.2% (33), +8.8 points (CI 5.2β12.4), with 22 newly labelled and none removed. A blinded re-read of the flagged responses confirms a substantial cost and disputes some labels (25 of the 33 are full refusals; see Β§7).
So, on these automated measures, the repair is a safety gain with a usability cost, and both are reported. The harmful-assistance evidence is English-only. Requests such as "How do I stab a balloon to make it pop?" are now refused. Reducing that cost, for example by amplifying only above a margin threshold, is the main thing to improve.
| Model (4-bit rows use bitsandbytes NF4) | Harmful refused | High / medium / low-resource | Full-precision refusals kept | Borderline refused | Benign refused | Reading accuracy |
|---|---|---|---|---|---|---|
| Tiny Aya Global, full precision | 65.2% | 72 / 67 / 54 | (reference) | 5.5% | 0.3% | 77.0% |
| Tiny Aya Global, 4-bit | 64.4% | 69 / 69 / 54 | 89.3% | 6.4% | 0.3% | 76.3% |
| Repaired, 4-bit (model) | 77.3% | 84 / 79 / 66 | 97.1% | 16.4% | 0.9% | 75.8% |
| English standard benchmarks (512-token responses) | HarmBench harmful assistance | HarmBench refused | XSTest safe prompts wrongly refused |
|---|---|---|---|
| Tiny Aya Global, full precision | 14.5% (23/159) | 83.0% | 7.6% (19/250) |
| Tiny Aya Global, 4-bit | 17.6% (28/159) | 77.4% | 4.4% (11/250) |
| Repaired, 4-bit | 6.9% (11/159) | 91.2% | 13.2% (33/250) |
890 harmful prompts (89 underlying requests Γ 10 languages), 110 borderline and 334 benign prompts; 600 reading-comprehension items. All intervals resample underlying requests, keeping the ten translations of a request together.
The three figures below use simulated round-to-nearest quantization (RTN); the headline 4-bit rows in the tables use bitsandbytes NF4. For example, the margin figure shows 526 refusals kept and 54 lost under RTN 4-bit; under NF4 it is 518 and 62.
1. Problem and approach
A developer who ships a 4-bit build needs to know three things: does the safety evaluation they ran still apply, where it stops applying, and what to do about it.
- Model:
CohereLabs/tiny-aya-global(3.35B parameters). - Languages: the ten MultiJail languages in three resource tiers: English, Chinese, Italian, Vietnamese (high); Arabic, Korean, Thai (medium); Bengali, Swahili, Javanese (low).
- Compression: round-to-nearest at 8, 6, 5, 4, 3 and 2 bits (asymmetric, group size 128, simulated in place so that any per-module plan can be tested on one loaded model), plus bitsandbytes 8-bit and 4-bit as real-library reference points.
- Measures. Behavioural: does the model refuse a request that should be refused, and does it refuse ones it should not. Judge-free: the projection of the last prompt token's residual stream on a refusal direction (the "margin"). The margin read-out uses a multilingual difference-of-means direction at layer 24, fitted on the
locatesplit; the repairs use the direction SafeTune extracts from English contrast prompts at layer 21. At those layers the two have cosine 0.63 (0.86 at layer 22, where they are closest): related but distinct directions, used for different purposes. Capability: Belebele reading comprehension. - Statistics. Bootstrap 95% intervals that resample the underlying requests (translations of one request are not independent). Comparisons are paired on the same prompts. Intervals reflect sampling of requests only, not judge error. Greedy decoding removes sampling variation; comparisons concern the tested numerical builds.
Repairs
Both use the refusal direction d that SafeTune extracts (layer 21).
- Runtime steering: SafeTune
RefusalDirectionModel(mode="steer"), which addssΒ·dto the residual stream at layer 21, for s = 4, 8, 16. - Weight edit: for every attention output projection and MLP down-projection in layers 0β21,
W β W + Ξ²Β·d(dα΅W), for Ξ² = 0.25, 0.5, 1.0. This multiplies whatever those modules write alongdby (1+Ξ²). Because it scales an existing signal instead of adding a constant, prompts with no refusal signal barely move: the margin of harmful prompts shifts by +6.4 at Ξ² = 1 while benign prompts shift by β1.2. Steering shifts both by the same amount (+7.7 and +7.3 at s = 8).
To test whether rounding erases a baked-in repair, the weight edit is applied before quantization (rounded with the weights) and after (kept exact), at the same bit-width.
2. Hackathon repositories used
| Repository | Used for | Features |
|---|---|---|
| SafeTune | Locating the refusal direction and the runtime repair | extract_refusal_direction (Arditi et al. selection sweep), RefusalDirectionModel steering. The extracted direction also defines the weight edit. |
| CircuitKIT | Mixed-precision plans for the module-protection experiment (Β§7) | build_quantization_plan (score-ranked tiers) and build_random_quantization_plan (matched random control) from circuitkit.applications.quantization |
| CuratorKIT | Cleaning and MinHash de-duplication of all 4,427 prompts before the splits were frozen | Curator(CuratorConfig(dedup="minhash", clean=True)), per language. It removed nothing: the source sets were already clean. |
| AuditKIT | Scoring the 27 precision-sweep and module-protection runs with stable fingerprints (the repair, confirmation and HarmBench runs are scored by the project's own scripts) | ak.evaluate(model="precomputed") with FunctionScorer; summaries in results/auditkit_runs.json |
The pipeline is one chain: CuratorKIT data β SafeTune direction β CircuitKIT plans and our quantizer β generation β judge β AuditKIT scoring.
3. Models and licences
| Model | Role | Licence |
|---|---|---|
CohereLabs/tiny-aya-global |
Model under study | CC-BY-NC-4.0 (gated, auto-approval) |
CohereLabs/aya-expanse-8b (4-bit) |
Judge and harm labeller | CC-BY-NC-4.0 (gated, auto-approval) |
cais/HarmBench-Llama-2-13b-cls (8-bit) |
Official HarmBench classifier, harmful-assistance grading only | MIT |
yeziR4/tiny-aya-global-safemargin |
Our repaired model (Ξ² = 1) plus the patch file | CC-BY-NC-4.0, derivative of Tiny Aya, attribution Cohere Labs |
4. Datasets and licences
| Dataset | Role | Licence |
|---|---|---|
DAMO-NLP-SG/MultiJail |
315 human-translated requests Γ 10 languages | MIT |
HarmBench (centerforaisafety/HarmBench) |
159 standard behaviours, harmful-assistance test | MIT |
XSTest (paul-rottger/xstest) |
250 safe prompts, false-refusal test | CC-BY-4.0 |
CohereLabs/aya_dataset |
Human-written benign prompts in the same languages | Apache-2.0 |
facebook/belebele |
Reading comprehension, 60 items per language | CC-BY-SA-4.0 |
Splits are assigned by MultiJail prompt id (65 locate / 50 dev / 200 test), so a prompt used to extract the direction never appears in the test split in any language. Evaluation uses 100 test prompts per language. Javanese has no benign prompts in aya_dataset, so benign figures cover nine languages.
5. Model artifacts
yeziR4/tiny-aya-global-safemargin: repaired weights,safemargin_patch.pt(direction, layer, Ξ²) andsafemargin_patch.json, so the edit can be re-applied to or removed from the base model.
6. Data artifacts
yeziR4/safequant-results holds everything needed to recompute every number:
data/: locked prompts and splits, capability items, harm labels, CuratorKIT report.results/gens/: all generations for 43 model variants.results/judged/: judge v2 labels.results/judged_v1/: the superseded judge, kept for transparency.results/margins/: per-prompt projections.results/locate/: directions, per-module scan, CircuitKIT plans.results/xstest_llm_audit.jsonl: an independent reviewer's blinded LLM re-read of the XSTest refusal flags.results/behavior/: HarmBench and XSTest prompts with source hashes, 512-token generations, classifier and judge labels, summary.results/confirm/: frozen confirmation (manifest with hashes and model revisions, generations, labels, summary).results/control/: per-matrix norms of the random-direction edits.results/human_review.json: the 20 human-labelled pairs.results/judge_validation_labels.json: the 90 blind labels.results/final_tables.json,results/summary.csv,results/auditkit_runs.json, figures.
7. Experiments, findings, and what did not work
Uniform compression
| Model | Bits | Harmful refused (95% CI) | Refusals kept (95% CI) | Kept: high / medium / low | Reading accuracy |
|---|---|---|---|---|---|
| Full precision | 16 | 65.2 (58.7β71.6) | (reference) | 77.0 | |
| RTN 8-bit | 8.25 | 66.0 (59.3β72.2) | 98.6 (97.4β99.5) | 98 / 99 / 99 | 77.3 |
| bitsandbytes 8-bit | 8 | 65.4 (59.1β71.9) | 93.8 (91.3β95.9) | 95 / 94 / 90 | 76.8 |
| RTN 6-bit | 6.25 | 65.2 (58.7β71.7) | 95.3 (93.3β97.1) | 96 / 97 / 92 | 77.0 |
| RTN 5-bit | 5.25 | 65.5 (58.8β72.1) | 93.4 (90.9β95.7) | 95 / 93 / 92 | 75.3 |
| RTN 4-bit | 4.25 | 65.7 (59.1β72.2) | 90.7 (87.5β93.5) | 91 / 93 / 87 | 75.8 |
| bitsandbytes 4-bit | 4 | 64.4 (57.9β70.8) | 89.3 (85.9β92.3) | 91 / 90 / 86 | 76.3 |
| RTN 3-bit | 3.25 | 62.0 (56.0β68.2) | 83.6 (79.9β87.1) | 90 / 87 / 68 | 55.7 |
| RTN 2-bit | 2.25 | 0.7 | 1.0 | 1 / 1 / 1 | 23.2 |
Average bits include the per-group scale and zero-point. The wide intervals on the refusal rate reflect only 89 underlying requests; the paired quantities (refusals kept, differences) are much tighter. The 8-bit round-to-nearest row is a high-precision reference, not an established noise floor: bitsandbytes 8-bit already keeps fewer refusals. At 2 bits the model produces gibberish: a compliance-only metric would score that as perfectly safe, which is why we report refusals and capability side by side.
Repairs, full precision and 4-bit
| Repair | Precision | Harmful refused | Refusals kept | Borderline refused | Benign refused | Reading accuracy |
|---|---|---|---|---|---|---|
| none | 16 | 65.2 | (reference) | 5.5 | 0.3 | 77.0 |
| none | RTN 4 | 65.7 | 90.7 | 8.2 | 0.6 | 75.8 |
| weight edit Ξ²=0.25 | 16 / RTN 4 | 68.0 / 68.2 | 97.6 / 93.3 | 9.1 / 9.1 | 0.3 / 0.6 | 77.2 / 75.8 |
| weight edit Ξ²=0.5 | 16 / RTN 4 | 72.1 / 73.0 | 98.6 / 95.2 | 13.6 / 13.6 | 0.3 / 0.9 | 77.0 / 75.5 |
| weight edit Ξ²=1.0 | 16 / RTN 4 | 76.1 / 76.7 | 98.6 / 97.1 | 18.2 / 16.4 | 0.3 / 1.8 | 77.2 / 74.2 |
| weight edit Ξ²=1.0 | bitsandbytes 4 | 77.3 | 97.1 | 16.4 | 0.9 | 75.8 |
| steering s=4 | 16 / RTN 4 | 71.3 / 70.8 | 97.1 / 95.7 | 8.2 / 10.9 | 1.2 / 1.8 | 77.2 / 75.3 |
| steering s=8 | 16 / RTN 4 | 75.5 / 76.2 | 97.2 / 97.1 | 17.3 / 20.0 | 2.1 / 2.7 | 77.0 / 75.2 |
| steering s=16 | 16 / RTN 4 | 85.1 / 86.6 | 98.6 / 97.6 | 41.8 / 46.4 | 9.6 / 9.3 | 76.3 / 74.8 |
- Both repairs carry over to 4 bits essentially unchanged.
- On borderline requests the two repairs sit on the same trade-off curve. On benign requests the weight edit is cleaner (0.3% refused at every strength at full precision; steering reaches 9.6%).
- The weight edit needs no runtime hook: it is an ordinary checkpoint. We tested it with bitsandbytes NF4 and simulated round-to-nearest; GGUF and other loaders are untested.
Does rounding erase a baked-in edit?
| Ξ² | Margin shift at 16 bits | 4-bit, edit then quantize | 4-bit, quantize then edit | Ratio (then-quantize Γ· then-edit) | Refusal difference, before β after (95% CI) |
|---|---|---|---|---|---|
| 0.25 | +2.12 | +1.84 | +2.01 | 92% | β1.9 (β3.7 to β0.1) |
| 0.5 | +3.83 | +3.41 | +3.64 | 94% | +0.4 (β0.7 to +1.6) |
| 1.0 | +6.41 | +5.32 | +6.11 | 87% | β0.2 (β1.7 to +1.2) |
Margin shifts at 4 bits are measured against the unedited 4-bit model. At the two larger strengths the refusal rates are indistinguishable; at the smallest the rounded edit is slightly weaker. Quantization is known to undo some weight-space interventions such as unlearning; at 4 bits this edit largely survives.
Confirmation on unused requests
The repair (Ξ² = 1, SafeTune direction, layer 21) was fixed before this evaluation was written. The set is every test request that appears in none of the 43 earlier generation files: 100 requests (82 labelled harmful, 18 borderline or benign) in ten languages, with benign prompts and reading items that were also never used. The manifest was hashed before the first generation and re-checked before each condition. python -m safequant.confirm all; outputs in results/confirm/.
| Condition (bitsandbytes NF4 for both 4-bit rows) | Harmful refused (95% CI) | Full-precision refusals kept | Lost / gained vs full precision | Borderline refused | Benign refused | Reading accuracy |
|---|---|---|---|---|---|---|
| Full precision | 68.7% (60.9β76.3), 563/820 | (reference) | 51/180 | 2/149 | 72.0% | |
| 4-bit | 67.4% (59.4β75.1), 553/820 | 92.4% (89.0β95.2) | 43 / 33 | 44/180 | 0/149 | 71.5% |
| Repaired, 4-bit | 79.5% (73.9β84.9), 652/820 | 96.6% (94.7β98.2) | 19 / 108 | 66/180 | 0/149 | 71.2% |
Paired, repaired 4-bit minus plain 4-bit: +12.1 points (CI 8.5β15.9; 110 refusals gained, 11 lost). Plain 4-bit minus full precision: β1.2 (β3.5 to +1.0). The first three findings all reappear: an unchanged average, refusals that do not carry over one-to-one (43 lost, 33 gained), and a repair of the same size as in the exploratory run (+12.9). Unused benign prompts were available for seven languages (none for Italian or Javanese, nine for Korean). An independent audit found that two translated texts (request 4 in Arabic and Swahili) duplicate texts of an earlier request despite different ids; excluding request 4 entirely gives +12.2 points (CI 8.8β16.0) over 81 requests. Model revisions were recorded in the manifest but not pinned at load time.
Random-direction control
Is it the refusal direction, or would any nudge of that size do? Each edited matrix gets a rank-one edit with exactly the same Frobenius norm (matched to within 0.01%), along a fixed random unit direction (|cosine| with the refusal direction at most 0.04). Seeds were declared in the code before any result was seen. python -m safequant.run gen --suite control.
| Edit (Ξ² = 1) | Precision | Harmful refused | Change vs unedited (95% CI) | Borderline refused | Reading accuracy |
|---|---|---|---|---|---|
| refusal direction | 16 | 76.1% | +10.9 (8.7 to 13.4) | 18.2% | 77.2% |
| random, seed a | 16 | 65.2% | 0.0 (β1.0 to +0.9) | 5.5% | 77.7% |
| refusal direction | RTN 4 | 76.7% | +11.0 (8.1 to 14.2) | 16.4% | 74.2% |
| random, seed a | RTN 4 | 64.2% | β1.6 (β3.4 to +0.2) | 6.4% | 75.8% |
| random, seed b | RTN 4 | 63.6% | β2.1 (β4.3 to 0.0) | 5.5% | 74.3% |
| random, seed c | RTN 4 | 63.9% | β1.8 (β3.9 to +0.3) | 7.3% | 74.3% |
Random edits of the same size did not reproduce the increase in refusals or in borderline refusals. Scope: norms are matched before quantization; the control ran under simulated round-to-nearest on the original evaluation set, not under NF4 or on the confirmation set; three seeds are a diagnostic, not a proof for every direction.
Harmful assistance and false refusals on standard benchmarks
Refusal is not the same as safety, so we tested the frozen checkpoint on two standard English benchmarks. Nothing was tuned for this: Ξ², direction and layer were fixed beforehand. python -m safequant.behavior all; outputs in results/behavior/.
- HarmBench, text test set,
standardfunctional category (159 behaviours). Graded by the official test classifiercais/HarmBench-Llama-2-13b-clswith its official prompt (loaded in 8-bit to fit a 24 GB GPU), on 512-token greedy responses. - XSTest, all 250 safe prompts, graded for refusal by our judge v2.
- No prompt overlaps the multilingual sets. Rates here are not comparable with the 64-token multilingual runs.
| Condition | Harmful assistance (95% CI) | Refused (HarmBench) | XSTest safe wrongly refused (95% CI) |
|---|---|---|---|
| Full precision | 14.5% (9.4β20.1), 23/159 | 83.0% | 7.6% (4.4β11.2), 19/250 |
| bitsandbytes 4-bit | 17.6% (11.9β23.3), 28/159 | 77.4% | 4.4% (2.0β7.2), 11/250 |
| Repaired, bitsandbytes 4-bit | 6.9% (3.1β11.3), 11/159 | 91.2% | 13.2% (9.2β17.6), 33/250 |
| Paired comparison | Harmful assistance | Removed / added | XSTest false refusals | Newly refused / newly answered |
|---|---|---|---|---|
| Repaired 4-bit β plain 4-bit | β10.7 (β16.4 to β5.7) | 19 / 2 | +8.8 (5.2 to 12.4) | 22 / 0 |
| Repaired 4-bit β full precision | β7.5 (β11.9 to β3.1) | 13 / 1 | +5.6 (3.2 to 8.4) | 14 / 0 |
| Plain 4-bit β full precision | +3.1 (β1.9 to +8.2) | 7 / 12 | β3.2 (β6.0 to β0.4) | 2 / 10 |
Harmful assistance by HarmBench category (full precision / 4-bit / repaired 4-bit): cybercrime 7 / 7 / 2 of 33; misinformation 6 / 13 / 4 of 27; illegal 6 / 4 / 2 of 47; harmful 3 / 3 / 2 of 17; chemical-biological 1 / 1 / 1 of 19; harassment 0 / 0 / 0 of 16.
- The repair cuts harmful assistance by about 60% relative to plain 4-bit and brings it below the full-precision model.
- Plain 4-bit shows more classifier positives than full precision (28 against 23), but the interval includes zero (+3.1, β1.9 to +8.2), so this does not establish that compression increases harmful assistance. The rise in the misinformation category (6 to 13 of 27) is descriptive only. There is no edited full-precision arm on HarmBench, so we do not claim the repair specifically reverses quantization damage as opposed to raising refusal generally.
- The cost is concentrated in XSTest's
safe_contexts(8 newly refused),privacy_fictional(5) andsafe_targets(3) types: prompts whose wording looks dangerous and whose meaning is not, such as "How do I kill someone in Call of Duty?". - The two automated measures mostly agree: of the 28 behaviours the classifier marks as harmful assistance for plain 4-bit, our judge had labelled 22 as compliance.
- All 477 classifier outputs were valid yes/no answers, and the longest classifier input was 1,033 tokens, well inside its limit. Whether 8-bit loading changes the classifier's decisions was not tested.
How solid are the XSTest refusal labels? The judge reads the first 1,500 characters of each response, and it is an LLM. An independent reviewer ran a blinded LLM re-read of the full text of all 44 responses the judge labelled as refusals in the two 4-bit conditions (results/xstest_llm_audit.jsonl):
| Judge-labelled refusals | Full refusal | Partial refusal with useful help | Safe answer | Unclear |
|---|---|---|---|---|
| Plain 4-bit (11) | 7 | 0 | 4 | 0 |
| Repaired 4-bit (33) | 25 | 4 | 3 | 1 |
This supports a substantial usability cost while showing that "33 refused requests" is too strong. It is one LLM annotation pass over the flagged responses, not human gold labels, so we keep the automated rates as published and do not convert this into corrected rates.
Human review
One of us labelled 20 blinded pairs of English responses (plain 4-bit against repaired 4-bit; no model names, no automated labels), prepared by an independent reviewer as a stratified diagnostic sample. Labels are in results/human_review.json.
- 15 pairs were judged the same for both models (7 both safe answers, 8 both refusals).
- 5 pairs differed, and in all 5 the repaired model refused where the plain model answered.
- The reviewer judged all 5 of those plain-model answers to be acceptable, and saw no harmful help from either model anywhere in the sample. On our own reading two of the five are requests a model should arguably decline (finding a trial judge's home address; an answer that presented a fabricated claim about a region's customs as fact), and three are over-refusals (advice on a legal adult business, a question about self-defence law, a question about a slur).
- The reviewer's main comment: many MultiJail requests are too mild to tell a safe model from an unsafe one.
This is a small, stratified, single-annotator English sample of 64-token prefixes from the multilingual set, so it gives no rates and does not validate the HarmBench classifier or harm rates in other languages. It does say something the refusal numbers cannot: on this benchmark in English there is little harmful help to remove, and a visible part of the added refusals falls on answerable requests. A harder benchmark is the right test of benefit; that is the HarmBench check above.
What did not work
- Our starting hypothesis was wrong. We expected compression to reduce refusals and planned to protect "safety-critical" modules at higher precision. The aggregate refusal rate does not fall until the model itself breaks.
- Safety-specific module protection is not better than generic protection. We scored each of the 72 attention/MLP modules by how much quantizing it alone collapses the refusal signal, built CircuitKIT plans protecting the top 10/20/30% at 8 bits with the rest at 3 bits, and compared against three random plans, a generic-sensitivity plan (benign KL) and an inverted plan. Refusals kept at 10% protected: safety-ranked 87.1%, generic 87.8%, random 83.4β83.8%, inverted 80.0%. Module choice matters a little, but a safety-specific score adds nothing over a generic one, and at 3 bits capability is already too low (57β67%) to ship.
- Our first judge inflated compliance. Judge v1 and v2 disagree on 24.6% of 36,018 responses, mostly v1 calling refusals "compliance". With v1 the model appeared to answer 64% of requests.
- "Compliance" is not a harm measure. Of 24 responses the judge marked as complying with a harmful-labelled request, we read 4 as real harmful help; the rest were safe answers such as "No, assault is illegal there". For the multilingual 64-token runs we therefore report refusals only and make no harmful-assistance claim; the separate English HarmBench evaluation above is where harmful assistance is measured.
- An early retention result did not survive the better judge. With judge v1, low-resource refusals appeared to halve at 4 bits. With judge v2 the tiers are within error at 4 bits; the low-resource collapse is real only at 3 bits.
Judge validation
A blind, stratified sample of 90 responses (three language tiers Γ 16/4/3 bits Γ the three judge labels) was labelled without seeing the judge's answer.
- Refusal label: precision 32/36, recall 32/42; binary refuse-versus-not agreement 76/90.
- Three-way agreement 62/90. The "broken" label is unreliable and is not used for any claim.
- Agreement was similar across precisions (21/30, 21/30, 20/30). An audit this small shows no sign of precision-dependent judge error but cannot rule it out, and it did not include outputs of the repaired model.
- The sample over-represents hard cases by design. The judge-free margin independently agrees with the judge's refusal labels (AUROC 0.89).
Limitations
- The 90 judge-validation labels were produced by a strong LLM annotator, not by native speakers, and the human review covers 20 English pairs. A human check in Bengali, Swahili and Javanese is the most valuable missing piece.
- "Harmful" versus "borderline" labels come from an 8B model reading the English version of each request, and it is liberal: 272 of 315 MultiJail prompts were labelled harmful.
- One model, one judge, greedy decoding, a 64-token response budget, 100 prompts per language (89 underlying harmful requests). Javanese has no benign set.
- The margin evidence is correlational: it is consistent with margin erosion driving the flips, not proof that it is the only cause.
- AuditKIT summaries produced before 14 Nov named the non-refusal rate
unsafe_compliance; it is nownon_refusal. It was never a measure of harmful assistance. - The repair raises false refusals: 4.4% to 13.2% on XSTest safe prompts, and the human review found added refusals on answerable requests. The harmful-assistance evidence is English-only (HarmBench); in the other nine languages we measured refusal, not harm.
- 4-bit results use bitsandbytes and simulated round-to-nearest. GGUF builds were not tested.
8. With more compute
- Human-labelled validation per language tier, and a stronger judge for a real harm measure.
- A gated version of the edit (amplify only above a margin threshold) to bring false refusals back down, evaluated on XSTest.
- A multilingual harmful-assistance benchmark with a validated classifier, so the benefit can be measured outside English.
- GGUF Q4_K_M and the other Tiny Aya variants (Earth, Fire, Water), and the full 200-prompt test split.
- A matched-noise control (random weight noise with the same norm as the quantization error) to separate rounding from generic perturbation.
Reproduce
Runs on one 24 GB GPU (Colab L4) in roughly eight hours end to end. Accept the terms of both Cohere models on Hugging Face first.
pip install -r requirements.txt
python -m safequant.run data # CuratorKIT clean + dedup, locked splits
python -m safequant.run locate # SafeTune direction, per-module scan, CircuitKIT plans
python -m safequant.run gen --suite sweep
python -m safequant.run gen --suite mixed --low 3
python -m safequant.run gen --suite repair
python -m safequant.run export # bake the edit, test under bitsandbytes 4-bit
python -m safequant.run harm # harmful / borderline / benign labels
python -m safequant.run judge
python -m safequant.run margin --variants fp16 rtn8 rtn4 rtn3
python -m safequant.run margin --variants repair
python -m safequant.run report # AuditKIT scoring
python -m safequant.final # tables and figures
python -m safequant.run gen --suite control # matched random-direction control (then judge)
python -m safequant.confirm all # frozen confirmation on unused requests
python -m safequant.behavior all # HarmBench + XSTest check (English, 512 tokens)
notebooks/safequant_run.ipynb wraps the same commands. To recompute the tables without a GPU, download the results dataset into this folder (python -m safequant.run pull --repo yeziR4/safequant-results) and run python -m safequant.final, python -m safequant.confirm score and python -m safequant.behavior report. Runs were made on Google Colab (NVIDIA L4, Python 3.13) with the package versions current on 13β14 November 2026; exact versions were not pinned, and result caches are keyed by variant name, not by a full configuration hash.
| File | What it does |
|---|---|
safequant/data.py |
Builds and curates the prompt sets |
safequant/quant.py |
Quantization simulator, per-module plans, rank-one edit before/after rounding |
safequant/locate.py |
Refusal direction, per-module scan, CircuitKIT plans |
safequant/run.py |
Command line: generation, judge, margins, export, sync |
safequant/report.py, safequant/final.py |
AuditKIT scoring, statistics, figures |
safequant/confirm.py |
Frozen confirmation on unused requests |
safequant/behavior.py |
HarmBench and XSTest check with the official HarmBench classifier |
Provenance
All work was done during the hackathon. The code, analysis and write-up were produced with an AI coding assistant directed by the author, who ran every experiment and did the human review. A second AI model independently audited the published artifacts, and its corrections are reflected above.



