Bussynfrfrfr's picture
|
download
raw
4.54 kB
---
base_model: google/gemma-4-26B-A4B-it
pipeline_tag: text-generation
library_name: transformers
language:
- en
license: apache-2.0
tags:
- abliteration
- uncensored
- gemma-4
---
# gemma-4-26B-A4B-it-uncensored
Uncensored version of [google/gemma-4-26B-A4B-it](https://huggingface.co/google/gemma-4-26B-A4B-it) with refusal behavior removed.
## Results
| | Before | After |
|--|--------|-------|
| **Refusals (mlabonne, 100 prompts)** | 98/100 | **1/100 effective (3 flagged, 2 refusal-then-comply)** |
| **Refusals (cross-dataset, 686 prompts)** | — | **5/686 (0.7%)** |
| **KL Divergence** | 0 (baseline) | **0.09** |
| **Quality (harmless response length ratio)** | 1.0 | **~1.01** (no degradation) |
### Cross-Dataset Validation
Tested against 4 independent prompt datasets to verify generalization:
| Dataset | Prompts | Refusals |
|---------|---------|----------|
| [JailbreakBench](https://huggingface.co/datasets/JailbreakBench/JBB-Behaviors) | 100 | 1/100 |
| [tulu-harmbench](https://huggingface.co/datasets/allenai/tulu-3-harmbench-eval) | 320 | 1/320 |
| [NousResearch/RefusalDataset](https://huggingface.co/datasets/NousResearch/RefusalDataset) | 166 | 0/166 |
| [mlabonne/harmful_behaviors](https://huggingface.co/datasets/mlabonne/harmful_behaviors) | 100 | 3/100 |
| **Total** | **686** | **5/686 (0.7%)** |
Every flagged refusal was manually audited. Most are "refusal-then-comply" false positives where the model
adds an AI identity disclaimer then answers the question anyway.
## Method
Norm-preserving biprojected abliteration on the dense pathway (o_proj + shared mlp.down_proj),
plus **Expert-Granular Abliteration (EGA)** on all 128 MoE expert down_proj slices per layer.
EGA ([OBLITERATUS](https://github.com/elder-plinius/OBLITERATUS)) hooks the MoE routers during probing
to compute per-expert routing weights for harmful vs harmless prompts, then applies norm-preserving
projection ([grimjim](https://huggingface.co/blog/grimjim/norm-preserving-biprojected-abliteration)) to each expert
individually. Dense-only abliteration leaves 29/100 refusals; adding EGA drops it to 3/100.
### Pipeline
1. Load model in bf16 with LoRA adapters on `o_proj` and `mlp.down_proj`
2. Collect residual activations for 400 harmful + 400 harmless prompts ([mlabonne](https://huggingface.co/mlabonne) datasets)
3. Winsorize activations at 99.5th percentile (clamps GeGLU outlier activations in Gemma family)
4. Compute per-layer refusal direction: `normalize(mean(harmful) - mean(harmless))`
5. Orthogonalize each direction against harmless mean (double-pass Gram-Schmidt)
6. Apply norm-preserving weight modification to `o_proj` and `down_proj` in all layers
7. Hook MoE routers, collect per-expert routing weights for harmful vs harmless prompts
8. Apply same norm-preserving modification to all 128 expert `down_proj` slices per layer
9. Merge LoRA adapters into base weights for clean tensor names
### Parameters
| Parameter | Value |
|-----------|-------|
| Layers abliterated | 100% |
| Scale | 1.0 |
| Winsorization | 0.995 |
| Experts abliterated | 100% (128/128 per layer) |
| Expert scale | 1.0 |
### How this differs from vanilla [heretic](https://github.com/p-e-w/heretic)
- **Norm-preserving biprojection** instead of standard projection (preserves weight magnitudes)
- **Per-layer refusal directions** instead of one global direction
- **Deterministic single-pass** instead of 50-trial Optuna search (faster, same or better results)
- **LoRA merge before save** for clean GGUF-compatible tensor names
- **Expert-Granular Abliteration** for MoE expert weights (not supported in heretic)
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model = AutoModelForCausalLM.from_pretrained("TrevorJS/gemma-4-26B-A4B-it-uncensored", dtype=torch.bfloat16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("TrevorJS/gemma-4-26B-A4B-it-uncensored")
messages = [{"role": "user", "content": "Your prompt here"}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True)
outputs = model.generate(inputs.to(model.device), max_new_tokens=512)
print(tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True))
```
## Reproduction
Full code and experiment data: [abliteration research repo](https://github.com/TrevorS/gemma-4-abliteration)
```bash
python scripts/ega.py --model google/gemma-4-26B-A4B-it \
--top-pct 100 --strip-topic-markers --skip-prefix --batch-size 4 \
--save output_dir
```

Xet Storage Details

Size:
4.54 kB
·
Xet hash:
4c1935fbdb517b59540b1ee46eb145e0a787b5d9ae1e183d9b96bf075e720b1c

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.