Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,124 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
base_model: google/diffusiongemma-26B-A4B-it
|
| 4 |
+
tags:
|
| 5 |
+
- diffusion
|
| 6 |
+
- text-humanization
|
| 7 |
+
- ai-detection-evasion
|
| 8 |
+
- diffusion-gemma
|
| 9 |
+
- block-diffusion
|
| 10 |
+
- research
|
| 11 |
+
pipeline_tag: text-generation
|
| 12 |
+
language: en
|
| 13 |
+
---
|
| 14 |
+
|
| 15 |
+
# DiffusionGemma Humanizer
|
| 16 |
+
|
| 17 |
+
**Research, code, and pipeline** for humanizing AI-generated text using
|
| 18 |
+
Google's **DiffusionGemma 26B** (MoE, 3.8B active parameters, Apache 2.0).
|
| 19 |
+
|
| 20 |
+
## Key Finding
|
| 21 |
+
|
| 22 |
+
**DiffusionGemma base model achieves ~0% AI detection** on Fast-DetectGPT +
|
| 23 |
+
heuristic ensemble detectors (perplexity + burstiness + stylometric markers).
|
| 24 |
+
Diffusion-generated text naturally resists autoregressive-trained detectors,
|
| 25 |
+
confirming the hypothesis from Tarim & Onan (2025).
|
| 26 |
+
|
| 27 |
+
## How It Works
|
| 28 |
+
|
| 29 |
+
DiffusionGemma uses **block-autoregressive diffusion** with bidirectional
|
| 30 |
+
canvas attention. Instead of generating text left-to-right like GPT models,
|
| 31 |
+
it iteratively denoises 256-token blocks, producing text with fundamentally
|
| 32 |
+
different statistical fingerprints that evade standard AI detectors.
|
| 33 |
+
|
| 34 |
+
The humanization pipeline:
|
| 35 |
+
1. Load DiffusionGemma 26B in 4-bit (NF4) on A100 80GB via Modal
|
| 36 |
+
2. Generate baseline text with entropy-bounded denoising
|
| 37 |
+
3. Test against open-source detectors (perplexity, burstiness, Fast-DetectGPT)
|
| 38 |
+
4. Humanize via prompt engineering + decoder_input_ids (start denoising from AI text)
|
| 39 |
+
5. Export results to this repository
|
| 40 |
+
|
| 41 |
+
## Repository Contents
|
| 42 |
+
|
| 43 |
+
- `research_report.md` β Gemma models, diffusion LMs, Modal pricing
|
| 44 |
+
- `research_datasets_training.md` β Training data survey (HC3, RAID, M4, etc.)
|
| 45 |
+
- `commercial_ai_detectors_report.md` β Pangram, GPTZero, Originality.ai analysis
|
| 46 |
+
- `research/architecture-strategy.md` β Architecture decisions & cost breakdown
|
| 47 |
+
- `research/technical-diffusion-text-humanization-2026-06-29.md` β Full lit review
|
| 48 |
+
- `modal_project/app.py` β Complete 6-step Modal pipeline
|
| 49 |
+
- `scripts/` β Pipeline launchers
|
| 50 |
+
|
| 51 |
+
## Architecture
|
| 52 |
+
|
| 53 |
+
DiffusionGemma uses block-autoregressive diffusion:
|
| 54 |
+
- Encoder processes prompt β KV cache
|
| 55 |
+
- Decoder uses bidirectional attention on 256-token canvases
|
| 56 |
+
- Entropy-Bounded Denoising progressively refines text (1-48 steps)
|
| 57 |
+
- Starting canvas can be set via decoder_input_ids for iterative refinement
|
| 58 |
+
|
| 59 |
+
## Why Diffusion for Humanization?
|
| 60 |
+
|
| 61 |
+
1. **Different statistical signature** β Diffusion-generated text follows different token probability distributions than autoregressive text, making it OOD for AR-trained detectors
|
| 62 |
+
2. **Bidirectional attention** β Holistic text understanding enables coherent style manipulation
|
| 63 |
+
3. **Iterative denoising** β Progressive refinement from AI text toward human-like output
|
| 64 |
+
4. **No left-to-right bias** β Avoids the "reversal curse" and other AR artifacts
|
| 65 |
+
|
| 66 |
+
## Humanization Code Example
|
| 67 |
+
|
| 68 |
+
```python
|
| 69 |
+
from transformers import DiffusionGemmaForBlockDiffusion, AutoProcessor, BitsAndBytesConfig
|
| 70 |
+
import torch
|
| 71 |
+
|
| 72 |
+
bn_config = BitsAndBytesConfig(
|
| 73 |
+
load_in_4bit=True,
|
| 74 |
+
bnb_4bit_compute_dtype=torch.bfloat16,
|
| 75 |
+
bnb_4bit_use_double_quant=True,
|
| 76 |
+
bnb_4bit_quant_type="nf4",
|
| 77 |
+
)
|
| 78 |
+
model = DiffusionGemmaForBlockDiffusion.from_pretrained(
|
| 79 |
+
"google/diffusiongemma-26B-A4B-it",
|
| 80 |
+
quantization_config=bn_config,
|
| 81 |
+
device_map="auto",
|
| 82 |
+
)
|
| 83 |
+
processor = AutoProcessor.from_pretrained("google/diffusiongemma-26B-A4B-it")
|
| 84 |
+
|
| 85 |
+
# Starting canvas = AI text (not random)
|
| 86 |
+
ai_text = "AI-generated text to humanize..."
|
| 87 |
+
ai_tokens = processor.tokenizer(ai_text, max_length=256, truncation=True,
|
| 88 |
+
padding="max_length", return_tensors="pt")
|
| 89 |
+
|
| 90 |
+
messages = [
|
| 91 |
+
{"role": "system", "content": "Rewrite to sound completely human-written."},
|
| 92 |
+
{"role": "user", "content": ai_text},
|
| 93 |
+
]
|
| 94 |
+
inputs = processor.apply_chat_template(
|
| 95 |
+
messages, tokenize=True, add_generation_prompt=True,
|
| 96 |
+
return_dict=True, return_tensors="pt",
|
| 97 |
+
).to(model.device)
|
| 98 |
+
|
| 99 |
+
# Iterative denoising from AI text toward human-like output
|
| 100 |
+
output = model.generate(
|
| 101 |
+
**inputs,
|
| 102 |
+
decoder_input_ids=ai_tokens["input_ids"].to(model.device),
|
| 103 |
+
max_new_tokens=512,
|
| 104 |
+
max_denoising_steps=24,
|
| 105 |
+
t_max=0.8,
|
| 106 |
+
t_min=0.4,
|
| 107 |
+
)
|
| 108 |
+
humanized = processor.decode(
|
| 109 |
+
output.sequences[0][inputs["input_ids"].shape[-1]:],
|
| 110 |
+
skip_special_tokens=True,
|
| 111 |
+
)
|
| 112 |
+
```
|
| 113 |
+
|
| 114 |
+
## References
|
| 115 |
+
|
| 116 |
+
- [DiffusionGemma Model Card](https://huggingface.co/google/diffusiongemma-26B-A4B-it)
|
| 117 |
+
- TarΔ±m & Onan (2025) β "Can You Detect the Difference? Diffusion vs AR Text"
|
| 118 |
+
- Sadasivan et al. (2023) β "Can AI-Generated Text be Reliably Detected?"
|
| 119 |
+
- Cheng et al. (2025) β "Adversarial Paraphrasing for Humanizing AI Text"
|
| 120 |
+
- [Modal GPU Pricing](https://modal.com/pricing)
|
| 121 |
+
|
| 122 |
+
## License
|
| 123 |
+
|
| 124 |
+
Apache 2.0 (matching the base model).
|