simonlesaumon commited on
Commit
e359422
Β·
verified Β·
1 Parent(s): 826eda7

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +124 -0
README.md ADDED
@@ -0,0 +1,124 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model: google/diffusiongemma-26B-A4B-it
4
+ tags:
5
+ - diffusion
6
+ - text-humanization
7
+ - ai-detection-evasion
8
+ - diffusion-gemma
9
+ - block-diffusion
10
+ - research
11
+ pipeline_tag: text-generation
12
+ language: en
13
+ ---
14
+
15
+ # DiffusionGemma Humanizer
16
+
17
+ **Research, code, and pipeline** for humanizing AI-generated text using
18
+ Google's **DiffusionGemma 26B** (MoE, 3.8B active parameters, Apache 2.0).
19
+
20
+ ## Key Finding
21
+
22
+ **DiffusionGemma base model achieves ~0% AI detection** on Fast-DetectGPT +
23
+ heuristic ensemble detectors (perplexity + burstiness + stylometric markers).
24
+ Diffusion-generated text naturally resists autoregressive-trained detectors,
25
+ confirming the hypothesis from Tarim & Onan (2025).
26
+
27
+ ## How It Works
28
+
29
+ DiffusionGemma uses **block-autoregressive diffusion** with bidirectional
30
+ canvas attention. Instead of generating text left-to-right like GPT models,
31
+ it iteratively denoises 256-token blocks, producing text with fundamentally
32
+ different statistical fingerprints that evade standard AI detectors.
33
+
34
+ The humanization pipeline:
35
+ 1. Load DiffusionGemma 26B in 4-bit (NF4) on A100 80GB via Modal
36
+ 2. Generate baseline text with entropy-bounded denoising
37
+ 3. Test against open-source detectors (perplexity, burstiness, Fast-DetectGPT)
38
+ 4. Humanize via prompt engineering + decoder_input_ids (start denoising from AI text)
39
+ 5. Export results to this repository
40
+
41
+ ## Repository Contents
42
+
43
+ - `research_report.md` β€” Gemma models, diffusion LMs, Modal pricing
44
+ - `research_datasets_training.md` β€” Training data survey (HC3, RAID, M4, etc.)
45
+ - `commercial_ai_detectors_report.md` β€” Pangram, GPTZero, Originality.ai analysis
46
+ - `research/architecture-strategy.md` β€” Architecture decisions & cost breakdown
47
+ - `research/technical-diffusion-text-humanization-2026-06-29.md` β€” Full lit review
48
+ - `modal_project/app.py` β€” Complete 6-step Modal pipeline
49
+ - `scripts/` β€” Pipeline launchers
50
+
51
+ ## Architecture
52
+
53
+ DiffusionGemma uses block-autoregressive diffusion:
54
+ - Encoder processes prompt β†’ KV cache
55
+ - Decoder uses bidirectional attention on 256-token canvases
56
+ - Entropy-Bounded Denoising progressively refines text (1-48 steps)
57
+ - Starting canvas can be set via decoder_input_ids for iterative refinement
58
+
59
+ ## Why Diffusion for Humanization?
60
+
61
+ 1. **Different statistical signature** β€” Diffusion-generated text follows different token probability distributions than autoregressive text, making it OOD for AR-trained detectors
62
+ 2. **Bidirectional attention** β€” Holistic text understanding enables coherent style manipulation
63
+ 3. **Iterative denoising** β€” Progressive refinement from AI text toward human-like output
64
+ 4. **No left-to-right bias** β€” Avoids the "reversal curse" and other AR artifacts
65
+
66
+ ## Humanization Code Example
67
+
68
+ ```python
69
+ from transformers import DiffusionGemmaForBlockDiffusion, AutoProcessor, BitsAndBytesConfig
70
+ import torch
71
+
72
+ bn_config = BitsAndBytesConfig(
73
+ load_in_4bit=True,
74
+ bnb_4bit_compute_dtype=torch.bfloat16,
75
+ bnb_4bit_use_double_quant=True,
76
+ bnb_4bit_quant_type="nf4",
77
+ )
78
+ model = DiffusionGemmaForBlockDiffusion.from_pretrained(
79
+ "google/diffusiongemma-26B-A4B-it",
80
+ quantization_config=bn_config,
81
+ device_map="auto",
82
+ )
83
+ processor = AutoProcessor.from_pretrained("google/diffusiongemma-26B-A4B-it")
84
+
85
+ # Starting canvas = AI text (not random)
86
+ ai_text = "AI-generated text to humanize..."
87
+ ai_tokens = processor.tokenizer(ai_text, max_length=256, truncation=True,
88
+ padding="max_length", return_tensors="pt")
89
+
90
+ messages = [
91
+ {"role": "system", "content": "Rewrite to sound completely human-written."},
92
+ {"role": "user", "content": ai_text},
93
+ ]
94
+ inputs = processor.apply_chat_template(
95
+ messages, tokenize=True, add_generation_prompt=True,
96
+ return_dict=True, return_tensors="pt",
97
+ ).to(model.device)
98
+
99
+ # Iterative denoising from AI text toward human-like output
100
+ output = model.generate(
101
+ **inputs,
102
+ decoder_input_ids=ai_tokens["input_ids"].to(model.device),
103
+ max_new_tokens=512,
104
+ max_denoising_steps=24,
105
+ t_max=0.8,
106
+ t_min=0.4,
107
+ )
108
+ humanized = processor.decode(
109
+ output.sequences[0][inputs["input_ids"].shape[-1]:],
110
+ skip_special_tokens=True,
111
+ )
112
+ ```
113
+
114
+ ## References
115
+
116
+ - [DiffusionGemma Model Card](https://huggingface.co/google/diffusiongemma-26B-A4B-it)
117
+ - TarΔ±m & Onan (2025) β€” "Can You Detect the Difference? Diffusion vs AR Text"
118
+ - Sadasivan et al. (2023) β€” "Can AI-Generated Text be Reliably Detected?"
119
+ - Cheng et al. (2025) β€” "Adversarial Paraphrasing for Humanizing AI Text"
120
+ - [Modal GPU Pricing](https://modal.com/pricing)
121
+
122
+ ## License
123
+
124
+ Apache 2.0 (matching the base model).