catplusplus/Qwen21_Text_Encoder_Heretic

Qwen3-VL-8B Heretic Text & Vision Encoder (Prompt Adherence & Geometric Alignment Edition) 🌺✨

This repository provides an optimized, abliterated checkpoint of the Qwen3-VL-8B text and vision encoder from Qwen-Image-2.1, processed with Norm-Preserving Biprojected Abliteration.

The primary purpose of this model is maximum instruction following and prompt adherence: it banishes geometric representation deflection ("internal blush" / hesitation vectors) that otherwise causes safety-tuned VLMs to corrupt diffusion conditioning on dynamic poses, human figures, athletic wear, and complex scenes.

---

🔬 The Core Problem: Why VLM Safety Alignment Degrades Diffusion Conditioning

In text-generation tasks, safety alignment mechanisms steer models to emit refusal text (e.g. "I cannot fulfill this request..."). However, modern multimodal diffusion architectures like Qwen-Image-2.1 do not generate text tokens:

$$\text{DiT Conditioning}⟵{\mathbf{h}}_L=\text{TextEncoder}(\text{tokens})[-1]\backslash text\{DiT\; Conditioning\}\; \backslash longleftarrow\; \backslash mathbf\{h\}\_L\; =\; \backslash text\{TextEncoder\}(\backslash text\{tokens\})[-1]$$

The diffusion transformer taps the raw pre-RMSNorm residual hidden states $\mathbf{h}_L$ directly from the text encoder to drive cross-attention.

The "Internal Blush" / Hesitation Deflection Phenomenon

When prompts describe human subjects, dynamic physical actions, athletic attire (e.g. swimwear, volleyball, gymnastics), or expressive emotions, safety-tuning vectors inside the language model activate even on completely benign, non-refusal prompts.

Because the model cannot output a refusal string, these alignment vectors manifest as a geometric rotation of the latent representation:

$${\mathbf{h}}_{\text{sensitive}}={\mathbf{h}}_{\text{clean}}+{\mathbf{v}}_{\text{refusal}}\backslash mathbf\{h\}\_\{\backslash text\{sensitive\}\}\; =\; \backslash mathbf\{h\}\_\{\backslash text\{clean\}\}\; +\; \backslash mathbf\{v\}\_\{\backslash text\{refusal\}\}$$

This hidden deflection rotates the conditioning signal by over 60% relative norm away from the prompt's intended semantic visual trajectory!

Visual Consequences in Image Generation

When cross-attention layers in the DiT receive a representation deflected into the refusal/modesty subspace, the model displays hesitation artifacts:

1. Modesty Hallucinations & Clothing Confusion: Spontaneous addition of mismatched cloth, awkward white ruffles, or extra fabric covering swimwear or sportswear.
2. Anatomical Occlusion: The model avoids rendering human limbs or athletic poses, awkwardly hiding arms behind character backs or contorting torsos.
3. Subject & Prop Merging: Equipment or background elements get fused into characters (e.g. sports balls bizarrely merged onto heads as hair ornaments).
4. Action Damping: Dynamic verbs ("jumping to spike the ball") are subdued into passive, static standing postures.

---

📊 Quantitative Measurement of Representation Deflection

Using contrastive prompt pairs across benign and sensitive subjects, we measured the layer-by-layer cosine similarity and relative deflection norm across all 37 positions (input embeddings + 36 decoder layers) of Qwen3-VL-8B:

Layer Index	Position	Cosine Similarity ($\cos \theta$)	Relative Deflection ($|\Delta \mathbf{h}| / |\mathbf{h}|$)	Deflection Norm $|\Delta \mathbf{h}|$
0	Input Embeddings	1.0000	0.00%	0.00
8	Early Transformer	0.9991	3.82%	18.24
16	Mid-Low (Deflection Onset)	0.9943	10.64%	52.88
20	Mid-High Divergence	0.9780	21.05%	114.73
24	Refusal Vector Surge	0.9414	34.25%	192.40
28	Acceleration Peak	0.8842	47.19%	275.31
32	Late Transformer	0.8350	56.12%	331.05
36	Final Conditioning Layer	0.8110	60.30%	357.94

Between Layer 20 and Layer 36, representation deflection accelerates rapidly, culminating in a 60.3% vector distortion. By surgically neutralizing this direction, the text encoder reflects the exact intended prompt semantics.

---

🛠️ Methodology: Norm-Preserving Biprojected Abliteration

To eliminate hesitation deflection without degrading general language comprehension, we applied Norm-Preserving Biprojected Abliteration (create_heretic_text_encoder.py):

1. Refusal Subspace Extraction: Difference-of-means vectors were extracted across contrastive prompt sets: $$\mathbf{r}l = \boldsymbol{\mu}{\text{sensitive}}^{(l)} - \boldsymbol{\mu}_{\text{benign}}^{(l)}$$
2. Benign Subspace Orthogonalization: The general semantic direction was stripped from the refusal vector: $$\mathbf{v}l = \mathbf{r}l - \text{proj}{\mathbf{u}{\text{benign}}}(\mathbf{r}_l)$$
3. Norm-Preserving Rank-1 Projection: Across 54 linear projection matrices (self_attn.o_proj and mlp.down_proj in layers 9–35, centered at layer 26 with Gaussian falloff $\lambda \in [0.10, 1.00]$): $$W_{\text{norm}} = \text{normalize}(W, p=2, \text{dim}=1)$$ $$W' = \text{normalize}\Big(W_{\text{norm}} - \lambda \mathbf{v}l (\mathbf{v}l^T W{\text{norm}})\Big) \cdot |W|{\text{row}}$$

Because exact row norms ($|W|_{\text{row}}$) are strictly preserved, the network's overall activation scales and general reasoning capabilities remain completely intact.

---

🤝 Pairing with Quantized DiT (nunchaku-qwen-image-2.1)

This text encoder is specifically engineered to be paired with nunchaku-qwen-image-2.1 for consumer GPU setups:

• Resident DiT + Streamed Text Encoder:
  • DiT (best_quality_fp4.safetensors): 4.08 GB resident VRAM.
  • VAE (AutoencoderKLQwenImage21): 0.64 GB resident VRAM.
  • Qwen3-VL-8B ViT Vision Encoder: 1.07 GB resident VRAM.
  • Qwen3-VL-8B Language Model: Streamed layer-by-layer through a static 368 MB GPU buffer over PCIe at ~28.7 GB/s via stream_encoder.py.
• Total VRAM Footprint: ~6.17 GB active VRAM, leaving ~9.5 GB free headroom on a single 16 GB GPU (such as RTX 5060 Ti or RTX 4080)!
• Inference Speed: Multimodal prompt encoding completes in 1.06s (saving 16s vs CPU), and 25-step image generation runs in ~20s.

---

🚀 Quickstart Usage

1. Installation

pip install diffusers transformers accelerate torch sentencepiece

2. Loading with Diffusers

import torch
from diffusers import QwenImage21Pipeline
from transformers import Qwen3VLForConditionalGeneration, Qwen3VLProcessor

# 1. Load Heretic text encoder and processor
text_encoder = Qwen3VLForConditionalGeneration.from_pretrained(
    "models/Qwen21_Text_Encoder_Heretic",
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
)
processor = Qwen3VLProcessor.from_pretrained("models/Qwen21_Text_Encoder_Heretic")

# 2. Assemble into pipeline
pipe = QwenImage21Pipeline.from_pretrained(
    "Qwen/Qwen-Image-2.1",
    text_encoder=text_encoder,
    processor=processor,
    torch_dtype=torch.bfloat16,
)
pipe.enable_sequential_cpu_offload(gpu_id=0)

# 3. Generate with precise prompt adherence
image = pipe(
    prompt="Two cute anime girls in colorful bikinis playing beach volleyball on a sunny tropical beach, dynamic action pose, jumping to spike the ball, sharp focus",
    height=1024,
    width=1024,
    num_inference_steps=25,
    true_cfg_scale=1.0,
).images[0]

image.save("beach_volleyball.png")

3. High-Throughput Server Usage

Run the bundled ImageEditServer with NVFP4 DiT and Heretic text encoder:

# Start server on port 4500 (uses Heretic text encoder by default)
./extras/imagegen_qwen21_nvfp4.sh 4500

---

📦 Packaged Sources (extras/)

• create_heretic_text_encoder.py: Complete script used to measure refusal vectors and perform norm-preserving biprojected abliteration.
• stream_encoder.py: Zero-quality-loss layerwise weight streaming engine for Qwen3-VL-8B.
• test_heretic_beach_volleyball.py: Empirical verification script comparing stock vs Heretic encoders.
• QwenImage21NVFP4Backend.py: Diffusers + Nunchaku backend supporting custom text encoder overrides.
• ImageEditServer.py & imagegen_qwen21_nvfp4.sh: Resident image generation server.

---

📜 Citation & Credits

• Qwen-Image-2.1 & Qwen3-VL: Qwen Team, Alibaba Cloud.
• Abliteration Principles: Arditi et al. (Refusal in Language Models Is Mediated by a Single Direction).
• Heretic LLM: Heretic project (Directional Abliteration Toolkit).
• Abliteration & Diffusion Conditioning Optimization: Oleg K. / Nikola Seeker Project.