Instructions to use culturerevolt/gemma-4-12b-heretic-abliterated-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use culturerevolt/gemma-4-12b-heretic-abliterated-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="culturerevolt/gemma-4-12b-heretic-abliterated-GGUF",
	filename="gemma-4-12b-heretic-IQ3_XS.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use culturerevolt/gemma-4-12b-heretic-abliterated-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf culturerevolt/gemma-4-12b-heretic-abliterated-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf culturerevolt/gemma-4-12b-heretic-abliterated-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf culturerevolt/gemma-4-12b-heretic-abliterated-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf culturerevolt/gemma-4-12b-heretic-abliterated-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf culturerevolt/gemma-4-12b-heretic-abliterated-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf culturerevolt/gemma-4-12b-heretic-abliterated-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf culturerevolt/gemma-4-12b-heretic-abliterated-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf culturerevolt/gemma-4-12b-heretic-abliterated-GGUF:Q4_K_M

Use Docker

docker model run hf.co/culturerevolt/gemma-4-12b-heretic-abliterated-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use culturerevolt/gemma-4-12b-heretic-abliterated-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "culturerevolt/gemma-4-12b-heretic-abliterated-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "culturerevolt/gemma-4-12b-heretic-abliterated-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/culturerevolt/gemma-4-12b-heretic-abliterated-GGUF:Q4_K_M

Ollama
How to use culturerevolt/gemma-4-12b-heretic-abliterated-GGUF with Ollama:
```
ollama run hf.co/culturerevolt/gemma-4-12b-heretic-abliterated-GGUF:Q4_K_M
```

Unsloth Studio

How to use culturerevolt/gemma-4-12b-heretic-abliterated-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for culturerevolt/gemma-4-12b-heretic-abliterated-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for culturerevolt/gemma-4-12b-heretic-abliterated-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for culturerevolt/gemma-4-12b-heretic-abliterated-GGUF to start chatting

How to use culturerevolt/gemma-4-12b-heretic-abliterated-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf culturerevolt/gemma-4-12b-heretic-abliterated-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "culturerevolt/gemma-4-12b-heretic-abliterated-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use culturerevolt/gemma-4-12b-heretic-abliterated-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf culturerevolt/gemma-4-12b-heretic-abliterated-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default culturerevolt/gemma-4-12b-heretic-abliterated-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use culturerevolt/gemma-4-12b-heretic-abliterated-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf culturerevolt/gemma-4-12b-heretic-abliterated-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "culturerevolt/gemma-4-12b-heretic-abliterated-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use culturerevolt/gemma-4-12b-heretic-abliterated-GGUF with Docker Model Runner:
```
docker model run hf.co/culturerevolt/gemma-4-12b-heretic-abliterated-GGUF:Q4_K_M
```

Lemonade

How to use culturerevolt/gemma-4-12b-heretic-abliterated-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull culturerevolt/gemma-4-12b-heretic-abliterated-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.gemma-4-12b-heretic-abliterated-GGUF-Q4_K_M

List all available models

lemonade list

Gemma-4-12B-Heretic-Abliterated-iMatrix-GGUF (Complete Suite)

This repository hosts a complete, professional-grade suite of iMatrix (Importance Matrix) GGUF quantizations of culturerevolt/gemma-4-12b-heretic-abliterated.

The base model is an abliterated, fully decensored variant of Google's gemma-4-12b-it architecture, stripped of categorical refusal strings via norm-preserving directional ablation. This comprehensive GGUF suite is calibrated to preserve maximum linguistic entropy, narrative depth for creative fiction, and strict syntax parsing for agentic tools across multiple hardware profiles.

📊 Quick-Reference: Which Quant Should You Download?

All models (except the standard Q8_0 baseline) utilize a custom calculated importance matrix to prevent the loss of reasoning performance that standard static quants suffer from. Use this table to match the model files to your available hardware footprint:

File Name	Quant Type	Sizing / VRAM Target	Best Use Case
`gemma-4-12b-heretic-Q8_0.gguf`	Q8_0	24GB VRAM (RTX 3090 / 4090 / Mac Studio)	The Master Reference. Runs at a near-lossless INT8 resolution. Best suited for high-end setups where maximum possible precision is required.
`gemma-4-12b-heretic-Q6_K.gguf`	Q6_K	20GB–24GB VRAM (RTX 3090 / Dual GPUs)	The Near-Lossless Sweet Spot. Offers an incredibly high quality retention while saving significant space over the raw weights.
`gemma-4-12b-heretic-Q5_K_M.gguf`	Q5_K_M	16GB+ VRAM (RTX 4080 / 3090 / 4090)	The Enthusiast Tier. The highest fidelity compression level before hitting severe diminishing returns. Retains near-flawless parity with the source weights.
`gemma-4-12b-heretic-Q4_K_M.gguf`	Q4_K_M	16GB VRAM (RTX 4080 / 4070 Ti Super)	The Classic Standard. Features a balanced distribution of bit weights. Outstanding general purpose reasoning, chat response times, and stability.
`gemma-4-12b-heretic-IQ4_XS.gguf`	IQ4_XS	12GB–16GB VRAM (RTX 4070 / 4080)	The Optimizer's Choice. Custom tailored to maximize token throughput while protecting memory headroom. Fits alongside large context windows (up to 131k+) and MCP pipelines without aggressive system RAM spilling.
`gemma-4-12b-heretic-IQ3_XS.gguf`	IQ3_XS	8GB–12GB VRAM / Integrated Graphics / Mobile	The Lightweight Champion. Drastically reduces the memory footprint to run on lower-spec hardware, laptops, or mobile backends. The iMatrix lifeline keeps it coherent where flat 3-bit models collapse.

🧠 Technical Methodology: The iMatrix Advantage

When compressing a model down to 3, 4, or 5 bits, a basic, uncalibrated static quantization acts like a global color reduction on an image because it destroys critical structural details. Large Language Models contain sensitive outlier weights that dictate logic patterns, punctuation structure, and instruction adherence. Indiscriminately compressing these weights leads to specialization collapse, which causes a model to loop tokens, output formatting gibberish, or hallucinate heavily.

To protect this repo from degradation, these files were compiled using a Custom Hybrid Importance Matrix (imatrix.dat) strategy.

1. Calibration Profile

The calibration matrix was generated over a high-entropy, multi-domain text dataset specifically balanced for the local LLM stack:

Core Logic & Reasoning: Grounded using community standard English linguistic corpuses (text_en_large) to lock down general intelligence, deductive consistency, and vocabulary breadth.
Agentic Punctuation: Interwoven with structured tool-calling strings (tools_large) to safeguard brackets, colons, nested variables, and JSON structures necessary for automated file parsing pipelines like VaultForge.
Narrative Fidelity: Infused with creative prose samples to explicitly train the importance matrix to prioritize complex character interiority and vivid world-building pacing.

2. Execution Parameters

To avoid calculation errors or PCIe data-transfer distortions, the matrix was evaluated over a tight, isolated memory profile:

Total Cycles: 200 high-entropy chunks (representing roughly 100,000 tokens of diverse data)
Chunk Boundary: 512 tokens
Synchronization Sizing: -b 512 -ub 512 (Forcing physical batch alignment to protect mathematical scaling)

The resulting map acts as an explicit instruction booklet during quantization. It forces the llama-quantize tool to fiercely protect sensitive reasoning neurons while aggressively compressing the easy background language weights.

⚙️ Recommended Inference Settings

The Gemma-4 family is a highly capable but precise architecture. To avoid text stuttering, formatting drops, or interface crashes in backends like LM Studio, AnythingLLM, or llama-server, implement the following configurations:

1. Multi-Modal Vision Execution

Gemma-4 features a unified, encoder-free architecture, projecting raw image patches straight into the embedding layers. To handle vision tasks inside llama.cpp based frontends, you must load the companion multimodal projector file alongside your chosen text quant.

The verified companion file gemma-4-12b-it-mmproj-f16.gguf is hosted right here in the repository root. Ensure you map this file inside your backend Vision Adapter settings slot to seamlessly initialize image ingestion.

📜 Acknowledgements

Google DeepMind for pioneering the unified Gemma-4 architecture.
Philipp Emanuel Weidmann for developing the underlying Heretic abliteration framework.
Massive thanks to the open-source local AI community for continuously pushing the boundaries of what is possible on local consumer hardware.

⚠️ Disclaimer & Boundary Limits

This model is completely unaligned. It will output text without filtering, judgment, or warning labels. By downloading this model, you accept full responsibility for the prompts fed to it and the text generated by it. Use responsibly within local sandbox development setups.

🎛️ Streamlined Jinja Chat Template

If you encounter interface parsing issues with heavy multi-turn configurations or want to maximize token efficiency during rapid back-and-forth chat sessions, use this clean, hyper-efficient template:

{%- for message in messages %}
<|turn|>{{ message['role'] }}
{{ message['content'] }}<|turn|>
{%- endfor %}
{%- if add_generation_prompt %}
<|turn|>assistant<|channel>thought <channel|>
{%- endif %}

Downloads last month: 167,056

GGUF

Model size

12B params

Architecture

gemma4

Hardware compatibility

3-bit

4-bit

5-bit

6-bit

8-bit

Model tree for culturerevolt/gemma-4-12b-heretic-abliterated-GGUF

Base model

culturerevolt/gemma-4-12b-heretic-abliterated

Quantized

(5)

this model