Instructions to use Genrius-AI/Qwen3.5-4B-Pingala8-2bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Genrius-AI/Qwen3.5-4B-Pingala8-2bit with Trellis:

# No code snippets available yet for this library.

# To use this model, check the repository files and the library's documentation.

# Want to help? PRs adding snippets are welcome at:
# https://github.com/huggingface/huggingface.js

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use Genrius-AI/Qwen3.5-4B-Pingala8-2bit with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Genrius-AI/Qwen3.5-4B-Pingala8-2bit
# Run inference directly in the terminal:
llama cli -hf Genrius-AI/Qwen3.5-4B-Pingala8-2bit

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Genrius-AI/Qwen3.5-4B-Pingala8-2bit
# Run inference directly in the terminal:
llama cli -hf Genrius-AI/Qwen3.5-4B-Pingala8-2bit

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Genrius-AI/Qwen3.5-4B-Pingala8-2bit
# Run inference directly in the terminal:
./llama-cli -hf Genrius-AI/Qwen3.5-4B-Pingala8-2bit

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Genrius-AI/Qwen3.5-4B-Pingala8-2bit
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Genrius-AI/Qwen3.5-4B-Pingala8-2bit

Use Docker

docker model run hf.co/Genrius-AI/Qwen3.5-4B-Pingala8-2bit

LM Studio
Jan

vLLM

How to use Genrius-AI/Qwen3.5-4B-Pingala8-2bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Genrius-AI/Qwen3.5-4B-Pingala8-2bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Genrius-AI/Qwen3.5-4B-Pingala8-2bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Genrius-AI/Qwen3.5-4B-Pingala8-2bit

Ollama
How to use Genrius-AI/Qwen3.5-4B-Pingala8-2bit with Ollama:
```
ollama run hf.co/Genrius-AI/Qwen3.5-4B-Pingala8-2bit
```
Unsloth Desktop

How to use Genrius-AI/Qwen3.5-4B-Pingala8-2bit with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Genrius-AI/Qwen3.5-4B-Pingala8-2bit

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Genrius-AI/Qwen3.5-4B-Pingala8-2bit"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Genrius-AI/Qwen3.5-4B-Pingala8-2bit with Docker Model Runner:
```
docker model run hf.co/Genrius-AI/Qwen3.5-4B-Pingala8-2bit
```

Lemonade

How to use Genrius-AI/Qwen3.5-4B-Pingala8-2bit with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Genrius-AI/Qwen3.5-4B-Pingala8-2bit

Run and chat with the model

lemonade run user.Qwen3.5-4B-Pingala8-2bit-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use Genrius-AI/Qwen3.5-4B-Pingala8-2bit with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Genrius-AI/Qwen3.5-4B-Pingala8-2bit

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Genrius-AI/Qwen3.5-4B-Pingala8-2bit

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Genrius-AI/Qwen3.5-4B-Pingala8-2bit with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Genrius-AI/Qwen3.5-4B-Pingala8-2bit

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Genrius-AI/Qwen3.5-4B-Pingala8-2bit" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3.5-4B Pingala-8 2-bit

Qwen3.5-4B in 1.51 GB, 2-bit: WikiText-2 perplexity 10.97 vs 14.75 for llama.cpp IQ2_XXS, in a file 26.8 MB smaller (lower is better).

Needs the patched llama.cpp build (CUDA, NVIDIA RTX 30/40/50); stock llama.cpp, Ollama and LM Studio cannot load it. The patch is the same one the Qwen3-4B Pingala-8 file uses (patches/llama_cuda_vs_7ac59a6.patch, llama.cpp commit 7ac59a6); no new runtime is needed for this file. Tested in the v2.5 one-click build: double-click run_qwen35_4b.bat (Linux: ./run.sh qwen35-4b); the model downloads once, then a chat page opens in your browser.

File (pass part 1 to llama.cpp)	Quant	Size	WikiText-2 PPL
`Qwen3.5-4B-Pingala8-2bit-00001-of-00002.gguf` + `00002`	Pingala-8, 2-bit	1.51 GB	10.97
for comparison: IQ2_XXS (llama.cpp, same imatrix)	2-bit	1.54 GB	14.75
for comparison: IQ2_KT (ik_llama.cpp, default recipe, smaller file)	2-bit	1.44 GB	12.07
for comparison: Q8_0 (llama.cpp)	8-bit	4.5 GB	8.91

This is the text model only: the vision tower and the multi-token-prediction (MTP) block of Qwen3.5-4B are not included.

Author: Harshit Goyal, Genrius AI (harshit@genrius.in). Base model: Qwen/Qwen3.5-4B (Apache-2.0). Method paper: Small-State Coset Trellis Coding at IQ2_XXS Size: An Equal-Bytes Study in llama.cpp (preprint, v2.4), DOI 10.5281/zenodo.23481757. The paper covers Qwen3-4B, Llama-3.1-8B and Qwen3-14B; the Qwen3.5-4B numbers on this page are new measurements and are not in the paper. Code and runtime patch: https://github.com/Genrius-AI/pingala8-llm

What this is

A 2-bit-class quantization of Qwen3.5-4B (hybrid model: 24 Gated DeltaNet layers and 8 gated full-attention layers) at no more bytes than llama.cpp's IQ2_XXS file of the same model. Every linear weight (224 tensors: attn_qkv, attn_gate, ssm_beta, ssm_out in the DeltaNet layers; attn_q, attn_k, attn_v, attn_output in the attention layers; the three FFN matrices everywhere) is stored with Ungerboeck's 8-state set-partitioned trellis code over 4-weight symbols, behind a random Hadamard rotation and QuIP's LDLQ rounding, one rate class per tensor and incoherence-bound column scales: the B4 recipe of the Qwen3-4B release, unchanged. Layer Hessians come from 256 sequences of 2048 tokens of C4 text, propagated through the bf16 model one layer at a time. The other tensors (token embedding and tied output, norms, conv1d, the DeltaNet gate parameters) are copied from the stock IQ2_XXS file, except ssm_alpha, which is stored as Q4_0 (0.6 MB, paid from this file's byte budget).

This was a single build with the recipe and the release rule written down before any 2-bit number was measured; it is not the best of several tries on WikiText-2.

Files

file	bytes	sha256
`Qwen3.5-4B-Pingala8-2bit-00001-of-00002.gguf`	985,477,344	`2d78d7d4eccd4da1a2777ccc26ae399d262205b3fe5468fd30feb12105291050`
`Qwen3.5-4B-Pingala8-2bit-00002-of-00002.gguf`	526,138,464	`caacb42845e6ac3da4bdecec8ae92d157308b56d628eb6feee32a1cca33b1810`

Total 1,511,615,808 bytes (IQ2_XXS of the same model: 1,538,393,280 bytes). Pass the first shard to llama.cpp; it finds the second.

Which file should I pick?

If you...	Use
have an NVIDIA RTX 30/40/50 GPU and want Qwen3.5-4B text generation in about 1.5 GB	this file with the patched build or the v2.5 one-click build
use stock llama.cpp, Ollama, LM Studio, a Mac, an AMD GPU or only a CPU	a standard GGUF from the base model's Quantizations list; this file will not load there
need images or video input	a standard GGUF with its mmproj file; this file is text only
serve many requests at once or process long prompts	a standard quant: this file processes prompts at 0.79x the speed of IQ2_XXS

Prompt format

The Qwen3.5-4B chat template is stored in the GGUF (ChatML, <|im_start|> / <|im_end|>, with Qwen's thinking switch).

Results

WikiText-2 test, llama-perplexity -c 2049 --ppl-stride 4095 (windows of 4096 tokens; the Qwen3.5 tokenizer gives 72 windows on this text). Paired difference: this file minus the row, nats per token, 95% interval, and the windows on which this file is better.

format	file bytes	PPL	ratio to Q8_0	paired, Pingala-8 - row
Q8_0 (llama.cpp)	4.5 GB	8.9149	1.000	+0.207 +/- 0.017 (0/72)
IQ2_XXS (llama.cpp, imatrix)	1,538,393,280	14.7526	1.655	-0.296 +/- 0.010 (72/72)
IQ2_KT (ik_llama.cpp, default recipe, imatrix)	1,442,067,648	12.0709	1.354	-0.096 +/- 0.006 (72/72)
Pingala-8 (this file)	1,511,615,808	10.9693	1.230	0

The IQ2_KT file is 4.6% smaller than this one, so that row is not an equal-bytes comparison; an IQ2_KT recipe at this file's size was not built. All three 2-bit files use the same importance matrix (C4 text, context 512, 200 chunks).

Zero-shot accuracy, % (0-shot continuation scores from llama-perplexity; HellaSwag 1,000 items, MMLU all 14,042 test items):

format	HellaSwag	WinoGrande	ARC-Easy	ARC-Challenge	MMLU	average
Pingala-8 (this file)	61.9	65.19	62.98	43.48	35.51	53.81
IQ2_XXS	58.3	60.54	55.96	37.12	33.68	49.12

Standard errors: about +/-1.5 (HellaSwag), +/-1.34 (WinoGrande), +/-2.0 (ARC-Easy), +/-2.8 (ARC-Challenge), +/-0.4 (MMLU).

Speed, same-round ratio to IQ2_XXS on one RTX 5070 Ti Laptop GPU, batch 1 (llama-bench -ngl 99 -fa 1, median of 3 rounds of -r 5): token generation (tg128) 1.092x (167-173 against 152-162 tokens/s), prompt processing (pp512) 0.793x.

How to run

Build llama.cpp at commit 7ac59a6e3ad851cd41af00f678effab0598ba9a8 with the CUDA patch from the code repository:

git clone https://github.com/Genrius-AI/pingala8-llm
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout 7ac59a6e3ad851cd41af00f678effab0598ba9a8
git apply ../pingala8-llm/patches/llama_cuda_vs_7ac59a6.patch
cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120 -DCMAKE_BUILD_TYPE=Release -DLLAMA_OPENSSL=OFF
cmake --build build -j8 --target llama-cli llama-perplexity llama-bench

Then (all layers on the GPU):

hf download Genrius-AI/Qwen3.5-4B-Pingala8-2bit --local-dir qwen3.5-4b-pingala8
export GGML_F44=4 GGML_F44_BATCH=2 GGML_VR_NATIVE=1 GGML_VRC_V=5 GGML_VRC_NWP=8 GGML_VR_RL=1 GGML_VR_PRMG=1 GGML_VR_PPC=2
./build/bin/llama-cli -m qwen3.5-4b-pingala8/Qwen3.5-4B-Pingala8-2bit-00001-of-00002.gguf -ngl 99
./build/bin/llama-perplexity -m qwen3.5-4b-pingala8/Qwen3.5-4B-Pingala8-2bit-00001-of-00002.gguf -f wikitext2_test_hfjoin.raw -c 2049 --ppl-stride 4095 -ngl 99

The perplexity command is the measured protocol (expected final value [72]10.9693); the code repository rebuilds the text file. The llama-cli line is a usage example and was not part of the measured runs. Set CMAKE_CUDA_ARCHITECTURES for your GPU.

Limitations

CUDA only, measured on one laptop GPU (RTX 5070 Ti Laptop, WSL2); no Metal, Vulkan or CPU path for this file.
Text only: no vision tower, no MTP block (so no MTP speculative decoding).
Prompt processing is slower than IQ2_XXS (0.79x). Batched decoding was not measured on this model; on Qwen3-4B it was much slower than IQ2_XXS (0.18-0.34x for 2 to 8 sequences).
At 2 bits the model is clearly worse than the 8-bit model (perplexity 1.23x Q8_0).
Perplexity was measured at context 4096 only; long-context behaviour of the quantized DeltaNet layers was not tested.
One model of the Qwen3.5 family at this point; with the same recipe the method lost to IQ2_KT on Qwen3-14B (paper, Section 4.3).
Refusal, bias and hallucination behaviour after quantization were not evaluated. Evaluate the model for your own use.

Citation

@misc{goyal2026pingala8,
  title        = {Small-State Coset Trellis Coding at {IQ2\_XXS} Size: An Equal-Bytes Study in {llama.cpp}},
  author       = {Goyal, Harshit},
  year         = {2026},
  month        = oct,
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.23481757},
  url          = {https://doi.org/10.5281/zenodo.23481757},
  note         = {Preprint, version 2.4. Code: https://github.com/Genrius-AI/pingala8-llm}
}

License and notice

Apache-2.0, inherited from Qwen3.5-4B (Qwen team, Alibaba Cloud); see LICENSE and NOTICE. The weights are a quantized derivative of Qwen3.5-4B made by Genrius AI. "Pingala-8" and "Genrius AI" are names of Genrius AI; the license grants no trademark rights.