Instructions to use Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit with Trellis:

# No code snippets available yet for this library.

# To use this model, check the repository files and the library's documentation.

# Want to help? PRs adding snippets are welcome at:
# https://github.com/huggingface/huggingface.js

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit
# Run inference directly in the terminal:
llama cli -hf Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit
# Run inference directly in the terminal:
llama cli -hf Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit
# Run inference directly in the terminal:
./build/bin/llama cli -hf Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit

Use Docker

docker model run hf.co/Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit

Llama new
LM Studio
Jan

vLLM

How to use Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit

Ollama
How to use Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit with Ollama:
```
ollama run hf.co/Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit
```
Unsloth Desktop

How to use Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit with Docker Model Runner:
```
docker model run hf.co/Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit
```

Lemonade

How to use Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit

Run and chat with the model

lemonade run user.Qwen3.5-0.8B-Pingala8-2bit-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3.5-0.8B Pingala-8 2-bit

Like it? Star the code on GitHub: Genrius-AI/pingala8-llm (it helps others find it)

Try it in your browser: https://huggingface.co/spaces/harshit-genrius/pingala8-demo (live chat with Qwen3-1.7B-Pingala8-2bit on CPU)

Qwen3.5-0.8B in 0.35 GB, 2-bit: WikiText-2 perplexity 25.97 vs 85.09 for llama.cpp IQ2_XXS, in a file 5.5 MB smaller (lower is better).

Stock llama.cpp, Ollama and LM Studio will refuse this file with a load error; use one of these instead: the one-click build for Windows or Linux or the Linux one-liner curl -fsSL https://genrius-ai.github.io/pingala8-llm/install.sh | bash (Run it).

Use the v2.5 one-click build (Windows and Linux, NVIDIA RTX 30/40/50): download it from the latest release, unzip it and double-click run_qwen35_0_8b.bat (Linux: ./run.sh qwen35-0.8b). The model downloads once, then a chat page opens in your browser. Stock llama.cpp, Ollama and LM Studio cannot load this file, and neither can Pingala-8 runtimes before v2.5: their loader expects 7 coded linear tensors per layer, and the small Qwen3.5 models have 6 in each DeltaNet layer. The v2.5 runtime is llama.cpp 7ac59a6 + the released CUDA patch + a one-line loader change (patches/q35_loader_count_delta.patch: the coded-tensor count must equal the file's vr.n_linear).

File	Quant	Size	WikiText-2 PPL
`Qwen3.5-0.8B-Pingala8-2bit.gguf`	Pingala-8, 2-bit	0.35 GB	25.97
for comparison: IQ2_XXS (llama.cpp, same imatrix)	2-bit	0.35 GB	85.09
for comparison: IQ2_KT (ik_llama.cpp, default recipe, smaller file)	2-bit	0.31 GB	35.39
for comparison: Q8_0 (llama.cpp)	8-bit	0.80 GB	15.67

This is the text model only: the vision tower and the multi-token-prediction (MTP) block of Qwen3.5-0.8B are not included.

Author: Harshit Goyal, Genrius AI (harshit@genrius.in). Base model: Qwen/Qwen3.5-0.8B (Apache-2.0). Method paper: Small-State Coset Trellis Coding at IQ2_XXS Size: An Equal-Bytes Study in llama.cpp (preprint, v2.4), DOI 10.5281/zenodo.23481757. The Qwen3.5 numbers on this page are new measurements and are not in the paper. Larger sibling: Qwen3.5-4B Pingala-8. Code and runtime patch: https://github.com/Genrius-AI/pingala8-llm

What this is

A 2-bit-class quantization of Qwen3.5-0.8B (hybrid model: 18 Gated DeltaNet layers and 6 gated full-attention layers) at no more bytes than llama.cpp's IQ2_XXS file of the same model. Every linear weight that the format can hold (150 tensors: attn_qkv, attn_gate, ssm_out in the DeltaNet layers; attn_q, attn_k, attn_v, attn_output in the attention layers; the three FFN matrices everywhere) is stored with Ungerboeck's 8-state set-partitioned trellis code over 4-weight symbols, behind a random Hadamard rotation and QuIP's LDLQ rounding, one rate class per tensor and incoherence-bound column scales (the B4 recipe of the Qwen3-4B release, unchanged). Layer Hessians come from 256 sequences of 2048 tokens of C4 text. ssm_alpha and ssm_beta have only 16 rows (the format needs a multiple of 32), so they are stored as Q4_0 (paid from this file's byte budget); the other tensors (token embedding and tied output, norms, conv1d, the DeltaNet gate parameters) are copied from the stock IQ2_XXS file.

This was a single build with the recipe and the release rule written down before any 2-bit number was measured.

Files

file	bytes	sha256
`Qwen3.5-0.8B-Pingala8-2bit.gguf`	346,833,696	`8c80d48787d0dcd1a8aafb4d47d3e0e00c73214b103b376bfa764cc4c771df27`

IQ2_XXS of the same model: 352,301,856 bytes.

Which file should I pick?

If you...	Use
have an NVIDIA RTX 30/40/50 GPU and want Qwen3.5-0.8B text generation in 0.35 GB	this file with the v2.5 one-click build or your own build of the runtime
use stock llama.cpp, Ollama, LM Studio, a one-click build older than v2.5, a Mac, an AMD GPU or only a CPU	a standard GGUF from the base model's Quantizations list; this file will not load there
need images or video input	a standard GGUF with its mmproj file; this file is text only
serve many requests at once or process long prompts	a standard quant: this file processes prompts at 0.886x the speed of IQ2_XXS

Prompt format

The Qwen3.5-0.8B chat template is stored in the GGUF (ChatML, <|im_start|> / <|im_end|>, with Qwen's thinking switch).

Results

WikiText-2 test, llama-perplexity -c 2049 --ppl-stride 4095 (windows of 4096 tokens; the Qwen3.5 tokenizer gives 72 windows on this text). Paired difference: this file minus the row, nats per token, 95% interval, and the windows on which this file is better.

format	file bytes	PPL	ratio to Q8_0	paired, Pingala-8 - row
Q8_0 (llama.cpp)	0.80 GB	15.6652	1.000	+0.506 +/- 0.014 (0/72)
IQ2_XXS (llama.cpp, imatrix)	352,301,856	85.0892	5.432	-1.187 +/- 0.024 (72/72)
IQ2_KT (ik_llama.cpp, default recipe, imatrix)	312,659,520	35.3860	2.259	-0.309 +/- 0.012 (72/72)
Pingala-8 (this file)	346,833,696	25.9736	1.658	0

The IQ2_KT file is 9.9% smaller than this one, so that row is not an equal-bytes comparison; an IQ2_KT recipe at this file's size was not built. All three 2-bit files use the same importance matrix (C4 text, context 512, 200 chunks).

Zero-shot accuracy, % (0-shot continuation scores from llama-perplexity; HellaSwag 1,000 items, MMLU all 14,042 test items):

format	HellaSwag	WinoGrande	ARC-Easy	ARC-Challenge	MMLU	average
Pingala-8 (this file)	40.0	55.8	48.77	25.42	29.43	39.88
IQ2_XXS	31.4	51.07	38.42	25.42	28.04	34.87

Standard errors: about +/-1.6 (HellaSwag), +/-1.38 (WinoGrande), +/-2.1 (ARC-Easy), +/-2.7 (ARC-Challenge), +/-0.39 (MMLU).

Speed, same-round ratio to IQ2_XXS on one RTX 5070 Ti Laptop GPU, batch 1 (llama-bench -ngl 99 -fa 1, median of 3 rounds of -r 5): token generation (tg128) 1.002x, prompt processing (pp512) 0.886x. Generation speed is a tie and noisy at this size: per round 1.002, 0.934 and 1.105 (this file 379.9, 422.8, 388.3 tokens/s; IQ2_XXS 379.2, 452.8, 351.6).

How to run

The easiest way is the v2.5 one-click build (run_qwen35_0_8b.bat on Windows, ./run.sh qwen35-0.8b on Linux). To build it yourself, use llama.cpp at commit 7ac59a6e3ad851cd41af00f678effab0598ba9a8 with the released CUDA patch plus the loader change from the code repository (patches/q35_loader_count_delta.patch):

git clone https://github.com/Genrius-AI/pingala8-llm
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout 7ac59a6e3ad851cd41af00f678effab0598ba9a8
git apply ../pingala8-llm/patches/llama_cuda_vs_7ac59a6.patch
git apply ../pingala8-llm/patches/q35_loader_count_delta.patch
cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120 -DCMAKE_BUILD_TYPE=Release -DLLAMA_OPENSSL=OFF
cmake --build build -j8 --target llama-cli llama-perplexity llama-bench

The loader change keeps every earlier Pingala-8 file working: on the rebuilt runtime the Qwen3-4B file gives the same perplexity ([73]16.6413) and the same speed (tg128 1.01x, pp512 1.01x of the released build). Then (all layers on the GPU):

hf download Genrius-AI/Qwen3.5-0.8B-Pingala8-2bit --local-dir qwen3.5-0.8b-pingala8
export GGML_F44=4 GGML_F44_BATCH=2 GGML_VR_NATIVE=1 GGML_VRC_V=5 GGML_VRC_NWP=8 GGML_VR_RL=1 GGML_VR_PRMG=1 GGML_VR_PPC=2
./build/bin/llama-cli -m qwen3.5-0.8b-pingala8/Qwen3.5-0.8B-Pingala8-2bit.gguf -ngl 99
./build/bin/llama-perplexity -m qwen3.5-0.8b-pingala8/Qwen3.5-0.8B-Pingala8-2bit.gguf -f wikitext2_test_hfjoin.raw -c 2049 --ppl-stride 4095 -ngl 99

The perplexity command is the measured protocol (expected final value [72]25.9736); the code repository rebuilds the text file. The llama-cli line is a usage example and was not part of the measured runs. Set CMAKE_CUDA_ARCHITECTURES for your GPU.

Limitations

Needs the v2.5 runtime (the v2.5 one-click build, or your own build with patches/q35_loader_count_delta.patch); one-click builds up to v2.4.1 cannot load this file.
CUDA only, measured on one laptop GPU (RTX 5070 Ti Laptop, WSL2); no Metal, Vulkan or CPU path for this file.
Text only: no vision tower, no MTP block (so no MTP speculative decoding).
Prompt processing is slower than IQ2_XXS (0.886x). Batched decoding was not measured on this model; on Qwen3-4B it was much slower than IQ2_XXS (0.18-0.34x for 2 to 8 sequences).
At 2 bits the model is clearly worse than the 8-bit model (perplexity 1.658x Q8_0).
Perplexity was measured at context 4096 only; long-context behaviour of the quantized DeltaNet layers was not tested.
With the same recipe the method lost to IQ2_KT on Qwen3-14B (paper, Section 4.3); results do not transfer to every model.
Refusal, bias and hallucination behaviour after quantization were not evaluated. Evaluate the model for your own use.

Citation

@misc{goyal2026pingala8,
  title        = {Small-State Coset Trellis Coding at {IQ2\_XXS} Size: An Equal-Bytes Study in {llama.cpp}},
  author       = {Goyal, Harshit},
  year         = {2026},
  month        = oct,
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.23481757},
  url          = {https://doi.org/10.5281/zenodo.23481757},
  note         = {Preprint, version 2.4. Code: https://github.com/Genrius-AI/pingala8-llm}
}

License and notice

Apache-2.0, inherited from Qwen3.5-0.8B (Qwen team, Alibaba Cloud); see LICENSE and NOTICE. The weights are a quantized derivative of Qwen3.5-0.8B made by Genrius AI. "Pingala-8" and "Genrius AI" are names of Genrius AI; the license grants no trademark rights.