Instructions to use guideboardlabs/Nanbeige4.2-3B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use guideboardlabs/Nanbeige4.2-3B-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="guideboardlabs/Nanbeige4.2-3B-GGUF",
	filename="nanbeige-3b-q4_k_m.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use guideboardlabs/Nanbeige4.2-3B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf guideboardlabs/Nanbeige4.2-3B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf guideboardlabs/Nanbeige4.2-3B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf guideboardlabs/Nanbeige4.2-3B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf guideboardlabs/Nanbeige4.2-3B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf guideboardlabs/Nanbeige4.2-3B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf guideboardlabs/Nanbeige4.2-3B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf guideboardlabs/Nanbeige4.2-3B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf guideboardlabs/Nanbeige4.2-3B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/guideboardlabs/Nanbeige4.2-3B-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use guideboardlabs/Nanbeige4.2-3B-GGUF with Ollama:
```
ollama run hf.co/guideboardlabs/Nanbeige4.2-3B-GGUF:Q4_K_M
```

Unsloth Studio

How to use guideboardlabs/Nanbeige4.2-3B-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for guideboardlabs/Nanbeige4.2-3B-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for guideboardlabs/Nanbeige4.2-3B-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for guideboardlabs/Nanbeige4.2-3B-GGUF to start chatting

How to use guideboardlabs/Nanbeige4.2-3B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf guideboardlabs/Nanbeige4.2-3B-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "guideboardlabs/Nanbeige4.2-3B-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use guideboardlabs/Nanbeige4.2-3B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf guideboardlabs/Nanbeige4.2-3B-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default guideboardlabs/Nanbeige4.2-3B-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use guideboardlabs/Nanbeige4.2-3B-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf guideboardlabs/Nanbeige4.2-3B-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "guideboardlabs/Nanbeige4.2-3B-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use guideboardlabs/Nanbeige4.2-3B-GGUF with Docker Model Runner:
```
docker model run hf.co/guideboardlabs/Nanbeige4.2-3B-GGUF:Q4_K_M
```

Lemonade

How to use guideboardlabs/Nanbeige4.2-3B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull guideboardlabs/Nanbeige4.2-3B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Nanbeige4.2-3B-GGUF-Q4_K_M

List all available models

lemonade list

Nanbeige4.2-3B — Looped Transformer GGUF

2.4GB Q4_K_M GGUF for AMD ROCm / llama.cpp

A 3B parameter model using a looped transformer architecture (22 physical layers × 2 loops = 44 effective layers). Built and benchmarked on an AMD RX 5700 XT (8GB VRAM) via the ROCmFPX fork of llama.cpp.

Download

File	Size	Quant	Link
`nanbeige-3b-q4_k_m.gguf`	2.4 GB	Q4_K_M	Download

Hardware Requirements

Minimum: 8GB VRAM GPU (fits entirely at Q4_K_M with -ngl 99)
Recommended: Any AMD GPU with ROCm support, or any Vulkan-capable GPU
RAM: 4GB+ system RAM
Storage: 2.4GB for the GGUF file

Build (ROCmFPX fork)

This GGUF was built and tested with a custom fork of llama.cpp that adds Nanbeige architecture support. The fork is at GuideboardLabs/ROCmFPX.

git clone https://github.com/GuideboardLabs/ROCmFPX.git
cd ROCmFPX
mkdir build && cd build

# For AMD ROCm (tested on ROCm 5.7 with RX 5700 XT)
cmake .. -DCMAKE_BUILD_TYPE=Release -DLLAMA_HIPBLAS=ON

# For Vulkan (any GPU)
cmake .. -DCMAKE_BUILD_TYPE=Release -DLLAMA_VULKAN=ON

# For CPU-only
cmake .. -DCMAKE_BUILD_TYPE=Release

make -j$(nproc)

Server Command

./bin/llama-server \
  --model /path/to/nanbeige-3b-q4_k_m.gguf \
  --port 8100 \
  --host 0.0.0.0 \
  --gpu-layers 99 \
  --mlock \
  --threads 6 \
  --threads-batch 6 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --temp 0.7 \
  --min-p 0.05 \
  --parallel 1

Key flags explained

Flag	Why
`--gpu-layers 99`	Offload all layers to GPU. Model is 2.4GB, fits entirely in 8GB VRAM.
`--mlock`	Lock memory to prevent swapping. Critical for consistent inference speed.
`--cache-type-k q8_0`	KV cache in Q8_0 to save VRAM. The model has 262K context — this matters.
`--cache-type-v q8_0`	Same for value cache.
`--threads 6`	Match your CPU core count (6 physical cores on this test system).
`--temp 0.7`	Standard sampling temperature for agent/code tasks.
`--min-p 0.05`	Min-p sampling to filter low-probability tokens.

Performance (AMD RX 5700 XT, 8GB VRAM)

Metric	Value
Prompt processing	169.25 tok/s
Text generation	44.88 tok/s
VRAM usage	2.39 GB (fits entirely)
Context window	262,144 tokens

Agon-Bench Results

Full benchmark: 3 events × 3 runs each on AMD RX 5700 XT via llama.cpp + Vulkan.

Event	Run 1	Run 2	Run 3	Avg	Pct
Agent	23/24	22/24	24/24	23.0	95.8%
Code	18/25	16/25	16/25	16.7	66.7%
Reasoning	15/17	15/17	16/17	15.3	90.2%
Composite					84.2%

Leaderboard position

Rank	Model	Size	Composite	tok/s
1	Gemma4-26B-A4B	16.9 GB	84.6%	13.5
2	Nanbeige4.2-3B	2.4 GB	84.2%	44.9
3	Ornith-1.0-9B	5.3 GB	82.7%	48.0
4	Bonsai-27B-Q1_0	3.8 GB	82.2%	12.5

Per-task breakdown

Agent (95.8%) — Near-perfect across the board. Tool Calling, Error Handling, Planning, Parsing, Decision-Making, and Multi-Step Research all at 100%. Only Orchestration (78%) and Memory-Augmented Agent (89%) show minor weakness.

Code (66.7%) — Inconsistent. Log Parser and LRU Cache at 100%. Regex Engine (33%) and String Cleaner (53%) are the weak points — the 3B training corpus limits breadth of coding knowledge.

Reasoning (90.2%) — Strong. Math Word Problems, Constraint Satisfaction, Counterfactual Reasoning, and Multi-Hop Synthesis all at 100%. Logical Deduction (67%) and Analytical Explanation (67%) are the only gaps.

Architecture Notes

Nanbeige uses a looped transformer design:

22 physical transformer layers
Each layer processes the hidden state twice (2 loops)
Effective depth: 44 layers
Total parameters: ~3B (non-embedding)
Context window: 262,144 tokens
Rope theta: 70,000,000 (supports the long context)

This gives a 3B model the reasoning depth of a 6-7B model, which is why it competes with models 5-7x its size on agent and reasoning tasks. The tradeoff is that code tasks benefit more from training corpus breadth than architectural depth.

Conversion Notes

The GGUF was converted from the original safetensors using the ROCmFPX fork's converter:

python3 convert_hf_to_gguf.py /path/to/Nanbeige4.2-3B/ --outfile nanbeige-3b-f16.gguf --model nanbeige
./bin/llama-quantize nanbeige-3b-f16.gguf nanbeige-3b-q4_k_m.gguf Q4_K_M

The converter handles the looped architecture metadata automatically:

num_loops=2
skip_loop_final_norm=false
Rope theta scaled for 262K context

License

Apache-2.0 (same as the original Nanbeige4.2-3B model)

Model tree for guideboardlabs/Nanbeige4.2-3B-GGUF

Base model

Nanbeige/Nanbeige4.2-3B-Base

Finetuned

Nanbeige/Nanbeige4.2-3B

Quantized

(22)

this model

guideboardlabs
/

Nanbeige4.2-3B-GGUF

Nanbeige4.2-3B — Looped Transformer GGUF

Download

Hardware Requirements

Build (ROCmFPX fork)

Server Command

Key flags explained

Performance (AMD RX 5700 XT, 8GB VRAM)

Agon-Bench Results

Leaderboard position

Per-task breakdown

Architecture Notes

Conversion Notes

License

Links

Model tree for guideboardlabs/Nanbeige4.2-3B-GGUF