Instructions to use RenormalizationTech/Qwen3-4B-tuneLLM-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use RenormalizationTech/Qwen3-4B-tuneLLM-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf RenormalizationTech/Qwen3-4B-tuneLLM-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf RenormalizationTech/Qwen3-4B-tuneLLM-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf RenormalizationTech/Qwen3-4B-tuneLLM-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf RenormalizationTech/Qwen3-4B-tuneLLM-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf RenormalizationTech/Qwen3-4B-tuneLLM-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf RenormalizationTech/Qwen3-4B-tuneLLM-GGUF:Q4_K_M
Use Docker
docker model run hf.co/RenormalizationTech/Qwen3-4B-tuneLLM-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use RenormalizationTech/Qwen3-4B-tuneLLM-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "RenormalizationTech/Qwen3-4B-tuneLLM-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "RenormalizationTech/Qwen3-4B-tuneLLM-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/RenormalizationTech/Qwen3-4B-tuneLLM-GGUF:Q4_K_M
- Ollama
How to use RenormalizationTech/Qwen3-4B-tuneLLM-GGUF with Ollama:
ollama run hf.co/RenormalizationTech/Qwen3-4B-tuneLLM-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use RenormalizationTech/Qwen3-4B-tuneLLM-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf RenormalizationTech/Qwen3-4B-tuneLLM-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "RenormalizationTech/Qwen3-4B-tuneLLM-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use RenormalizationTech/Qwen3-4B-tuneLLM-GGUF with Docker Model Runner:
docker model run hf.co/RenormalizationTech/Qwen3-4B-tuneLLM-GGUF:Q4_K_M
- Lemonade
How to use RenormalizationTech/Qwen3-4B-tuneLLM-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull RenormalizationTech/Qwen3-4B-tuneLLM-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Qwen3-4B-tuneLLM-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use RenormalizationTech/Qwen3-4B-tuneLLM-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf RenormalizationTech/Qwen3-4B-tuneLLM-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default RenormalizationTech/Qwen3-4B-tuneLLM-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use RenormalizationTech/Qwen3-4B-tuneLLM-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf RenormalizationTech/Qwen3-4B-tuneLLM-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "RenormalizationTech/Qwen3-4B-tuneLLM-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Qwen3-4B tuneLLM GGUF
Portfolio summary
Quantization experiment: converted the pinned Qwen3-4B checkpoint to BF16 GGUF, Q8_0 and Q4_K_M, then measured file size, generation throughput and fixed capability subsets. Weight sizes were 7.50 / 3.99 / 2.33 GiB; the short A6000 generation benchmark measured 79.1 / 130.5 / 186.0 tokens/s. These are microbenchmark results, not production latency. Capability counts and reproduction instructions below define the evaluation scope. No weights changed in this documentation update.
This artifact is part of tuneLLM, a public research portfolio covering quantization, LoRA/QLoRA, refusal steering and trait steering. Results and saved examples ยท Reproduction and environment setup ยท Methods and references.
Related releases: GGUF, NF4 base, physics QLoRA, trait controls, refusal experiment. Each release has its own verified environment; activation steering and adapter training use different mechanisms.
Separate 27B work: the text-only NF4 load/save/reload smoke is complete on Qwen3.8-27B, using two simple prompts. It is a functional check, not a quality benchmark. A one-step cache check exceeded its numerical tolerance; that result is retained. The NF4 checkpoint remains local; 27B sarcasm steering and QLoRA training are planned. Recorded scope.
An unofficial learning release of Qwen3-4B converted to BF16 GGUF and quantized to Q8_0 and Q4_K_M. These files exercise the conversion and evaluation workflow; they do not introduce new training or claim improved model capability.
Upstream model: Qwen/Qwen3-4B, revision
1cfa9a7208912126459214e8b04321603b3df60c.
Original model and copyright belong to their upstream authors. The upstream
Apache-2.0 license is included in LICENSE. Changes by tuneLLM consist of
GGUF conversion and numerical quantization using llama.cpp commit
8ca5a8008b57aa39f2601546c0d07912045080a5. No importance matrix was used.
The implementation and reproduction notes are included in
reproduction-dab1dae.zip, a source archive of local workflow commit dab1dae.
The archive preserves that implementation version; the public
project repository also contains later work.
The manifest records the full implementation commit, source-archive hash, model
file hashes and evaluation settings. Training datasets and intermediate model
checkpoints are not included.
Files and initial measurements
| File | Size (GiB) | Generation tokens/s | Small-text perplexity |
|---|---|---|---|
Qwen3-4B-BF16.gguf |
7.498 | 79.1 | 2.6299 |
Qwen3-4B-Q8_0.gguf |
3.986 | 130.5 | 2.6282 |
Qwen3-4B-Q4_K_M.gguf |
2.326 | 186.0 | 2.6608 |
Throughput is from a short llama-bench test on one NVIDIA RTX A6000: three repetitions, 64 generated tokens, full GPU offload, flash attention enabled and four CPU threads. Prompt processing was measured separately with 128 prompt tokens. These figures describe this microbenchmark, not production throughput, chat latency, or a comparison with Transformers.
Perplexity used llama-perplexity on the same text for every precision: the first
32 prepared GSM8K test question/solution pairs, context 512, four chunks,
batch/microbatch 512. The tool-reported uncertainties were about 0.175โ0.177.
This small sample does not establish general capability preservation. GSM8K is
an MIT-licensed dataset by Cobbe et al. (2021),
Training Verifiers to Solve Math Word Problems.
Dataset revision: 740312add88f781978c0658806c59bc2815b9866.
The observed perplexity differences are smaller than the tool's reported uncertainty on the individual estimates. These estimates do not establish a capability ranking or equivalence between formats.
All three formats passed exact input-token parity against Transformers on an
eight-item project-authored smoke fixture. CPU scores were 7/8, 7/8 and 8/8
respectively. A one-item difference is not evidence that Q4 is more capable.
BF16 and Q8_0 produced token-identical responses to Transformers on this fixture.
Complete compact measurements are included as evaluation.json.
Completed fixed-subset benchmark screen
The later screen evaluated the same published weight bytes on 384 items per
condition. This is separate from the initial eight-item smoke check and tiny
perplexity/microbenchmark above; the historical evaluation.json and provenance
remain unchanged. The additive benchmark_summary.json records the new screen,
paired counts, evaluated source commits and scorer hashes.
| GGUF condition | General MMLU /128 | Physics MMLU /96 | GSM8K /128 | IFEval /32 |
|---|---|---|---|---|
| BF16 | 92 | 62 | 119 | 25 |
| Q8_0 | 89 | 61 | 121 | 25 |
| Q4_K_M | 89 | 61 | 116 | 24 |
General/physics are custom zero-shot MMLU four-choice likelihood subsets, not
full standard leaderboard scores. GSM8K uses last-number fallback correctness;
strict final-line correctness and truncation are reported separately. IFEval
uses its pinned official prompt-level strict scorer. Dataset revisions:
MMLU c30699e8356da336a370243923dbaf21066bb9fe,
GSM8K 740312add88f781978c0658806c59bc2815b9866, and
IFEval 966cd89545d6b6acfd7638bc708b98261ca58e84; official scorer source
f8d6d14a1fc6eaef606d7d1e1fa4e96ee09afb81.
The screen uses llama.cpp 8ca5a8008b57aa39f2601546c0d07912045080a5,
greedy non-thinking decoding, seed 42, batch one, and output budgets of 512 tokens
for GSM8K and 1024 for IFEval. Input-token parity is checked against the pinned
upstream tokenizer. The implementation commits and source SHA256s are in the
new report; the original reproduction archive still describes the initial cycle.
| Published weight | SHA256 |
|---|---|
Qwen3-4B-BF16.gguf |
5dbbfbb5ccea81a811ef8346248bbf8f808c94477feb84415699d632798925b9 |
Qwen3-4B-Q8_0.gguf |
4b2b96c36712c42d902fb23cb361ed8ebf838a4342f77cedb87afe99e291775f |
Qwen3-4B-Q4_K_M.gguf |
f7a3480949df3ae406a9661aae19e34b6c6ae344e7b6b8edd38b6185b786f339 |
These are small fixed subsets with unknown base-pretraining exposure. The paired tests are exploratory and unadjusted; equal totals or nonsignificant differences do not establish equivalence, preserved general capability, or a universal quantization ranking. IFEval language-detection random-state sensitivity limits close comparisons. No full benchmark or production-performance claim is made.
Use
Use a llama.cpp build that supports Qwen3 and these GGUF formats. For example:
llama-cli -m Qwen3-4B-Q4_K_M.gguf --single-turn --reasoning off \
--chat-template-kwargs '{"enable_thinking":false}' \
-p "Explain why the sky is blue."
The pinned build above supports these flags. Other releases may expose different flags; consult their help. The saved chat template comes from the upstream tokenizer. Non-thinking mode must be explicit for comparisons with the included evaluations.
The experiments used greedy decoding for a reproducible pipeline comparison. The upstream model card recommends sampling and warns that greedy generation can repeat. Consult the upstream card for normal conversational generation settings, supported languages and intended use.
Limitations
This is a small educational quantization experiment. It does not establish accuracy on comprehensive benchmarks, robustness across prompts, safety performance, or suitability for a particular application. No domain adaptation or activation steering was applied. Quantization and runtime changes can alter individual answers even when input token IDs match.
The full source model's capabilities and limitations also apply. Report this release as an unofficial derivative of the pinned Qwen model rather than an official Qwen checkpoint.
- Downloads last month
- 550
4-bit
8-bit
16-bit