Instructions to use kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF:Q4_K_M
Use Docker
docker model run hf.co/kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF:Q4_K_M
- Ollama
How to use kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF with Ollama:
ollama run hf.co/kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF with Docker Model Runner:
docker model run hf.co/kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF:Q4_K_M
- Lemonade
How to use kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Qwen2.5-Coder-7B-Instruct-Edge-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Qwen2.5-Coder-7B-Instruct โ Edge-Calibrated imatrix GGUF Suite
Importance-matrix (llama-imatrix) GGUF quantizations of Qwen/Qwen2.5-Coder-7B-Instruct, calibrated specifically for agentic tool-calling, code review, and declarative infrastructure reasoning and benchmarked across 8 GiB NVIDIA Jetson Orin Nano (CUDA 12.6 sm_87) and x86_64 AVX2.
- Interactive Pareto & 8 GiB Edge KV-Cache Explorer:
spaces/kleinbem/edge-llm-pareto - Full Parquet Telemetry Dataset:
datasets/kleinbem/edge-llm-benchmarks - Reproducible Nix Flake & Pipeline Source:
github.com/kleinbem/hf-bench
Quality (FP16 Logit KL-Divergence) & Hardware Telemetry
Unlike perplexity-only tables, every quantization below was scored directly against the exact FP16 base model probability distribution (llama-perplexity --kl-divergence), alongside real hardware throughput and INA3221/RAPL power telemetry:
| Quant | imatrix |
BPW | Size (GiB) | Mean KLD (nats) | Top-1 Match | PPL | Orin CUDA tg128 |
Orin Power / Energy | x86 AVX2 tg128 |
|---|---|---|---|---|---|---|---|---|---|
Q8_0 |
Yes | 8.50 | 7.54 | 0.00026 | 99.61% | 8.276 | OOM (>8 GiB) | โ | 2.2 tok/s |
Q6_K |
Yes | 6.57 | 5.83 | 0.00121 | 98.69% | 8.278 | OOM (>8 GiB) | โ | 2.6 tok/s |
Q5_K_M |
Yes | 5.72 | 5.07 | 0.00286 | 97.97% | 8.268 | OOM (>8 GiB) | โ | 2.0 tok/s |
Q4_K_M |
Yes | 4.92 | 4.36 | 0.01005 | 96.08% | 8.356 | 13.0 tok/s | 20.4 W (1.570 J/tok) | 2.8 tok/s |
Q4_K_M-std |
No (control) | 4.92 | 4.36 | 0.01321 | 95.49% | 8.281 | 13.0 tok/s | 20.3 W (1.565 J/tok) | 3.0 tok/s |
IQ4_XS |
Yes | 4.43 | 3.93 | 0.01287 | 95.36% | 8.307 | 16.3 tok/s | 20.5 W (1.256 J/tok) | 2.4 tok/s |
IQ3_M |
Yes | 3.75 | 3.33 | 0.04187 | 90.52% | 8.374 | 11.9 tok/s | 20.9 W (1.756 J/tok) | 1.1 tok/s |
imatrixAblation (Q4_K_MvsQ4_K_M-std): Calibrating second-moment channel activations on agentic tool-calling + systems code changed Mean KL-Divergence from0.01321to0.01005(+23.9% KLD reduction at identical byte size).
GQA KV-Cache Sizing Formula (Qwen2.5-Coder-7B-Instruct)
Qwen2.5-Coder-7B-Instruct uses Grouped-Query Attention with 28 layers, 4 KV heads, and head_dim = 128 (56.0 KiB/token in FP16 KV cache, or 448 MiB at 8,192 context). Use the interactive Edge Unified-Memory Sizer to verify exact headroom for your context length and KV quantization (FP16 / Q8_0 / Q4_0).
Reproducing with Nix
nix run github:kleinbem/hf-bench#hf-bench -- run-local --model qwen2.5-coder-7b
nix run github:kleinbem/hf-bench#hf-bench -- bench-orin --model qwen2.5-coder-7b --host orin-nano
- Downloads last month
- 600