Text Generation
Transformers
GGUF
English
code
agentic
tool-use
agent
minicpm
full-fine-tune
on-cpu
text-generation-inference
unsloth
llama
conversational
Instructions to use ITLL/MiniCPM5-1B-Agent-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ITLL/MiniCPM5-1B-Agent-GGUF with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="ITLL/MiniCPM5-1B-Agent-GGUF")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("ITLL/MiniCPM5-1B-Agent-GGUF", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use ITLL/MiniCPM5-1B-Agent-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf ITLL/MiniCPM5-1B-Agent-GGUF:Q8_0 # Run inference directly in the terminal: llama cli -hf ITLL/MiniCPM5-1B-Agent-GGUF:Q8_0
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf ITLL/MiniCPM5-1B-Agent-GGUF:Q8_0 # Run inference directly in the terminal: llama cli -hf ITLL/MiniCPM5-1B-Agent-GGUF:Q8_0
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf ITLL/MiniCPM5-1B-Agent-GGUF:Q8_0 # Run inference directly in the terminal: ./build/bin/llama cli -hf ITLL/MiniCPM5-1B-Agent-GGUF:Q8_0
Use Docker
docker model run hf.co/ITLL/MiniCPM5-1B-Agent-GGUF:Q8_0
- Llama new
- LM Studio
- Jan
- vLLM
How to use ITLL/MiniCPM5-1B-Agent-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "ITLL/MiniCPM5-1B-Agent-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ITLL/MiniCPM5-1B-Agent-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/ITLL/MiniCPM5-1B-Agent-GGUF:Q8_0
- SGLang
How to use ITLL/MiniCPM5-1B-Agent-GGUF with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "ITLL/MiniCPM5-1B-Agent-GGUF" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ITLL/MiniCPM5-1B-Agent-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "ITLL/MiniCPM5-1B-Agent-GGUF" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ITLL/MiniCPM5-1B-Agent-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Ollama
How to use ITLL/MiniCPM5-1B-Agent-GGUF with Ollama:
ollama run hf.co/ITLL/MiniCPM5-1B-Agent-GGUF:Q8_0
- Unsloth Desktop
- Pi
How to use ITLL/MiniCPM5-1B-Agent-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf ITLL/MiniCPM5-1B-Agent-GGUF:Q8_0
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "ITLL/MiniCPM5-1B-Agent-GGUF:Q8_0" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use ITLL/MiniCPM5-1B-Agent-GGUF with Docker Model Runner:
docker model run hf.co/ITLL/MiniCPM5-1B-Agent-GGUF:Q8_0
- Lemonade
How to use ITLL/MiniCPM5-1B-Agent-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull ITLL/MiniCPM5-1B-Agent-GGUF:Q8_0
Run and chat with the model
lemonade run user.MiniCPM5-1B-Agent-GGUF-Q8_0
List all available models
lemonade list
- Hermes Agent
How to use ITLL/MiniCPM5-1B-Agent-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf ITLL/MiniCPM5-1B-Agent-GGUF:Q8_0
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default ITLL/MiniCPM5-1B-Agent-GGUF:Q8_0
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use ITLL/MiniCPM5-1B-Agent-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf ITLL/MiniCPM5-1B-Agent-GGUF:Q8_0
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "ITLL/MiniCPM5-1B-Agent-GGUF:Q8_0" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
File size: 6,021 Bytes
0467408 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 | """On-policy preference capture via transformers on GPU (llama-server is CPU-only -> too slow).
Samples K completions per prompt from the bf16 sft_v2_ablit (the EXACT model DPO trains -> truly
on-policy) and harvests v2's REAL failures to emit a valid tool call:
VALID -> parses to a correct <function name><param> XML call
BAD -> WRONG (markdown/Claude/JSON/broken-XML attempt) OR NOCALL (punted, no call) = the eval `calls=0` failure
Pairs: chosen = a VALID sample (the model's own correct format) else the GOLD tool-call from the SFT row
rejected= a BAD sample (the model's real mistake)
-> chosen/rejected target exactly "emit a valid tool call vs not", the failure that costs eval points.
Outputs: dpo_format_onpolicy.jsonl, kto_format_onpolicy.jsonl + the real BAD rate (decision signal).
python data/build_prefs_onpolicy_gpu.py [--prompts N] [--k 4] [--temp 0.8] [--maxprompt 13312] [--stride 3]
"""
import os, sys, json, argparse
HERE = os.path.dirname(os.path.abspath(__file__)); PROJ = os.path.dirname(HERE)
sys.path.insert(0, HERE); sys.path.insert(0, os.path.join(PROJ, "backend"))
import schema
from build_prefs_onpolicy import classify
import torch
# Blackwell sm_120 SDPA: force O(L) mem-efficient (math = O(L^2) -> OOM at long ctx); repeat_kv over GQA.
torch.backends.cuda.enable_flash_sdp(False)
torch.backends.cuda.enable_mem_efficient_sdp(True)
torch.backends.cuda.enable_cudnn_sdp(False)
torch.backends.cuda.enable_math_sdp(False)
torch.set_float32_matmul_precision("high")
import transformers.integrations.sdpa_attention as _sdpa_attn
_sdpa_attn.use_gqa_in_sdpa = lambda *a, **k: False
from transformers import AutoTokenizer, AutoModelForCausalLM
MODEL_DIR = os.path.join(PROJ, "train", "outputs", "sft_v2_ablit")
SRC = os.path.join(HERE, "built", "dataset_golden.jsonl")
TOK = AutoTokenizer.from_pretrained(os.path.join(PROJ, "model", "final"), trust_remote_code=True)
ASSIST = "<|im_start|>assistant\n"
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--prompts", type=int, default=800)
ap.add_argument("--k", type=int, default=4)
ap.add_argument("--temp", type=float, default=0.8)
ap.add_argument("--maxprompt", type=int, default=13312)
ap.add_argument("--npred", type=int, default=320)
ap.add_argument("--stride", type=int, default=3)
ap.add_argument("--model", default=MODEL_DIR) # point at the DPO base (e.g. sft_v3/checkpoint-5900)
ap.add_argument("--out", default=os.path.join(HERE, "built", "dpo_format_onpolicy.jsonl"))
ap.add_argument("--src", default=SRC) # prompt source (dataset_golden was cleaned -> pass train_v4.jsonl)
a = ap.parse_args()
if TOK.pad_token_id is None:
TOK.pad_token = TOK.eos_token
print(f"[gpu] loading {a.model} bf16 (mem-efficient SDPA) ...", flush=True)
model = AutoModelForCausalLM.from_pretrained(a.model, dtype=torch.bfloat16,
trust_remote_code=True, attn_implementation="sdpa").to("cuda").eval()
pool = []; seen = toolong = 0
for line in open(a.src, encoding="utf-8"):
row = json.loads(line); msgs, tools = row.get("messages", []), row.get("tools")
ti = next((i for i, m in enumerate(msgs) if m["role"] == "assistant" and m.get("tool_calls")), None)
if ti is None:
continue
seen += 1
if seen % a.stride:
continue
try:
prompt = schema.render(msgs[:ti], tools, TOK, enable_thinking=True, add_generation_prompt=True)
full = schema.render(msgs[:ti + 1], tools, TOK, enable_thinking=True, add_generation_prompt=False)
except Exception:
continue
sp = full.rfind(ASSIST)
gold = full[sp + len(ASSIST):] if sp >= 0 else ""
if "<function name=" not in gold:
continue
if len(TOK(prompt, add_special_tokens=False)["input_ids"]) > a.maxprompt:
toolong += 1
continue
pool.append((prompt, gold))
if len(pool) >= a.prompts:
break
print(f"[gpu] {len(pool)} prompts (skipped {toolong} >{a.maxprompt} tok); k={a.k} temp={a.temp}", flush=True)
dpo_f = open(a.out, "w", encoding="utf-8")
kto_f = open(a.out.replace("dpo_", "kto_"), "w", encoding="utf-8")
nv = nb = ns = ndpo = 0
for pi, (prompt, gold) in enumerate(pool):
ids = TOK(prompt, return_tensors="pt", add_special_tokens=False).to("cuda")
try:
with torch.no_grad():
out = model.generate(**ids, do_sample=True, temperature=a.temp, top_p=0.95,
num_return_sequences=a.k, max_new_tokens=a.npred, pad_token_id=TOK.pad_token_id)
except torch.cuda.OutOfMemoryError:
torch.cuda.empty_cache(); continue
gens = [g.split("<|im_end|>")[0] for g in TOK.batch_decode(out[:, ids["input_ids"].shape[1]:], skip_special_tokens=False)]
valids, bads = [], []
for g in gens:
ok = classify(g) == "VALID"; ns += 1
(valids if ok else bads).append(g)
kto_f.write(json.dumps({"prompt": prompt, "completion": g, "label": ok}, ensure_ascii=False) + "\n")
nv += ok; nb += (not ok)
kto_f.write(json.dumps({"prompt": prompt, "completion": gold, "label": True}, ensure_ascii=False) + "\n") # gold = known-good
if bads:
chosen = valids[0] if valids else gold
dpo_f.write(json.dumps({"prompt": prompt, "chosen": chosen, "rejected": bads[0]}, ensure_ascii=False) + "\n"); ndpo += 1
if (pi + 1) % 50 == 0:
print(f" {pi+1}/{len(pool)} valid={nv} bad={nb} dpo={ndpo}", flush=True)
dpo_f.close(); kto_f.close()
print(f"\n=== ON-POLICY (GPU) REPORT ===")
print(f"samples={ns} VALID={nv} ({100*nv/max(1,ns):.1f}%) BAD={nb} ({100*nb/max(1,ns):.1f}%) DPO pairs={ndpo}")
print(f"--> real failure rate {100*nb/max(1,ns):.1f}% (chosen=valid-call / rejected=model's real miss)")
if __name__ == "__main__":
main()
|