Instructions to use peasantsmith/Inkling-2-Small-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use peasantsmith/Inkling-2-Small-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf peasantsmith/Inkling-2-Small-GGUF:IQ2_XXS # Run inference directly in the terminal: llama cli -hf peasantsmith/Inkling-2-Small-GGUF:IQ2_XXS
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf peasantsmith/Inkling-2-Small-GGUF:IQ2_XXS # Run inference directly in the terminal: llama cli -hf peasantsmith/Inkling-2-Small-GGUF:IQ2_XXS
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf peasantsmith/Inkling-2-Small-GGUF:IQ2_XXS # Run inference directly in the terminal: ./build/bin/llama cli -hf peasantsmith/Inkling-2-Small-GGUF:IQ2_XXS
Use Docker
docker model run hf.co/peasantsmith/Inkling-2-Small-GGUF:IQ2_XXS
- Llama new
- LM Studio
- Jan
- vLLM
How to use peasantsmith/Inkling-2-Small-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "peasantsmith/Inkling-2-Small-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "peasantsmith/Inkling-2-Small-GGUF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/peasantsmith/Inkling-2-Small-GGUF:IQ2_XXS
- Ollama
How to use peasantsmith/Inkling-2-Small-GGUF with Ollama:
ollama run hf.co/peasantsmith/Inkling-2-Small-GGUF:IQ2_XXS
- Unsloth Desktop
- Pi
How to use peasantsmith/Inkling-2-Small-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf peasantsmith/Inkling-2-Small-GGUF:IQ2_XXS
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "peasantsmith/Inkling-2-Small-GGUF:IQ2_XXS" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use peasantsmith/Inkling-2-Small-GGUF with Docker Model Runner:
docker model run hf.co/peasantsmith/Inkling-2-Small-GGUF:IQ2_XXS
- Lemonade
How to use peasantsmith/Inkling-2-Small-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull peasantsmith/Inkling-2-Small-GGUF:IQ2_XXS
Run and chat with the model
lemonade run user.Inkling-2-Small-GGUF-IQ2_XXS
List all available models
lemonade list
- Hermes Agent
How to use peasantsmith/Inkling-2-Small-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf peasantsmith/Inkling-2-Small-GGUF:IQ2_XXS
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default peasantsmith/Inkling-2-Small-GGUF:IQ2_XXS
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use peasantsmith/Inkling-2-Small-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf peasantsmith/Inkling-2-Small-GGUF:IQ2_XXS
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "peasantsmith/Inkling-2-Small-GGUF:IQ2_XXS" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Inkling-2-Small GGUF (~150 GB)
A ~150 GB GGUF of thinkingmachines/Inkling-2-Small for llama.cpp, made to keep as much of the original model's quality as possible at this size. All credit for the model goes to Thinking Machines; it is released under the Apache-2.0 license, and so is this quant.
Files
| File | Size | What |
|---|---|---|
Inkling-2-Small-IQ2_XXS-mix-00001-of-00004.gguf ... -00004-of-00004.gguf |
150.0 GB in total | the language model (pass the first shard; keep all four in one folder) |
mmproj-Inkling-2-Small-F16.gguf |
0.66 GB | vision projector (--mmproj) |
Inkling-2-Small-MTP-BF16.gguf |
3.4 GB | the model's multi-token-prediction block, weights unchanged (BF16) |
The MTP file holds Inkling's draft block (its DSpark speculator), weights unchanged. These files run with llama.cpp PR 32187 (text and vision), which does not load the draft block. llama.cpp's DSpark support for Inkling 2 (PR 32218, on top of PR 32217) uses a different GGUF layout for both the language model and the draft; files for that layout will follow once it is final.
How it was made
- Routed experts: per-expert error-feedback (GPTQ-style) rounding, each layer calibrated on the output of the already-quantized layers before it, with the rounding settings tuned for this model on held-out data.
- Bits per layer from a measured per-layer sensitivity sweep: gate/up at IQ2_XXS everywhere; down projections set per layer by that sweep (IQ1_S in 30 layers, IQ1_M in 2, IQ2_XXS in 14, IQ2_XS in the 2 most sensitive).
- Every non-expert tensor kept high: attention Q8_0, attention output and output head Q6_K, embeddings Q4_K, router and norms F32.
- Calibration: a neutral mix of chat, code, tool use and long-form text in the model's own chat format, with a held-out set never used for calibration.
Quality
This GGUF, run in llama.cpp, measured against the original model on held-out text never used for calibration (32 x 2048 tokens of chat, code, tool use and long-form text; the reference is the original model's top-64 log-probabilities at every scored position):
| Quant | Size (GB) | Mean KLD | Median KLD | Same top token | PPL vs original |
|---|---|---|---|---|---|
| this quant | 150.0 | 0.114 | 0.0026 | 94.3% | +3.6% |
Running
Inkling 2 needs a llama.cpp build with Inkling 2 support. Until it is merged upstream, build the pull request (ggml-org/llama.cpp PR 32187):
git clone https://github.com/ggml-org/llama.cpp
git -C llama.cpp fetch origin pull/32187/head:inkling2
git -C llama.cpp checkout inkling2
cmake -S llama.cpp -B llama.cpp/build -DGGML_CUDA=ON && cmake --build llama.cpp/build -j --target llama-server
hf download peasantsmith/Inkling-2-Small-GGUF --local-dir Inkling-2-Small-GGUF
./llama.cpp/build/bin/llama-server \
-m Inkling-2-Small-GGUF/Inkling-2-Small-IQ2_XXS-mix-00001-of-00004.gguf \
--mmproj Inkling-2-Small-GGUF/mmproj-Inkling-2-Small-F16.gguf \
--jinja -ngl 99 -c 32768 \
--temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0
Keep all shards in one folder. If the model does not fit in your GPU memory, keep some or all experts in system
RAM with --n-cpu-moe N (the experts of the first N layers) or --cpu-moe (all of them). The chat template is
embedded; --chat-template-kwargs '{"reasoning_effort":"medium"}' changes the thinking effort (none ... max,
default high).
License
Apache-2.0, like the original model. Model by Thinking Machines.
- Downloads last month
- 1,997
2-bit
16-bit
Model tree for peasantsmith/Inkling-2-Small-GGUF
Base model
thinkingmachines/Inkling-2-Small