Text Generation
GGUF
rocmfpx
amd
vulkan
rocm
strix-halo
quantized
Mixture of Experts
inkling
inkling2
imatrix
conversational
Instructions to use Harsha001/Inkling-2-Mini-ROCmFPX with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use Harsha001/Inkling-2-Mini-ROCmFPX with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf Harsha001/Inkling-2-Mini-ROCmFPX:Q4_0_ROCMFP # Run inference directly in the terminal: llama cli -hf Harsha001/Inkling-2-Mini-ROCmFPX:Q4_0_ROCMFP
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf Harsha001/Inkling-2-Mini-ROCmFPX:Q4_0_ROCMFP # Run inference directly in the terminal: llama cli -hf Harsha001/Inkling-2-Mini-ROCmFPX:Q4_0_ROCMFP
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf Harsha001/Inkling-2-Mini-ROCmFPX:Q4_0_ROCMFP # Run inference directly in the terminal: ./build/bin/llama cli -hf Harsha001/Inkling-2-Mini-ROCmFPX:Q4_0_ROCMFP
Use Docker
docker model run hf.co/Harsha001/Inkling-2-Mini-ROCmFPX:Q4_0_ROCMFP
- Llama new
- LM Studio
- Jan
- vLLM
How to use Harsha001/Inkling-2-Mini-ROCmFPX with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Harsha001/Inkling-2-Mini-ROCmFPX" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Harsha001/Inkling-2-Mini-ROCmFPX", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Harsha001/Inkling-2-Mini-ROCmFPX:Q4_0_ROCMFP
- Ollama
How to use Harsha001/Inkling-2-Mini-ROCmFPX with Ollama:
ollama run hf.co/Harsha001/Inkling-2-Mini-ROCmFPX:Q4_0_ROCMFP
- Unsloth Desktop
- Pi
How to use Harsha001/Inkling-2-Mini-ROCmFPX with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Harsha001/Inkling-2-Mini-ROCmFPX:Q4_0_ROCMFP
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "Harsha001/Inkling-2-Mini-ROCmFPX:Q4_0_ROCMFP" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use Harsha001/Inkling-2-Mini-ROCmFPX with Docker Model Runner:
docker model run hf.co/Harsha001/Inkling-2-Mini-ROCmFPX:Q4_0_ROCMFP
- Lemonade
How to use Harsha001/Inkling-2-Mini-ROCmFPX with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull Harsha001/Inkling-2-Mini-ROCmFPX:Q4_0_ROCMFP
Run and chat with the model
lemonade run user.Inkling-2-Mini-ROCmFPX-Q4_0_ROCMFP
List all available models
lemonade list
- Hermes Agent
How to use Harsha001/Inkling-2-Mini-ROCmFPX with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Harsha001/Inkling-2-Mini-ROCmFPX:Q4_0_ROCMFP
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default Harsha001/Inkling-2-Mini-ROCmFPX:Q4_0_ROCMFP
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use Harsha001/Inkling-2-Mini-ROCmFPX with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Harsha001/Inkling-2-Mini-ROCmFPX:Q4_0_ROCMFP
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "Harsha001/Inkling-2-Mini-ROCmFPX:Q4_0_ROCMFP" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
|
Download README.md from Harsha001/Inkling-2-Mini-ROCmFPX: direct link, hf CLI and curl.
- Browser
- Download file 4.49 kB
-
https://huggingface.co/Harsha001/Inkling-2-Mini-ROCmFPX/resolve/main/README.md
- Command line
-
hf download hf://Harsha001/Inkling-2-Mini-ROCmFPX/README.md
-
curl -L -o README.md https://huggingface.co/Harsha001/Inkling-2-Mini-ROCmFPX/resolve/main/README.md
4.49 kB
| license: apache-2.0 | |
| base_model: thinkingmachines/Inkling-2-Mini | |
| base_model_relation: quantized | |
| pipeline_tag: text-generation | |
| library_name: gguf | |
| tags: [gguf, rocmfpx, amd, vulkan, rocm, strix-halo, quantized, moe, inkling, inkling2] | |
| # Inkling 2 Mini ROCmFPX (AMD GPUs and Strix Halo) | |
| Hardware-aware GGUF quants of [Inkling 2 Mini by Thinking Machines](https://huggingface.co/thinkingmachines/Inkling-2-Mini), | |
| built with the [ROCmFPX](https://github.com/HarshaVardhan3002/ROCmFPX) weight formats for AMD GPUs (Vulkan and ROCm). | |
| Inkling 2 Mini: 26.1B parameters, 24 layers, 128 experts per layer with 8 active per token (about 3.1B active | |
| parameters per layer pass). It is a looped model: layers 6 to 17 run three times per token, so a token passes through | |
| 48 layers. Text only here: the vision encoder and the multi-token prediction drafter are not included. | |
| > Not an official release. All credit for the model goes to Thinking Machines. | |
| ## Files | |
| | File | Size | Weights | Fits | | |
| |---|---:|---|---| | |
| | `Inkling-2-Mini-Q8_0_ROCMFPX.gguf` | 26.9 GB | everything 8-bit ROCmFPX (token embedding Q8_0) | best quality: Strix Halo, 32 GB AMD GPUs | | |
| | `Inkling-2-Mini-Q4_0_ROCMFP4_STRIX_O8.gguf` | 15.5 GB | experts ROCmFP4, everything else 8-bit ROCmFPX | best smaller: 24 GB AMD GPUs, Strix Halo with room for long context | | |
| ## Quality | |
| KL divergence against the original weights (Thinking Machines' NVFP4 experts kept as they are, the MXFP8 layers 20 to | |
| 23 stored as BF16), 40 chunks of 512 tokens of English Wikipedia text (wikitext-2), CPU. Lower is better. | |
| "Same top token" is how often the file's most likely next token matches the original. | |
| | Build | Size | Perplexity | KL to original | Same top token | | |
| |---|---:|---:|---:|---:| | |
| | Original weights (reference, not published) | 22.9 GB | 6.743 | | | | |
| | **Q8_0_ROCMFPX** | 26.9 GB | 6.741 | 0.0039 | 97.2% | | |
| | **Q4_0_ROCMFP4_STRIX_O8** | 15.5 GB | 6.784 | 0.0120 | 95.4% | | |
| | stock Q8_0 (comparison, not uploaded) | 27.7 GB | 6.739 | 0.0035 | 97.5% | | |
| | stock Q4_K_M (comparison, not uploaded) | 17.4 GB | 6.819 | 0.0174 | 94.1% | | |
| At 4 bits the ROCmFP4 file is smaller than stock Q4_K_M and closer to the original on every measure. At 8 bits stock | |
| Q8_0 is a little closer to the original (0.0035 against 0.0039) and 0.8 GB larger; the ROCmFPX file is the one built for | |
| the AMD kernels. | |
| The Q8 file is larger than the original checkpoint because the original experts are 4-bit NVFP4. 8 bits keep them | |
| almost exactly; the 4-bit ROCmFP4 file is the size to pick when memory is tight. | |
| Every file also passed a greedy chat check before upload ("What is the capital of France? Answer in one word." gives | |
| "Paris" after a short thinking part). | |
| ## Running it | |
| Inkling 2 Mini needs the looped `inkling2` graph, which is only in the ROCmFPX fork of llama.cpp (branch | |
| `inkling2-mini`). Build it with Vulkan or ROCm/HIP, then for example: | |
| ``` | |
| llama-server -m Inkling-2-Mini-Q4_0_ROCMFP4_STRIX_O8.gguf -ngl 99 -c 16384 --jinja | |
| ``` | |
| Known limits of this first version: | |
| - Only the CPU path has been checked against the original PyTorch model (it matches to about 5e-6 on a small test | |
| model, and the tokenizer matches exactly). Vulkan and ROCm have not been tried yet. Please report in the Community | |
| tab whether it runs, how fast, and how the answers look. | |
| - llama.cpp has no parser for Inkling's message format yet. Answers come back with Inkling's own markers | |
| (`<|content_thinking|>`, `<|end_message|>` and so on): the thinking is not split from the answer and tool calls are | |
| not parsed. Plain chat works; agent use does not yet. | |
| - The chat template accepts `reasoning_effort` as `none`, `minimal`, `low`, `medium`, `high`, `xhigh` or `max`, | |
| or as a number from 0 to 0.99 (default 0.9). | |
| ## How it was made | |
| - Converted from the original checkpoint with the NVFP4 experts kept as they are, then requantized with | |
| `llama-quantize`. The per-expert NVFP4 scale is folded into the weights before requantizing. | |
| - ROCmFPX tensor scales on (a power-of-two scale per tensor, stored as `.scale`), token embedding Q8_0. | |
| - Importance matrix from a general text mix (prose, code, chat) on the original weights. | |
| ## Credits and license | |
| - Model: [thinkingmachines/Inkling-2-Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini), Apache-2.0. | |
| - Runtime: [llama.cpp](https://github.com/ggml-org/llama.cpp), with Inkling 2 and ROCmFPX support in | |
| [ROCmFPX](https://github.com/HarshaVardhan3002/ROCmFPX). | |
| Apache-2.0. See `LICENSE`. | |