Instructions to use cafonez/Agent-Nemotron-ROCmFP6 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use cafonez/Agent-Nemotron-ROCmFP6 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf cafonez/Agent-Nemotron-ROCmFP6 # Run inference directly in the terminal: llama cli -hf cafonez/Agent-Nemotron-ROCmFP6
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf cafonez/Agent-Nemotron-ROCmFP6 # Run inference directly in the terminal: llama cli -hf cafonez/Agent-Nemotron-ROCmFP6
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf cafonez/Agent-Nemotron-ROCmFP6 # Run inference directly in the terminal: ./build/bin/llama cli -hf cafonez/Agent-Nemotron-ROCmFP6
Use Docker
docker model run hf.co/cafonez/Agent-Nemotron-ROCmFP6
- Llama new
- LM Studio
- Jan
- vLLM
How to use cafonez/Agent-Nemotron-ROCmFP6 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "cafonez/Agent-Nemotron-ROCmFP6" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "cafonez/Agent-Nemotron-ROCmFP6", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/cafonez/Agent-Nemotron-ROCmFP6
- Ollama
How to use cafonez/Agent-Nemotron-ROCmFP6 with Ollama:
ollama run hf.co/cafonez/Agent-Nemotron-ROCmFP6
- Unsloth Desktop
- Pi
How to use cafonez/Agent-Nemotron-ROCmFP6 with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf cafonez/Agent-Nemotron-ROCmFP6
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "cafonez/Agent-Nemotron-ROCmFP6" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use cafonez/Agent-Nemotron-ROCmFP6 with Docker Model Runner:
docker model run hf.co/cafonez/Agent-Nemotron-ROCmFP6
- Lemonade
How to use cafonez/Agent-Nemotron-ROCmFP6 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull cafonez/Agent-Nemotron-ROCmFP6
Run and chat with the model
lemonade run user.Agent-Nemotron-ROCmFP6-{{QUANT_TAG}}List all available models
lemonade list
- Hermes Agent
How to use cafonez/Agent-Nemotron-ROCmFP6 with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf cafonez/Agent-Nemotron-ROCmFP6
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default cafonez/Agent-Nemotron-ROCmFP6
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use cafonez/Agent-Nemotron-ROCmFP6 with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf cafonez/Agent-Nemotron-ROCmFP6
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "cafonez/Agent-Nemotron-ROCmFP6" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Update README.md
Browse files---
base_model: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
license: other
license_name: nvidia-nemotron-open-model-license
license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/
tags:
- gguf
- quantized
- rocm
- rocmfpx
- agentic
- tool-calling
- nemotron
- nemotron-3
- nvidia
- text-generation
- llama.cpp
---
# Agent-Nemotron-ROCmFP6
**Q6_0_ROCMFPX_AGENT** (ROCmFP6 Agent) quantized GGUF of NVIDIA's Nemotron-3-Nano-30B-A3B.
- **Base model**: [nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16)
- **Quantization**: `Q6_0_ROCMFPX_AGENT` — ROCm-optimized 6-bit format with **agent/tool-call coherent routing**
- **Size**: ~27.4 GiB (21 GB + 6.4 GB shards)
- **Parameters**: ~30B total / 3.5B active (hybrid Mamba-2 + MoE)
- **Optimized for**: Agentic workflows, tool calling, reasoning on **AMD ROCm** and **Vulkan** backends
This quantization uses custom ROCmFPX kernels (part of experimental ROCmFPx family in llama.cpp) that provide better performance/quality on ROCm hardware for agent-style workloads. The `_AGENT` preset protects and enhances routing for tool use (Hermes-style / OpenClaw / BFCL etc.).

## Files
| File | Size | Description |
|------|------|-------------|
| `Nemotron-3-Nano-30B-A3B-Q6_0_ROCMFPX_AGENT-00001-of-00002.gguf` | 21 GB | Main weights shard |
| `Nemotron-3-Nano-30B-A3B-Q6_0_ROCMFPX_AGENT-00002-of-00002.gguf` | 6.4 GB | Second shard |
## Recommended Usage (llama.cpp)
Use a **ROCmFPX-enabled** build of llama.cpp (see ROCmFPX projects / strix builds).
### Quick server (recommended flags)
```bash
# Using the convenience wrapper (if installed)
HERMES_NEMOTRON_NANO_FP6_MODEL=/path/to/Nemotron-3-Nano-30B-A3B-Q6_0_ROCMFPX_AGENT-00001-of-00002.gguf \
hermes-nemotron-nano-30b-rocmfp6-agent-server
```
Direct `llama-server`:
```bash
llama-server \
-m /path/to/Nemotron-3-Nano-30B-A3B-Q6_0_ROCMFPX_AGENT-00001-of-00002.gguf \
--alias nemotron-nano-30b-rocmfp6-agent \
--host 0.0.0.0 --port 8101 \
-dev ROCm0 \
-ngl 999 \
-fa on \
--mmap \
--jinja \
-c 131072 \
-b 512 -ub 512 \
--reasoning off \
--slots \
--metrics
```
For best agent/tool performance use `--jinja` (the GGUF embeds a strong Nemotron tool calling template).
### Key notes
- `Q6_0_ROCMFPX_AGENT` spends a few extra bits on agent routing tensors compared to plain `Q6_0_ROCMFPX`.
- Excellent balance of quality vs size for agentic use on high-end AMD GPUs (Strix Halo, etc.).
- Supports very long context (tested high values).
- Tool calling format is the Nemotron `<tool_call>` style (also compatible with many frameworks via parsers).
## Chat Template
The GGUF includes the official Nemotron-3 tool-aware chat template. Use `--jinja` (or equivalent) with your loader.
## Benchmarks (example from development)
Typical token/s on ROCm0 (full offload) for this quant:
- ~650+ t/s prompt eval (pp512)
- ~53 t/s generation (tg128)
Results vary by hardware + context.
## License
- Original weights: [NVIDIA Nemotron Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/)
- This is a derived quantized artifact. You must comply with the base model's license terms.
---
**Model page**: https://huggingface.co/cafonez/Agent-Nemotron-ROCmFP6
For questions or issues with the quantization, refer to the ROCmFPX documentation in the corresponding development repositories.
|
@@ -29,6 +29,8 @@ tags:
|
|
| 29 |
|
| 30 |
This quantization uses custom ROCmFPX kernels (part of experimental ROCmFPx family in llama.cpp) that provide better performance/quality on ROCm hardware for agent-style workloads. The `_AGENT` preset protects and enhances routing for tool use (Hermes-style / OpenClaw / BFCL etc.).
|
| 31 |
|
|
|
|
|
|
|
| 32 |
## Files
|
| 33 |
|
| 34 |
| File | Size | Description |
|
|
@@ -93,11 +95,6 @@ Results vary by hardware + context.
|
|
| 93 |
- Original weights: [NVIDIA Nemotron Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/)
|
| 94 |
- This is a derived quantized artifact. You must comply with the base model's license terms.
|
| 95 |
|
| 96 |
-
## Acknowledgements
|
| 97 |
-
|
| 98 |
-
- NVIDIA for the outstanding Nemotron-3-Nano open model family
|
| 99 |
-
- Contributors to ROCmFPX / custom quantization formats in llama.cpp for AMD performance
|
| 100 |
-
|
| 101 |
---
|
| 102 |
|
| 103 |
**Model page**: https://huggingface.co/cafonez/Agent-Nemotron-ROCmFP6
|
|
|
|
| 29 |
|
| 30 |
This quantization uses custom ROCmFPX kernels (part of experimental ROCmFPx family in llama.cpp) that provide better performance/quality on ROCm hardware for agent-style workloads. The `_AGENT` preset protects and enhances routing for tool use (Hermes-style / OpenClaw / BFCL etc.).
|
| 31 |
|
| 32 |
+
|
| 33 |
+
|
| 34 |
## Files
|
| 35 |
|
| 36 |
| File | Size | Description |
|
|
|
|
| 95 |
- Original weights: [NVIDIA Nemotron Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/)
|
| 96 |
- This is a derived quantized artifact. You must comply with the base model's license terms.
|
| 97 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 98 |
---
|
| 99 |
|
| 100 |
**Model page**: https://huggingface.co/cafonez/Agent-Nemotron-ROCmFP6
|