cafonez commited on
Commit
1e4a324
·
verified ·
1 Parent(s): b7151e6

Update README.md

Browse files

---
base_model: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
license: other
license_name: nvidia-nemotron-open-model-license
license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/
tags:
- gguf
- quantized
- rocm
- rocmfpx
- agentic
- tool-calling
- nemotron
- nemotron-3
- nvidia
- text-generation
- llama.cpp
---

# Agent-Nemotron-ROCmFP6

**Q6_0_ROCMFPX_AGENT** (ROCmFP6 Agent) quantized GGUF of NVIDIA's Nemotron-3-Nano-30B-A3B.

- **Base model**: [nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16)
- **Quantization**: `Q6_0_ROCMFPX_AGENT` — ROCm-optimized 6-bit format with **agent/tool-call coherent routing**
- **Size**: ~27.4 GiB (21 GB + 6.4 GB shards)
- **Parameters**: ~30B total / 3.5B active (hybrid Mamba-2 + MoE)
- **Optimized for**: Agentic workflows, tool calling, reasoning on **AMD ROCm** and **Vulkan** backends

This quantization uses custom ROCmFPX kernels (part of experimental ROCmFPx family in llama.cpp) that provide better performance/quality on ROCm hardware for agent-style workloads. The `_AGENT` preset protects and enhances routing for tool use (Hermes-style / OpenClaw / BFCL etc.).

![c0f43f9f-4a7b-47df-a0a5-ac9b79fb68ec](https://cdn-uploads.huggingface.co/production/uploads/66c3d6e296583c59b06a1dd3/elqta73JLCv9LWByfNkco.png)

## Files

| File | Size | Description |
|------|------|-------------|
| `Nemotron-3-Nano-30B-A3B-Q6_0_ROCMFPX_AGENT-00001-of-00002.gguf` | 21 GB | Main weights shard |
| `Nemotron-3-Nano-30B-A3B-Q6_0_ROCMFPX_AGENT-00002-of-00002.gguf` | 6.4 GB | Second shard |

## Recommended Usage (llama.cpp)

Use a **ROCmFPX-enabled** build of llama.cpp (see ROCmFPX projects / strix builds).

### Quick server (recommended flags)

```bash
# Using the convenience wrapper (if installed)
HERMES_NEMOTRON_NANO_FP6_MODEL=/path/to/Nemotron-3-Nano-30B-A3B-Q6_0_ROCMFPX_AGENT-00001-of-00002.gguf \
hermes-nemotron-nano-30b-rocmfp6-agent-server
```

Direct `llama-server`:

```bash
llama-server \
-m /path/to/Nemotron-3-Nano-30B-A3B-Q6_0_ROCMFPX_AGENT-00001-of-00002.gguf \
--alias nemotron-nano-30b-rocmfp6-agent \
--host 0.0.0.0 --port 8101 \
-dev ROCm0 \
-ngl 999 \
-fa on \
--mmap \
--jinja \
-c 131072 \
-b 512 -ub 512 \
--reasoning off \
--slots \
--metrics
```

For best agent/tool performance use `--jinja` (the GGUF embeds a strong Nemotron tool calling template).

### Key notes

- `Q6_0_ROCMFPX_AGENT` spends a few extra bits on agent routing tensors compared to plain `Q6_0_ROCMFPX`.
- Excellent balance of quality vs size for agentic use on high-end AMD GPUs (Strix Halo, etc.).
- Supports very long context (tested high values).
- Tool calling format is the Nemotron `<tool_call>` style (also compatible with many frameworks via parsers).

## Chat Template

The GGUF includes the official Nemotron-3 tool-aware chat template. Use `--jinja` (or equivalent) with your loader.

## Benchmarks (example from development)

Typical token/s on ROCm0 (full offload) for this quant:
- ~650+ t/s prompt eval (pp512)
- ~53 t/s generation (tg128)

Results vary by hardware + context.

## License

- Original weights: [NVIDIA Nemotron Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/)
- This is a derived quantized artifact. You must comply with the base model's license terms.

---

**Model page**: https://huggingface.co/cafonez/Agent-Nemotron-ROCmFP6

For questions or issues with the quantization, refer to the ROCmFPX documentation in the corresponding development repositories.

Files changed (1) hide show
  1. README.md +2 -5
README.md CHANGED
@@ -29,6 +29,8 @@ tags:
29
 
30
  This quantization uses custom ROCmFPX kernels (part of experimental ROCmFPx family in llama.cpp) that provide better performance/quality on ROCm hardware for agent-style workloads. The `_AGENT` preset protects and enhances routing for tool use (Hermes-style / OpenClaw / BFCL etc.).
31
 
 
 
32
  ## Files
33
 
34
  | File | Size | Description |
@@ -93,11 +95,6 @@ Results vary by hardware + context.
93
  - Original weights: [NVIDIA Nemotron Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/)
94
  - This is a derived quantized artifact. You must comply with the base model's license terms.
95
 
96
- ## Acknowledgements
97
-
98
- - NVIDIA for the outstanding Nemotron-3-Nano open model family
99
- - Contributors to ROCmFPX / custom quantization formats in llama.cpp for AMD performance
100
-
101
  ---
102
 
103
  **Model page**: https://huggingface.co/cafonez/Agent-Nemotron-ROCmFP6
 
29
 
30
  This quantization uses custom ROCmFPX kernels (part of experimental ROCmFPx family in llama.cpp) that provide better performance/quality on ROCm hardware for agent-style workloads. The `_AGENT` preset protects and enhances routing for tool use (Hermes-style / OpenClaw / BFCL etc.).
31
 
32
+
33
+
34
  ## Files
35
 
36
  | File | Size | Description |
 
95
  - Original weights: [NVIDIA Nemotron Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/)
96
  - This is a derived quantized artifact. You must comply with the base model's license terms.
97
 
 
 
 
 
 
98
  ---
99
 
100
  **Model page**: https://huggingface.co/cafonez/Agent-Nemotron-ROCmFP6