---
license: mit
base_model: baselabs/elephant-1
base_model_relation: quantized
pipeline_tag: text-classification
library_name: gguf
tags: [gguf, rocmfpx, rocmfp4, amd, rocm, strix-halo, quantized, decision-model, classification, calibration]
---

# elephant-1 — ROCmFPX GGUF

[baselabs/elephant-1](https://huggingface.co/baselabs/elephant-1), Base Labs' 9B **decision model** (post-trained
from Qwen3.5-9B), quantized to the [ROCmFPX](https://github.com/charlie12345/ROCmFPX) formats for AMD GPUs. Give it
a state and a set of questions; it returns a probability for every option of every question from one shared prefill.

These files run with **`llama-decide`**, a small tool in
[HarshaVardhan3002/ROCmFPX](https://github.com/HarshaVardhan3002/ROCmFPX) that reads the same request format as
Base Labs' own runtime. A normal chat server is not the way to use this model.

**Start with `Q4_0_ROCMFP4_STRIX`.** It is the closest 4-bit file to the original.

> These files only load in a ROCmFPX build of llama.cpp. **They have not been tested on AMD hardware yet.**
>
> Not an official release of Base Labs or Baseten. All credit for the model goes to Base Labs.

## Files and measured quality

Measured against the BF16 model (perplexity 7.5591) on WikiText-2 test, 40 chunks of 512 tokens.
**KLD** is the mean KL divergence of the next-token distribution from BF16 (lower is better). **Same top** is how
often the quant picks the same next token as BF16. These numbers use ordinary left-to-right attention on plain
text; they show how much each file changes the weights, not decision accuracy, which was not measured.

| File | Size | PPL | KLD | Same top | |
|---|---:|---:|---:|---:|---|
| `elephant-1-Q8_0_ROCMFPX.gguf` | 9.2 GB | 7.5678 | 0.0054 | 96.47% | 8-bit, closest to BF16 |
| `elephant-1-Q6_K_ROCMFPX.gguf` | 7.4 GB | 7.5805 | 0.0067 | 95.96% | 6-bit, almost as close as 8-bit |
| `elephant-1-Q4_0_ROCMFP4_STRIX.gguf` | 5.1 GB | 7.7795 | 0.0478 | 88.63% | **Recommended.** 4-bit recipe tuned for Strix Halo (`gfx1151`) |
| `elephant-1-Q4_0_ROCMFP4_FAST.gguf` | 4.8 GB | 7.8060 | 0.0496 | 88.54% | smallest, speed-first 4-bit (one scale per block) |

The 6-bit file is named `elephant-1-Q6_K_ROCMFPX.gguf` so that Hugging Face lists it as a 6-bit quant. Its weights are the ROCmFPX `Q6_0_ROCMFPX` type, not a stock llama.cpp `Q6_K`, and it needs the same fork as the other files.

Not published: `Q3_0_ROCMFPX` (4.9 GB, PPL 7.9236, KLD 0.0750, same top 86.74%) is larger than `Q4_0_ROCMFP4_FAST` and
further from BF16.

## Decisions

The GGUF files carry the decision settings from the original checkpoint (`qwen35.decision.*`: block-bidirectional
attention on the full-attention layers and the 256 answer labels), so `llama-decide` needs no extra files.

```bash
git clone https://github.com/HarshaVardhan3002/ROCmFPX && cd ROCmFPX
cmake -B build -DGGML_VULKAN=ON          # or -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 (gfx1100 for RX 7900)
cmake --build build --target llama-decide -j
build/bin/llama-decide -m elephant-1-Q4_0_ROCMFP4_STRIX.gguf -ngl 999 -f request.json
```

`request.json`: a `state` (text or JSON) and `questions`, as in Base Labs' card:

```json
{
  "state": {
    "task": "Pick up the red mug from the table and place it on the shelf.",
    "observations": ["The gripper closed around the handle of the red mug.",
                     "The arm lifted the mug 12 cm above the table.",
                     "The mug slipped and fell back onto the table, upright and intact."],
    "gripper_force_newton": 4.1
  },
  "questions": {
    "success": {"type": "noul", "instructions": "Was the task completed successfully?"},
    "failure_mode": {"type": "choice", "instructions": "What is the main reason for the outcome?",
      "criteria": {"grasp_slip": "The object slipped out of the gripper.", "collision": "The arm or the object hit something.",
                   "wrong_object": "The robot handled the wrong object.", "no_failure": "Nothing went wrong."}},
    "progress": {"type": "score", "instructions": "How far did the robot get with the task?",
      "criteria": ["did not start", "grasped the object", "lifted the object", "placed the object on the shelf"]}
  }
}
```

Answer from `elephant-1-Q4_0_ROCMFP4_STRIX.gguf` (CPU backend): `success` 0.020; `failure_mode` `grasp_slip` 0.996;
`progress` most likely "grasped the object" (0.719), then "lifted the object" (0.223).

| `type` | `criteria` | answer fields |
|---|---|---|
| `noul`: yes or no | optional | `noul`: P(yes) |
| `choice`: one of named options | `{name: description}` | `choice`, `probabilities` |
| `score`: ordered levels | list of levels, lowest first | `score`, `probabilities` |

Options: `--debias N` (average over N cyclic shifts of the answer labels), `--causal` (plain causal attention instead
of the trained block-bidirectional mask). `llama-decide` was checked against Base Labs' PyTorch runtime on this model
(probabilities within 0.004).

## How these were made

1. `convert_hf_to_gguf.py --outtype bf16` from the fork, which also writes the decision settings.
2. Importance matrix from the BF16 model on the GPU over a general calibration text (`calibration_datav3`), 512-token
   chunks.
3. `llama-quantize --imatrix`, one run per type (ROCmFPX `fb08d7c` merged onto upstream llama.cpp).
4. Quality: the BF16 reference ran on an NVIDIA GPU with stock BF16 kernels; each quant was scored with the fork's
   CPU backend, which runs the ROCmFPX types directly. AMD GPU kernels compute the same math in a different order, so
   results on an AMD GPU can differ slightly.

The 8-bit file is not as close to BF16 as a stock `Q8_0` usually is: ROCmFPX block scales have a fixed smallest step,
so 8-bit blocks of small weights keep fewer than 8 bits
([ROCmFPX#118](https://github.com/charlie12345/ROCmFPX/issues/118)).

## License

MIT, same as the base model.