| --- |
| license: mit |
| base_model: baselabs/elephant-1 |
| base_model_relation: quantized |
| pipeline_tag: text-classification |
| library_name: gguf |
| tags: [gguf, rocmfpx, rocmfp4, amd, rocm, strix-halo, quantized, decision-model, classification, calibration] |
| --- |
| |
| # elephant-1 — ROCmFPX GGUF |
|
|
| [baselabs/elephant-1](https://huggingface.co/baselabs/elephant-1), Base Labs' 9B **decision model** (post-trained |
| from Qwen3.5-9B), quantized to the [ROCmFPX](https://github.com/charlie12345/ROCmFPX) formats for AMD GPUs. Give it |
| a state and a set of questions; it returns a probability for every option of every question from one shared prefill. |
|
|
| These files run with **`llama-decide`**, a small tool in |
| [HarshaVardhan3002/ROCmFPX](https://github.com/HarshaVardhan3002/ROCmFPX) that reads the same request format as |
| Base Labs' own runtime. A normal chat server is not the way to use this model. |
|
|
| **Start with `Q4_0_ROCMFP4_STRIX`.** It is the closest 4-bit file to the original. |
| |
| > These files only load in a ROCmFPX build of llama.cpp. **They have not been tested on AMD hardware yet.** |
| > |
| > Not an official release of Base Labs or Baseten. All credit for the model goes to Base Labs. |
| |
| ## Files and measured quality |
| |
| Measured against the BF16 model (perplexity 7.5591) on WikiText-2 test, 40 chunks of 512 tokens. |
| **KLD** is the mean KL divergence of the next-token distribution from BF16 (lower is better). **Same top** is how |
| often the quant picks the same next token as BF16. These numbers use ordinary left-to-right attention on plain |
| text; they show how much each file changes the weights, not decision accuracy, which was not measured. |
| |
| | File | Size | PPL | KLD | Same top | | |
| |---|---:|---:|---:|---:|---| |
| | `elephant-1-Q8_0_ROCMFPX.gguf` | 9.2 GB | 7.5678 | 0.0054 | 96.47% | 8-bit, closest to BF16 | |
| | `elephant-1-Q6_K_ROCMFPX.gguf` | 7.4 GB | 7.5805 | 0.0067 | 95.96% | 6-bit, almost as close as 8-bit | |
| | `elephant-1-Q4_0_ROCMFP4_STRIX.gguf` | 5.1 GB | 7.7795 | 0.0478 | 88.63% | **Recommended.** 4-bit recipe tuned for Strix Halo (`gfx1151`) | |
| | `elephant-1-Q4_0_ROCMFP4_FAST.gguf` | 4.8 GB | 7.8060 | 0.0496 | 88.54% | smallest, speed-first 4-bit (one scale per block) | |
|
|
| The 6-bit file is named `elephant-1-Q6_K_ROCMFPX.gguf` so that Hugging Face lists it as a 6-bit quant. Its weights are the ROCmFPX `Q6_0_ROCMFPX` type, not a stock llama.cpp `Q6_K`, and it needs the same fork as the other files. |
|
|
| Not published: `Q3_0_ROCMFPX` (4.9 GB, PPL 7.9236, KLD 0.0750, same top 86.74%) is larger than `Q4_0_ROCMFP4_FAST` and |
| further from BF16. |
|
|
| ## Decisions |
|
|
| The GGUF files carry the decision settings from the original checkpoint (`qwen35.decision.*`: block-bidirectional |
| attention on the full-attention layers and the 256 answer labels), so `llama-decide` needs no extra files. |
|
|
| ```bash |
| git clone https://github.com/HarshaVardhan3002/ROCmFPX && cd ROCmFPX |
| cmake -B build -DGGML_VULKAN=ON # or -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 (gfx1100 for RX 7900) |
| cmake --build build --target llama-decide -j |
| build/bin/llama-decide -m elephant-1-Q4_0_ROCMFP4_STRIX.gguf -ngl 999 -f request.json |
| ``` |
|
|
| `request.json`: a `state` (text or JSON) and `questions`, as in Base Labs' card: |
|
|
| ```json |
| { |
| "state": { |
| "task": "Pick up the red mug from the table and place it on the shelf.", |
| "observations": ["The gripper closed around the handle of the red mug.", |
| "The arm lifted the mug 12 cm above the table.", |
| "The mug slipped and fell back onto the table, upright and intact."], |
| "gripper_force_newton": 4.1 |
| }, |
| "questions": { |
| "success": {"type": "noul", "instructions": "Was the task completed successfully?"}, |
| "failure_mode": {"type": "choice", "instructions": "What is the main reason for the outcome?", |
| "criteria": {"grasp_slip": "The object slipped out of the gripper.", "collision": "The arm or the object hit something.", |
| "wrong_object": "The robot handled the wrong object.", "no_failure": "Nothing went wrong."}}, |
| "progress": {"type": "score", "instructions": "How far did the robot get with the task?", |
| "criteria": ["did not start", "grasped the object", "lifted the object", "placed the object on the shelf"]} |
| } |
| } |
| ``` |
|
|
| Answer from `elephant-1-Q4_0_ROCMFP4_STRIX.gguf` (CPU backend): `success` 0.020; `failure_mode` `grasp_slip` 0.996; |
| `progress` most likely "grasped the object" (0.719), then "lifted the object" (0.223). |
|
|
| | `type` | `criteria` | answer fields | |
| |---|---|---| |
| | `noul`: yes or no | optional | `noul`: P(yes) | |
| | `choice`: one of named options | `{name: description}` | `choice`, `probabilities` | |
| | `score`: ordered levels | list of levels, lowest first | `score`, `probabilities` | |
|
|
| Options: `--debias N` (average over N cyclic shifts of the answer labels), `--causal` (plain causal attention instead |
| of the trained block-bidirectional mask). `llama-decide` was checked against Base Labs' PyTorch runtime on this model |
| (probabilities within 0.004). |
|
|
| ## How these were made |
|
|
| 1. `convert_hf_to_gguf.py --outtype bf16` from the fork, which also writes the decision settings. |
| 2. Importance matrix from the BF16 model on the GPU over a general calibration text (`calibration_datav3`), 512-token |
| chunks. |
| 3. `llama-quantize --imatrix`, one run per type (ROCmFPX `fb08d7c` merged onto upstream llama.cpp). |
| 4. Quality: the BF16 reference ran on an NVIDIA GPU with stock BF16 kernels; each quant was scored with the fork's |
| CPU backend, which runs the ROCmFPX types directly. AMD GPU kernels compute the same math in a different order, so |
| results on an AMD GPU can differ slightly. |
|
|
| The 8-bit file is not as close to BF16 as a stock `Q8_0` usually is: ROCmFPX block scales have a fixed smallest step, |
| so 8-bit blocks of small weights keep fewer than 8 bits |
| ([ROCmFPX#118](https://github.com/charlie12345/ROCmFPX/issues/118)). |
|
|
| ## License |
|
|
| MIT, same as the base model. |
|
|
