johnsonchromia commited on
Commit
c7bb155
路
verified 路
1 Parent(s): 44a00e4

Add Q8_0 and F16 GGUF builds

Browse files
Files changed (4) hide show
  1. .gitattributes +2 -0
  2. README.md +9 -7
  3. decision-4b-F16.gguf +3 -0
  4. decision-4b-Q8_0.gguf +3 -0
.gitattributes CHANGED
@@ -35,3 +35,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  benchmark.png filter=lfs diff=lfs merge=lfs -text
37
  decision-4b-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
 
 
 
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  benchmark.png filter=lfs diff=lfs merge=lfs -text
37
  decision-4b-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
38
+ decision-4b-F16.gguf filter=lfs diff=lfs merge=lfs -text
39
+ decision-4b-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -14,17 +14,19 @@ tags:
14
 
15
  # Decision-4B GGUF
16
 
17
- **Q4_K_M GGUF of [Decision-4B](https://huggingface.co/evalengine/decision-4b), an open-weight Jev-like decision model from [Eval Engine](https://evalengine.ai), the AI arm of Chromia.**
18
 
19
- Give it a state, a question, and a list of options. It answers with one letter. 2.7 GB, runs in llama.cpp, Ollama, and on your phone. This is the file that powers Decision-4B in the Unbound app.
20
 
21
  **Try it now:** [Unbound on the App Store](https://apps.apple.com/us/app/unbound-ai/id6769727542) 路 [Unbound on the web](https://unbound.evalengine.ai/chat)
22
 
23
- | File | Size | Notes |
24
- |---|---:|---|
25
- | `decision-4b-Q4_K_M.gguf` | 2.71 GB | LoRA merged into Qwen3.5-4B, quantized Q4_K_M |
 
 
26
 
27
- On our 892-case development panel the Q4_K_M file scores 88.9%, matching the BF16 adapter (87.3%).
28
 
29
  ## Benchmark
30
 
@@ -35,7 +37,7 @@ Full table and details on the [adapter card](https://huggingface.co/evalengine/d
35
  ## Run with llama.cpp
36
 
37
  ```bash
38
- llama-server -m decision-4b-Q4_K_M.gguf -c 2048
39
  ```
40
 
41
  ```bash
 
14
 
15
  # Decision-4B GGUF
16
 
17
+ **GGUF builds of [Decision-4B](https://huggingface.co/evalengine/decision-4b), an open-weight Jev-like decision model from [Eval Engine](https://evalengine.ai), the AI arm of Chromia.**
18
 
19
+ Give it a state, a question, and a list of options. It answers with one letter. Runs in llama.cpp, Ollama, and on your phone. The Q4_K_M file is the one that powers Decision-4B in the Unbound app.
20
 
21
  **Try it now:** [Unbound on the App Store](https://apps.apple.com/us/app/unbound-ai/id6769727542) 路 [Unbound on the web](https://unbound.evalengine.ai/chat)
22
 
23
+ | File | Size | Dev accuracy (892 cases) |
24
+ |---|---:|---:|
25
+ | `decision-4b-Q4_K_M.gguf` | 2.71 GB | 88.9% |
26
+ | `decision-4b-Q8_0.gguf` | 4.48 GB | 87.7% |
27
+ | `decision-4b-F16.gguf` | 8.42 GB | 87.9% |
28
 
29
+ All three are the LoRA merged into Qwen3.5-4B. The BF16 adapter scores 87.3% on the same panel. Use Q4_K_M for phones and laptops, Q8_0 or F16 when you have the memory.
30
 
31
  ## Benchmark
32
 
 
37
  ## Run with llama.cpp
38
 
39
  ```bash
40
+ llama-server -m decision-4b-Q4_K_M.gguf -c 2048 # or Q8_0 / F16
41
  ```
42
 
43
  ```bash
decision-4b-F16.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:481875895114e7f96061df2d86fb08666f6a5ff4ef08c1dbafb74256b44f99d1
3
+ size 8424393152
decision-4b-Q8_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f2519bcdc945291974c81a37ff92417b22df27154c9df2564420d08576b907ce
3
+ size 4482402752