Add full BFCL test results, per-category breakdown, architecture notes
Browse files
README.md
CHANGED
|
@@ -22,21 +22,57 @@ Qwen3.5-4B quantized to Q3_K_M (2.3 GB) for on-device MCP tool routing on iPhone
|
|
| 22 |
|
| 23 |
```bash
|
| 24 |
ollama pull dcostenco/prism-coder:2b
|
|
|
|
| 25 |
```
|
| 26 |
|
| 27 |
## Download
|
| 28 |
|
| 29 |
| File | Size | BFCL | SHA256 |
|
| 30 |
|------|------|------|--------|
|
| 31 |
-
| [Qwen3.5-4B-Q3_K_M.gguf](https://huggingface.co/dcostenco/prism-coder-4b/resolve/main/Qwen3.5-4B-Q3_K_M.gguf) | 2.3 GB | 99.1% | `
|
| 32 |
|
| 33 |
-
|
| 34 |
|
| 35 |
-
|
| 36 |
|
| 37 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 38 |
|
| 39 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
|
| 41 |
## Fleet Position
|
| 42 |
|
|
|
|
| 22 |
|
| 23 |
```bash
|
| 24 |
ollama pull dcostenco/prism-coder:2b
|
| 25 |
+
ollama run dcostenco/prism-coder:2b
|
| 26 |
```
|
| 27 |
|
| 28 |
## Download
|
| 29 |
|
| 30 |
| File | Size | BFCL | SHA256 |
|
| 31 |
|------|------|------|--------|
|
| 32 |
+
| [Qwen3.5-4B-Q3_K_M.gguf](https://huggingface.co/dcostenco/prism-coder-4b/resolve/main/Qwen3.5-4B-Q3_K_M.gguf) | 2.3 GB | 99.1% | `d6981ab4d77ba712b48ef69d69042d75b5e39b9dce5fb5a5b054fd08e06afb95` |
|
| 33 |
|
| 34 |
+
## BFCL Benchmark — 99.1% × 3 Seeds
|
| 35 |
|
| 36 |
+
**114/115 × 3 shuffled runs = 99.1%, 1 flaky case, stable across all seeds**
|
| 37 |
|
| 38 |
+
| Category | Count | Description | Accuracy |
|
| 39 |
+
|----------|------:|-------------|:--------:|
|
| 40 |
+
| save | 17 | session_save_ledger logging | 100% |
|
| 41 |
+
| smem | 17 | Session memory search/recall | 100% |
|
| 42 |
+
| aac | 12 | AAC phrase/communication routing | 100% |
|
| 43 |
+
| hand | 12 | Agent handoff (session_save_handoff) | 100% |
|
| 44 |
+
| irrel | 10 | Irrelevant/general questions → plain text | 90% |
|
| 45 |
+
| load | 9 | Context loading (session_load_context) | 100% |
|
| 46 |
+
| pred | 8 | Prediction/task routing | 100% |
|
| 47 |
+
| know | 7 | Knowledge search/recall | 100% |
|
| 48 |
+
| cmpct | 6 | Compact/prune ledger | 100% |
|
| 49 |
+
| edge | 6 | Edge cases (self-description, ambiguous) | 100% |
|
| 50 |
+
| tran | 6 | Translation routing | 100% |
|
| 51 |
+
| info | 5 | Info/capability queries → plain text | 100% |
|
| 52 |
|
| 53 |
+
Single failure: `"Write a regex to match email addresses"` → knowledge_search instead of plain text. All other 114 cases correct across all 3 seeds (2027/2028/2029).
|
| 54 |
+
|
| 55 |
+
### Comparison: Q3_K_M vs Q4_K_M vs old 1.7B
|
| 56 |
+
|
| 57 |
+
| Model | GGUF Size | BFCL | Avg Latency | Failures |
|
| 58 |
+
|-------|-----------|------|-------------|----------|
|
| 59 |
+
| **prism-coder:2b (Q3_K_M)** | **2.3 GB** | **99.1%** | **1.79s** | **1** |
|
| 60 |
+
| prism-coder:4b (Q4_K_M) | 3.4 GB | 100% | 1.37s | 0 |
|
| 61 |
+
| old prism-coder:1b7 (Qwen3 SFT) | 2.2 GB | 90.4% | 0.30s | 11 |
|
| 62 |
+
|
| 63 |
+
The Q3_K_M replaces the old 1.7B SFT: same size (+100 MB), +9 percentage points accuracy, 10 fewer failures.
|
| 64 |
+
|
| 65 |
+
## Architecture
|
| 66 |
+
|
| 67 |
+
Qwen3.5-4B uses a hybrid attention architecture:
|
| 68 |
+
- **24 linear attention layers** (Gated DeltaNet) — O(n) inference, efficient on-device
|
| 69 |
+
- **8 full attention layers** (standard softmax) — precise routing decisions
|
| 70 |
+
|
| 71 |
+
The `2b` tag reflects the memory footprint (2.3 GB), not the parameter count (4B). No fine-tuning — the system prompt is baked into the Ollama Modelfile.
|
| 72 |
+
|
| 73 |
+
### Why not SFT?
|
| 74 |
+
|
| 75 |
+
Qwen3.5's DeltaNet linear attention layers are incompatible with LoRA fine-tuning — every SFT attempt regressed accuracy below the base model. The hybrid architecture means LoRA only reaches 8 of 32 layers. Prompt engineering achieves 99.1-100% without any weight modification.
|
| 76 |
|
| 77 |
## Fleet Position
|
| 78 |
|