dcostenco commited on
Commit
41e04f6
·
verified ·
1 Parent(s): b03f587

Add full BFCL test results, per-category breakdown, architecture notes

Browse files
Files changed (1) hide show
  1. README.md +41 -5
README.md CHANGED
@@ -22,21 +22,57 @@ Qwen3.5-4B quantized to Q3_K_M (2.3 GB) for on-device MCP tool routing on iPhone
22
 
23
  ```bash
24
  ollama pull dcostenco/prism-coder:2b
 
25
  ```
26
 
27
  ## Download
28
 
29
  | File | Size | BFCL | SHA256 |
30
  |------|------|------|--------|
31
- | [Qwen3.5-4B-Q3_K_M.gguf](https://huggingface.co/dcostenco/prism-coder-4b/resolve/main/Qwen3.5-4B-Q3_K_M.gguf) | 2.3 GB | 99.1% | `d6981ab4d77b...` |
32
 
33
- The GGUF is hosted in the [prism-coder-4b](https://huggingface.co/dcostenco/prism-coder-4b) repo alongside the Q4_K_M variant.
34
 
35
- ## Why 2B?
36
 
37
- This is a Qwen3.5-**4B** model quantized to Q3_K_M. The `2b` tag reflects its memory footprint (2.3 GB), not the parameter count. It fits in the same RAM as a 2B model while delivering 4B-class routing accuracy.
 
 
 
 
 
 
 
 
 
 
 
 
 
38
 
39
- No fine-tuning — the system prompt IS the specialization, baked into the Ollama Modelfile.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
40
 
41
  ## Fleet Position
42
 
 
22
 
23
  ```bash
24
  ollama pull dcostenco/prism-coder:2b
25
+ ollama run dcostenco/prism-coder:2b
26
  ```
27
 
28
  ## Download
29
 
30
  | File | Size | BFCL | SHA256 |
31
  |------|------|------|--------|
32
+ | [Qwen3.5-4B-Q3_K_M.gguf](https://huggingface.co/dcostenco/prism-coder-4b/resolve/main/Qwen3.5-4B-Q3_K_M.gguf) | 2.3 GB | 99.1% | `d6981ab4d77ba712b48ef69d69042d75b5e39b9dce5fb5a5b054fd08e06afb95` |
33
 
34
+ ## BFCL Benchmark — 99.1% × 3 Seeds
35
 
36
+ **114/115 × 3 shuffled runs = 99.1%, 1 flaky case, stable across all seeds**
37
 
38
+ | Category | Count | Description | Accuracy |
39
+ |----------|------:|-------------|:--------:|
40
+ | save | 17 | session_save_ledger logging | 100% |
41
+ | smem | 17 | Session memory search/recall | 100% |
42
+ | aac | 12 | AAC phrase/communication routing | 100% |
43
+ | hand | 12 | Agent handoff (session_save_handoff) | 100% |
44
+ | irrel | 10 | Irrelevant/general questions → plain text | 90% |
45
+ | load | 9 | Context loading (session_load_context) | 100% |
46
+ | pred | 8 | Prediction/task routing | 100% |
47
+ | know | 7 | Knowledge search/recall | 100% |
48
+ | cmpct | 6 | Compact/prune ledger | 100% |
49
+ | edge | 6 | Edge cases (self-description, ambiguous) | 100% |
50
+ | tran | 6 | Translation routing | 100% |
51
+ | info | 5 | Info/capability queries → plain text | 100% |
52
 
53
+ Single failure: `"Write a regex to match email addresses"` → knowledge_search instead of plain text. All other 114 cases correct across all 3 seeds (2027/2028/2029).
54
+
55
+ ### Comparison: Q3_K_M vs Q4_K_M vs old 1.7B
56
+
57
+ | Model | GGUF Size | BFCL | Avg Latency | Failures |
58
+ |-------|-----------|------|-------------|----------|
59
+ | **prism-coder:2b (Q3_K_M)** | **2.3 GB** | **99.1%** | **1.79s** | **1** |
60
+ | prism-coder:4b (Q4_K_M) | 3.4 GB | 100% | 1.37s | 0 |
61
+ | old prism-coder:1b7 (Qwen3 SFT) | 2.2 GB | 90.4% | 0.30s | 11 |
62
+
63
+ The Q3_K_M replaces the old 1.7B SFT: same size (+100 MB), +9 percentage points accuracy, 10 fewer failures.
64
+
65
+ ## Architecture
66
+
67
+ Qwen3.5-4B uses a hybrid attention architecture:
68
+ - **24 linear attention layers** (Gated DeltaNet) — O(n) inference, efficient on-device
69
+ - **8 full attention layers** (standard softmax) — precise routing decisions
70
+
71
+ The `2b` tag reflects the memory footprint (2.3 GB), not the parameter count (4B). No fine-tuning — the system prompt is baked into the Ollama Modelfile.
72
+
73
+ ### Why not SFT?
74
+
75
+ Qwen3.5's DeltaNet linear attention layers are incompatible with LoRA fine-tuning — every SFT attempt regressed accuracy below the base model. The hybrid architecture means LoRA only reaches 8 of 32 layers. Prompt engineering achieves 99.1-100% without any weight modification.
76
 
77
  ## Fleet Position
78