Upload BENCHMARK.md with huggingface_hub

BENCHMARK.md

@@ -54,20 +54,20 @@ reproducibility and audit.
 
 | Rank | Model | Params | Size (MB) | JSON Valid | Op Correct | Exact Match | Latency (ms) |
 |---|---|---|---|---|---|---|---|
-| 1 | **[SAM]** SAM | 35.9M | 137.0 | 84.0% | 31.5% | 8.5% | 691.6 |
-| 2 |  Qwen2.5-3B | 3085.9M | 5886.0 | 100.0% | 81.5% | 17.5% | 2716.7 |
-| 3 |  DeepSeek-R1-Qwen3-8B-GGUF | 8000.0M | 4794.9 | 98.5% | 79.5% | 11.5% | 37642.4 |
-| 4 |  Qwen2.5-1.5B | 1543.7M | 2944.4 | 100.0% | 75.0% | 9.5% | 1976.9 |
-| 5 |  TinyLlama-1.1B | 1100.0M | 2098.2 | 93.5% | 57.0% | 9.0% | 4484.0 |
-| 6 |  Qwen2.5-0.5B | 494.0M | 942.3 | 71.5% | 28.5% | 2.5% | 2108.7 |
-| 7 |  SmolLM2-1.7B | 1711.4M | 3264.2 | 85.0% | 47.0% | 2.5% | 1884.5 |
-| 8 |  SmolLM2-360M | 361.8M | 690.1 | 98.5% | 0.0% | 0.0% | 1027.1 |
+| 1 | **[SAM]** SAM | 33.4M | 127.4 | 88.0% | 63.5% | 18.0% | 715.4 |
+| 2 |  Qwen2.5-3B | 3085.9M | 5886.0 | 100.0% | 82.5% | 16.0% | 2061.1 |
+| 3 |  DeepSeek-R1-Qwen3-8B-GGUF | 8000.0M | 4794.9 | 98.0% | 79.0% | 11.5% | 31737.9 |
+| 4 |  TinyLlama-1.1B | 1100.0M | 2098.2 | 92.0% | 55.5% | 9.0% | 4199.3 |
+| 5 |  Qwen2.5-1.5B | 1543.7M | 2944.4 | 100.0% | 78.5% | 9.0% | 1852.4 |
+| 6 |  Qwen2.5-0.5B | 494.0M | 942.3 | 70.5% | 30.0% | 2.5% | 1937.2 |
+| 7 |  SmolLM2-1.7B | 1711.4M | 3264.2 | 89.0% | 38.5% | 2.5% | 1694.9 |
+| 8 |  SmolLM2-360M | 361.8M | 690.1 | 89.0% | 0.0% | 0.0% | 2042.9 |
 | 9 |  Llama-3.2-1B | ? | 0.0 | 0.0% | 0.0% | 0.0% | 0.0 |
 | 10 |  Gemma-2-2B | ? | 0.0 | 0.0% | 0.0% | 0.0% | 0.0 |
 | 11 |  Llama-3.2-3B | ? | 0.0 | 0.0% | 0.0% | 0.0% | 0.0 |
 | 12 |  Phi-3.5-mini-3.8B | ? | 0.0 | 0.0% | 0.0% | 0.0% | 0.0 |
 | 13 |  Llama-3.2-3B-uns | ? | 0.0 | 0.0% | 0.0% | 0.0% | 0.0 |
-| 14 |  Gemma-3-270M-GGUF | 270.0M | 241.4 | 92.0% | 2.5% | 0.0% | 6871.5 |
+| 14 |  Gemma-3-270M-GGUF | 270.0M | 241.4 | 92.0% | 2.0% | 0.0% | 6322.5 |
 | 15 |  Ministral-3-8B-GGUF | ? | 0.0 | 0.0% | 0.0% | 0.0% | 0.0 |
 
 ## Per-tier exact-match breakdown (all models)
@@ -76,11 +76,11 @@ _This is the breakdown that matters for fair comparison. Each model's exact-matc
 
 | Model | T1 atomic | T2 compound | T3 noisy | T4 long | T5 cross |
 |---|---|---|---|---|---|
-| **[SAM]** SAM | 20.0% (n=40) | 0.0% (n=40) | 22.5% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
-| Qwen2.5-3B | 40.0% (n=40) | 20.0% (n=40) | 27.5% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
+| **[SAM]** SAM | 37.5% (n=40) | 27.5% (n=40) | 25.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
+| Qwen2.5-3B | 40.0% (n=40) | 12.5% (n=40) | 27.5% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
 | DeepSeek-R1-Qwen3-8B-GGUF | 27.5% (n=40) | 7.5% (n=40) | 22.5% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
-| Qwen2.5-1.5B | 30.0% (n=40) | 15.0% (n=40) | 2.5% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
 | TinyLlama-1.1B | 27.5% (n=40) | 12.5% (n=40) | 5.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
+| Qwen2.5-1.5B | 30.0% (n=40) | 12.5% (n=40) | 2.5% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
 | Qwen2.5-0.5B | 12.5% (n=40) | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
 | SmolLM2-1.7B | 12.5% (n=40) | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
 | SmolLM2-360M | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
@@ -90,25 +90,25 @@ _This is the breakdown that matters for fair comparison. Each model's exact-matc
 
 | Model | T1 atomic | T2 compound | T3 noisy | T4 long | T5 cross |
 |---|---|---|---|---|---|
-| **[SAM]** SAM | 92.5% | 80.0% | 77.5% | 70.0% | 100.0% |
+| **[SAM]** SAM | 100.0% | 100.0% | 100.0% | 75.0% | 65.0% |
 | Qwen2.5-3B | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% |
-| DeepSeek-R1-Qwen3-8B-GGUF | 100.0% | 95.0% | 100.0% | 97.5% | 100.0% |
+| DeepSeek-R1-Qwen3-8B-GGUF | 100.0% | 95.0% | 100.0% | 95.0% | 100.0% |
+| TinyLlama-1.1B | 100.0% | 90.0% | 100.0% | 70.0% | 100.0% |
 | Qwen2.5-1.5B | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% |
-| TinyLlama-1.1B | 100.0% | 90.0% | 100.0% | 77.5% | 100.0% |
-| Qwen2.5-0.5B | 60.0% | 65.0% | 67.5% | 100.0% | 65.0% |
-| SmolLM2-1.7B | 92.5% | 72.5% | 100.0% | 60.0% | 100.0% |
-| SmolLM2-360M | 100.0% | 100.0% | 100.0% | 92.5% | 100.0% |
+| Qwen2.5-0.5B | 60.0% | 65.0% | 67.5% | 95.0% | 65.0% |
+| SmolLM2-1.7B | 92.5% | 80.0% | 100.0% | 72.5% | 100.0% |
+| SmolLM2-360M | 100.0% | 100.0% | 100.0% | 45.0% | 100.0% |
 | Gemma-3-270M-GGUF | 85.0% | 97.5% | 90.0% | 87.5% | 100.0% |
 
 ## Per-domain exact-match breakdown (all models)
 
 | Model | CAL (n=2) | DB (n=27) | ECOMMERCE (n=8) | FILE (n=4) | HOME (n=7) | HTTP (n=25) | MQTT (n=15) | ROS (n=46) | VEHICLE (n=17) | WORKFLOW (n=49) |
 |---|---|---|---|---|---|---|---|---|---|---|
-| **[SAM]** SAM | 50.0% | 3.7% | 0.0% | 0.0% | 0.0% | 40.0% | 0.0% | 4.3% | 17.6% | 0.0% |
-| Qwen2.5-3B | 50.0% | 18.5% | 0.0% | 0.0% | 0.0% | 8.0% | 73.3% | 28.3% | 17.6% | 0.0% |
+| **[SAM]** SAM | 100.0% | 11.1% | 0.0% | 75.0% | 0.0% | 44.0% | 6.7% | 10.9% | 17.6% | 16.3% |
+| Qwen2.5-3B | 50.0% | 18.5% | 0.0% | 0.0% | 0.0% | 8.0% | 73.3% | 21.7% | 17.6% | 0.0% |
 | DeepSeek-R1-Qwen3-8B-GGUF | 50.0% | 0.0% | 0.0% | 25.0% | 0.0% | 8.0% | 73.3% | 10.9% | 17.6% | 0.0% |
-| Qwen2.5-1.5B | 0.0% | 18.5% | 0.0% | 0.0% | 0.0% | 4.0% | 0.0% | 21.7% | 17.6% | 0.0% |
 | TinyLlama-1.1B | 0.0% | 18.5% | 0.0% | 25.0% | 0.0% | 0.0% | 0.0% | 17.4% | 23.5% | 0.0% |
+| Qwen2.5-1.5B | 0.0% | 18.5% | 0.0% | 0.0% | 0.0% | 4.0% | 0.0% | 19.6% | 17.6% | 0.0% |
 | Qwen2.5-0.5B | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 4.3% | 17.6% | 0.0% |
 | SmolLM2-1.7B | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 4.3% | 17.6% | 0.0% |
 | SmolLM2-360M | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |