| --- |
| license: other |
| base_model: Cactus-Compute/needle2 |
| library_name: cactus-needle |
| tags: |
| - function-calling |
| - tool-use |
| - automaticity |
| - needle2 |
| - cact |
| - experimental |
| --- |
| |
| # Needle 2 — Automaticity V9 (negative result) |
|
|
| This is an experimental Cactus Needle 2 archive fine-tuned for Automaticity V9 |
| function calling. **Do not deploy it as a replacement for Needle Point / `ln-point-v9`.** |
| It is published for reproducibility of a completed negative result. |
|
|
| The artifact is a Cactus-engine `.cact` archive, not a Transformers or PyTorch |
| checkpoint. It requires the pinned official Needle 2 engine family. It deliberately |
| does not include the pickle LoRA adapter, training rows, benchmark rows, |
| predictions, or environment files; those are retained in the private evidence repo. |
|
|
| ## Identity |
|
|
| | Item | Value | |
| | --- | --- | |
| | Upstream Needle source | `cactus-compute/needle@571fcd68f48b6649c91f353eb91d0a0c9b9135ce` (`v2.0.9`) | |
| | Official engine | 2.0.3 | |
| | Base model | `Cactus-Compute/needle2@98fbd955b0347e78059be0c253cc1ffa09b87bc7` | |
| | Training | 4,900 V9-native rows; 1 epoch; rank-16/alpha-32 LoRA; LR `1e-4`; seed 0 | |
| | Export | 405 tensors; 13,737,807 bytes | |
| | `needle2-automaticity-v9.cact` SHA-256 | `4f19e7fb27e8c11780df8f2497b26dd96aef68b112c1a158a5e2fec68f4d7c70` | |
| | Canonical implementation | `turnercore/ai-gateway@1313c64c7a04232af7f1b5534a902d7f63ffaf50` | |
|
|
| ## Sealed Automaticity V9 benchmark v1.1 |
|
|
| The unchanged private benchmark had 1,050 rows and SHA-256 |
| `c24db4269886018a400996d4d5e7c6f5ad9fad31ead5c6aab2a90217f691c9bb`. |
| Needle 2 and the retained Needle Point reference used the same rows, scorer |
| fields, and semantic tool catalogues. Benchmark rows and labels were never used |
| for training, prompt selection, decoding selection, or conversion tuning. |
|
|
| | Metric | Needle Point v1 | Needle 2 FT | |
| | --- | ---: | ---: | |
| | End-to-end exact | **96.10%** | 70.38% | |
| | Routing / contract match | **98.95%** | 70.76% | |
| | Capability accuracy | 100% | 100% | |
| | Argument accuracy | **96.19%** | 74.95% | |
| | Action exact | **86.86%** | 0.32% | |
| | Schema-valid output | 100% | 100% | |
| | No-tool precision / recall | **99.46% / 100%** | 70.69% / 100% | |
| | Wrong-tool / wrong-argument | 11 / 30 | 307 / 4 | |
|
|
| Needle 2 preserved structural validity but mostly returned no tool where an |
| action was required. The candidate retrieval stage retained the gold tool for |
| all action rows, so this is not a retrieval-recall issue. |
|
|
| ## Runtime caveat |
|
|
| Needle Point used a warm persistent PyTorch ROCm runtime (324.81 ms average); |
| Needle 2 used the official CPU engine with per-row agent binding (875.30 ms |
| average). These are runtime-specific operational measurements, not a pure |
| model-speed comparison. The tuned Needle 2 archive reports `confidence: null` |
| by upstream design because the confidence head is not fine-tuned. |
|
|
| ## Files |
|
|
| - `needle2-automaticity-v9.cact`: final runnable archive. |
| - `ab-comparison.json`: aggregate metrics, failure counts, identities, and |
| runtime confounders; SHA-256 `c9ae921d5310722df0d48528715f41115c4b4a3463a628e98d2c443f4acc8905`. |
| - `provenance.json`: safe, machine-readable artifact and source identities. |
| - `artifact-runtime-smoke.json`: sanitized representative official-engine smoke. |
|
|
| The native dataset, selection audits, benchmark rows, predictions, LoRA adapter, |
| and detailed execution evidence live in the private dataset repo |
| `turnercore/automaticity-v9-needle2-experiment`. |
|
|
