Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,32 @@
|
|
| 1 |
-
-
|
| 2 |
-
|
| 3 |
-
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Terminal-SLM: Learning Tool Actions Is Not the Same as Completing Tasks
|
| 2 |
+
|
| 3 |
+
**Track 3 — Agentic Fine-tuning**
|
| 4 |
+
|
| 5 |
+
Terminal-SLM tests whether reviewed supervised demonstrations can improve a small model's reliability on executable terminal tasks. The project uses CohereLabs Tiny Aya Global, AgentTune for its model/action loop and fine-tuning, and AuditKIT to audit recorded episodes. Five workspace-scoped tools operate on isolated repository snapshots; executable validators determine task success.
|
| 6 |
+
|
| 7 |
+
## Evaluation
|
| 8 |
+
|
| 9 |
+
The benchmark contains 50 deterministic tasks: 40 support tasks and 10 held-out tasks. The split was fixed before training-data preparation. Only reviewed support demonstrations were admitted for SFT; held-out tasks, outcomes, patches, and rewards were excluded from training.
|
| 10 |
+
|
| 11 |
+
We first evaluated the untouched base model. After support-only fine-tuning and support-based candidate selection, we ran a matched comparison using the same held-out tasks and inference configuration. We retained complete trajectories and independently checked outcomes against the task contracts.
|
| 12 |
+
|
| 13 |
+
## Results
|
| 14 |
+
|
| 15 |
+
The untouched base passed **1/10** held-out task contracts; the selected adapter passed **2/10**. However, both completed **0/5** code-repair or update tasks. Trace review also found the adapter's additional contract pass contained an inaccurate explanation. We therefore do not claim a reliable end-to-end improvement.
|
| 16 |
+
|
| 17 |
+
A later, separate task-start experiment scored **2/4** versus **0/4** for the base on four known support tasks. Its two successful full-file edits matched reviewed references and passed the named tests plus a regression test. The other two tasks failed. This follow-up was not evaluated on held-out tasks and is not evidence of generalization.
|
| 18 |
+
|
| 19 |
+
## What we learned
|
| 20 |
+
|
| 21 |
+
The adapter learned some actions when given reviewed contexts, but those gains did not translate into dependable completion from fresh task conversations. Tool-choice accuracy and training loss alone were insufficient indicators of success. The small benchmark, shared synthetic repository, and narrow validators also limit the conclusions.
|
| 22 |
+
|
| 23 |
+
This negative result is useful: it separates conditional action learning from full-agent task completion and shows why trajectory audits and executable checks matter. DPO, GRPO, and local quantized deployment were outside this submission's scope.
|
| 24 |
+
|
| 25 |
+
## Artifacts
|
| 26 |
+
|
| 27 |
+
- Code: https://github.com/devjuneja789/terminal-agent
|
| 28 |
+
- Models and adapters: https://huggingface.co/collections/devjuneja789/terminal-slm-models-and-adapters
|
| 29 |
+
- Datasets and evaluation traces: https://huggingface.co/collections/devjuneja789/terminal-slm-data-and-evaluation
|
| 30 |
+
- Results and reproduction notes: https://github.com/devjuneja789/terminal-agent/tree/main/submission
|
| 31 |
+
|
| 32 |
+
The model collection should link to the pinned base model and versioned adapters; it must not rehost the gated base weights. Dataset entries should label held-out records as evaluation-only. Tiny Aya is licensed CC-BY-NC-4.0 and requires following Cohere Labs' acceptable-use policy. Replace the placeholders only after the repositories and collections are public.
|