|
Download README.md from devjuneja789/Terminal-SLM: direct link, hf CLI and curl.
- Browser
- Download file 3.19 kB
-
https://huggingface.co/devjuneja789/Terminal-SLM/resolve/main/README.md
- Command line
-
hf download hf://devjuneja789/Terminal-SLM/README.md
-
curl -L -o README.md https://huggingface.co/devjuneja789/Terminal-SLM/resolve/main/README.md
3.19 kB
| license: cc-by-nc-4.0 | |
| base_model: CohereLabs/tiny-aya-global | |
| tags: [terminal-slm, agenttune, auditkit, hackathon, track-3] | |
| # Terminal-SLM: Learning Tool Actions Is Not the Same as Completing Tasks | |
| **Track 3 — Agentic Fine-tuning** | |
| Terminal-SLM tests whether reviewed supervised demonstrations can improve a small model's reliability on executable terminal tasks. The project uses CohereLabs Tiny Aya Global, AgentTune for its model/action loop and fine-tuning, and AuditKIT to audit recorded episodes. Five workspace-scoped tools operate on isolated repository snapshots; executable validators determine task success. | |
| ## Evaluation | |
| The benchmark contains 50 deterministic tasks: 40 support tasks and 10 held-out tasks. The split was fixed before training-data preparation. Only reviewed support demonstrations were admitted for SFT; held-out tasks, outcomes, patches, and rewards were excluded from training. | |
| We first evaluated the untouched base model. After support-only fine-tuning and support-based candidate selection, we ran a matched comparison using the same held-out tasks and inference configuration. We retained complete trajectories and independently checked outcomes against the task contracts. | |
| ## Results | |
| The untouched base passed **1/10** held-out task contracts; the selected adapter passed **2/10**. However, both completed **0/5** code-repair or update tasks. Trace review also found the adapter's additional contract pass contained an inaccurate explanation. We therefore do not claim a reliable end-to-end improvement. | |
| A later, separate task-start experiment scored **2/4** versus **0/4** for the base on four known support tasks. Its two successful full-file edits matched reviewed references and passed the named tests plus a regression test. The other two tasks failed. This follow-up was not evaluated on held-out tasks and is not evidence of generalization. | |
| ## What we learned | |
| The adapter learned some actions when given reviewed contexts, but those gains did not translate into dependable completion from fresh task conversations. Tool-choice accuracy and training loss alone were insufficient indicators of success. The small benchmark, shared synthetic repository, and narrow validators also limit the conclusions. | |
| This negative result is useful: it separates conditional action learning from full-agent task completion and shows why trajectory audits and executable checks matter. DPO, GRPO, and local quantized deployment were outside this submission's scope. | |
| ## Artifacts | |
| - Code: https://github.com/devjuneja789/terminal-agent | |
| - Models and adapters: https://huggingface.co/collections/devjuneja789/terminal-slm-models-and-adapters | |
| - Datasets and evaluation traces: https://huggingface.co/collections/devjuneja789/terminal-slm-data-and-evaluation | |
| - Results and reproduction notes: https://github.com/devjuneja789/terminal-agent/tree/main/submission | |
| The model collection should link to the pinned base model and versioned adapters; it must not rehost the gated base weights. Dataset entries should label held-out records as evaluation-only. Tiny Aya is licensed CC-BY-NC-4.0 and requires following Cohere Labs' acceptable-use policy. | |