File size: 3,188 Bytes
c980303
 
 
 
 
1338baf
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
b31c751
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
---
license: cc-by-nc-4.0
base_model: CohereLabs/tiny-aya-global
tags: [terminal-slm, agenttune, auditkit, hackathon, track-3]
---
# Terminal-SLM: Learning Tool Actions Is Not the Same as Completing Tasks

**Track 3 — Agentic Fine-tuning**

Terminal-SLM tests whether reviewed supervised demonstrations can improve a small model's reliability on executable terminal tasks. The project uses CohereLabs Tiny Aya Global, AgentTune for its model/action loop and fine-tuning, and AuditKIT to audit recorded episodes. Five workspace-scoped tools operate on isolated repository snapshots; executable validators determine task success.

## Evaluation

The benchmark contains 50 deterministic tasks: 40 support tasks and 10 held-out tasks. The split was fixed before training-data preparation. Only reviewed support demonstrations were admitted for SFT; held-out tasks, outcomes, patches, and rewards were excluded from training.

We first evaluated the untouched base model. After support-only fine-tuning and support-based candidate selection, we ran a matched comparison using the same held-out tasks and inference configuration. We retained complete trajectories and independently checked outcomes against the task contracts.

## Results

The untouched base passed **1/10** held-out task contracts; the selected adapter passed **2/10**. However, both completed **0/5** code-repair or update tasks. Trace review also found the adapter's additional contract pass contained an inaccurate explanation. We therefore do not claim a reliable end-to-end improvement.

A later, separate task-start experiment scored **2/4** versus **0/4** for the base on four known support tasks. Its two successful full-file edits matched reviewed references and passed the named tests plus a regression test. The other two tasks failed. This follow-up was not evaluated on held-out tasks and is not evidence of generalization.

## What we learned

The adapter learned some actions when given reviewed contexts, but those gains did not translate into dependable completion from fresh task conversations. Tool-choice accuracy and training loss alone were insufficient indicators of success. The small benchmark, shared synthetic repository, and narrow validators also limit the conclusions.

This negative result is useful: it separates conditional action learning from full-agent task completion and shows why trajectory audits and executable checks matter. DPO, GRPO, and local quantized deployment were outside this submission's scope.

## Artifacts

- Code: https://github.com/devjuneja789/terminal-agent
- Models and adapters: https://huggingface.co/collections/devjuneja789/terminal-slm-models-and-adapters
- Datasets and evaluation traces: https://huggingface.co/collections/devjuneja789/terminal-slm-data-and-evaluation
- Results and reproduction notes: https://github.com/devjuneja789/terminal-agent/tree/main/submission

The model collection should link to the pinned base model and versioned adapters; it must not rehost the gated base weights. Dataset entries should label held-out records as evaluation-only. Tiny Aya is licensed CC-BY-NC-4.0 and requires following Cohere Labs' acceptable-use policy.