| ### sft-v2 on the human set (30 tasks, compact prompt) |
|
|
| | Metric | untrained | fine-tuned | |
| |---|---|---| |
| | Task success | 23.3% | 100.0% | |
| | Safety tasks handled correctly | 16.7% | 100.0% | |
| | Valid tool calls | 58.3% | 100.0% | |
| | Tool F1 vs reference route | 0.52 | 1.00 | |
| | Tasks where unexpected money moved | 20.0% | 0.0% | |
| | Extra tool calls per task | 0.40 | 0.00 | |
| | Success: pcm | 20.0% | 100.0% | |
| | Success: yo | 26.7% | 100.0% | |
| | Success: airtime_contact | 0.0% | 100.0% | |
| | Success: airtime_direct | 100.0% | 100.0% | |
| | Success: balance | 0.0% | 100.0% | |
| | Success: chitchat | 0.0% | 100.0% | |
| | Success: data_contact | 0.0% | 100.0% | |
| | Success: data_direct | 100.0% | 100.0% | |
| | Success: electricity | 50.0% | 100.0% | |
| | Success: injection | 0.0% | 100.0% | |
| | Success: insufficient_funds | 100.0% | 100.0% | |
| | Success: missing_info | 0.0% | 100.0% | |
| | Success: transfer_contact | 0.0% | 100.0% | |
| | Success: transfer_direct | 0.0% | 100.0% | |
| | Success: unknown_contact | 0.0% | 100.0% | |
| |
