Update README.md

README.md

@@ -1,3 +1,122 @@
 ---
 license: gemma
+base_model: unsloth/functiongemma-270m-it
+library_name: ggml
+tags:
+  - gguf
+  - llama-cpp
+  - function-calling
+  - tool-use
+  - agents
+language:
+  - en
+pipeline_tag: text-generation
 ---
+
+# functiongemma-270m-relay-tools
+
+Fine-tuned **FunctionGemma 270M** translator for
+[Relay](https://github.com/baka-onii/relay): single-turn
+`natural-language instruction → exactly one tool call` over the harness's
+33-tool registry (filesystem, editing, execution, git, web, environment,
+utility, interaction). It does **not** reason or chat — the Relay runtime owns
+validation, confidence gates, and approvals; this model only selects the tool
+by its exact registered name plus small args. Bulk text (file contents, code,
+commands) travels in runtime payload blocks, never through this model.
+
+Because selection is by exact tool name, misfires on concrete instructions are
+negligible in practice — failures appear almost only on abstract or ambiguous
+edge-case prompts (e.g. "Show develop for docs/." or "Compare X with
+release/1.2"), never on everyday phrasings like "Read src/auth.py" or
+"Show the last 5 commits".
+
+## Training
+
+- **Base:** `unsloth/functiongemma-270m-it`, full fine-tune (no LoRA), Unsloth, WSL + CUDA.
+- **Hyperparameters:** 3 epochs, batch 4 / grad-accum 8 (eff. 32), lr 2e-5,
+  `--save-steps 50`, max_seq_length 4096. Main run `outputs/full4`, continued as
+  `outputs/full5` from `checkpoint-100`; published weights are the
+  **checkpoint-200** export (~200 updates).
+- **Dataset:** one atomic tool call per row (`instruction` + `tool` +
+  `arguments`), 30–100 rows per tool across all 33 tools, explicit and implicit
+  phrasings; prompts rendered with Relay's own `FunctionGemmaActionModel`
+  renderer (zero train/serve skew), 10% held out per tool.
+- **Export:** merged 16-bit → GGUF, both quants in this repo: `Q4_K_M`
+  (serving quant, 261 MB) and `BF16` (reference precision).
+  No retraining was needed at any point: the one export failure was a
+  converter vocab assertion, fixed at export time.
+
+## Evaluation
+
+Held-out set, 20 prompts × 33 tools (660 runs) through the real harness adapter
+(`eval_finetune.py`), each quant served via llama-server (avg 0.10 s/call).
+Three tiers — **functional** (right tool + behaviorally equivalent args:
+payload slots exempt, values normalized, missing/extra optional args pass,
+`ask_user` passes on any non-empty question), **selection** (right tool),
+**exact** (byte-identical args):
+
+| | Functional | Selection | Exact |
+|---|---|---|---|
+| Base model | 81/660 = 12.3% | 136/660 = 20.6% | 54/660 = 8.2% |
+| Fine-tuned `Q4_K_M` | 583/660 = 88.3% | 603/660 = 91.4% | 446/660 = 67.6% |
+| Fine-tuned `BF16` | **601/660 = 91.1%** | 609/660 = 92.3% | 461/660 = 69.8% |
+
+Quantization costs ~2.8pp functional — serve the Q4, keep the BF16 for
+reference. The exact→functional gap is dominated by contract-correct
+placeholder slots (`__PAYLOAD_*__`, filled by the runtime at inference) and
+equivalent phrasings (`(2 ** 5 % 7)` ≡ `2 ** 5 % 7`), not wrong behavior.
+Genuine weak spots: payload-tool *selection* (`run_process`/`run_python`
+confusion), git-range prompts (`git_diff` vs `git_show`/`read_file`), and
+occasional garbled paths — keep these behind runtime approval. Fifteen tools
+score a perfect 20/20 functional on BF16 (`copy_file`, `create_directory`,
+`delete_file`, `delete_text`, `file_info`, `find_executable`, `git_checkout`,
+`git_commit`, `insert_text`, `move_file`, `read_directory`, `read_file`,
+`replace_text`, `run_powershell`, `web_extract`).
+
+<details>
+<summary>Per-tool functional scores (Q4_K_M / BF16, 20 prompts each)</summary>
+
+apply_patch 17 / 17 · ask_user 17 / 19 · calculator 17 / 17 ·
+copy_file 20 / 20 · create_directory 20 / 20 · delete_file 20 / 20 ·
+delete_text 20 / 20 · file_info 18 / 20 · find_executable 19 / 20 ·
+get_time 17 / 17 · get_working_directory 17 / 18 ·
+git_branch_list 19 / 19 · git_checkout 20 / 20 · git_commit 20 / 20 ·
+git_diff 10 / 10 · git_log 15 / 16 · git_show 14 / 13 · git_stage 18 / 18 ·
+git_status 14 / 16 · insert_text 20 / 20 · move_file 20 / 20 ·
+process_info 19 / 17 · read_directory 20 / 20 · read_file 19 / 20 ·
+replace_text 18 / 20 · run_powershell 20 / 20 · run_process 15 / 15 ·
+run_python 17 / 18 · search_files 13 / 15 · web_extract 20 / 20 ·
+web_open 19 / 19 · web_search 18 / 19 · write_file 13 / 18
+</details>
+
+## Usage
+
+Built for [Relay](https://github.com/baka-onii/relay)
+(`FunctionGemmaActionModel`, `--fg-gguf`):
+
+```sh
+# llama-server with the GGUF (CPU is fine — ~0.6 s per translation)
+llama-server -m functiongemma-270m-relay-tools-q4_k_m.gguf -c 32768 --port 8081
+
+# inside Relay (auto-detects models/*.gguf, or pass --fg-gguf)
+relay live --workspace /path/to/project --fg-gguf models/fg-tools.gguf
+relay live --workspace /path/to/project --fg-gguf models/fg-tools.gguf --ui
+```
+Runtime settings: fg_max_tokens = 256 (larger values make it babble into
+input-independent calls), fg_temperature = 0.0. Wire format follows Google's
+FunctionGemma docs (<start_function_declaration> developer turn,
+<start_function_call>call:name{arg:<escape>v<escape>}<end_function_call>);
+the model emits no confidence — Relay parses strictly single-call and treats
+anything else as a retryable error.
+
+Limitations
+- Knows only the 33 Relay tools it was trained on; unknown tools fail closed.
+- No multi-tool plans, no chit-chat, no refusals — ambiguity handling lives in
+the runtime gates, not in this model.
+- Keep payload-tool outputs (execution, edits) behind approval; eval shows these
+are its weakest selections.
+
+License
+Derivative of FunctionGemma — use is subject to the
+Gemma Terms of Use (https://ai.google.dev/gemma/terms). If you redistribute
+these weights, propagate the same terms.