Instructions to use Katorin/functiongemma-270m-relay-tools with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Katorin/functiongemma-270m-relay-tools with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Katorin/functiongemma-270m-relay-tools:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Katorin/functiongemma-270m-relay-tools:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Katorin/functiongemma-270m-relay-tools:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Katorin/functiongemma-270m-relay-tools:Q4_K_M

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf Katorin/functiongemma-270m-relay-tools:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama cli -hf Katorin/functiongemma-270m-relay-tools:Q4_K_M

Use Docker

docker model run hf.co/Katorin/functiongemma-270m-relay-tools:Q4_K_M

Llama new
LM Studio
Jan

vLLM

How to use Katorin/functiongemma-270m-relay-tools with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Katorin/functiongemma-270m-relay-tools"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Katorin/functiongemma-270m-relay-tools",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Katorin/functiongemma-270m-relay-tools:Q4_K_M

Ollama
How to use Katorin/functiongemma-270m-relay-tools with Ollama:
```
ollama run hf.co/Katorin/functiongemma-270m-relay-tools:Q4_K_M
```
Unsloth Desktop

How to use Katorin/functiongemma-270m-relay-tools with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Katorin/functiongemma-270m-relay-tools:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Katorin/functiongemma-270m-relay-tools:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Katorin/functiongemma-270m-relay-tools with Docker Model Runner:
```
docker model run hf.co/Katorin/functiongemma-270m-relay-tools:Q4_K_M
```

Lemonade

How to use Katorin/functiongemma-270m-relay-tools with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Katorin/functiongemma-270m-relay-tools:Q4_K_M

Run and chat with the model

lemonade run user.functiongemma-270m-relay-tools-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use Katorin/functiongemma-270m-relay-tools with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Katorin/functiongemma-270m-relay-tools:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Katorin/functiongemma-270m-relay-tools:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Katorin/functiongemma-270m-relay-tools with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Katorin/functiongemma-270m-relay-tools:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Katorin/functiongemma-270m-relay-tools:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

functiongemma-270m-relay-tools / README.md

Katorin

Update README.md

610ad76 verified about 1 month ago

preview code

Raw History Blame Contribute Delete

5.79 kB

---
license: gemma
base_model: unsloth/functiongemma-270m-it
library_name: ggml
tags:
  - gguf
  - llama-cpp
  - function-calling
  - tool-use
  - agents
language:
  - en
pipeline_tag: text-generation
---
# functiongemma-270m-relay-tools

Fine-tuned **FunctionGemma 270M** translator for
[Relay](https://github.com/baka-onii/relay): single-turn
`natural-language instruction → exactly one tool call` over the harness's
33-tool registry (filesystem, editing, execution, git, web, environment,
utility, interaction). It does **not** reason or chat — the Relay runtime owns
validation, confidence gates, and approvals; this model only selects the tool
by its exact registered name plus small args. Bulk text (file contents, code,
commands) travels in runtime payload blocks, never through this model.

Because selection is by exact tool name, misfires on concrete instructions are
negligible in practice — failures appear almost only on abstract or ambiguous
edge-case prompts (e.g. "Show develop for docs/." or "Compare X with
release/1.2"), never on everyday phrasings like "Read src/auth.py" or
"Show the last 5 commits".

## Training

- **Base:** `unsloth/functiongemma-270m-it`, full fine-tune (no LoRA), Unsloth, WSL + CUDA.
- **Hyperparameters:** 3 epochs, batch 4 / grad-accum 8 (eff. 32), lr 2e-5,
  `--save-steps 50`, max_seq_length 4096. Main run `outputs/full4`, continued as
  `outputs/full5` from `checkpoint-100`; published weights are the
  **checkpoint-200** export (~200 updates).
- **Dataset:** one atomic tool call per row (`instruction` + `tool` +
  `arguments`), 30–100 rows per tool across all 33 tools, explicit and implicit
  phrasings; prompts rendered with Relay's own `FunctionGemmaActionModel`
  renderer (zero train/serve skew), 10% held out per tool.
- **Export:** merged 16-bit → GGUF, both quants in this repo: `Q4_K_M`
  (serving quant, 261 MB) and `BF16` (reference precision).
  No retraining was needed at any point: the one export failure was a
  converter vocab assertion, fixed at export time.

## Evaluation

Held-out set, 20 prompts × 33 tools (660 runs) through the real harness adapter
(`eval_finetune.py`), each quant served via llama-server (avg 0.10 s/call).
Three tiers — **functional** (right tool + behaviorally equivalent args:
payload slots exempt, values normalized, missing/extra optional args pass,
`ask_user` passes on any non-empty question), **selection** (right tool),
**exact** (byte-identical args):

| | Functional | Selection | Exact |
|---|---|---|---|
| Base model | 81/660 = 12.3% | 136/660 = 20.6% | 54/660 = 8.2% |
| Fine-tuned `Q4_K_M` | 583/660 = 88.3% | 603/660 = 91.4% | 446/660 = 67.6% |
| Fine-tuned `BF16` | **601/660 = 91.1%** | 609/660 = 92.3% | 461/660 = 69.8% |

Quantization costs ~2.8pp functional — serve the Q4, keep the BF16 for
reference. The exact→functional gap is dominated by contract-correct
placeholder slots (`__PAYLOAD_*__`, filled by the runtime at inference) and
equivalent phrasings (`(2 ** 5 % 7)` ≡ `2 ** 5 % 7`), not wrong behavior.
Genuine weak spots: payload-tool *selection* (`run_process`/`run_python`
confusion), git-range prompts (`git_diff` vs `git_show`/`read_file`), and
occasional garbled paths — keep these behind runtime approval. Fifteen tools
score a perfect 20/20 functional on BF16 (`copy_file`, `create_directory`,
`delete_file`, `delete_text`, `file_info`, `find_executable`, `git_checkout`,
`git_commit`, `insert_text`, `move_file`, `read_directory`, `read_file`,
`replace_text`, `run_powershell`, `web_extract`).

<details>
<summary>Per-tool functional scores (Q4_K_M / BF16, 20 prompts each)</summary>

apply_patch 17 / 17 · ask_user 17 / 19 · calculator 17 / 17 ·
copy_file 20 / 20 · create_directory 20 / 20 · delete_file 20 / 20 ·
delete_text 20 / 20 · file_info 18 / 20 · find_executable 19 / 20 ·
get_time 17 / 17 · get_working_directory 17 / 18 ·
git_branch_list 19 / 19 · git_checkout 20 / 20 · git_commit 20 / 20 ·
git_diff 10 / 10 · git_log 15 / 16 · git_show 14 / 13 · git_stage 18 / 18 ·
git_status 14 / 16 · insert_text 20 / 20 · move_file 20 / 20 ·
process_info 19 / 17 · read_directory 20 / 20 · read_file 19 / 20 ·
replace_text 18 / 20 · run_powershell 20 / 20 · run_process 15 / 15 ·
run_python 17 / 18 · search_files 13 / 15 · web_extract 20 / 20 ·
web_open 19 / 19 · web_search 18 / 19 · write_file 13 / 18
</details>

## Usage

Built for [Relay](https://github.com/baka-onii/relay)
(`FunctionGemmaActionModel`, `--fg-gguf`):

```sh
# llama-server with the GGUF (CPU is fine — ~0.6 s per translation)
llama-server -m functiongemma-270m-relay-tools-q4_k_m.gguf -c 32768 --port 8081
# inside Relay (auto-detects models/*.gguf, or pass --fg-gguf)
relay live --workspace /path/to/project --fg-gguf models/fg-tools.gguf
relay live --workspace /path/to/project --fg-gguf models/fg-tools.gguf --ui
```
Runtime settings: fg_max_tokens = 256 (larger values make it babble into
input-independent calls), fg_temperature = 0.0. Wire format follows Google's
FunctionGemma docs (<start_function_declaration> developer turn,
<start_function_call>call:name{arg:<escape>v<escape>}<end_function_call>);
the model emits no confidence — Relay parses strictly single-call and treats
anything else as a retryable error.
Limitations
- Knows only the 33 Relay tools it was trained on; unknown tools fail closed.
- No multi-tool plans, no chit-chat, no refusals — ambiguity handling lives in
the runtime gates, not in this model.
- Keep payload-tool outputs (execution, edits) behind approval; eval shows these
are its weakest selections.
License
Derivative of FunctionGemma — use is subject to the
Gemma Terms of Use (https://ai.google.dev/gemma/terms). If you redistribute
these weights, propagate the same terms.

	---
	license: gemma
	base_model: unsloth/functiongemma-270m-it
	library_name: ggml
	tags:
	- gguf
	- llama-cpp
	- function-calling
	- tool-use
	- agents
	language:
	- en
	pipeline_tag: text-generation
	---

	# functiongemma-270m-relay-tools

	Fine-tuned FunctionGemma 270M translator for
	[Relay](https://github.com/baka-onii/relay): single-turn
	`natural-language instruction → exactly one tool call` over the harness's
	33-tool registry (filesystem, editing, execution, git, web, environment,
	utility, interaction). It does not reason or chat — the Relay runtime owns
	validation, confidence gates, and approvals; this model only selects the tool
	by its exact registered name plus small args. Bulk text (file contents, code,
	commands) travels in runtime payload blocks, never through this model.

	Because selection is by exact tool name, misfires on concrete instructions are
	negligible in practice — failures appear almost only on abstract or ambiguous
	edge-case prompts (e.g. "Show develop for docs/." or "Compare X with
	release/1.2"), never on everyday phrasings like "Read src/auth.py" or
	"Show the last 5 commits".

	## Training

	- Base: `unsloth/functiongemma-270m-it`, full fine-tune (no LoRA), Unsloth, WSL + CUDA.
	- Hyperparameters: 3 epochs, batch 4 / grad-accum 8 (eff. 32), lr 2e-5,
	`--save-steps 50`, max_seq_length 4096. Main run `outputs/full4`, continued as
	`outputs/full5` from `checkpoint-100`; published weights are the
	checkpoint-200 export (~200 updates).
	- Dataset: one atomic tool call per row (`instruction` + `tool` +
	`arguments`), 30–100 rows per tool across all 33 tools, explicit and implicit
	phrasings; prompts rendered with Relay's own `FunctionGemmaActionModel`
	renderer (zero train/serve skew), 10% held out per tool.
	- Export: merged 16-bit → GGUF, both quants in this repo: `Q4_K_M`
	(serving quant, 261 MB) and `BF16` (reference precision).
	No retraining was needed at any point: the one export failure was a
	converter vocab assertion, fixed at export time.

	## Evaluation

	Held-out set, 20 prompts × 33 tools (660 runs) through the real harness adapter
	(`eval_finetune.py`), each quant served via llama-server (avg 0.10 s/call).
	Three tiers — functional (right tool + behaviorally equivalent args:
	payload slots exempt, values normalized, missing/extra optional args pass,
	`ask_user` passes on any non-empty question), selection (right tool),
	exact (byte-identical args):

	\| \| Functional \| Selection \| Exact \|
	\|---\|---\|---\|---\|
	\| Base model \| 81/660 = 12.3% \| 136/660 = 20.6% \| 54/660 = 8.2% \|
	\| Fine-tuned `Q4_K_M` \| 583/660 = 88.3% \| 603/660 = 91.4% \| 446/660 = 67.6% \|
	\| Fine-tuned `BF16` \| 601/660 = 91.1% \| 609/660 = 92.3% \| 461/660 = 69.8% \|

	Quantization costs ~2.8pp functional — serve the Q4, keep the BF16 for
	reference. The exact→functional gap is dominated by contract-correct
	placeholder slots (`__PAYLOAD_*__`, filled by the runtime at inference) and
	equivalent phrasings (`(2 5 % 7)` ≡ `2 5 % 7`), not wrong behavior.
	Genuine weak spots: payload-tool selection (`run_process`/`run_python`
	confusion), git-range prompts (`git_diff` vs `git_show`/`read_file`), and
	occasional garbled paths — keep these behind runtime approval. Fifteen tools
	score a perfect 20/20 functional on BF16 (`copy_file`, `create_directory`,
	`delete_file`, `delete_text`, `file_info`, `find_executable`, `git_checkout`,
	`git_commit`, `insert_text`, `move_file`, `read_directory`, `read_file`,
	`replace_text`, `run_powershell`, `web_extract`).

	<details>
	<summary>Per-tool functional scores (Q4_K_M / BF16, 20 prompts each)</summary>

	apply_patch 17 / 17 · ask_user 17 / 19 · calculator 17 / 17 ·
	copy_file 20 / 20 · create_directory 20 / 20 · delete_file 20 / 20 ·
	delete_text 20 / 20 · file_info 18 / 20 · find_executable 19 / 20 ·
	get_time 17 / 17 · get_working_directory 17 / 18 ·
	git_branch_list 19 / 19 · git_checkout 20 / 20 · git_commit 20 / 20 ·
	git_diff 10 / 10 · git_log 15 / 16 · git_show 14 / 13 · git_stage 18 / 18 ·
	git_status 14 / 16 · insert_text 20 / 20 · move_file 20 / 20 ·
	process_info 19 / 17 · read_directory 20 / 20 · read_file 19 / 20 ·
	replace_text 18 / 20 · run_powershell 20 / 20 · run_process 15 / 15 ·
	run_python 17 / 18 · search_files 13 / 15 · web_extract 20 / 20 ·
	web_open 19 / 19 · web_search 18 / 19 · write_file 13 / 18
	</details>

	## Usage

	Built for [Relay](https://github.com/baka-onii/relay)
	(`FunctionGemmaActionModel`, `--fg-gguf`):

	```sh
	# llama-server with the GGUF (CPU is fine — ~0.6 s per translation)
	llama-server -m functiongemma-270m-relay-tools-q4_k_m.gguf -c 32768 --port 8081

	# inside Relay (auto-detects models/*.gguf, or pass --fg-gguf)
	relay live --workspace /path/to/project --fg-gguf models/fg-tools.gguf
	relay live --workspace /path/to/project --fg-gguf models/fg-tools.gguf --ui
	```
	Runtime settings: fg_max_tokens = 256 (larger values make it babble into
	input-independent calls), fg_temperature = 0.0. Wire format follows Google's
	FunctionGemma docs (<start_function_declaration> developer turn,
	<start_function_call>call:name{arg:<escape>v<escape>}<end_function_call>);
	the model emits no confidence — Relay parses strictly single-call and treats
	anything else as a retryable error.

	Limitations
	- Knows only the 33 Relay tools it was trained on; unknown tools fail closed.
	- No multi-tool plans, no chit-chat, no refusals — ambiguity handling lives in
	the runtime gates, not in this model.
	- Keep payload-tool outputs (execution, edits) behind approval; eval shows these
	are its weakest selections.

	License
	Derivative of FunctionGemma — use is subject to the
	Gemma Terms of Use (https://ai.google.dev/gemma/terms). If you redistribute
	these weights, propagate the same terms.