FunctionGemma-270M — Mobile Actions
A fine-tune of unsloth/functiongemma-270m-it
that turns a natural-language instruction into a single, validated mobile-action tool call.
It targets two uses:
- Mobile test automation — drive an Appium / UiAutomator / Espresso / XCUITest layer from plain-English steps.
- Agentic workflows — let an on-device agent pick the next action as a structured, executable call.
At 270M parameters it runs on CPU, a modest GPU, or fully on-device (LiteRT-LM / Google AI Edge).
Repository layout
| Path | Contents |
|---|---|
merged_16bit/ |
Standalone 16-bit model (load this for inference) |
lora/ |
LoRA adapter only (apply on top of the base model) |
Output format
The model emits FunctionGemma's call DSL, e.g.:
<start_function_call>call:tap{selector:<escape>login_button<escape>}<end_function_call>
Parse it with:
import re
CALL = re.compile(r"<start_function_call>call:(\w+)\{(.*?)\}<end_function_call>", re.DOTALL)
ARG = re.compile(r"(\w+):(?:<escape>(.*?)<escape>|([^,}]*))")
def parse(text):
out = []
for name, raw in CALL.findall(text):
args = {k: (e or p).strip() for k, e, p in ARG.findall(raw)}
out.append({"name": name, "arguments": args})
return out
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
REPO = "ahmadw/functiongemma-270m-mobile"
tok = AutoTokenizer.from_pretrained(REPO, subfolder="merged_16bit")
model = AutoModelForCausalLM.from_pretrained(REPO, subfolder="merged_16bit").eval()
prompt = tok.apply_chat_template(
[{"role": "user", "content": "Tap the login button."}],
tools=TOOLS, # the schema below
tokenize=False, add_generation_prompt=True,
).removeprefix("<bos>")
inputs = tok(prompt, return_tensors="pt")
with torch.no_grad():
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
# -> <start_function_call>call:tap{selector:<escape>login_button<escape>}<end_function_call>
You must pass the tools schema (below) so the chat template declares the available actions.
Reliable tool calls (recommended)
For production, constrain decoding so the model can only emit a syntactically valid call naming one
of your tools (prevents hallucinated names). Pair the model with
xgrammar — build an EBNF grammar whose function name is
restricted to your tool list, pass it as a LogitsProcessor to generate(...), and snap any
out-of-schema name to the nearest valid tool as a dependency-free fallback.
Wiring into an automation driver
The model decides what; your driver decides how. Map each tool name to a concrete action:
HANDLERS = {
"launch_app": lambda d, app_name: d.activate_app(app_name),
"tap": lambda d, selector: d.find(selector).click(),
"type_text": lambda d, selector, text: d.find(selector).send_keys(text),
"swipe": lambda d, direction: d.swipe(direction),
"wait": lambda d, duration_ms: d.sleep(duration_ms / 1000),
# ... one entry per tool ...
}
Selectors come from the live UI. The model infers a reasonable id (e.g.
login_button) from the wording; resolve it against the real accessibility tree, or pass the available element ids in the instruction so the model can pick an exact match.
Supported actions (38 tools)
| Category | Tools |
|---|---|
| App lifecycle | launch_app, close_app |
| Touch | tap, double_tap, long_press, tap_coordinates, drag, swipe, pinch, scroll_to, pull_to_refresh |
| Text input | type_text, clear_text, paste_text, copy_text, dismiss_keyboard |
| Controls | toggle_switch, set_checkbox, select_option, set_slider, set_date, set_time, set_volume |
| Navigation | navigate_back, go_home, open_app_switcher, open_notifications, open_quick_settings, press_keyboard_key, rotate_screen |
| Synchronization | wait, wait_for_element |
| Assertions | assert_element_visible, assert_element_not_visible, assert_text_equals, assert_text_contains |
| Utility | take_screenshot, open_url |
Full tool schema (copy into TOOLS)
[
{
"type": "function",
"function": {
"name": "launch_app",
"description": "Launch a mobile application by its visible name.",
"parameters": {
"type": "object",
"properties": {
"app_name": {
"type": "string",
"description": "Visible name of the app to open."
}
},
"required": [
"app_name"
]
}
}
},
{
"type": "function",
"function": {
"name": "close_app",
"description": "Close or force stop a running application.",
"parameters": {
"type": "object",
"properties": {
"app_name": {
"type": "string",
"description": "Visible name of the app to close."
}
},
"required": [
"app_name"
]
}
}
},
{
"type": "function",
"function": {
"name": "tap",
"description": "Tap a single UI element once by its selector. Use tap_coordinates for raw screen pixels.",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "Logical id or label of the element."
}
},
"required": [
"selector"
]
}
}
},
{
"type": "function",
"function": {
"name": "double_tap",
"description": "Double tap a UI element.",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "Logical id or label of the element."
}
},
"required": [
"selector"
]
}
}
},
{
"type": "function",
"function": {
"name": "long_press",
"description": "Press and hold a UI element.",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "Logical id or label of the element."
}
},
"required": [
"selector"
]
}
}
},
{
"type": "function",
"function": {
"name": "tap_coordinates",
"description": "Tap the screen at absolute pixel x/y coordinates, with no element selector.",
"parameters": {
"type": "object",
"properties": {
"x": {
"type": "integer",
"description": "Horizontal pixel coordinate."
},
"y": {
"type": "integer",
"description": "Vertical pixel coordinate."
}
},
"required": [
"x",
"y"
]
}
}
},
{
"type": "function",
"function": {
"name": "drag",
"description": "Drag one element onto another.",
"parameters": {
"type": "object",
"properties": {
"source": {
"type": "string",
"description": "Logical id of the element to drag."
},
"target": {
"type": "string",
"description": "Logical id of the drop target."
}
},
"required": [
"source",
"target"
]
}
}
},
{
"type": "function",
"function": {
"name": "swipe",
"description": "Swipe across the screen in a direction.",
"parameters": {
"type": "object",
"properties": {
"direction": {
"type": "string",
"description": "One of: up, down, left, right."
}
},
"required": [
"direction"
]
}
}
},
{
"type": "function",
"function": {
"name": "pinch",
"description": "Pinch with two fingers to zoom in or out on an element. Does not rotate the screen.",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "Logical id of the element to zoom."
},
"direction": {
"type": "string",
"description": "Zoom direction: in or out."
}
},
"required": [
"selector",
"direction"
]
}
}
},
{
"type": "function",
"function": {
"name": "scroll_to",
"description": "Scroll until an element becomes visible.",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "Logical id of the target element."
}
},
"required": [
"selector"
]
}
}
},
{
"type": "function",
"function": {
"name": "pull_to_refresh",
"description": "Pull down from the top to refresh the screen.",
"parameters": {
"type": "object",
"properties": {},
"required": []
}
}
},
{
"type": "function",
"function": {
"name": "type_text",
"description": "Type text into an input field.",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "Logical id of the input field."
},
"text": {
"type": "string",
"description": "Text to enter into the field."
}
},
"required": [
"selector",
"text"
]
}
}
},
{
"type": "function",
"function": {
"name": "clear_text",
"description": "Clear the contents of an input field.",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "Logical id of the input field."
}
},
"required": [
"selector"
]
}
}
},
{
"type": "function",
"function": {
"name": "paste_text",
"description": "Paste the clipboard contents into an input field.",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "Logical id of the input field."
}
},
"required": [
"selector"
]
}
}
},
{
"type": "function",
"function": {
"name": "copy_text",
"description": "Copy the text of an element to the clipboard.",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "Logical id of the element to copy from."
}
},
"required": [
"selector"
]
}
}
},
{
"type": "function",
"function": {
"name": "dismiss_keyboard",
"description": "Hide the on-screen keyboard.",
"parameters": {
"type": "object",
"properties": {},
"required": []
}
}
},
{
"type": "function",
"function": {
"name": "toggle_switch",
"description": "Set a toggle or switch to a desired state.",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "Logical id of the switch."
},
"state": {
"type": "string",
"description": "Desired state: on or off."
}
},
"required": [
"selector",
"state"
]
}
}
},
{
"type": "function",
"function": {
"name": "set_checkbox",
"description": "Check or uncheck a checkbox.",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "Logical id of the checkbox."
},
"state": {
"type": "string",
"description": "Desired state: checked or unchecked."
}
},
"required": [
"selector",
"state"
]
}
}
},
{
"type": "function",
"function": {
"name": "select_option",
"description": "Select an option from a dropdown or picker.",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "Logical id of the dropdown."
},
"option": {
"type": "string",
"description": "Visible option to select."
}
},
"required": [
"selector",
"option"
]
}
}
},
{
"type": "function",
"function": {
"name": "set_slider",
"description": "Set a slider to a numeric value.",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "Logical id of the slider."
},
"value": {
"type": "integer",
"description": "Target value for the slider."
}
},
"required": [
"selector",
"value"
]
}
}
},
{
"type": "function",
"function": {
"name": "set_date",
"description": "Set a date picker to a specific date.",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "Logical id of the date field."
},
"date": {
"type": "string",
"description": "Date in YYYY-MM-DD format."
}
},
"required": [
"selector",
"date"
]
}
}
},
{
"type": "function",
"function": {
"name": "set_time",
"description": "Set a time picker to a specific time.",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "Logical id of the time field."
},
"time": {
"type": "string",
"description": "Time in HH:MM 24-hour format."
}
},
"required": [
"selector",
"time"
]
}
}
},
{
"type": "function",
"function": {
"name": "navigate_back",
"description": "Go back one step to the previous screen. Does not return to the home screen.",
"parameters": {
"type": "object",
"properties": {},
"required": []
}
}
},
{
"type": "function",
"function": {
"name": "go_home",
"description": "Return to the device home screen (launcher). This is not a single back step.",
"parameters": {
"type": "object",
"properties": {},
"required": []
}
}
},
{
"type": "function",
"function": {
"name": "open_app_switcher",
"description": "Open the recent apps or app switcher.",
"parameters": {
"type": "object",
"properties": {},
"required": []
}
}
},
{
"type": "function",
"function": {
"name": "press_keyboard_key",
"description": "Press a soft keyboard action key.",
"parameters": {
"type": "object",
"properties": {
"key": {
"type": "string",
"description": "One of: enter, search, done, next, go."
}
},
"required": [
"key"
]
}
}
},
{
"type": "function",
"function": {
"name": "open_notifications",
"description": "Open the notifications panel.",
"parameters": {
"type": "object",
"properties": {},
"required": []
}
}
},
{
"type": "function",
"function": {
"name": "open_quick_settings",
"description": "Open the quick toggles or control panel.",
"parameters": {
"type": "object",
"properties": {},
"required": []
}
}
},
{
"type": "function",
"function": {
"name": "rotate_screen",
"description": "Rotate the screen orientation between portrait and landscape. Does not zoom.",
"parameters": {
"type": "object",
"properties": {
"orientation": {
"type": "string",
"description": "Orientation: portrait or landscape."
}
},
"required": [
"orientation"
]
}
}
},
{
"type": "function",
"function": {
"name": "set_volume",
"description": "Set the media volume to a level.",
"parameters": {
"type": "object",
"properties": {
"level": {
"type": "integer",
"description": "Volume level from 0 to 100."
}
},
"required": [
"level"
]
}
}
},
{
"type": "function",
"function": {
"name": "wait_for_element",
"description": "Wait until a specific element appears, up to a timeout. Requires an element selector.",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "Logical id of the element."
},
"timeout_ms": {
"type": "integer",
"description": "Maximum wait time in milliseconds."
}
},
"required": [
"selector",
"timeout_ms"
]
}
}
},
{
"type": "function",
"function": {
"name": "wait",
"description": "Pause execution for a fixed duration in milliseconds. Does not wait for any element.",
"parameters": {
"type": "object",
"properties": {
"duration_ms": {
"type": "integer",
"description": "Duration to wait in milliseconds."
}
},
"required": [
"duration_ms"
]
}
}
},
{
"type": "function",
"function": {
"name": "assert_element_visible",
"description": "Assert that an element is visible on screen.",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "Logical id of the element."
}
},
"required": [
"selector"
]
}
}
},
{
"type": "function",
"function": {
"name": "assert_element_not_visible",
"description": "Assert that an element is not visible on screen.",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "Logical id of the element."
}
},
"required": [
"selector"
]
}
}
},
{
"type": "function",
"function": {
"name": "assert_text_equals",
"description": "Assert that an element's text exactly equals an expected value (full match).",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "Logical id of the element."
},
"expected_text": {
"type": "string",
"description": "Expected text content."
}
},
"required": [
"selector",
"expected_text"
]
}
}
},
{
"type": "function",
"function": {
"name": "assert_text_contains",
"description": "Assert that an element's text contains a substring (partial match).",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "Logical id of the element."
},
"text": {
"type": "string",
"description": "Substring expected within the element text."
}
},
"required": [
"selector",
"text"
]
}
}
},
{
"type": "function",
"function": {
"name": "take_screenshot",
"description": "Capture a screenshot of the current screen.",
"parameters": {
"type": "object",
"properties": {
"name": {
"type": "string",
"description": "File name for the screenshot."
}
},
"required": [
"name"
]
}
}
},
{
"type": "function",
"function": {
"name": "open_url",
"description": "Open a URL in the default browser.",
"parameters": {
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "Fully qualified URL to open."
}
},
"required": [
"url"
]
}
}
}
]
Evaluation
Greedy + grammar-constrained decoding, on held-out paraphrase sets the model never saw in training:
| Set | Tool selection | Exact match (name + args) |
|---|---|---|
| Wild (colloquial paraphrases) | 97.7% | 90.7% |
| Stress (high-variance, 64 cases) | 93.8% | 92.2% |
Hallucinated tool names: 0. Remaining misses cluster on indirect phrasings with no action keyword
(e.g. "make the picture smaller" -> pinch out). Accuracy is highest when the instruction names the action.
Training
- Base:
unsloth/functiongemma-270m-it - Method: 16-bit LoRA (r=16, alpha=16; q/k/v/o + gate/up/down) via Unsloth, on a free Tesla T4
- LR 2e-4, linear schedule,
adamw_8bit, 3 epochs, response-only loss masking - Synthetic instruction -> tool-call data with contrastive confusable pairs and broad phrasing variety
License
Governed by the Gemma Terms of Use. Inherits the base model license.
Model tree for ahmadw/functiongemma-270m-mobile
Base model
google/functiongemma-270m-it Finetuned
unsloth/functiongemma-270m-it