File size: 5,594 Bytes
d599418 03061db d599418 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 | ---
language:
- my
- en
license: apache-2.0
tags:
- functiongemma
- function-calling
- tool-use
- burmese
- myanmar
- on-device
- gemma
base_model: google/functiongemma-270m-it
datasets:
- thantzinphyo/Hey-Function-Burmese
pipeline_tag: text-generation
---
<div align="center">
<img src="hey_avatar.png" width="108" alt="Hey-Function-270M Avatar">
<h1>Hey-Function-270M</h1>
<p><strong>A Lightweight On-Device Burmese & English Function-Calling Agent</strong></p>
</div>
---
## Overview
Hey-Function-270M Alpha Version is an on-device function-calling agent designed for lightweight platforms (smartphones, IoT, edge boards). It enables applications to map Burmese, English, and Code-Switching natural language instructions into structured tool calls.
> [!NOTE]
> **Release:** This checkpoint serves as an initial proof-of-concept to evaluate Burmese function-calling feasibility on a compact 270M-parameter architecture.
- **Base Model:** Google/FunctionGemma-270M-IT
- **Dataset:** [thantzinphyo/Hey-Function-Burmese](https://huggingface.co/datasets/thantzinphyo/Hey-Function-Burmese)
- **Training Stage:** Initial Baseline (1 Epoch)
---
## Weight Adaptation & Training Methodology
How the model weights were updated and retrained:
- **Full-Parameter Tuning:**
- All 270M transformer decoder weights were unfrozen and updated without parameter-efficient adapters (such as LoRA).
- This was done because the original base model's tool-calling capability was trained almost exclusively in English.
- Full adaptation allowed attention representations to align with Myanmar Subject-Object-Verb (SOV) sentence structures and entity schemas.
- **Dynamic Candidate Collation:**
- Paired target ground-truth tools with 2 negative distractors per sample to prevent 61-tool schema overflow.
- Fitted comfortably within an extended 2,800-token context budget.
- Achieved 0.0% sequence truncation across the entire 42.6K dataset.
- **Response-Only Loss Masking:**
- Masked all system prompt, tool schema, and user query tokens with label -100.
- Gradients were computed strictly on the model's generated function call tokens (<start_function_call>call:...<end_function_call>).
- **Optimization:**
- Trained on a NVIDIA Tesla T4 GPU (16 GB VRAM).
- Effective batch size of 32 (micro-batch size 1 with 32 gradient accumulation steps).
- AdamW optimizer with FP16 mixed precision.
- Learning rate of 2e-5 with 36-step linear warmup and linear decay across 1 full epoch (1,214 steps).
- Zero-copy chunked cross-entropy (chunk_size=256) kept peak VRAM usage under 5.0 GB.
---
## Evaluation Benchmark
| Register / Language | Samples | Fn Acc (%) | Arg Acc (%) | Exact Match (EM) |
| :--- | :---: | :---: | :---: | :---: |
| **English** | 71 | 69.0% | 67.6% | 67.6% |
| **Burmese** | 47 | 44.7% | 44.7% | 44.7% |
| **Code-Switch** | 73 | 52.1% | 52.1% | 52.1% |
| **Overall** | **191** | **56.5%** | **56.0%** | **56.0%** |
- **Fn Acc (Function Accuracy):** Percentage of samples where all predicted tool names match ground truth.
- **Arg Acc (Argument Accuracy):** Percentage of samples where tool names AND all arguments match ground truth.
- **Exact Match (EM):** Strict end-to-end match.
### Per-Tool Accuracy
*Evaluated at tool invocation level (minimum 3 samples):*
| Tool Name | Correct | Total | Accuracy (%) |
| :--- | :---: | :---: | :---: |
| `turn_off_flashlight` | 12 | 12 | 100.0% |
| `check_traffic` | 3 | 3 | 100.0% |
| `open_website` | 4 | 4 | 100.0% |
| `open_wifi_settings` | 8 | 9 | 88.9% |
| `create_contact` | 30 | 35 | 85.7% |
| `turn_on_flashlight` | 15 | 18 | 83.3% |
| `navigate_to` | 13 | 16 | 81.2% |
| `send_email` | 18 | 26 | 69.2% |
| `show_map` | 17 | 29 | 58.6% |
| `create_calendar_event` | 33 | 61 | 54.1% |
| `set_volume_level` | 2 | 4 | 50.0% |
| `play_music` | 12 | 26 | 46.2% |
| `control_media` | 1 | 6 | 16.7% |
---
## Future Roadmap & Planned Improvements
As this is an initial release, future iterations are planned:
- **Dataset Expansion:**
Expanding [thantzinphyo/Hey-Function-Burmese](https://huggingface.co/datasets/thantzinphyo/Hey-Function-Burmese) with more conversational Burmese dialogue, regional slang, complex compound commands, and multi-turn function calls.
- **Improving Argument Extraction:**
Training for additional epochs (3–5 epochs) to boost Argument Accuracy (Arg Acc), enabling precise extraction of Burmese dates, times, currency amounts, and entity names into JSON arguments.
- **On-Device Quantization:**
Exporting to GGUF and LiteRT formats (int4 / int8 quantization) for offline execution inside Android and Flutter mobile apps.
---
## License
This project is licensed under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0), in accordance with the base model's [Gemma Terms of Use](https://ai.google.dev/gemma/terms).
---
## Citation & Resources
- **Model Repository:** [thantzinphyo/Hey-Function-270M](https://huggingface.co/thantzinphyo/Hey-Function-270M)
- **Dataset:** [thantzinphyo/Hey-Function-Burmese](https://huggingface.co/datasets/thantzinphyo/Hey-Function-Burmese)
- **Codebase:** [GitHub](https://github.com/thantzinphyo-gg/Hey-Function-Burmese)
```bibtex
@misc{hey_270m_alpha_2026,
title={Hey-Function-270M: A Lightweight On-Device Burmese & English Function-Calling Agent},
author={Thant Zin Phyo},
year={2026},
publisher={Hugging Face},
howpublished={\url{https://huggingface.co/thantzinphyo/Hey-Function-270M}}
}
```
|