--- language: - en - es tags: - iot - smart-home - home-assistant - tool-calling - agent - cybersecurity - devops - ollama - inference-pipeline - autonomous-agent - raspberry-pi - qwen3-coder license: mit library_name: transformers pipeline_tag: text-generation --- # TokioAI-Coder IoT Package **Turn a 30B open-weight model into a fully autonomous IoT/smart-home agent.** This package contains everything needed to deploy TokioAI-Coder as an autonomous IoT controller: - The complete **TokioAI Engine** -- inference pipeline that wraps any Ollama model and adds reliable tool calling - A production-tested **IoT system prompt** with a full Home Assistant cookbook - **12 native tools** for system administration, file operations, network diagnostics, and robot vision - **Server-side investigation** -- the engine autonomously chains 8+ tool calls to answer a single question ## What It Does You say: *"is Home Assistant running?"* The engine: 1. Checks `systemctl status home-assistant` -- not found 2. Checks `docker ps | grep home` -- not found 3. Checks `ss -tlnp | grep 8123` -- port listening 4. Checks `curl -s http://localhost:8123/api/` -- HA API responds 5. Answers: "Yes, HA is running on port 8123 via Docker" All 5 steps happen **in a single API call**. The client sees only the final answer. You say: *"play some blues"* The engine: 1. Calls HA API to send `play_media` to the Alexa 2. Waits 3 seconds 3. Queries the media player state to verify the music actually changed 4. Reports: "Now playing: Blues Essentials on Jarvis" ## Architecture ``` TokioAI Engine (OpenAI-compatible API) ==================================== User Query ──> Router ──> Ollama Model ──> Verifier ──> Tool Executor ──> | | | | Intent System Retry Local Classifier Prompt + Loop (2x) Execution (0ms regex) Few-Shot (read-only safe) | | | | v v v v tool_call Guided Validates Feeds result / text / Generation JSON + back to model decision (schema- correct for next step constrained) tool names | v Investigation Loop (up to 8 rounds, 90s timeout) | v Final Answer ``` ## Components ### 1. TokioAI Engine (`engine/`) The core inference pipeline (3,774 lines of Python): | Module | Lines | Purpose | |---|---|---| | `server.py` | 864 | OpenAI-compatible API with server-side investigation | | `schemas.py` | 497 | Tool definitions, JSON validation, parameter schemas | | `pipeline.py` | 476 | Main orchestrator: Router -> Memory -> Model -> Verifier | | `router.py` | 382 | Intent classifier (regex) + system prompt builder | | `memory.py` | 353 | Adaptive few-shot memory (learns from successes) | | `executor.py` | 331 | Safe server-side tool execution | | `guided.py` | 228 | JSON schema constraints for Ollama | | `verifier.py` | 186 | Output validation + retry logic | ### 2. IoT System Prompt (`prompts/iot_system_prompt.md`) A production-tested system prompt that includes: - Complete Home Assistant API cookbook (media, lights, switches, sensors, vacuum, TTS) - Alexa voice control via `alexa_media` integration - Robot vision via moondream (1.4B vision model) - Investigation playbooks for multi-step problem solving - Anti-hallucination rules (never simulate tool execution) - Verification protocol (always confirm state changes actually happened) ### 3. Tool Definitions (`tools/`) 12 native tools in OpenAI function-calling format: - `execute_local` -- Run shell commands - `execute_raspi` / `execute_gcp` / `execute_router` -- Remote execution - `read_file` / `write_file` / `edit_file` -- File operations - `search_files` -- grep across codebases - `diagnose` -- System health checks - `robot_vision` -- Camera capture + AI analysis - `memory` -- Persistent memory across sessions - `task` -- Task tracking for multi-session projects ## Quick Start ### Prerequisites - Python 3.8+ - [Ollama](https://ollama.ai/) with a model (recommended: `qwen3-coder` 30B MoE) - Home Assistant instance (for IoT features) ### Install ```bash # Clone this repo git clone https://huggingface.co/MrMoz33/tokioai-coder-iot cd tokioai-coder-iot # Install dependencies pip install flask requests # Pull a model ollama pull qwen3-coder # 30B MoE (3B active), recommended # or: ollama pull qwen2.5-coder:7b # lighter alternative ``` ### Configure Home Assistant Set your HA token: ```bash export HA_TOKEN="your-long-lived-access-token" export HA_URL="http://localhost:8123" # or your HA URL ``` ### Start ```bash # Start the engine python -m engine serve --model qwen3-coder --port 8080 # Use it (any OpenAI-compatible client works) curl http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "tokioai-engine", "messages": [{"role": "user", "content": "is home assistant running?"}] }' ``` ### With TokioAI CLI ```bash pip install git+https://github.com/TokioAI/tokioai.git export TOKIOAI_ENGINE_HOST=http://localhost:8080 tokioai # Interactive agent with full tool calling ``` ## Customization ### Adding Your Own Home Assistant Entities Edit `prompts/iot_system_prompt.md` and add your entities to the cookbook section: ```markdown LIGHTS: light.living_room, light.bedroom, light.kitchen SWITCHES: switch.fan, switch.heater SENSORS: sensor.temperature, sensor.humidity MEDIA: media_player.living_room_speaker ``` ### Adding Custom Tools Edit `engine/schemas.py` to add new tool definitions: ```python { "type": "function", "function": { "name": "my_custom_tool", "description": "What this tool does", "parameters": { "type": "object", "properties": { "param1": {"type": "string", "description": "First param"} }, "required": ["param1"] } } } ``` Then add the executor in `engine/executor.py`. ### Adapting for Different Smart Home Platforms The system prompt is platform-agnostic in concept. To use with: - **Google Home**: Replace HA API calls with Google Home API - **Apple HomeKit**: Use HomeKit Accessory Protocol - **MQTT Direct**: Replace curl commands with `mosquitto_pub/sub` The engine and tool pipeline remain identical -- only the system prompt changes. ## Performance Tested on Raspberry Pi 5 (8GB) + L4 GPU (via SSH tunnel): | Metric | Value | |---|---| | Model | Qwen3-Coder 30B (3B active MoE) | | GPU VRAM | 19.4 GB (L4 24GB) | | Response time | 3-8s (simple), 15-45s (investigation) | | Investigation depth | Up to 8 rounds per query | | Tool calling accuracy | 94%+ with engine pipeline | | Uptime | 30+ days continuous (systemd service) | ### Cost Comparison | Approach | Monthly Cost | Latency | Privacy | |---|---|---|---| | GPT-4o API | $50-200 | 2-5s | Data sent to cloud | | Claude API | $50-200 | 2-5s | Data sent to cloud | | **TokioAI-Coder (self-hosted)** | **$0** | **3-8s** | **100% local** | After initial GPU investment ($0.50-1.50/hr for L4 on GCP, or free on local hardware), all inference is free and private. ## Model Compatibility | Model | Params | Tool Calling | IoT Quality | Notes | |---|---|---|---|---| | qwen3-coder | 30B (3B active) | Excellent | Excellent | Recommended. MoE = fast inference | | qwen2.5-coder:7b | 7B | Good | Good | Budget option | | llama3.1:8b | 8B | Good | Fair | General purpose | | deepseek-coder-v2:16b | 16B | Good | Good | Strong code understanding | | mistral:7b | 7B | OK | Fair | Needs guided gen | The engine's guided generation + retry logic compensates for weaker models. ## Related Projects - [TokioAI CLI](https://github.com/TokioAI/tokioai) -- Multi-provider AI CLI with native tool calling - [TokioAI Fine-Tuning Guide](https://github.com/daletoniris/tokioai-fine-tuning-guide) -- Train your own tool-calling model - [TokioAI Fine-Tuning Paper](https://github.com/daletoniris/tokioai-fine-tuning-paper) -- Research paper on LLM fine-tuning for tool use - [TokioNav](https://github.com/daletoniris/tokionav) -- AI-controlled autonomous robot navigation ## License MIT ## Credits Built by Daniel Dieser ([daletoniris](https://github.com/daletoniris)) and the TokioAI team. The TokioAI Engine was developed to make small open models competitive with GPT-4 and Claude for autonomous tool calling -- specifically for IoT, cybersecurity, and DevOps tasks. The key insight: you don't need a bigger model, you need a smarter pipeline around it.