Create README.md

README.md

@@ -0,0 +1,180 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen3.6-27B
+base_model_relation: finetune
+pipeline_tag: text-generation
+library_name: transformers
+language:
+- en
+- vi
+tags:
+- function-calling
+- tool-use
+- tool-calling
+- agent
+- agentic
+- workflow-automation
+- vllm
+- sglang
+- qwen3.6
+- vietnamese
+- enterprise-automation
+---
+
+# Ops-I — a 27B agent for business workflow automation
+
+**Ops-I** is a fine-tune of **Qwen3.6-27B** aimed at *workflow* agents: episodes where the
+model reads a business request, discovers the right tool out of a large surface, chains
+several calls together, and — just as importantly — knows when **not** to act.
+
+On the public 600-task [AutomationBench](https://github.com/zapier/AutomationBench)
+(Zapier), Ops-I is the **strongest of four open-weight models** we measured, ahead of the
+Qwen3.6-27B base it was tuned from.
+
+**Use it if you need:** an open-weight backend for business automation (finance, HR,
+operations, sales, support) that holds up across long multi-step tool episodes.
+
+| | |
+|---|---|
+| Base | `Qwen/Qwen3.6-27B` (dense, VL architecture) |
+| Precision / size | bf16, 27.8B params, 15 shards (~56 GB) |
+| Context | base supports 262 144; 16K validated in our runs |
+| Reasoning | thinking **on** — how it was evaluated |
+| Tool-call format | Qwen3.5-style XML (`<function=...>`), **not** JSON — see gotchas |
+| License | Apache-2.0 |
+
+---
+
+## Results — AutomationBench 600
+
+Official metric is **pass rate**: a task counts only if *every* assertion passes.
+`partial credit` = fraction of assertions satisfied. Benchmark v1.0.5, `--toolset api`,
+served locally on 8×H200. Full per-task data:
+[Hert4/automationbench-opera-gemma4](https://github.com/Hert4/automationbench-opera-gemma4).
+
+| Model | Pass rate | Passed / 600 | Partial credit | Wall clock |
+|---|---|---|---|---|
+| **beyoru/Ops-I** | **10.17%** | **61** | **45.54%** | 1.96 h |
+| gemma-4-31B-it (thinking on) | 10.17% | 61 | 41.64% | 2.15 h |
+| Qwen3.6-27B (base) | 9.67% | 58 | 46.00% | 1.95 h |
+
+**Read this honestly:** at n=600 one task is worth 0.17 points, so Ops-I, Gemma 4 and the
+base model sit within three tasks of each other — that spread is noise, not a ranking.
+What the numbers *do* support is that Ops-I is at least not worse than its base while
+being the fastest of the tuned models, and that it beats Opera by 10 tasks.
+
+For scale, the reference models in the upstream README (run at their highest reasoning
+effort) land at Claude Opus 4.8 30.33%, GPT-5.6 Sol 29.17%, GLM 5.2 20.33%,
+Gemini 3.5 Flash 14.83%.
+
+### Per domain
+
+Tasks passed out of 100, partial credit in parentheses.
+
+| Model | Finance | HR | Marketing | Operations | Sales | Support |
+|---|---|---|---|---|---|---|
+| **beyoru/Ops-I** | **20** (57.0) | 3 (10.1) | 5 (51.4) | **18** (60.5) | **10** (38.2) | 5 (55.9) |
+| Qwen3.6-27B base | 21 (57.7) | 3 (12.4) | 3 (45.1) | 17 (60.9) | 8 (41.4) | 6 (58.4) |
+| gemma-4-31B-it | 18 (48.7) | 1 (5.2) | **14** (56.3) | 15 (56.2) | 9 (34.3) | 4 (49.1) |
+
+Finance and operations carry the score. HR is the floor for every model tested (1-3 / 100).
+
+---
+
+## Known weakness: it acts too much
+
+The dominant failure mode is **not** inability to act — it is failure to *refrain*.
+
+| Assertion type | Ops-I |
+|---|---|
+| Positive ("do send / do create") | 3028 / 5636 = **53.7%** |
+| Negative ("do **not** send / do **not** create") | 65 / 585 = **11.1%** |
+
+Some negative assertion types never pass, aggregated across all four models tested:
+`gmail_message_not_sent` 0/203, `slack_message_not_in_channel` 0/166,
+`google_sheets_row_not_exists` 0/161.
+
+This is uniform across two model families and four checkpoints, so treat it as a property
+of the current open-weight generation rather than of Ops-I specifically. **If you deploy
+this in a workflow that can send email, post to chat, or write to a sheet, gate those
+actions behind confirmation.**
+
+---
+
+## Quickstart
+
+### Serve with SGLang
+
+```bash
+python3 -m sglang.launch_server --model-path beyoru/Ops-I --served-model-name Ops-I \
+  --host 0.0.0.0 --port 30014 --tp 4 \
+  --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
+  --mem-fraction-static 0.85
+```
+
+### Serve with vLLM
+
+```bash
+vllm serve beyoru/Ops-I --served-model-name Ops-I \
+  -tp 4 --max-model-len 16384 \
+  --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3
+```
+
+### Call it
+
+```python
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:30014/v1", api_key="EMPTY")
+
+tools = [{
+    "type": "function",
+    "function": {
+        "name": "crm_update_contact",
+        "description": "Update a contact's details in the CRM.",
+        "parameters": {
+            "type": "object",
+            "properties": {"contact_id": {"type": "string"}, "phone": {"type": "string"}},
+            "required": ["contact_id", "phone"],
+        },
+    },
+}]
+
+r = client.chat.completions.create(
+    model="Ops-I",
+    messages=[{"role": "user", "content": "Update Le Van A's phone to 0912345678."}],
+    tools=tools,
+    tool_choice="auto",
+)
+print(r.choices[0].message.tool_calls)
+```
+
+---
+
+## Training
+
+
+---
+
+## Evaluation notes
+
+- All numbers above come from a single run per model (k=1) on the full public 600-task
+  set, temperature left at server default. No task was excluded.
+- AutomationBench has **no separate train split** — the 600 public tasks are both the
+  dataset and the scoring set. Ops-I was **not** trained on them; if you fine-tune against
+  this benchmark, its scores stop being comparable to anything.
+- Numbers were produced with SGLang 0.5.12. A vLLM baseline was not measured, so do not
+  mix serving stacks when comparing.
+
+## Citation
+
+Benchmark tasks, tools and assertions are Zapier's:
+
+```bibtex
+@misc{automationbench,
+  title  = {AutomationBench},
+  author = {Zapier},
+  year   = {2026},
+  url    = {https://github.com/zapier/AutomationBench}
+}
+```