Ops-I — a 27B agent for business workflow automation

Ops-I is a fine-tune of Qwen3.6-27B aimed at workflow agents: episodes where the model reads a business request, discovers the right tool out of a large surface, chains several calls together, and — just as importantly — knows when not to act.

On the public 600-task AutomationBench (Zapier), Ops-I is the strongest of four open-weight models we measured, ahead of the Qwen3.6-27B base it was tuned from.

Use it if you need: an open-weight backend for business automation that holds up across long multi-step tool episodes. Really want to hear about

Base	Qwen/Qwen3.6-27B (dense, VL architecture)
Precision / size	bf16, 27.8B params, 15 shards (~56 GB)
Context	base supports 262 144; 16K validated in our runs
Reasoning	thinking on — how it was evaluated
Tool-call format	Qwen3.5-style XML (<function=...>), not JSON — see gotchas
License	Apache-2.0

---

Results — AutomationBench 600

Model	Pass rate	Passed / 600	Partial credit	Wall clock
beyoru/Ops-I	10.17%	61	45.54%	1.96 h
gemma-4-31B-it (thinking on)	10.17%	61	41.64%	2.15 h
Qwen3.6-27B (base)	9.67%	58	46.00%	1.95 h

For scale, the reference models in the upstream README (run at their highest reasoning effort) land at Claude Opus 4.8 30.33%, GPT-5.6 Sol 29.17%, GLM 5.2 20.33%, Gemini 3.5 Flash 14.83%.

Per domain

Tasks passed out of 100, partial credit in parentheses.

Model	Finance	HR	Marketing	Operations	Sales	Support
beyoru/Ops-I	20 (57.0)	3 (10.1)	5 (51.4)	18 (60.5)	10 (38.2)	5 (55.9)
Qwen3.6-27B base	21 (57.7)	3 (12.4)	3 (45.1)	17 (60.9)	8 (41.4)	6 (58.4)
gemma-4-31B-it	18 (48.7)	1 (5.2)	14 (56.3)	15 (56.2)	9 (34.3)	4 (49.1)

Finance and operations carry the score. HR is the floor for every model tested (1-3 / 100).

---

Known weakness: it acts too much

The dominant failure mode is not inability to act — it is failure to refrain.

Assertion type	Ops-I
Positive ("do send / do create")	3028 / 5636 = 53.7%
Negative ("do not send / do not create")	65 / 585 = 11.1%

Some negative assertion types never pass, aggregated across all four models tested: gmail_message_not_sent 0/203, slack_message_not_in_channel 0/166, google_sheets_row_not_exists 0/161.

This is uniform across two model families and four checkpoints, so treat it as a property of the current open-weight generation rather than of Ops-I specifically. If you deploy this in a workflow that can send email, post to chat, or write to a sheet, gate those actions behind confirmation.

---

Quickstart

Serve with SGLang

python3 -m sglang.launch_server --model-path beyoru/Ops-I --served-model-name Ops-I \
  --host 0.0.0.0 --port 30014 --tp 4 \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
  --mem-fraction-static 0.85

Serve with vLLM

vllm serve beyoru/Ops-I --served-model-name Ops-I \
  -tp 4 --max-model-len 16384 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3

Call it

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30014/v1", api_key="EMPTY")

tools = [{
    "type": "function",
    "function": {
        "name": "crm_update_contact",
        "description": "Update a contact's details in the CRM.",
        "parameters": {
            "type": "object",
            "properties": {"contact_id": {"type": "string"}, "phone": {"type": "string"}},
            "required": ["contact_id", "phone"],
        },
    },
}]

r = client.chat.completions.create(
    model="Ops-I",
    messages=[{"role": "user", "content": "Update Le Van A's phone to 0912345678."}],
    tools=tools,
    tool_choice="auto",
)
print(r.choices[0].message.tool_calls)

---

Training

---

Evaluation notes

• All numbers above come from a single run per model (k=1) on the full public 600-task set, temperature left at server default. No task was excluded.
• AutomationBench has no separate train split — the 600 public tasks are both the dataset and the scoring set. Ops-I was not trained on them; if you fine-tune against this benchmark, its scores stop being comparable to anything.
• Numbers were produced with SGLang 0.5.12. A vLLM baseline was not measured, so do not mix serving stacks when comparing.

Limitaion

The model only take a few steps training because of LIMITATION gpu usage in our org

Community Feedback

Ops-I is an ongoing project, and community feedback is invaluable.

If you discover failure cases, benchmark results, or have ideas for future improvements, feel free to start a Discussion. Every report helps improve future releases.

Citation

Benchmark tasks, tools and assertions are Zapier's:

@misc{automationbench,
  title  = {AutomationBench},
  author = {Zapier},
  year   = {2026},
  url    = {https://github.com/zapier/AutomationBench}
}