Kumo-1.5B — A 1B-Class Coding Agent LLM

Kumo-1.5B is a 1.5B-parameter coding agent fine-tuned from Qwen/Qwen2.5-Coder-1.5B-Instruct for autonomous multi-step debugging and code generation.

It supports two modes over a single ChatML template:

1. Agent Mode (<read>, <file>, <py>, <final>, <observation>): Inspects files in a workspace, writes patches, executes tests in a sandbox, and iterates until all tests pass.
2. Direct Mode: Standard single-turn Python code generation and bug fixing.

---

Benchmark Results (Greedy pass@1, temperature=0.0)

Model	HumanEval pass@1 (164)	MBPP pass@1 (374)	Bug-Fix Zero-Shot (60)	Agent Tool-Loop pass@1 (60)	Avg Agent Steps
Qwen/Qwen2.5-Coder-1.5B-Instruct (base)	69.5%	55.9%	56.7%	10.0%	2.4
Kumo-1.5B (Tomopas/Kumo-1.5B)	66.5%	58.3%	71.7%	73.3%	4.7

Highlights

• +63.3% on Autonomous Multi-Step Workspace Debugging (10.0% $\to$ 73.3%, solving 44/60 held-out buggy workspaces in an average of 4.7 steps).
• +15.0% on Zero-Shot Bug-Fixing (56.7% $\to$ 71.7%).
• +2.4% on Full Held-Out MBPP (55.9% $\to$ 58.3%).
• Multi-Step Agent Beats Zero-Shot Prompting: Allowing Kumo-1.5B to inspect and test code interactively in a workspace (73.3%) outperforms giving it the buggy code and failing test in a single prompt (71.7%).

---

Training Recipe

1. On-Policy Rejection Sampling (RFT): Sampled candidate solutions and multi-step agent traces from the base model and filtered strictly by sandboxed unit-test execution.
2. 100% Bug-Free-Assistant SFT (sft_clean): Reformulated multi-turn code repair so that buggy code appears only in user or <observation> turns—never in assistant turns—preventing assistant_only_loss from training the model to inject operator mutations on Turn 1.
3. WiSE-FT Weight-Space Interpolation ($\lambda = 0.55$): Merged the rank-32 LoRA adapter into the base model with scale $\lambda = 0.55$ ($W_{\text{kumo}} = W_{\text{base}} + 0.55,\Delta W_{\text{sft_clean}}$), combining the base model's algorithmic synthesis with the learned agent tool-use protocol.

---

Quickstart (Transformers)

1. Direct Code Generation

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Tomopas/Kumo-1.5B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")

messages = [
    {
        "role": "system",
        "content": (
            "You are Kumo, an expert Python programmer. Solve the task. "
            "Put the complete, runnable solution in a single ```python code block. "
            "Do not include tests or example usage in the code block."
        ),
    },
    {"role": "user", "content": "Write a Python function `top_k_frequent(nums, k)` that returns the k most frequent integers sorted by frequency descending (and value ascending on ties)."},
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

2. Multi-Step Coding Agent Prompt Protocol

SYSTEM_AGENT = """You are Kumo, a coding agent working in a sandboxed workspace (files are in the current directory).
Act with these tags:
  <py>python code</py>              -> runs the code in the workspace
  <file path="foo.py">content</file> -> writes a file
  <read path="foo.py"/>             -> reads a file
  <final>answer</final>             -> finishes the task
Observations arrive as <observation>...</observation>.
Inspect the code, make a focused change, then verify by running the tests. Only finish after the tests pass."""