You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

This repository contains benchmark task definitions together with their grading logic and expected answers. Access is reviewed manually. Please state who you are and what you intend to use it for.

kimi-harness

Running Kimi-K3 with Moonshot's own agent harness against a GUI-agent benchmark, instead of forcing it through a generic PromptAgent contract.

This is the wiring we used to evaluate kimi-k3 on a QuPath (digital pathology) OSWorld-style benchmark. Kimi is the only model in that evaluation that runs its own harness; every other arm goes through a shared prompt/parse contract. The point of this package is that the upstream adapter is not modified — every alignment happens from the outside, via constructor arguments and by wrapping agent._payload / agent.call_llm.

⚠️ This repo contains benchmark ground truth. bench/tasks*/ hold expected answers (exact object counts, measurement windows, answer hashes) and bench/evaluators/ hold the grading code. Do not redistribute, and do not train on them.

Layout

agent/                        the model-facing layer
  mm_agents/kimi/             Moonshot's official adapter — BYTE-IDENTICAL to upstream
  desktop_env/                OSWorld's pyautogui controller (one import patched, see below)
  kimi_loop.py                the agent loop: screenshot → predict → execute → feed result back
  guest_control.py            HTTP control plane to the VM (/execute, /setup/upload_file)
adapters/qupath/              driver for the QuPath benchmark
  run_kimi.py                 builds the instruction, calls the loop, writes meta.json
  run_task_kimi.sh            one task end-to-end: reset VM → setup → run → judge
  run_kimi_23.sh              23-task batch dispatcher with a per-VM lock
bench/                        the benchmark itself (tasks, grader, evaluators)
reference/vm/                 VM lifecycle scripts, for reference
LICENSES/                     Apache-2.0 (OSWorld) and MIT (our code)

Install

pip install -r requirements.txt     # requests, loguru, Pillow

Python ≥ 3.11. Tested with requests 2.34.2, loguru 0.7.3, Pillow 12.3.0.

You also need a running OSWorld-style guest VM exposing the standard HTTP control API (POST /execute, POST /setup/upload_file, GET /screenshot). This package does not ship a VM image. reference/vm/start_env.sh shows how we started ours (Docker + QEMU).

Configure

Copy .env.example and fill it in — or just export the variables.

Variable	Required	What it does
`ORION_ROOT`	yes	Root that contains `code/.env` with your `OPENROUTER_API_KEY`
`KIMI_CHAT_URL`	no	Full `.../chat/completions` URL. Default: OpenRouter
`KIMI_KEY_FILE`	no	Read the key from a file instead of the environment
`KIMI_PROVIDER`	no	`none` = don't send OpenRouter's `provider` field. Default `moonshotai`
`KIMI_BUDGET`	no	`matched` (default) or anything else for Moonshot factory defaults
`KIMI_MODE`	no	`gui` (default) or `hybrid` (adds a direct shell tool)
`KIMI_TRACE_HTTP`	no	`1` logs every HTTP attempt to `http_attempts.jsonl`
`OSWORLD_VM_PASSWORD`	no	Guest sudo password. Default is upstream's public test value
`ORION_HARNESS`	for the shell drivers	Path to `bench/`
`ORION_PY`	no	Interpreter for the shell drivers. Default `python3`

⚠️ KIMI_CHAT_URL must end in /chat/completions — the loop does cfg["url"].rsplit("/chat/completions", 1)[0] to derive the adapter's base_url.

Run

One task:

export ORION_ROOT=/path/to/root ORION_HARNESS=$PWD/bench
python adapters/qupath/run_kimi.py QP-002-T1 --port 5060 --outdir ./out/QP-002-T1

End-to-end with VM reset and grading:

ORION_HARNESS=$PWD/bench ./adapters/qupath/run_task_kimi.sh full QP-002-T1 \
    --arm osworld --port 5060 --run-id my_run --model moonshotai/kimi-k3

CLI: task (positional), --port (required), --model (default moonshotai/kimi-k3), --max-steps 100, --max-tokens 16384, --outdir. (--max-trajectory is parsed but unused — the real history window comes from the budget below.)

The budget alignment — why this exists

Moonshot's factory defaults give Kimi a much larger budget than a like-for-like comparison allows. KIMI_BUDGET=matched (the default here) overrides five constructor arguments:

Setting	Upstream default	`matched`	Why
`max_history_length`	1000	5	match a 5-screenshot history window
`history_slide_chunk`	10	1	with a window of 5, a chunk of 10 empties the history
`num_max_images`	100	6	5 history + 1 current, just under the compaction threshold
`num_compact_images`	10	1	—
`fail_on_max_steps`	`True`	False	don't burn the last step on a forced FAIL
`max_tokens`	65536	16384	same output cap as every other arm

matched additionally strips temperature / top_p from the payload (other arms don't send them either), and pins the OpenRouter provider so repeated runs don't land on different implementations.

None of this edits mm_agents/kimi/. Budget changes go through the public constructor; sampling and provider changes wrap agent._payload from outside.

Four things that will bite you

Feed the tool result back. The loop passes obs["last_tool_result"] on the next step. Without it the model can't see its own command's returncode/stderr and will repeat a failing command forever.
Set the real guest password. The adapter's default is upstream's public evaluation password; if your VM differs, set OSWORLD_VM_PASSWORD or the wrong password goes into the system prompt.
RUN_COMMAND / WAIT_FOR_STATE are dict actions, not strings — they need their own branch (only reachable in hybrid mode).
Screen size is a constructor argument (screen_size=(1920, 1080)). Kimi has no resolution sentence in its system prompt; it takes the frame size from here. Get it wrong and every click lands in the wrong place.

Outputs

Per run, in --outdir: meta.json (status, steps, tokens, cost, budget knobs actually used), trace.jsonl (per-step response/action/exec_result), llm_raw.jsonl (per-call usage and cost), shots/step_NNN.png, prompt_sent.txt, system_prompt.txt.

Cost comes from OpenRouter's per-call usage.cost when available. Do not compute cost from an account balance delta — under parallel runs we measured it inflated 5.6–11.8×.

Known limitations

bench/ will not grade out of the box: the evaluators reference a reference_private/ tree and per-task private_gt/ directories that are not in this package, and the WSI slide inputs and VM image are not shipped either. Treat bench/ as the specification of the tasks and the grading contract, not as a turnkey runner.
Paths inside bench/ still refer to the machine we ran on.
run_kimi.py requires OPENROUTER_API_KEY in $ORION_ROOT/code/.env even when you point KIMI_CHAT_URL elsewhere — the credential check runs first. Put any placeholder there if you are using a different endpoint.

Attribution

agent/mm_agents/kimi/ and agent/desktop_env/ come from xlang-ai/OSWorld @ 84aee655c2afb6b77ecf39884432615ba345c031, Apache-2.0, Copyright 2024 XLANG NLP Lab. See ATTRIBUTION.md for the exact list of files and the one modification we made. Our own code is MIT — see LICENSES/.

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support