--- title: Rosdiff emoji: πŸš€ colorFrom: blue colorTo: red sdk: docker pinned: false license: mit --- # RosDiff β€” Stage 1 (internal tester build) > **Handover:** [`docs/PROJECT_REPORT.md`](docs/PROJECT_REPORT.md) summarises everything built so far, the decisions > behind it, and how to move the project to another environment. Orcanion internal build for a small, hand-picked group of testers. It exists to measure one thing: **how good the generated simulation scenes and ROS 2 code are**, before any commercial infrastructure gets built. No payments, no tiers, no signup β€” a handful of API keys, one model, and a log of everything. | Endpoint | Input | Output | |---|---|---| | `POST /simulate/generate` | scenario description + robot | MuJoCo MJCF scene around the robot, load-tested in MuJoCo | | `POST /code/generate` | ROS 2 request (Humble/Jazzy, Python/C++) | a ROS 2 package, statically validated against the real message definitions (`check: true` also runs it) | | `POST /code/check` | a generated package | built, linted, tested and launched in a ROS 2 sandbox; report with logs and the recorded bag | | `POST /simulate/run` | a valid scene (`generation_id`, or raw MJCF) | a cloud GPU run (Modal, or RunPod): video, Foxglove replay, trajectory; reviewed by Cosmos Reason | | `POST /simulate/sweep` | a run, generation or MJCF | the scene Γ—N with randomized friction, masses and pushes (MuJoCo Warp): fall rate | | `POST /simulate/run/{id}/review` | a finished run | Cosmos Reason's verdict on the video (opt-in; `AUTO_REVIEW=true` reviews every run) | | `POST /simulate/run/{id}/isaac` | a finished run | the same motion rendered with Isaac Sim RTX (10 a day) | | `POST /datasets/export` | finished runs | a LeRobot dataset on the Hugging Face Hub | | `GET /report/{id}` | (signed link) | HTML report for any job, with its follow-up jobs | | `GET /jobs`, `GET /jobs/{id}` | β€” | your jobs, their status, and this month's budget | | `POST /feedback/{id}` | `works` / `partly` / `broken` + note | the tester's verdict after running the output | | `GET /health` | β€” | model, index size, OpenCode availability (no key needed) | Robots: Unitree **G1** and **H1** humanoids, Unitree **Go2** quadruped, **Franka Panda** arm (all from MuJoCo Menagerie). Generated ROS 2 code only ever runs inside the sandbox (no network, no secrets). ### Models, budgets and prompt enhancement Testers never pick a model. **Automatic routing** (`api/routing.py`) rates every request's complexity and sends it to the cheapest capable tier; a draft that fails validation is repaired by a stronger one. Scenes and code have their own model lists: capable Chinese open-weight models on OpenRouter, chosen by benchmark and price rather than release date (override with `SCENE_MODELS` / `CODE_MODELS`): | Tier | Code | Scenes | |---|---|---| | low | `deepseek/deepseek-v4-flash` (79.0% SWE-bench Verified, $0.09/$0.18 per M) | (not used: scenes start at mid) | | mid | `deepseek/deepseek-v4-pro` (80.6% SWE-bench Verified) | `z-ai/glm-5.2` (62.1% SWE-bench Pro, close to Claude Opus 4.8 on FrontierSWE) | | high | `z-ai/glm-5.2` | `moonshotai/kimi-k2.6` (80.2% SWE-bench Verified, top open-weight Intelligence Index) | **Free debugger.** The first repair after a failed check goes to a free DeepSeek V4 Flash through OpenCode Zen (`DEBUG_MODEL`, default `opencode/deepseek-v4-flash-free`), not OpenRouter, so it costs nothing and is logged as exempt. If that call fails (rate limit, the free offer withdrawn, OpenCode missing) the same repair goes to the next paid tier at once; a further repair keeps climbing. Code gets 3 attempts (`CODE_MAX_ATTEMPTS`) so the free try never replaces the stronger paid repair. `FREE_DEBUG=false` turns it off. The web app shows the choice as information ("AI Routing: automatic Β· Standard tier Β· Why?"), plus the simulator it picked (MuJoCo, MuJoCo Warp for "how often does it…" questions, Isaac Sim for photoreal renders) with a **Use another simulator** menu. Gazebo, Webots and PyBullet are listed as not available yet. **Prompt enhancement (free).** Before generating, the tester's words are rewritten into a precise brief by OpenCode with a free OpenCode Zen model (`ENHANCE_MODEL`, default `opencode/deepseek-v4-flash-free`). It never goes through OpenRouter and costs nothing; the call is logged as exempt. The brief must keep every number the tester wrote, or it is thrown away, and any failure (OpenCode missing, timeout, a rotated-out free model) just means the original text is used. Routing and cost estimates still use the tester's own words. The app shows the brief under "✨ Enhanced prompt". It needs `opencode` installed on the API server (`npm i -g opencode-ai`); `PROMPT_ENHANCE=false` switches it off, `ZEN_API_KEY` supplies a Zen key if the free model asks for one. **AI budget (rupees).** One pool, `BUDGET_TOTAL_INR` (β‚Ή500), and an equal allocation per tester, `BUDGET_PER_USER_INR` (β‚Ή125). It covers every billable OpenRouter call: coding, simulation AI, automatic repairs ("debugging") and testing. Costs come from OpenRouter's reported usage (list prices otherwise). Free and OpenRouter-exempt calls are recorded but not deducted. Tasks whose worst case is above `CONFIRM_ABOVE_INR` show an estimate and wait for "Run". Testers see their balance in the top bar and under *My usage*; testers listed in `ADMIN_TESTERS` also see the whole team. **GPU budget (dollars).** Simulations, sweeps, renders and build checks run on Modal's credit, `MONTHLY_CREDIT_LIMIT_USD` (default $29.98), split equally between `COMPUTE_TESTERS` (4): $7.495 each per month (`COMPUTE_PER_TESTER_USD` overrides the share). The limit is strict: a job starts only if its **worst case** (the cost of running until its timeout) still fits in both the tester's share and the team's limit, and jobs still running count at that worst case, so parallel jobs can never overshoot. Finished jobs count at their measured cost. ## How it works ``` β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ RAG (local Chroma, data/chroma) ────────────┐ β”‚ mujoco: MuJoCo XML reference + modeling docs, MJCF β”‚ β”‚ schema (every element's valid attributes), β”‚ β”‚ Menagerie unitree_g1 β”‚ β”‚ ros2: ROS 2 Humble + Jazzy tutorials & launch docs, β”‚ β”‚ .msg/.srv definitions (common_interfaces, β”‚ β”‚ rcl_interfaces, example_interfaces) β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ /simulate/generate ─► retrieve ─► LLM ─────► extract MJCF ─► static safety check ─► MuJoCo load + 2 s step ─┐ β–² (subprocess, timeout) β”‚ └───────────── repair: MuJoCo's error + valid attributes β—„β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ /code/generate ─────► retrieve ─► OpenCode (same LLM, headless) ─► files ─► ROS 2 validator ─┐ β–² β”‚ └──────── repair in the same OpenCode session β—„β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ every request ──► data/generations.sqlite3 ──► python -m api.review ``` **MuJoCo scenes.** The model never writes the robot. Scenes `` from MuJoCo Menagerie, exactly like Menagerie's own `scene.xml`, and the model only builds the world around it. The prompt carries: robot facts read from the model file itself (bodies, joints with ranges, actuators, sites, keyframes), Menagerie's example scene, the exact attribute list of the ~16 elements scenes use (from MuJoCo's XML schema), and the retrieved documentation sections. The result is loaded with MuJoCo and stepped for 2 simulated seconds in a separate process; if loading fails, MuJoCo's error message (plus the valid attributes of the offending element) goes back to the model, up to 3 attempts. Scenes may not reference files other than `g1.xml`, absolute paths or `..`. **ROS 2 packages.** Named message/service types in the request (`LaserScan`, `nav_msgs/Odometry`, …) are looked up exactly, together with the types they contain; related definitions and distro-specific tutorials are retrieved from the vector store. OpenCode runs headless (`opencode run`) in a throwaway workspace with the configured LLM as its only provider, file edits allowed inside the workspace and shell/web access denied; the RAG context is attached to its message. The files it writes are validated without a ROS install: package layout (`package.xml`, `setup.py` / `CMakeLists.txt` with `ament_package()`), Python and launch-file syntax, `generate_launch_description()`, XML launch files β€” and **every message/service type used must exist in the requested distro** (types from packages that aren't indexed are flagged as unverified warnings). Problems go back to the same OpenCode session for one repair round. `CODEGEN_BACKEND=direct` uses a plain chat call instead of OpenCode. ## Tools Everything that runs on a GPU or in a container is a *job* on the backend (Modal by default). Each job has a report page, an estimated cost, and counts toward the monthly budget. | Tool | What it answers | Where it runs | Est. cost | |---|---|---|---| | **Simulation run** (MuJoCo) | does the scene work, does the robot stay up? | Modal T4 | ~1Β’ | | **ROS 2 sandbox** (colcon, ament linters, launch_testing, ros2_control, Nav2, MoveIt 2, xacro, robot_state_publisher, tf2, rosbag2 MCAP) | does the generated package build, pass its tests, and start? which nodes and topics come up? | Modal CPU, no network | <1Β’ | | **Robustness sweep** (MuJoCo Warp, NVIDIA Warp) | how often does the robot fall across 64–256 randomized copies? | Modal T4 | a few Β’ | | **Cosmos Reason review** (NVIDIA) | does the video match the description, is the physics plausible? | Modal L4 | ~2–3Β’ | | **LeRobot export** (Hugging Face) | runs β†’ a robot-learning dataset | Modal CPU | <1Β’ | | **Isaac Sim render** (NVIDIA) | the same motion, photoreal (RTX) | Modal L4, 10/day | ~5–10Β’ | - **ROS 2 sandbox** (`worker/ros/Dockerfile`, `worker/ros_runner.py`): the package is built with `colcon`, checked with `ament_flake8`/`pep257` (Python) or `ament_cpplint`/`cppcheck` (C++) and `ament_xmllint`, its tests run with `colcon test`, then its launch file (or each executable) runs for 10 s while `ros2 bag record -a` captures every topic to MCAP. Build errors, missing dependencies, crashed nodes and the live node/topic list are on the report; the bag opens in the embedded Foxglove viewer. Humble and Jazzy images; generated code gets no network and no secrets. CI checks four sample packages in both distros. - **Robustness sweep** (`worker/sweep.py`): every world gets its own friction scale, free-object mass scale, joint noise and a push on the robot's root. MuJoCo Warp steps all worlds together on the GPU; without CUDA the same randomization runs in plain MuJoCo (results agree within a millimetre; CI runs both). The report plots push against friction for every world. - **Cosmos Reason** (`worker/cosmos_review.py`): NVIDIA's physical-reasoning vision-language model watches the video at 2 fps with the scenario text and the simulator's facts, and answers matches / plausible / fell with its reasoning. A first pass for reviewers, not a verdict. Model: `COSMOS_MODEL` (default `nvidia/Cosmos-Reason1-7B`). - **LeRobot export** (`worker/lerobot_export.py`): each run becomes an episode with `observation.state` (robot joints), `observation.velocity`, `action` (commanded controls), `observation.images.main` (640Γ—360 video) and the scenario as the task. Pushed as a private dataset to `HF_NAMESPACE/rosdiff-`, or stored as a zip when no Hugging Face token is configured. - **Isaac Sim render** (`worker/usd_scene.py`, `worker/isaac_render.py`): the run is turned into a USD stage with every body's pose time-sampled and a tracking camera; Isaac Sim renders it with RTX. No second physics engine, so it is the same motion as the MuJoCo video. ## Simulation runs ``` POST /simulate/run ─► re-validate the scene ─► log (tester, time) ─► Modal GPU job (T4, or RunPod) β”‚ worker/: MuJoCo, robot holds its β”‚ standing pose (or passive), records β–Ό GET /simulate/report/{id} ◄── presigned R2 links ◄── Cloudflare R2: runs//run.mcap, video.mp4, status Β· summary Β· video Β· Foxglove iframe summary.json, scene.xml ``` The worker ([`worker/`](worker/)) loads the scene, resets to the robot's keyframe, and steps it for `duration_s` (≀ 60 s). With `controller: "hold"` the robot keeps its starting pose (position servos track it on the G1 and Panda; a PD law drives the H1's and Go2's torque motors); with `"passive"` actuation is off. It writes: - **`run.mcap`**, readable in Foxglove: `/scene` (`foxglove.SceneUpdate`, every visible geom; robot meshes built from the compiled model), `/tf` (`foxglove.FrameTransforms`, every body at 30 Hz), `/state` (JSON: time, root height, fallen, contacts) - **`video.mp4`** and **`video.webm`**: 1280Γ—720 from a camera tracking the robot, as H.264 and VP9 so every browser plays it. Rendered on the GPU via EGL (or on the CPU with `MODAL_GPU=none`). If EGL fails the worker re-renders on the CPU, and if rendering breaks mid-run the simulation and MCAP still complete and the summary says why there is no video. - **`summary.json`**: simulated vs. wall time, whether and when the robot fell, instabilities, MuJoCo warnings - **`trajectory.npz`**: the robot's joint positions, velocities and controls, and every body's pose, at 30 Hz (used by the dataset export and the Isaac Sim render) The report page refreshes itself while the job is queued or running. Report links are signed (`REPORT_SECRET`), so testers can open or share them in a browser without their API key. File links are presigned R2 URLs valid for 7 days. The 3D replay uses Foxglove's official embedding library ([`@foxglove/embed`](https://www.npmjs.com/package/@foxglove/embed), MIT, vendored in `api/static/foxglove-embed/`): an `embed.foxglove.dev` frame that loads the MCAP from R2 as a `remote-file` data source. Viewers need a free Foxglove account, and it prompts for sign-in the first time. Set `FOXGLOVE_ORG` to your Foxglove organization slug so testers are signed into it. The page also links the `.mcap` for download (it opens in any Foxglove app). ### One-time setup 1. **Cloudflare R2**: create a bucket (e.g. `rosdiff-runs`) and an R2 API token with *Object Read & Write* on it. Put `R2_ACCOUNT_ID`, `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY` and `R2_BUCKET` in `.env`, then: ```bash python -m api.storage check # uploads a test object, prints a presigned link python -m api.storage cors # lets embed.foxglove.dev / app.foxglove.dev read run files (GET, ranges) python -m api.storage expire --days 30 # delete run/job files after 30 days (stays inside R2's free 10 GB) python -m api.storage usage # how much the bucket holds ``` 2. **Modal** (default, `SIM_BACKEND=modal`). The free Starter plan includes $30 of compute credit a month (check [modal.com/pricing](https://modal.com/pricing): a T4 is roughly $0.59/hour, an L4 $0.80/hour). From the repository root: ```bash pip install modal && modal token new # once; or set MODAL_TOKEN_ID / MODAL_TOKEN_SECRET modal secret create rosdiff-r2 R2_ACCOUNT_ID=... R2_ACCESS_KEY_ID=... R2_SECRET_ACCESS_KEY=... R2_BUCKET=... modal deploy -m worker.modal_app # runs, sweeps, ROS 2 sandbox, LeRobot export (as zip) ``` That basic deploy stores nothing on Modal (no Volumes), so with a $0 spend limit nothing can be billed. Opt-in extras, each enabled by an environment variable at deploy time: ```bash modal secret create rosdiff-hf HF_TOKEN=hf_... # Hugging Face token (write access) MODAL_HF_SECRET=rosdiff-hf modal deploy -m worker.modal_app # exports go to the Hub ENABLE_COSMOS=1 MODAL_HF_SECRET=rosdiff-hf modal deploy -m worker.modal_app # + Cosmos Reason review ENABLE_COSMOS=1 modal run -m worker.modal_app::download_cosmos # once: ~16 GB into a Volume modal deploy -m worker.modal_isaac # Isaac Sim renders (large NGC image) ``` The Cosmos weights live in the Modal Volume `rosdiff-models`; Modal bills Volume storage even with a $0 spend limit. `modal volume delete rosdiff-models` removes it when you are done. Use the same variables on every later deploy, or the extras are removed again. Deploy-time settings (environment variables, read by the Modal files): `MODAL_GPU` (default `T4`; `none` renders on CPU, cheaper still), `MODAL_MAX_CONTAINERS` (default 2 at once; the rest queue), `RUN_EXECUTION_TIMEOUT_S` (default 600), `COSMOS_MODEL`, `ISAAC_IMAGE` (default `nvcr.io/nvidia/isaac-sim:5.0.0`) and `MODAL_NGC_SECRET` (a Modal secret with `REGISTRY_USERNAME=$oauthtoken` and `REGISTRY_PASSWORD=`, if NGC asks for a login). Deploying Isaac Sim means accepting NVIDIA's Isaac Sim license. The API server needs the same Modal token in its environment, and `HF_NAMESPACE` (your Hugging Face user or org) for dataset exports. 3. **RunPod instead** (`SIM_BACKEND=runpod`): runs and sweeps only (the other tools need Modal). Build `worker/Dockerfile` from the repository root, push it to a registry, and create a serverless endpoint from it (active workers 0, max workers 2, execution timeout 600 s, the four `R2_*` variables set on the endpoint). Put the endpoint ID and an API key in `.env` as `RUNPOD_ENDPOINT_ID` / `RUNPOD_API_KEY`. 4. **Report links**: set `REPORT_SECRET` (`openssl rand -hex 32`) and `PUBLIC_BASE_URL` (the address testers use). 5. **Spend limit**: every job's cost is estimated from its run time (`JOB_PRICES`, `JOB_OVERHEAD_S`). A new job is refused with HTTP 402 unless its worst case still fits in the tester's equal share and in `MONTHLY_CREDIT_LIMIT_USD` (default $29.98; see *GPU budget* above); everything else keeps working. Likewise, while the R2 bucket holds more than `STORAGE_LIMIT_GB` (default 9 GB for all testers together, under R2's free 10 GB) new jobs are refused with HTTP 507 until old files expire. The estimate is not Modal's bill, so also set a budget or usage alert in Modal's workspace settings (*Usage and Billing*). On RunPod, set a spend notification. `GET /health` shows `"runs_enabled": true` and the backend once it and R2 are configured. Otherwise `/simulate/run` answers 503 and everything else keeps working. ### Usage visibility Stage 1 runs are founder-funded for 4 testers, each with an equal GPU share (above) and an equal AI budget. On top of that: - every `/simulate/run` call is a row in the `runs` table: tester, time, source generation, duration, backend + job ID, status, GPU time (RunPod's billed figure, or the worker's own timing on Modal) - the server logs a warning when one tester starts more than `RUN_BURST_WARN` (10) runs within 10 minutes, e.g. a script looping by mistake. The run still goes through. - `python -m api.review runs` lists every job with its kind and estimated cost. `python -m api.review usage` shows this month's estimated spend against the limit, and per tester: total jobs, last 24 h, busiest 10-minute window, compute seconds and estimated cost. Testers see the same monthly figure with `GET /jobs`. To stop a runaway key immediately: `python -m api.keys revoke `. ## Setup Requirements: Python β‰₯ 3.10, git, and for the default ROS 2 backend Node.js + OpenCode (`npm install -g opencode-ai`). No ROS or GPU needed on the server. ```bash python3 -m venv .venv && . .venv/bin/activate pip install -r requirements.txt cp .env.example .env # set LLM_API_KEY (an OpenRouter key) set -a; source .env; set +a python -m rag.ingest # clones the sources, builds data/chroma (~2 min, ~90 MB + an 80 MB embedding model) pip install -r requirements-dev.txt && python -m pytest # offline tests, no key or network needed uvicorn api.main:app --host 0.0.0.0 --port 8000 curl localhost:8000/health ``` `python -m rag.ingest --only mujoco --reset` rebuilds one collection; what gets indexed is listed in [`rag/sources.yaml`](rag/sources.yaml). **Hosting for testers:** to keep RosDiff online while your PC is off, run it on a free Oracle Cloud machine: see [`deploy/oracle/README.md`](deploy/oracle/README.md) (one script, `deploy/oracle/setup.sh`). ## Adding a tester Keys are stored in `testers.yaml` as SHA-256 hashes only; the plain key is printed once. The server re-reads the file when it changes, so no restart is needed. ```bash python -m api.keys add alice --note "Alice, humanoid lab" # Added tester 'alice' to testers.yaml. # API key (shown once, send it to the tester privately): # # rd_7Qm... python -m api.keys list python -m api.keys revoke alice ``` Send the tester their key privately along with the tester instructions below. For a quick local session you can also set `ROSDIFF_API_KEYS="name:key,other:key2"` in the environment. ## For testers **Use the web app:** open the server's address in a browser (e.g. `http://:8000/`) and sign in with your key. Simulation is on the left, ROS 2 code on the right: describe what you want, press Enter, then **Run simulation** or **Build & run**. Everything you make is under History. The same works from the command line: ```bash export ROSDIFF_KEY=rd_... # your key export ROSDIFF_URL=https://... # the address you were given # A MuJoCo scene around the Unitree G1 curl -s $ROSDIFF_URL/simulate/generate -H "X-API-Key: $ROSDIFF_KEY" -H 'content-type: application/json' \ -d '{"description": "Unitree G1 facing a staircase of 5 steps, 15 cm high, 30 cm deep, starting 1.5 m ahead"}' \ | tee scene.json | python3 -c 'import json,sys; r=json.load(sys.stdin); print(r["id"], r["valid"]); open("scene_rosdiff.xml","w").write(r["mjcf"] or "")' # Run it: copy scene_rosdiff.xml into mujoco_menagerie/unitree_g1/ and # python -m mujoco.viewer --mjcf mujoco_menagerie/unitree_g1/scene_rosdiff.xml # A ROS 2 package curl -s $ROSDIFF_URL/code/generate -H "X-API-Key: $ROSDIFF_KEY" -H 'content-type: application/json' \ -d '{"request": "Stop the robot when a LaserScan range ahead is below 0.5 m; publish Twist on /cmd_vel", "distro": "humble", "language": "python"}' > code.json python3 -c 'import json,os; r=json.load(open("code.json")); print(r["id"], r["valid"], r["validation"]); [os.makedirs(os.path.dirname(p) or ".", exist_ok=True) or open(p,"w").write(c) for p,c in r["files"].items()]' # Build it in a colcon workspace: colcon build && source install/setup.bash && ros2 launch ... # Run a valid scene on a GPU (10 s, robot holding its standing pose) and open the report in a browser curl -s $ROSDIFF_URL/simulate/run -H "X-API-Key: $ROSDIFF_KEY" -H 'content-type: application/json' \ -d '{"generation_id": "", "duration_s": 10, "controller": "hold"}' # -> {"run_id": "...", "status": "IN_QUEUE", "report_url": "https://.../simulate/report/?token=..."} # Other tools on a finished run (each answers with a report_url): curl -s -X POST $ROSDIFF_URL/simulate/sweep -H "X-API-Key: $ROSDIFF_KEY" -H 'content-type: application/json' \ -d '{"run_id": "", "n_worlds": 64, "push_max_n": 80}' # how robust is it? curl -s -X POST $ROSDIFF_URL/simulate/run//isaac -H "X-API-Key: $ROSDIFF_KEY" # Isaac Sim render curl -s -X POST $ROSDIFF_URL/datasets/export -H "X-API-Key: $ROSDIFF_KEY" -H 'content-type: application/json' \ -d '{"run_ids": ["", ""], "name": "g1-stairs"}' # LeRobot dataset # Build, lint, test and launch a generated ROS 2 package in the sandbox: curl -s $ROSDIFF_URL/code/check -H "X-API-Key: $ROSDIFF_KEY" -H 'content-type: application/json' \ -d '{"generation_id": ""}' curl -s $ROSDIFF_URL/jobs -H "X-API-Key: $ROSDIFF_KEY" # your jobs + this month's budget # Then please tell us whether it actually worked: curl -s $ROSDIFF_URL/feedback/ -H "X-API-Key: $ROSDIFF_KEY" -H 'content-type: application/json' \ -d '{"rating": "works", "note": "robot stood on the second step"}' ``` `valid: true` means the automatic checks passed (MuJoCo loads and simulates the scene / the package is well-formed and uses real message types). It does not mean the scene matches your description or the code behaves correctly β€” that is what your feedback is for. ## Reviewing what testers generated Every request is one row in `data/generations.sqlite3`: tester, request, retrieved sources, every model attempt with its validation result, the final output, duration, the tester's feedback and your review. ```bash python -m api.review summary # per kind: total, auto-valid, tester "works", reviewed correct python -m api.review list --unreviewed # newest first python -m api.review show --attempts # request, sources, every raw attempt, output python -m api.review files review/ # write the scene / package to disk to run it yourself python -m api.review mark correct --note "stairs dimensions right, landing missing" python -m api.review export stage1.jsonl # everything, for analysis ``` Verdicts are `correct`, `partial` or `incorrect`. The summary line is the Stage 2 decision input: how often the output was automatically valid, how often testers say it works, and how often a reviewer agrees. ## Test scenarios `scenarios/mujoco/*.yaml` (5 scenes: pushable boxes, ramp, stairs, heightfield terrain, table with objects) and `scenarios/ros2/*.yaml` (4 packages: LaserScan safety stop, C++ IMU filter on Jazzy, service + client + launch, odometry-to-path). Each has a few structural checks (e.g. "at least 5 box geoms", "contains create_service"). ```bash python -m api.keys add orcanion-eval python -m scenarios.run --key rd_... --save eval_out/ # through the real API, so it is logged like any tester ``` ## CI `.github/workflows/ci.yml` runs on every push: - lint and the test suite (the Modal app definitions are import-checked there); - builds `worker/Dockerfile`, runs a real 2-second G1 job inside it (CPU rendering) and checks the MCAP, videos, summary and trajectory, rejects an unsafe scene, and runs a Go2 sweep with both engines (MuJoCo, MuJoCo Warp); - builds the ROS 2 sandbox for Humble and Jazzy and checks four sample packages (builds and runs, C++ with tests, does not compile, crashes on start); - simulates two Go2 runs, exports them with LeRobot and loads the dataset back. Not exercised in CI (needs a GPU and the model or NGC image): the Cosmos Reason inference and the Isaac Sim render. Their surrounding code (prompt, parsing, USD stage, endpoints) is tested. ## Repository layout ``` api/ FastAPI app, LLM client, MJCF + ROS 2 validation, OpenCode runner, jobs + budget, Modal/RunPod clients, R2 storage, report pages, key + review CLIs, SQLite log worker/ jobs: MuJoCo runs (sim_runner, handler), sweeps (sweep), ROS 2 sandbox (ros_runner, ros/), Cosmos review, LeRobot export, Isaac Sim replay (usd_scene, isaac_render); modal_app.py / modal_isaac.py for Modal, Dockerfile for RunPod rag/ sources.yaml, chunking, Chroma store, ingest CLI scenarios/ test cases + runner tests/ offline tests (fake robot, hash embeddings, scripted model, fake OpenCode binary) ``` ## Stage 1 boundaries In scope: the two generation endpoints, GPU runs of validated scenes and the tools above, one model (GLM-5.3 via OpenRouter) and one platform key for everyone, a key allowlist, logging, usage visibility and review. Deliberately **not** here: billing, tiers or model selection, signup/OAuth, per-tester run caps (one shared monthly limit protects the free credit), per-user OpenCode keys. Generated ROS 2 code runs only inside the network-less sandbox.