Spaces:
Paused
Paused
| title: Rosdiff | |
| emoji: 🚀 | |
| colorFrom: blue | |
| colorTo: red | |
| sdk: docker | |
| pinned: false | |
| license: mit | |
| # RosDiff — Stage 1 (internal tester build) | |
| > **Handover:** [`docs/PROJECT_REPORT.md`](docs/PROJECT_REPORT.md) summarises everything built so far, the decisions | |
| > behind it, and how to move the project to another environment. | |
| Orcanion internal build for a small, hand-picked group of testers. It exists to measure one thing: **how good | |
| the generated simulation scenes and ROS 2 code are**, before any commercial infrastructure gets built. | |
| No payments, no tiers, no signup — a handful of API keys, one model, and a log of everything. | |
| | Endpoint | Input | Output | | |
| |---|---|---| | |
| | `POST /simulate/generate` | scenario description + robot | MuJoCo MJCF scene around the robot, load-tested in MuJoCo | | |
| | `POST /code/generate` | ROS 2 request (Humble/Jazzy, Python/C++) | a ROS 2 package, statically validated against the real message definitions (`check: true` also runs it) | | |
| | `POST /code/check` | a generated package | built, linted, tested and launched in a ROS 2 sandbox; report with logs and the recorded bag | | |
| | `POST /simulate/run` | a valid scene (`generation_id`, or raw MJCF) | a cloud GPU run (Modal, or RunPod): video, Foxglove replay, trajectory; reviewed by Cosmos Reason | | |
| | `POST /simulate/sweep` | a run, generation or MJCF | the scene ×N with randomized friction, masses and pushes (MuJoCo Warp): fall rate | | |
| | `POST /simulate/run/{id}/review` | a finished run | Cosmos Reason's verdict on the video (opt-in; `AUTO_REVIEW=true` reviews every run) | | |
| | `POST /simulate/run/{id}/isaac` | a finished run | the same motion rendered with Isaac Sim RTX (10 a day) | | |
| | `POST /datasets/export` | finished runs | a LeRobot dataset on the Hugging Face Hub | | |
| | `GET /report/{id}` | (signed link) | HTML report for any job, with its follow-up jobs | | |
| | `GET /jobs`, `GET /jobs/{id}` | — | your jobs, their status, and this month's budget | | |
| | `POST /feedback/{id}` | `works` / `partly` / `broken` + note | the tester's verdict after running the output | | |
| | `GET /health` | — | model, index size, OpenCode availability (no key needed) | | |
| Robots: Unitree **G1** and **H1** humanoids, Unitree **Go2** quadruped, **Franka Panda** arm (all from MuJoCo | |
| Menagerie). Generated ROS 2 code only ever runs inside the sandbox (no network, no secrets). | |
| ### Models, budgets and prompt enhancement | |
| Testers never pick a model. **Automatic routing** (`api/routing.py`) rates every request's complexity and sends it | |
| to the cheapest capable tier; a draft that fails validation is repaired by a stronger one. Scenes and code have | |
| their own model lists: capable Chinese open-weight models on OpenRouter, chosen by benchmark and price rather | |
| than release date (override with `SCENE_MODELS` / `CODE_MODELS`): | |
| | Tier | Code | Scenes | | |
| |---|---|---| | |
| | low | `deepseek/deepseek-v4-flash` (79.0% SWE-bench Verified, $0.09/$0.18 per M) | (not used: scenes start at mid) | | |
| | mid | `deepseek/deepseek-v4-pro` (80.6% SWE-bench Verified) | `z-ai/glm-5.2` (62.1% SWE-bench Pro, close to Claude Opus 4.8 on FrontierSWE) | | |
| | high | `z-ai/glm-5.2` | `moonshotai/kimi-k2.6` (80.2% SWE-bench Verified, top open-weight Intelligence Index) | | |
| **Free debugger.** The first repair after a failed check goes to a free DeepSeek V4 Flash through OpenCode Zen | |
| (`DEBUG_MODEL`, default `opencode/deepseek-v4-flash-free`), not OpenRouter, so it costs nothing and is logged as | |
| exempt. If that call fails (rate limit, the free offer withdrawn, OpenCode missing) the same repair goes to the | |
| next paid tier at once; a further repair keeps climbing. Code gets 3 attempts (`CODE_MAX_ATTEMPTS`) so the free | |
| try never replaces the stronger paid repair. `FREE_DEBUG=false` turns it off. | |
| The web app shows the choice as information ("AI Routing: automatic · Standard tier · Why?"), plus the simulator | |
| it picked (MuJoCo, MuJoCo Warp for "how often does it…" questions, Isaac Sim for photoreal renders) with a | |
| **Use another simulator** menu. Gazebo, Webots and PyBullet are listed as not available yet. | |
| **Prompt enhancement (free).** Before generating, the tester's words are rewritten into a precise brief by | |
| OpenCode with a free OpenCode Zen model (`ENHANCE_MODEL`, default `opencode/deepseek-v4-flash-free`). It never | |
| goes through OpenRouter and costs nothing; the call is logged as exempt. The brief must keep every number the | |
| tester wrote, or it is thrown away, and any failure (OpenCode missing, timeout, a rotated-out free model) just | |
| means the original text is used. Routing and cost estimates still use the tester's own words. The app shows the | |
| brief under "✨ Enhanced prompt". It needs `opencode` installed on the API server (`npm i -g opencode-ai`); | |
| `PROMPT_ENHANCE=false` switches it off, `ZEN_API_KEY` supplies a Zen key if the free model asks for one. | |
| **AI budget (rupees).** One pool, `BUDGET_TOTAL_INR` (₹500), and an equal allocation per tester, | |
| `BUDGET_PER_USER_INR` (₹125). It covers every billable OpenRouter call: coding, simulation AI, automatic repairs | |
| ("debugging") and testing. Costs come from OpenRouter's reported usage (list prices otherwise). Free and | |
| OpenRouter-exempt calls are recorded but not deducted. Tasks whose worst case is above `CONFIRM_ABOVE_INR` show | |
| an estimate and wait for "Run". Testers see their balance in the top bar and under *My usage*; testers listed in | |
| `ADMIN_TESTERS` also see the whole team. | |
| **GPU budget (dollars).** Simulations, sweeps, renders and build checks run on Modal's credit, | |
| `MONTHLY_CREDIT_LIMIT_USD` (default $29.98), split equally between `COMPUTE_TESTERS` (4): $7.495 each per month | |
| (`COMPUTE_PER_TESTER_USD` overrides the share). The limit is strict: a job starts only if its **worst case** | |
| (the cost of running until its timeout) still fits in both the tester's share and the team's limit, and jobs | |
| still running count at that worst case, so parallel jobs can never overshoot. Finished jobs count at their | |
| measured cost. | |
| ## How it works | |
| ``` | |
| ┌──────────── RAG (local Chroma, data/chroma) ────────────┐ | |
| │ mujoco: MuJoCo XML reference + modeling docs, MJCF │ | |
| │ schema (every element's valid attributes), │ | |
| │ Menagerie unitree_g1 │ | |
| │ ros2: ROS 2 Humble + Jazzy tutorials & launch docs, │ | |
| │ .msg/.srv definitions (common_interfaces, │ | |
| │ rcl_interfaces, example_interfaces) │ | |
| └───────────────────────┬─────────────────────────────────┘ | |
| │ | |
| /simulate/generate ─► retrieve ─► LLM ─────► extract MJCF ─► static safety check ─► MuJoCo load + 2 s step ─┐ | |
| ▲ (subprocess, timeout) │ | |
| └───────────── repair: MuJoCo's error + valid attributes ◄───────────────┘ | |
| /code/generate ─────► retrieve ─► OpenCode (same LLM, headless) ─► files ─► ROS 2 validator ─┐ | |
| ▲ │ | |
| └──────── repair in the same OpenCode session ◄─────────┘ | |
| every request ──► data/generations.sqlite3 ──► python -m api.review | |
| ``` | |
| **MuJoCo scenes.** The model never writes the robot. Scenes `<include file="g1.xml"/>` from MuJoCo Menagerie, | |
| exactly like Menagerie's own `scene.xml`, and the model only builds the world around it. The prompt carries: | |
| robot facts read from the model file itself (bodies, joints with ranges, actuators, sites, keyframes), Menagerie's | |
| example scene, the exact attribute list of the ~16 elements scenes use (from MuJoCo's XML schema), and the | |
| retrieved documentation sections. The result is loaded with MuJoCo and stepped for 2 simulated seconds in a | |
| separate process; if loading fails, MuJoCo's error message (plus the valid attributes of the offending element) | |
| goes back to the model, up to 3 attempts. Scenes may not reference files other than `g1.xml`, absolute paths | |
| or `..`. | |
| **ROS 2 packages.** Named message/service types in the request (`LaserScan`, `nav_msgs/Odometry`, …) are looked | |
| up exactly, together with the types they contain; related definitions and distro-specific tutorials are | |
| retrieved from the vector store. OpenCode runs headless (`opencode run`) in a throwaway workspace with the | |
| configured LLM as its only provider, file edits allowed inside the workspace and shell/web access denied; the RAG | |
| context is attached to its message. The files it writes are validated without a ROS install: package layout | |
| (`package.xml`, `setup.py` / `CMakeLists.txt` with `ament_package()`), Python and launch-file syntax, | |
| `generate_launch_description()`, XML launch files — and **every message/service type used must exist in the | |
| requested distro** (types from packages that aren't indexed are flagged as unverified warnings). Problems go | |
| back to the same OpenCode session for one repair round. `CODEGEN_BACKEND=direct` uses a plain chat call instead | |
| of OpenCode. | |
| ## Tools | |
| Everything that runs on a GPU or in a container is a *job* on the backend (Modal by default). Each job has a | |
| report page, an estimated cost, and counts toward the monthly budget. | |
| | Tool | What it answers | Where it runs | Est. cost | | |
| |---|---|---|---| | |
| | **Simulation run** (MuJoCo) | does the scene work, does the robot stay up? | Modal T4 | ~1¢ | | |
| | **ROS 2 sandbox** (colcon, ament linters, launch_testing, ros2_control, Nav2, MoveIt 2, xacro, robot_state_publisher, tf2, rosbag2 MCAP) | does the generated package build, pass its tests, and start? which nodes and topics come up? | Modal CPU, no network | <1¢ | | |
| | **Robustness sweep** (MuJoCo Warp, NVIDIA Warp) | how often does the robot fall across 64–256 randomized copies? | Modal T4 | a few ¢ | | |
| | **Cosmos Reason review** (NVIDIA) | does the video match the description, is the physics plausible? | Modal L4 | ~2–3¢ | | |
| | **LeRobot export** (Hugging Face) | runs → a robot-learning dataset | Modal CPU | <1¢ | | |
| | **Isaac Sim render** (NVIDIA) | the same motion, photoreal (RTX) | Modal L4, 10/day | ~5–10¢ | | |
| - **ROS 2 sandbox** (`worker/ros/Dockerfile`, `worker/ros_runner.py`): the package is built with `colcon`, checked | |
| with `ament_flake8`/`pep257` (Python) or `ament_cpplint`/`cppcheck` (C++) and `ament_xmllint`, its tests run with | |
| `colcon test`, then its launch file (or each executable) runs for 10 s while `ros2 bag record -a` captures every | |
| topic to MCAP. Build errors, missing dependencies, crashed nodes and the live node/topic list are on the report; | |
| the bag opens in the embedded Foxglove viewer. Humble and Jazzy images; generated code gets no network and no | |
| secrets. CI checks four sample packages in both distros. | |
| - **Robustness sweep** (`worker/sweep.py`): every world gets its own friction scale, free-object mass scale, joint | |
| noise and a push on the robot's root. MuJoCo Warp steps all worlds together on the GPU; without CUDA the same | |
| randomization runs in plain MuJoCo (results agree within a millimetre; CI runs both). The report plots push | |
| against friction for every world. | |
| - **Cosmos Reason** (`worker/cosmos_review.py`): NVIDIA's physical-reasoning vision-language model watches the | |
| video at 2 fps with the scenario text and the simulator's facts, and answers matches / plausible / fell with its | |
| reasoning. A first pass for reviewers, not a verdict. Model: `COSMOS_MODEL` (default `nvidia/Cosmos-Reason1-7B`). | |
| - **LeRobot export** (`worker/lerobot_export.py`): each run becomes an episode with `observation.state` | |
| (robot joints), `observation.velocity`, `action` (commanded controls), `observation.images.main` (640×360 video) | |
| and the scenario as the task. Pushed as a private dataset to `HF_NAMESPACE/rosdiff-<name>`, or stored as a zip | |
| when no Hugging Face token is configured. | |
| - **Isaac Sim render** (`worker/usd_scene.py`, `worker/isaac_render.py`): the run is turned into a USD stage with | |
| every body's pose time-sampled and a tracking camera; Isaac Sim renders it with RTX. No second physics engine, | |
| so it is the same motion as the MuJoCo video. | |
| ## Simulation runs | |
| ``` | |
| POST /simulate/run ─► re-validate the scene ─► log (tester, time) ─► Modal GPU job (T4, or RunPod) | |
| │ worker/: MuJoCo, robot holds its | |
| │ standing pose (or passive), records | |
| ▼ | |
| GET /simulate/report/{id} ◄── presigned R2 links ◄── Cloudflare R2: runs/<id>/run.mcap, video.mp4, | |
| status · summary · video · Foxglove iframe summary.json, scene.xml | |
| ``` | |
| The worker ([`worker/`](worker/)) loads the scene, resets to the robot's keyframe, and steps it for | |
| `duration_s` (≤ 60 s). With `controller: "hold"` the robot keeps its starting pose (position servos track it on | |
| the G1 and Panda; a PD law drives the H1's and Go2's torque motors); with `"passive"` actuation is off. It writes: | |
| - **`run.mcap`**, readable in Foxglove: `/scene` (`foxglove.SceneUpdate`, every visible geom; robot meshes built | |
| from the compiled model), `/tf` (`foxglove.FrameTransforms`, every body at 30 Hz), `/state` (JSON: time, root | |
| height, fallen, contacts) | |
| - **`video.mp4`** and **`video.webm`**: 1280×720 from a camera tracking the robot, as H.264 and VP9 so every browser | |
| plays it. Rendered on the GPU via EGL (or on the CPU with `MODAL_GPU=none`). If EGL fails the worker re-renders on the CPU, and if rendering breaks | |
| mid-run the simulation and MCAP still complete and the summary says why there is no video. | |
| - **`summary.json`**: simulated vs. wall time, whether and when the robot fell, instabilities, MuJoCo warnings | |
| - **`trajectory.npz`**: the robot's joint positions, velocities and controls, and every body's pose, at 30 Hz | |
| (used by the dataset export and the Isaac Sim render) | |
| The report page refreshes itself while the job is queued or running. Report links are signed (`REPORT_SECRET`), so | |
| testers can open or share them in a browser without their API key. File links are presigned R2 URLs valid for | |
| 7 days. The 3D replay uses Foxglove's official embedding library ([`@foxglove/embed`](https://www.npmjs.com/package/@foxglove/embed), | |
| MIT, vendored in `api/static/foxglove-embed/`): an `embed.foxglove.dev` frame that loads the MCAP from R2 as a | |
| `remote-file` data source. Viewers need a free Foxglove account, and it prompts for sign-in the first time. Set | |
| `FOXGLOVE_ORG` to your Foxglove organization slug so testers are signed into it. The page also links the `.mcap` | |
| for download (it opens in any Foxglove app). | |
| ### One-time setup | |
| 1. **Cloudflare R2**: create a bucket (e.g. `rosdiff-runs`) and an R2 API token with *Object Read & Write* on it. | |
| Put `R2_ACCOUNT_ID`, `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY` and `R2_BUCKET` in `.env`, then: | |
| ```bash | |
| python -m api.storage check # uploads a test object, prints a presigned link | |
| python -m api.storage cors # lets embed.foxglove.dev / app.foxglove.dev read run files (GET, ranges) | |
| python -m api.storage expire --days 30 # delete run/job files after 30 days (stays inside R2's free 10 GB) | |
| python -m api.storage usage # how much the bucket holds | |
| ``` | |
| 2. **Modal** (default, `SIM_BACKEND=modal`). The free Starter plan includes $30 of compute credit a month | |
| (check [modal.com/pricing](https://modal.com/pricing): a T4 is roughly $0.59/hour, an L4 $0.80/hour). From the | |
| repository root: | |
| ```bash | |
| pip install modal && modal token new # once; or set MODAL_TOKEN_ID / MODAL_TOKEN_SECRET | |
| modal secret create rosdiff-r2 R2_ACCOUNT_ID=... R2_ACCESS_KEY_ID=... R2_SECRET_ACCESS_KEY=... R2_BUCKET=... | |
| modal deploy -m worker.modal_app # runs, sweeps, ROS 2 sandbox, LeRobot export (as zip) | |
| ``` | |
| That basic deploy stores nothing on Modal (no Volumes), so with a $0 spend limit nothing can be billed. | |
| Opt-in extras, each enabled by an environment variable at deploy time: | |
| ```bash | |
| modal secret create rosdiff-hf HF_TOKEN=hf_... # Hugging Face token (write access) | |
| MODAL_HF_SECRET=rosdiff-hf modal deploy -m worker.modal_app # exports go to the Hub | |
| ENABLE_COSMOS=1 MODAL_HF_SECRET=rosdiff-hf modal deploy -m worker.modal_app # + Cosmos Reason review | |
| ENABLE_COSMOS=1 modal run -m worker.modal_app::download_cosmos # once: ~16 GB into a Volume | |
| modal deploy -m worker.modal_isaac # Isaac Sim renders (large NGC image) | |
| ``` | |
| The Cosmos weights live in the Modal Volume `rosdiff-models`; Modal bills Volume storage even with a $0 spend | |
| limit. `modal volume delete rosdiff-models` removes it when you are done. Use the same variables on every | |
| later deploy, or the extras are removed again. | |
| Deploy-time settings (environment variables, read by the Modal files): `MODAL_GPU` (default `T4`; `none` | |
| renders on CPU, cheaper still), `MODAL_MAX_CONTAINERS` (default 2 at once; the rest queue), | |
| `RUN_EXECUTION_TIMEOUT_S` (default 600), `COSMOS_MODEL`, `ISAAC_IMAGE` (default | |
| `nvcr.io/nvidia/isaac-sim:5.0.0`) and `MODAL_NGC_SECRET` (a Modal secret with `REGISTRY_USERNAME=$oauthtoken` | |
| and `REGISTRY_PASSWORD=<NGC API key>`, if NGC asks for a login). Deploying Isaac Sim means accepting NVIDIA's | |
| Isaac Sim license. The API server needs the same Modal token in its environment, and `HF_NAMESPACE` (your | |
| Hugging Face user or org) for dataset exports. | |
| 3. **RunPod instead** (`SIM_BACKEND=runpod`): runs and sweeps only (the other tools need Modal). Build | |
| `worker/Dockerfile` from the repository root, push it to a registry, and create a serverless endpoint from it | |
| (active workers 0, max workers 2, execution timeout 600 s, the four `R2_*` variables set on the endpoint). Put | |
| the endpoint ID and an API key in `.env` as `RUNPOD_ENDPOINT_ID` / `RUNPOD_API_KEY`. | |
| 4. **Report links**: set `REPORT_SECRET` (`openssl rand -hex 32`) and `PUBLIC_BASE_URL` (the address testers use). | |
| 5. **Spend limit**: every job's cost is estimated from its run time (`JOB_PRICES`, `JOB_OVERHEAD_S`). A new job is | |
| refused with HTTP 402 unless its worst case still fits in the tester's equal share and in | |
| `MONTHLY_CREDIT_LIMIT_USD` (default $29.98; see *GPU budget* above); everything else keeps working. Likewise, while the R2 bucket holds more than `STORAGE_LIMIT_GB` | |
| (default 9 GB for all testers together, under R2's free 10 GB) new jobs are refused with HTTP 507 until old files expire. The estimate is not Modal's bill, so also set a budget | |
| or usage alert in Modal's workspace settings (*Usage and Billing*). On RunPod, set a spend notification. | |
| `GET /health` shows `"runs_enabled": true` and the backend once it and R2 are configured. Otherwise `/simulate/run` answers | |
| 503 and everything else keeps working. | |
| ### Usage visibility | |
| Stage 1 runs are founder-funded for 4 testers, each with an equal GPU share (above) and an equal AI budget. On top | |
| of that: | |
| - every `/simulate/run` call is a row in the `runs` table: tester, time, source generation, duration, backend + job | |
| ID, status, GPU time (RunPod's billed figure, or the worker's own timing on Modal) | |
| - the server logs a warning when one tester starts more than `RUN_BURST_WARN` (10) runs within 10 minutes, e.g. a | |
| script looping by mistake. The run still goes through. | |
| - `python -m api.review runs` lists every job with its kind and estimated cost. `python -m api.review usage` shows | |
| this month's estimated spend against the limit, and per tester: total jobs, last 24 h, busiest 10-minute window, | |
| compute seconds and estimated cost. Testers see the same monthly figure with `GET /jobs`. | |
| To stop a runaway key immediately: `python -m api.keys revoke <name>`. | |
| ## Setup | |
| Requirements: Python ≥ 3.10, git, and for the default ROS 2 backend Node.js + OpenCode | |
| (`npm install -g opencode-ai`). No ROS or GPU needed on the server. | |
| ```bash | |
| python3 -m venv .venv && . .venv/bin/activate | |
| pip install -r requirements.txt | |
| cp .env.example .env # set LLM_API_KEY (an OpenRouter key) | |
| set -a; source .env; set +a | |
| python -m rag.ingest # clones the sources, builds data/chroma (~2 min, ~90 MB + an 80 MB embedding model) | |
| pip install -r requirements-dev.txt && python -m pytest # offline tests, no key or network needed | |
| uvicorn api.main:app --host 0.0.0.0 --port 8000 | |
| curl localhost:8000/health | |
| ``` | |
| `python -m rag.ingest --only mujoco --reset` rebuilds one collection; what gets indexed is listed in | |
| [`rag/sources.yaml`](rag/sources.yaml). | |
| **Hosting for testers:** to keep RosDiff online while your PC is off, run it on a free Oracle Cloud machine: | |
| see [`deploy/oracle/README.md`](deploy/oracle/README.md) (one script, `deploy/oracle/setup.sh`). | |
| ## Adding a tester | |
| Keys are stored in `testers.yaml` as SHA-256 hashes only; the plain key is printed once. The server re-reads the | |
| file when it changes, so no restart is needed. | |
| ```bash | |
| python -m api.keys add alice --note "Alice, humanoid lab" | |
| # Added tester 'alice' to testers.yaml. | |
| # API key (shown once, send it to the tester privately): | |
| # | |
| # rd_7Qm... | |
| python -m api.keys list | |
| python -m api.keys revoke alice | |
| ``` | |
| Send the tester their key privately along with the tester instructions below. For a quick local session you can | |
| also set `ROSDIFF_API_KEYS="name:key,other:key2"` in the environment. | |
| ## For testers | |
| **Use the web app:** open the server's address in a browser (e.g. `http://<server>:8000/`) and sign in with your | |
| key. Simulation is on the left, ROS 2 code on the right: describe what you want, press Enter, then **Run | |
| simulation** or **Build & run**. Everything you make is under History. The same works from the command line: | |
| ```bash | |
| export ROSDIFF_KEY=rd_... # your key | |
| export ROSDIFF_URL=https://... # the address you were given | |
| # A MuJoCo scene around the Unitree G1 | |
| curl -s $ROSDIFF_URL/simulate/generate -H "X-API-Key: $ROSDIFF_KEY" -H 'content-type: application/json' \ | |
| -d '{"description": "Unitree G1 facing a staircase of 5 steps, 15 cm high, 30 cm deep, starting 1.5 m ahead"}' \ | |
| | tee scene.json | python3 -c 'import json,sys; r=json.load(sys.stdin); print(r["id"], r["valid"]); open("scene_rosdiff.xml","w").write(r["mjcf"] or "")' | |
| # Run it: copy scene_rosdiff.xml into mujoco_menagerie/unitree_g1/ and | |
| # python -m mujoco.viewer --mjcf mujoco_menagerie/unitree_g1/scene_rosdiff.xml | |
| # A ROS 2 package | |
| curl -s $ROSDIFF_URL/code/generate -H "X-API-Key: $ROSDIFF_KEY" -H 'content-type: application/json' \ | |
| -d '{"request": "Stop the robot when a LaserScan range ahead is below 0.5 m; publish Twist on /cmd_vel", | |
| "distro": "humble", "language": "python"}' > code.json | |
| python3 -c 'import json,os; r=json.load(open("code.json")); print(r["id"], r["valid"], r["validation"]); | |
| [os.makedirs(os.path.dirname(p) or ".", exist_ok=True) or open(p,"w").write(c) for p,c in r["files"].items()]' | |
| # Build it in a colcon workspace: colcon build && source install/setup.bash && ros2 launch ... | |
| # Run a valid scene on a GPU (10 s, robot holding its standing pose) and open the report in a browser | |
| curl -s $ROSDIFF_URL/simulate/run -H "X-API-Key: $ROSDIFF_KEY" -H 'content-type: application/json' \ | |
| -d '{"generation_id": "<id from /simulate/generate>", "duration_s": 10, "controller": "hold"}' | |
| # -> {"run_id": "...", "status": "IN_QUEUE", "report_url": "https://.../simulate/report/<id>?token=..."} | |
| # Other tools on a finished run (each answers with a report_url): | |
| curl -s -X POST $ROSDIFF_URL/simulate/sweep -H "X-API-Key: $ROSDIFF_KEY" -H 'content-type: application/json' \ | |
| -d '{"run_id": "<run_id>", "n_worlds": 64, "push_max_n": 80}' # how robust is it? | |
| curl -s -X POST $ROSDIFF_URL/simulate/run/<run_id>/isaac -H "X-API-Key: $ROSDIFF_KEY" # Isaac Sim render | |
| curl -s -X POST $ROSDIFF_URL/datasets/export -H "X-API-Key: $ROSDIFF_KEY" -H 'content-type: application/json' \ | |
| -d '{"run_ids": ["<run_id>", "<run_id>"], "name": "g1-stairs"}' # LeRobot dataset | |
| # Build, lint, test and launch a generated ROS 2 package in the sandbox: | |
| curl -s $ROSDIFF_URL/code/check -H "X-API-Key: $ROSDIFF_KEY" -H 'content-type: application/json' \ | |
| -d '{"generation_id": "<id from /code/generate>"}' | |
| curl -s $ROSDIFF_URL/jobs -H "X-API-Key: $ROSDIFF_KEY" # your jobs + this month's budget | |
| # Then please tell us whether it actually worked: | |
| curl -s $ROSDIFF_URL/feedback/<id> -H "X-API-Key: $ROSDIFF_KEY" -H 'content-type: application/json' \ | |
| -d '{"rating": "works", "note": "robot stood on the second step"}' | |
| ``` | |
| `valid: true` means the automatic checks passed (MuJoCo loads and simulates the scene / the package is | |
| well-formed and uses real message types). It does not mean the scene matches your description or the code | |
| behaves correctly — that is what your feedback is for. | |
| ## Reviewing what testers generated | |
| Every request is one row in `data/generations.sqlite3`: tester, request, retrieved sources, every model attempt | |
| with its validation result, the final output, duration, the tester's feedback and your review. | |
| ```bash | |
| python -m api.review summary # per kind: total, auto-valid, tester "works", reviewed correct | |
| python -m api.review list --unreviewed # newest first | |
| python -m api.review show <id> --attempts # request, sources, every raw attempt, output | |
| python -m api.review files <id> review/ # write the scene / package to disk to run it yourself | |
| python -m api.review mark <id> correct --note "stairs dimensions right, landing missing" | |
| python -m api.review export stage1.jsonl # everything, for analysis | |
| ``` | |
| Verdicts are `correct`, `partial` or `incorrect`. The summary line is the Stage 2 decision input: how often the | |
| output was automatically valid, how often testers say it works, and how often a reviewer agrees. | |
| ## Test scenarios | |
| `scenarios/mujoco/*.yaml` (5 scenes: pushable boxes, ramp, stairs, heightfield terrain, table with objects) and | |
| `scenarios/ros2/*.yaml` (4 packages: LaserScan safety stop, C++ IMU filter on Jazzy, service + client + launch, | |
| odometry-to-path). Each has a few structural checks (e.g. "at least 5 box geoms", "contains create_service"). | |
| ```bash | |
| python -m api.keys add orcanion-eval | |
| python -m scenarios.run --key rd_... --save eval_out/ # through the real API, so it is logged like any tester | |
| ``` | |
| ## CI | |
| `.github/workflows/ci.yml` runs on every push: | |
| - lint and the test suite (the Modal app definitions are import-checked there); | |
| - builds `worker/Dockerfile`, runs a real 2-second G1 job inside it (CPU rendering) and checks the MCAP, videos, | |
| summary and trajectory, rejects an unsafe scene, and runs a Go2 sweep with both engines (MuJoCo, MuJoCo Warp); | |
| - builds the ROS 2 sandbox for Humble and Jazzy and checks four sample packages (builds and runs, C++ with | |
| tests, does not compile, crashes on start); | |
| - simulates two Go2 runs, exports them with LeRobot and loads the dataset back. | |
| Not exercised in CI (needs a GPU and the model or NGC image): the Cosmos Reason inference and the Isaac Sim | |
| render. Their surrounding code (prompt, parsing, USD stage, endpoints) is tested. | |
| ## Repository layout | |
| ``` | |
| api/ FastAPI app, LLM client, MJCF + ROS 2 validation, OpenCode runner, jobs + budget, Modal/RunPod | |
| clients, R2 storage, report pages, key + review CLIs, SQLite log | |
| worker/ jobs: MuJoCo runs (sim_runner, handler), sweeps (sweep), ROS 2 sandbox (ros_runner, ros/), | |
| Cosmos review, LeRobot export, Isaac Sim replay (usd_scene, isaac_render); | |
| modal_app.py / modal_isaac.py for Modal, Dockerfile for RunPod | |
| rag/ sources.yaml, chunking, Chroma store, ingest CLI | |
| scenarios/ test cases + runner | |
| tests/ offline tests (fake robot, hash embeddings, scripted model, fake OpenCode binary) | |
| ``` | |
| ## Stage 1 boundaries | |
| In scope: the two generation endpoints, GPU runs of validated scenes and the tools above, one model (GLM-5.3 via OpenRouter) and | |
| one platform key for everyone, a key allowlist, logging, usage visibility and review. Deliberately **not** here: | |
| billing, tiers or model selection, signup/OAuth, per-tester run caps (one shared monthly limit protects the | |
| free credit), per-user OpenCode keys. Generated ROS 2 code runs only inside the network-less sandbox. | |