| # Contributing |
|
|
| AutoCAD Bench separates task data, model execution, trusted evaluation, and infrastructure. Keep changes inside the narrowest applicable boundary and preserve the scorer/model trust split. |
|
|
| ## Development setup |
|
|
| ```bash |
| uv sync --dev |
| cd apps/dashboard && npm ci |
| ``` |
|
|
| ## Common changes |
|
|
| ### Add or update a task |
|
|
| 1. Add the reference PNG and gold DWG under `tasks/corpus/2d/` or `tasks/corpus/3d/`. |
| 2. Add or update the canonical row in `tasks/manifest.jsonl`. |
| 3. Update `tasks/gold-audit.jsonl` with verified file metadata. |
| 4. Rebuild the gold cache with the pinned trusted evaluator. |
| 5. Run corpus, audit, cache, and test validation. |
|
|
| Never expose a gold path through `PublicTaskSpec`, `AutoCADTaskData`, a model prompt, or a model trace. |
|
|
| ### Change the harness |
|
|
| Model-provider logic and the agent loop belong under `src/autocad_bench/harness/`. Computer actions must remain typed and fully validated before any broker call. Preserve the 7,000-action rollout budget and the distinction between audit-only intermediate frames and model-visible observations. |
|
|
| ### Change evaluation |
|
|
| Trusted evaluation belongs under `src/autocad_bench/evaluation/`. A render-contract change requires a new evaluator version and gold cache; never reinterpret an old version in place. Windows evaluator changes also require a compiled fresh-clone gate before an AMI becomes active. |
|
|
| ### Change infrastructure |
|
|
| Backend selection, registration, and lifecycle boundaries belong under |
| `src/autocad_bench/infrastructure/`. Keep the batch controller |
| backend-neutral. New external backends should implement the public plugin |
| contract and register an `autocad_bench.infrastructure` entry point; see |
| `docs/infrastructure-plugins.md`. Backends that retain reconnectable workers |
| should also implement the optional recovery protocol; do not put |
| provider-specific worker inspection back into `orchestration/resume.py`. |
|
|
| The built-in AWS lifecycle and broker-client code remains under |
| `src/autocad_bench/sandbox/`. Windows runtime source belongs under |
| `sandbox/windows/`; cloud-controller assets belong under |
| `infra/cloud-controller/`. Do not launch instances, deploy infrastructure, or |
| replace the active AMI without explicit approval. |
|
|
| ## Validation |
|
|
| Run the complete offline gate before handing off a change: |
|
|
| ```bash |
| uv run autocad-bench validate |
| uv run autocad-bench validate-audit |
| uv run pytest |
| |
| cd apps/dashboard |
| npm test |
| npm run build |
| ``` |
|
|
| Before publishing, build the package and corpus into one new release directory, |
| then verify that complete directory: |
|
|
| ```bash |
| uv run autocad-bench-release build \ |
| --output dist/releases/0.1.0 |
| uv run autocad-bench-release verify \ |
| --root dist/releases/0.1.0 |
| ``` |
|
|
| The builder uses the allowlist-based bundle exporter and pinned PEP 517 |
| dependencies. It must never be expanded to include configs, environment files, |
| controller state, or run outputs. See `docs/releases.md` for the complete |
| manual publishing and reproducibility workflow. |
|
|
| When a current evaluator cache is available locally: |
|
|
| ```bash |
| uv run autocad-bench verify-gold-cache \ |
| --evaluator-version autocad-2019-r23.0.46-v4 \ |
| --split all |
| ``` |
|
|
| Live provider or AWS checks are separate from the offline suite and require explicit credentials. |
|
|
| ## Documentation |
|
|
| Update the relevant contract whenever behavior changes. Release evidence is append-only: add a new release record instead of editing historical results to describe a newer runtime. |
|
|
