File size: 3,440 Bytes
1866d85 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 | # Contributing
AutoCAD Bench separates task data, model execution, trusted evaluation, and infrastructure. Keep changes inside the narrowest applicable boundary and preserve the scorer/model trust split.
## Development setup
```bash
uv sync --dev
cd apps/dashboard && npm ci
```
## Common changes
### Add or update a task
1. Add the reference PNG and gold DWG under `tasks/corpus/2d/` or `tasks/corpus/3d/`.
2. Add or update the canonical row in `tasks/manifest.jsonl`.
3. Update `tasks/gold-audit.jsonl` with verified file metadata.
4. Rebuild the gold cache with the pinned trusted evaluator.
5. Run corpus, audit, cache, and test validation.
Never expose a gold path through `PublicTaskSpec`, `AutoCADTaskData`, a model prompt, or a model trace.
### Change the harness
Model-provider logic and the agent loop belong under `src/autocad_bench/harness/`. Computer actions must remain typed and fully validated before any broker call. Preserve the 7,000-action rollout budget and the distinction between audit-only intermediate frames and model-visible observations.
### Change evaluation
Trusted evaluation belongs under `src/autocad_bench/evaluation/`. A render-contract change requires a new evaluator version and gold cache; never reinterpret an old version in place. Windows evaluator changes also require a compiled fresh-clone gate before an AMI becomes active.
### Change infrastructure
Backend selection, registration, and lifecycle boundaries belong under
`src/autocad_bench/infrastructure/`. Keep the batch controller
backend-neutral. New external backends should implement the public plugin
contract and register an `autocad_bench.infrastructure` entry point; see
`docs/infrastructure-plugins.md`. Backends that retain reconnectable workers
should also implement the optional recovery protocol; do not put
provider-specific worker inspection back into `orchestration/resume.py`.
The built-in AWS lifecycle and broker-client code remains under
`src/autocad_bench/sandbox/`. Windows runtime source belongs under
`sandbox/windows/`; cloud-controller assets belong under
`infra/cloud-controller/`. Do not launch instances, deploy infrastructure, or
replace the active AMI without explicit approval.
## Validation
Run the complete offline gate before handing off a change:
```bash
uv run autocad-bench validate
uv run autocad-bench validate-audit
uv run pytest
cd apps/dashboard
npm test
npm run build
```
Before publishing, build the package and corpus into one new release directory,
then verify that complete directory:
```bash
uv run autocad-bench-release build \
--output dist/releases/0.1.0
uv run autocad-bench-release verify \
--root dist/releases/0.1.0
```
The builder uses the allowlist-based bundle exporter and pinned PEP 517
dependencies. It must never be expanded to include configs, environment files,
controller state, or run outputs. See `docs/releases.md` for the complete
manual publishing and reproducibility workflow.
When a current evaluator cache is available locally:
```bash
uv run autocad-bench verify-gold-cache \
--evaluator-version autocad-2019-r23.0.46-v4 \
--split all
```
Live provider or AWS checks are separate from the offline suite and require explicit credentials.
## Documentation
Update the relevant contract whenever behavior changes. Release evidence is append-only: add a new release record instead of editing historical results to describe a newer runtime.
|