| --- |
| tags: |
| - benchmark |
| - computer-use |
| - autocad |
| - cad |
| - multimodal |
| --- |
| |
| # AutoCAD Bench |
|
|
| AutoCAD Bench is a computer-use benchmark for evaluating whether multimodal agents can reproduce dimensioned 2D drawings and reconstruct dimensionally accurate 3D models in desktop AutoCAD. |
|
|
| Each rollout receives a task-specific instruction and reference image, operates an isolated AutoCAD session using mouse and keyboard actions, and is evaluated from the final DWG. Gold drawings and evaluator credentials never enter the model-visible session. |
|
|
| ## How it works |
|
|
| ```text |
| tasks/manifest.jsonl + reference PNG |
| │ |
| ▼ |
| model harness + agent loop |
| │ |
| ▼ |
| isolated Windows/AutoCAD sandbox |
| │ |
| ▼ |
| final attempt.dwg |
| │ |
| ▼ |
| trusted inspection + deterministic score + vision judgment |
| │ |
| ▼ |
| runs/<benchmark>/ + dashboard |
| ``` |
|
|
| The benchmark contains exactly 50 tasks, numbered `task-001` through |
| `task-050` with no gaps: 21 2D drawings and 29 3D drawings. A 2D task is |
| completed in Model Space. A 3D task requires true geometry in Model Space and |
| the complete reference sheet on `Layout1`; evaluator v4 renders and compares |
| that required layout. |
|
|
| ## Getting started |
|
|
| Requirements: |
|
|
| - Python 3.12 |
| - [uv](https://docs.astral.sh/uv/) |
| - Node.js 20+ for the dashboard |
| - Infrastructure-backend and model-provider credentials only for live runs |
|
|
| Install the Python environment and validate the corpus: |
|
|
| ```bash |
| # The development group includes the built-in AWS backend dependencies. |
| uv sync --dev |
| uv run autocad-bench validate |
| uv run autocad-bench validate-audit |
| uv run autocad-bench list --split all |
| ``` |
|
|
| Published installations that use the built-in backend should install |
| `autocad-bench[aws]`. Custom backends can depend on the base package and their |
| own infrastructure plugin without installing the AWS SDK. |
|
|
| The wheel contains code, while scorer-side benchmark data can come from either |
| a checkout or a checksum-bound bundle: |
|
|
| ```bash |
| autocad-bench bundle-validate --root /path/to/autocad-bench-corpus-v1 |
| export AUTOCAD_BENCH_ROOT=/path/to/autocad-bench-corpus-v1 |
| ``` |
|
|
| See [Portable benchmark bundles](docs/benchmark-bundles.md) for the data |
| format and [Manual releases](docs/releases.md) for reproducible package and |
| corpus publishing. |
|
|
| Run the offline tests: |
|
|
| ```bash |
| uv run pytest |
| cd apps/dashboard |
| npm ci |
| npm test |
| npm run build |
| ``` |
|
|
| ## Run a benchmark |
|
|
| Copy the AWS example and replace every infrastructure, bucket, and model |
| placeholder with resources you own: |
|
|
| ```bash |
| cp configs/aws.example.json configs/my-benchmark.json |
| |
| uv run autocad-bench-batch \ |
| --config configs/my-benchmark.json \ |
| --skip-direct-model-check |
| ``` |
|
|
| Execute only after provider credentials are exported and preflight passes: |
|
|
| ```bash |
| export OPENAI_API_KEY=... |
| |
| uv run autocad-bench-batch \ |
| --config configs/my-benchmark.json \ |
| --output-dir runs/benchmarks/<run-id> \ |
| --execute |
| ``` |
|
|
| Every rollout is evaluated automatically after its final DWG is retrieved and before its sandbox is cleaned up. Evaluation progress is written atomically for the dashboard. |
|
|
| ## Inspect results |
|
|
| ```bash |
| cd apps/dashboard |
| npm ci |
| npm run build |
| npm run preview |
| ``` |
|
|
| Open [http://127.0.0.1:4173](http://127.0.0.1:4173). The viewer reads local `runs/` data and can lazily retrieve missing trace artifacts from the encrypted benchmark bucket when configured. |
|
|
| ## Repository layout |
|
|
| ```text |
| autoCAD-bench/ |
| ├── tasks/ # Scorer-side manifest, audit inventory, and source corpus |
| ├── src/autocad_bench/ |
| │ ├── tasks/ # Task loading, validation, and public task models |
| │ ├── harness/ # Agent loop, model calls, and single-run execution |
| │ ├── evaluation/ # Trusted DWG evaluation and scoring |
| │ ├── infrastructure/ # Backend contract, registry, and built-in AWS plugin |
| │ ├── sandbox/ # Broker client and AWS/AutoCAD lifecycle |
| │ ├── orchestration/ # Parallel batches, resume, and recovery |
| │ └── common/ # Shared utilities |
| ├── sandbox/windows/ # Windows broker, evaluator, and AMI bootstrap source |
| ├── infra/cloud-controller/ # Persistent controller deployment assets |
| ├── configs/ # Bring-your-own infrastructure examples |
| ├── apps/dashboard/ # Local benchmark registry and trace viewer |
| ├── artifacts/gold-cache/ # Versioned trusted evaluator outputs |
| ├── docs/ # Contracts, architecture, operations, and releases |
| ├── tests/ # Offline regression suite |
| └── runs/ # Generated benchmark results; ignored by git |
| ``` |
|
|
| ## Active runtime contract |
|
|
| - Broker: `windows-autocad-2019-v10` |
| - Evaluator: `autocad-2019-r23.0.46-v4` |
| - Per-task action budget: 5,000 |
| - Action batch size: bounded only by the remaining rollout budget |
|
|
| Worker images are operator-owned and must satisfy the pinned broker contract. |
| Broker v11 adds controller-owned DWG save but remains a separate candidate |
| runtime. The complete evaluator-v4 cache contains and verifies all 50 source |
| DWGs. |
|
|
| ## Documentation |
|
|
| | Guide | Purpose | |
| |---|---| |
| | [Architecture](docs/architecture.md) | Component boundaries and data flow | |
| | [Benchmark contract](docs/benchmark-contract.md) | Model-visible and scorer-side task contract | |
| | [Sandbox contract](docs/deterministic-sandbox-contract.md) | AWS lifecycle, admission, and cleanup invariants | |
| | [Evaluator contract](docs/autocad-evaluator-contract.md) | DWG inspection and `Layout1` behavior | |
| | [Scoring](docs/final-dwg-scorer.md) | Deterministic and vision scoring | |
| | [Parallel rollouts](docs/parallel-rollouts.md) | Batch preflight and execution | |
| | [Infrastructure plugins](docs/infrastructure-plugins.md) | Bring-your-own worker backend contract | |
| | [Benchmark bundles](docs/benchmark-bundles.md) | Portable task and evaluator-data releases | |
| | [Manual releases](docs/releases.md) | Reproducible local assembly and verification | |
| | [Cloud controller](docs/cloud-controller.md) | Cloud-owned execution and recovery | |
| | [Contributing](CONTRIBUTING.md) | Development and validation workflow | |
|
|
| ## Security boundary |
|
|
| `tasks/manifest.jsonl` and the corpus are scorer-side inputs and contain paths |
| to gold DWGs that are private from the model session. Model-visible task |
| serialization excludes gold paths and DWG names. A rollout receives only its |
| reference image and task statement; trusted evaluator exchange uses |
| request-scoped presigned S3 objects after model execution. |
|
|
