Architecture

AutoCAD Bench is a filesystem-first benchmark with a trusted Windows execution boundary. Task definitions and configurations are versioned inputs; every rollout writes an immutable trace directory that the evaluator and dashboard consume.

Execution phases

Discover — load and validate the task corpus and a model/run configuration.
Preflight — verify credentials, model availability, backend capacity and worker identity, and every required gold-cache entry.
Run — launch or attach to one isolated AutoCAD sandbox, hand off the exact task, and execute the model's computer actions.
Finalize — retrieve the final DWG and complete trace evidence.
Evaluate — inspect the DWG inside trusted AutoCAD, compute deterministic diagnostics, and run the vision judge.
Report — expose normalized benchmark, task, scoring, usage, and trace data through the local dashboard.

tasks/                  configs/
   │                        │
   └──────────┬─────────────┘
              ▼
   orchestration/batch.py
              │
        bounded workers
              │
              ▼
 infrastructure plugin ─────── selected worker backend
              │                         │
              ▼                         ▼
       harness/run.py  ◄────── Windows broker + AutoCAD
              │                         │
              └──── final DWG ──────────┘
                          │
                          ▼
                 evaluation/run.py
                          │
                          ▼
                    runs/<run-id>/
                          │
                          ▼
                    apps/dashboard/

Task model

tasks/manifest.jsonl is the canonical catalog. Each row identifies one reference PNG, one private gold DWG, drawing type, difficulty, split, and execution budgets. The loader verifies:

exactly 50 entries with contiguous IDs from task-001 to task-050;
21 2D and 29 3D tasks;
valid PNGs and AutoCAD AC1032 DWGs;
paths that remain inside the repository;
a fixed seven-task development split; and
no reused reference or gold artifact.

tasks/gold-audit.jsonl records byte sizes, image dimensions, DWG signatures, and manual-review state. Public task models omit the gold path before any data reaches the model harness.

Package boundaries

Package	Responsibility
`autocad_bench.tasks`	Corpus validation, public task data, and prompts
`autocad_bench.harness`	Provider interaction, agent loop, and trace writing
`autocad_bench.infrastructure`	Worker backend registration, launch, cleanup, and optional recovery
`autocad_bench.sandbox`	Broker protocol, AWS lifecycle, handoff, and evidence retrieval
`autocad_bench.orchestration`	Model catalog checks, batch planning, concurrency, resume, and recovery
`autocad_bench.evaluation`	Trusted inspection, gold cache, deterministic scoring, and vision judgment
`autocad_bench.common`	Atomic filesystem utilities shared across boundaries

The Python code uses a src/ layout so imports always resolve through the installed package rather than the working directory. Commands normally discover the checkout from tasks/manifest.jsonl; an installed wheel can use AUTOCAD_BENCH_ROOT to point at a benchmark checkout.

Sandbox boundary

The model has no shell, filesystem, AutoLISP, API, MCP, or network tools. It can send only validated mouse and keyboard actions to the loopback broker. The broker captures every post-action frame for audit while the model sees only the observation returned at the end of its action batch.

The Windows source under sandbox/windows/ builds three runtime layers:

the authenticated loopback broker;
the PyAutoGUI interactive input worker; and
the trusted AutoCAD evaluator plug-in and runner.

The built-in AWS backend uses SSM and an inbound-free security group. Other backends must enforce an equivalent isolation and admission contract. Runtime admission verifies the current boot, deployment manifest, broker identity, display, input driver, and exact-task reset before the model receives control.

Evaluation boundary

The final DWG is scored only after model execution. For 2D tasks, evaluator v4 renders Model Space. For 3D tasks, it inspects true geometry and all paper layouts but renders the required Layout1 presentation. Gold-cache entries fail closed if their source checksum, evaluator version, render configuration, drawing type, or required 3D layout contract differs.

The dashboard-preferred score is the structured vision judgment. Deterministic geometry and rendering comparisons remain audit evidence and regression signals.

Results layout

runs/benchmarks/<benchmark-id>/
├── batch-state.json
└── rollouts/<task-id>/
    ├── run-state.json
    ├── reference.png
    ├── attempt.dwg
    ├── frames/
    ├── model_turns.jsonl
    ├── model_turn_summaries.jsonl
    ├── input_receipts.jsonl
    ├── events.json
    ├── frame_events.json
    └── evaluation/
        ├── status.json
        ├── metadata.json
        ├── candidate-render.png
        ├── score.json
        └── vision-judge.json

Writes that drive live status are atomic. Completed traces are immutable; recovery creates explicit evidence and promotion records rather than silently rewriting outcome history.