Snaptrude has developed AECBench, a benchmark designed to measure whether AI models can perform realistic architecture, engineering, and construction tasks. Unlike conventional AI benchmarks focused on reasoning, coding, or question answering, AECBench gives an AI a design brief, asks it to work through an architectural problem, and evaluates the resulting building model.
The benchmark addresses a basic problem facing AEC firms: there is no common way to determine whether an AI system can perform professional design work. Architectural problems rarely have a single correct answer, making traditional pass-or-fail testing inadequate. AECBench therefore divides evaluation into objective core requirements and qualitative design assessments.
Every task checks five fundamentals, including whether spaces overlap, stories stack correctly, all spaces are accessible, numerical requirements in the brief are satisfied, and building elements are parametric components. Additional evaluations cover space planning, BIM detailing, drawings, building services, and value engineering. Deterministic software graders measure objective requirements directly from the model. Design quality is evaluated by an LLM developed from reviews by practicing architects.
Snaptrude also tested its AI agent against Claude Code and Codex. Snaptrude’s agent works directly within a parametric building model, while the coding agents generated Three.js scenes. Snaptrude reported stronger results in areas including adjacency and zoning and BIM detailing, although Claude Code scored higher on site response and value engineering.
The distinction between visual output and usable BIM is central to the comparison. Snaptrude’s agent creates editable rooms, walls, openings, and other building components while maintaining a changelog of design decisions. It can also reason about circulation, privacy, acoustics, and relationships between spaces.
AECBench remains an internal benchmark, so its results should be viewed in that context. Snaptrude plans to open-source the tasks and provide a way for AI labs to run their models through the same evaluation framework.
