How BIM Bench measures results.

Rate one drawing at a time.

The rating page shows a full-sheet drawing with the model name, effort setting, token usage and original filename hidden. Choose how usable the visible drawing is:

1. Unusable.
2. Major rework.
3. Usable with substantial edits.
4. Usable with minor edits.
5. Ready to use as a drawing.

Choose Cannot judge when you cannot assess the drawing. It is excluded from the average and never treated as a zero.

The results table is visible before voting. Only the PDF gallery has a vote-first overlay, with a smaller Browse results option. Browsing remembers that choice for the experiment in the current browser tab.

Your selection can change until you press Submit & see results. Saving locks your response and reveals that drawing’s average and rating count. Next drawing advances when you are ready. The remaining count and progress bar show how many drawings are left.

Sign in with Google to save a response. Each account can respond once per drawing, experiment, artifact version and rating protocol, up to 40 drawings per experiment. Less-reviewed drawings receive priority, refreshing returns your pending drawing, and submission unlocks after two seconds. Server-verified Cloudflare Turnstile, request limits and database constraints help prevent duplicate and automated submissions.

Responses are anonymous to other voters. Internally, a pseudonymous account identifier enforces limits; your Google name and email are not stored. See privacy and deletion. Accounts do not establish unique people, and public original PDFs may be recognised.

Anyone with a verified Google account can sign in and save ratings. Visitors who are signed out see a blurred rating area with a Google sign-in button; published originals remain available on the results page. A usability rating describes the visible drawing; it does not certify native BIM correctness or approval for construction.

Score the model and effort together.

Each model and effort setting is a separate configuration. For example, Light and Ultra results from the same model receive separate averages. Different effort settings are not repeated trials of one configuration. Labels and effort bars use each model’s native settings, so the number of levels differs by model. These ordinal positions are not equivalent compute budgets across providers.

Usability / 5

The score is the arithmetic mean of counted ratings on the original 1–5 scale: add the scores and divide by the number of scored ratings. Ratings shows that sample size. Cannot judge and ratings held or excluded from results do not enter the average. Missing ratings show no score. The score tooltip includes the count for each of the five rating choices.

Ratings belong to a specific experiment, artifact version and rating protocol. Adding configurations to the same unchanged task preserves existing ratings and lets returning voters review the new drawings. Replaced drawings, a changed prompt or a changed rating scale require a separate rating version.

These are subjective, ordinal ratings. A small difference between averages, especially with few ratings, is not strong evidence that one configuration is better. The table does not estimate statistical confidence or adjust for reviewer severity.

Tokens, cost, time and steps

The results table places usability, rating count and recorded tokens together; experiments run by the automated harness also show estimated cost, time and steps. Its default order is highest usability, then lowest tokens. The Sort control lets you change the priorities. The drawing gallery keeps its own model and effort order.

Input includes cached input; output includes reasoning. These subsets are not added again. Missing values stay unreported, and calculated values are marked. Where the source supplies a combined total including helper agents, the table uses that total and preserves the separate chat-only record in the source data.

The token records are conversation snapshots, with different or unspecified agent scopes. They are not billing records or equal compute budgets. Read tokens alongside usability and sample size. Where shown, cost is an estimate from recorded tokens and list prices, not a bill (Claude estimates can run slightly low because cache writes cost more than modelled); time is the wall-clock length of the agent loop, including API latency and Revit; steps are model turns. There is no combined efficiency score.

What the experiments can show.

Experiment 001. All configurations received the same prompt: use pyRevit MCP to create a 1:10 Revit drafting view of a timber eaves junction, including a 30° roof, 600 mm overhang, 145 mm wall framing, 195 mm rafters, a birdsmouth on double top plates and insulation batting. The original prompt and token note remain available with the results.

Experiment 001 publishes the original PDFs and token records, with one output per configuration. Full-sheet previews preserve each PDF’s framing; varying paper sizes mean they are not displayed at a shared physical drawing scale. PDF hashes are included in the dataset.

Experiment 002 repeats the task with the automated BIM Bench harness: a fresh system prompt, a fixed set of Revit tools, no memory or skills, and a clean starting file for every run. Each configuration was run once, so differences between neighbouring settings can be ordinary run-to-run variation. The harness also has an automated check, but it is not published because it cannot reliably separate good drawings from poor ones; the community ratings are the quality measure.

Native RVT files, complete execution traces, repeated runs and billing records have not been published for either experiment. A printed dimension does not verify underlying geometry, and a drawn timber section does not establish family identity. For experiment 001, compliance with the prompt’s memory, computer-use and view-opening instructions needs execution evidence.

Model identity

In experiment 001, public model names are display mappings from the supplied aliases and export chronology, and exact historical model snapshot IDs remain unverified. In experiment 002, names are the API model identifiers the harness requested; providers may update a model behind an identifier. The original labels, filenames, reported values and source scopes are retained.

Prompt (001) · Prompt (002) · Original OpenAI token note · Original Gemini token note · Dataset and PDF hashes (001) · CSV (001) · Dataset (002) · CSV (002)

Future tests.

BIM Bench’s focus is work inside BIM applications, including 2D documentation and 3D authoring. The following extensions are planned directions. They are not part of the current usability score.

Native BIM checks

Use saved models and execution traces to check brief completion, measurable geometry, native families and element types, parameters, editability and successful reopen. For the eaves task, this would include measuring pitch, overhang, framing depth and birdsmouth bearing, and identifying the requested detail components and batting. Define the checks and tolerances before running models; report absent evidence as unverified.

Repeatability and resources

Repeat each model and effort configuration several times (so far each was run once) under a recorded application build, tool version, family library, units and starting file. Use fresh sessions, record interventions and retries, preserve failed runs and report the spread, not a single result.

New benchmark tasks

• Controlled revisions: change a dimension, pitch or opening while preserving unrelated elements and annotations.
• Documentation and quantities: create coordinated views, sheets, tags, schedules and quantity totals against a known baseline.
• Coordination and recovery: resolve planted clashes or recover from tool errors without duplicates or damaged elements.
• Cross-application exchange: test equivalent workflows and declared IFC checks while keeping application-specific results separate.
• Parametric behavior: exercise reusable families or objects across a predefined parameter matrix.

Each published experiment should include its task, prompt, environment, stable run IDs, original artifacts, resource records and evaluation method. Keep corrections traceable and ratings tied to the version reviewed.

The site’s presentation takes inspiration from Artificial Analysis and the supplied Veido palette. BIM Bench does not reuse their scores or claim an affiliation.

Back to results ↗