BIBIMBAP: A Benchmark for Instructional BIM-Based Automated Programming

Benchmark architecture: the Text-to-BIM system takes a prompt and an IFC file (with an output structure for retrieval tasks) and produces an edited IFC file and optional structured output, which are evaluated using the test file and the original IFC model.

Abstract

Large Language Models (LLMs) are emerging as natural-language interfaces for Building Information Modeling (BIM), successfully achieving information retrieval, but no benchmark exists to evaluate their ability to edit Industry Foundation Classes (IFC). We introduce BIBIMBAP, a Text-to-BIM benchmark with 100 curated tasks covering Create-Read-Update-Delete (CRUD) operations across spatial, geometric, topological, numeric, and conceptual categories. Each task includes a natural-language prompt, an IFC model, expected structured outputs, and test scripts for automated evaluation. Baseline results show limited performance (best: 50.2%) and frequent failures in constraint preservation and spatial reasoning, with some correct outputs achieved despite flawed reasoning.

Contributions

• Full CRUD Task Suite
  A set of 100 Text-to-BIM tasks spanning the full CRUD operations (Create, Read, Update, Delete) and multiple reasoning categories: direct, spatial, topological, numeric, and conceptual.
• Standardized Prompt Grammar
  A prompt grammar (operator / operand / context specifiers) that standardizes task formulation and supports systematic analysis and extensibility of the benchmark.
• Automated Evaluation Pipeline
  An automated pipeline with procedural tests and fine-grained evaluation metrics (existence, location, dimensions, correctness, and IFC integrity constraints) that enables reproducible, deterministic evaluation without LLM-as-a-judge.
• Baseline Experiments
  A validation of the benchmark across multiple LLMs (Claude Opus 4.6, GPT-5.4, Gemini 3 Flash, GPT-5.2, GPT-5 Mini, GPT-4.1), establishing baselines and analyzing common failure modes.

Task Categories

Tasks are organized by CRUD operation and further grouped by reasoning requirements:

• Direct: Tasks where all required information is explicitly provided in the prompt (e.g., create a wall at given coordinates).
• Spatial: Tasks requiring understanding of spatial/geometric concepts (e.g., place a window centered on the longest wall).
• Topological: Tasks requiring understanding of topological relationships between IFC elements (e.g., move a door from one wall to another).
• Numeric: Tasks requiring numerical computation or aggregation (e.g., compute total floor area).
• Conceptual: Tasks requiring architectural domain knowledge (e.g., identify the entrance door).

Prompt Grammar

Each task prompt follows a standardized grammar of Operator, Operand, Operation Specifiers, and Context Specifiers, enabling systematic analysis and extensibility:

The prompt grammar decomposes each task into an operator (CRUD action), an operand (target element or property), operation specifiers (parameters), and context specifiers (how to identify the target in the model).

Evaluation Metrics

Each task is evaluated using fine-grained, deterministic metrics:

• Object Exists (Create): Whether the requested element was created in the IFC model.
• Object Not Exists (Delete): Whether the targeted element was removed.
• Correct Location (Create, Update): Whether element placement matches the expected coordinates.
• Correct Dimensions (Create, Update): Whether element geometry matches specifications.
• Correct Answer (Read): Whether the structured retrieval output matches ground truth.
• Integrity Assured (Create, Update, Delete): Whether IFC model integrity constraints are preserved.

Task Examples

Spatial Update: "Rotate the wall by 45 degrees around its center which is parallel to wall with GlobalId = ..." showing before (left) and after (right).

Conceptual Read: "Provide the GlobalId of the door that is perceived as the entrance." Requires architectural domain knowledge.

Results

We evaluated six LLMs on BIBIMBAP using a simple code-execution setup with IfcOpenShell. Overall success remains limited: the best model, Claude Opus 4.6, reaches only 50.2% on average, while the remaining models score between 24% and 31%. The top three models (Claude Opus 4.6, GPT-5.4, Gemini 3 Flash) are averaged over three runs (mean ± std. dev.); the others are single-run.

Model Performance by Category (%)

* Single-run results. The top three models are averaged over three runs (mean ± std. dev.). Bold marks the best score in each column.

CRUD Operation Success Rate by Model (%)

* Single-run results. The top three models are averaged over three runs (mean ± std. dev.). Bold marks the best score in each column.

Each model's performance across the six evaluation criteria: Integrity Assured, Object Exists, Object Not Exists, Correct Answer, Correct Dimensions, and Correct Location.

Key Findings

• Claude Opus 4.6 achieves the highest overall score (50.2%), leading in every category and excelling at element creation (81.1%).
• Even the best model solves only about half the tasks, showing that reliable Text-to-BIM manipulation remains challenging.
• Numeric-calculative tasks are the most difficult category for all models, requiring multi-step aggregation and computation from geometry.
• Integrity constraints are the hardest evaluation criterion: models can create or delete entities but struggle to keep IFC relationships consistent.
• Iterative tool use helps: Claude Opus 4.6 makes the most tool calls (~4.9 per task) and achieves the strongest results.

BibTeX

Contact

For questions about our benchmark, please contact the authors via email.