After studying this material, students should be able to:

1. Explain what IDS and BIM are and why they matter in construction projects
2. Describe the challenges of authoring IDS files
3. Understand what a benchmark is and why Ishigaki-IDS-Bench was created
4. Interpret how the benchmark evaluates AI model performance
5. Analyze what the results tell us about current AI capabilities in this domain

---

Step-by-Step Teaching

Step 1: Foundation Concepts — What is BIM?

Building Information Modeling (BIM) is a digital process for creating and managing information about a building throughout its entire lifecycle.

Think of it like this:

🏗️ Instead of just drawing a wall on paper, BIM stores the wall as a smart object that knows its material, thickness, fire rating, cost, and more.

Why it matters:

• Multiple teams (architects, engineers, contractors) share one consistent digital model
• Reduces errors, saves time, and improves coordination

---

Step 2: What is IDS?

Information Delivery Specification (IDS) is a formal document that defines exactly what information must be present in a BIM model.

Analogy:

📋 Think of IDS as a checklist inspector — it says: "Every wall must have a fire rating. Every door must have a manufacturer name."

Key technical facts:

• IDS is written in XML format (a structured text format readable by computers)
• It is machine-checkable — software can automatically verify if a BIM model meets the requirements
• It is standardized by buildingSMART, the international BIM standards organization

Example IDS rule (conceptually):

IF element = Wall
THEN it MUST have property "FireRating" in PropertySet "Pset_WallCommon"

---

Step 3: What is IFC?

Industry Foundation Classes (IFC) is the open standard vocabulary/language used to describe BIM objects.

Why this matters for IDS:

IDS conditions must use correct IFC vocabulary. You cannot invent property names — they must match IFC standards exactly.

⚠️ This is a key difficulty: authoring IDS requires deep IFC expertise

---

Step 4: Why is IDS Authoring Difficult?

Three layers of expertise are required:

Layer 1: Understanding the BIM project requirements
         (What information do we actually need?)
              ↓
Layer 2: Knowing IFC vocabulary
         (What is the correct IFC term for that requirement?)
              ↓
Layer 3: Validation tool agreement
         (Does the IDS file pass automated checking tools?)

The problem this creates:

• Few people have expertise across all three layers
• Mistakes in IFC vocabulary make IDS files invalid or useless
• This bottleneck slows down BIM projects

This is why researchers are exploring whether AI (LLMs) can help automate IDS generation.

---

Step 5: What is a Benchmark and Why Do We Need One?

What is a benchmark?

A benchmark is a standardized test used to measure and compare performance — in this case, of AI models.

🏁 Like a standardized exam for students, a benchmark gives every AI model the same questions so results can be fairly compared.

Why existing benchmarks were insufficient:

Existing AI benchmarks tested general structured text generation but did not test:

• ✗ IFC vocabulary conformance
• ✗ Agreement with external validation tools
• ✗ Domain-specific BIM/IDS knowledge

Gap identified → New benchmark needed

---

Step 6: Introducing Ishigaki-IDS-Bench

What is it?

The first publicly released benchmark specifically designed to test AI models on generating IDS files from BIM information requirements.

Key statistics:

Structure of each example:

INPUT:  A BIM information requirement (written in natural language)
           +
        Metadata (language, IFC version, domain, conversation turns)
           ↓
OUTPUT: A gold-standard IDS XML file (created by experts)

💡 "Gold" file = the correct answer created by human experts, used as the reference for scoring AI outputs

---

Step 7: How Does the Benchmark Evaluate AI Performance?

Evaluation happens in two stages:

---

Stage 1: Formal Validity

Tool used: buildingSMART IDSAuditTool (official validation software)

Three sub-scores are measured:

📊 Think of it as: Can we open it? → Is it organized correctly? → Does it make sense in IFC terms?

---

Stage 2: Content Fidelity

Method: Facet-level macro-F1 score compared against the gold IDS file

What is a "facet"?

In IDS, a facet is one specific condition or requirement. For example:

• Entity facet: "Must be an IfcWall"
• Property facet: "Must have FireRating property"
• Material facet: "Must have concrete material"

What is macro-F1?

F1 score measures how well the AI's output matches the expert's gold answer:

F1 = Balance between:
     Precision (did AI include only correct facets?)
     +
     Recall (did AI include ALL required facets?)

Macro = averaged equally across all facet types

🎯 Higher F1 = AI output is closer to what the expert would write

---

Step 8: What Were the Results?

Experiment setup:

• 10 different LLMs (Large Language Models) were tested
• Zero-shot setting = AI received no examples, just the task description
• Both Japanese and English inputs were tested

Key results:

What do these numbers mean?

Facet F1 = 65.6%
→ Even the best AI only gets ~2/3 of the facets right
→ Significant room for improvement

Content Pass Rate = 33.1%
→ Only 1 in 3 generated IDS files fully pass validation
→ Most AI outputs have technical errors

🔍 Key insight: Current AI models struggle significantly with IDS generation, especially with IFC vocabulary precision and validation compliance. This benchmark reveals a real, unsolved technical challenge.

---

Step 9: Why Does This Matter? (Big Picture)

BIM Project Need
      ↓
Information Requirements (natural language)
      ↓
[Currently: Requires rare IFC expert]
[Future goal: AI generates IDS automatically]
      ↓
Valid IDS File
      ↓
Automated BIM model checking
      ↓
Higher quality, more compliant buildings

Ishigaki-IDS-Bench:

• Measures how close AI is to solving this problem
• Identifies where AI fails (vocabulary, structure, completeness)
• Enables researchers to improve AI models systematically

---

Summary Table

---

Self-Check Questions

1. What does IDS stand for and what problem does it solve?
2. Why does authoring IDS require IFC expertise?
3. What makes Ishigaki-IDS-Bench different from existing benchmarks?
4. Explain the two-stage evaluation process in your own words
5. What does a Content pass rate of 33.1% tell us about current AI capabilities?
6. Why is a high Facet F1 score important even if Content pass rate is low?