How AI Finds Errors in Seismic Data Workflows

🎯 Defined Learning Outcomes

After studying this material, students should be able to:

1. Explain how LLMs can be applied to analyze graph-based seismic workflows
2. Distinguish between embedding-based and fine-tuned LLM approaches for error detection
3. Evaluate the trade-offs between precision and explanation capability in AI workflow validation
4. Understand why pre-execution error detection matters in geophysical processing

---

📚 Step-by-Step Teaching Guide

---

STEP 1: Understanding the Problem Domain

What is Seismic Data Processing?

Raw Seismic Signal
        ↓
   [Filter 1] → [Filter 2] → [Filter 3] → Geological Information
        ↓
   This sequence = a WORKFLOW

Key concepts to grasp:

Why does this matter?

• Workflows can have hundreds of connected steps
• One misconfiguration can invalidate the entire process
• Running a broken workflow wastes significant computing resources

💡 Think of it like this: Imagine baking a cake but putting it in the oven before mixing ingredients. The error is obvious to humans early on, but in complex computational workflows, errors hide until execution.

---

STEP 2: Understanding the Core Challenge

Traditional Problem Flow

Design Workflow → Submit for Execution → FAIL → Diagnose → Fix → Repeat
     (hours)          (hours/days)      ❌        (hours)

Problems with this approach:

• ❌ Errors only discovered after expensive computation begins
• ❌ Expert diagnosis is time-consuming
• ❌ No automatic explanation of what went wrong
• ❌ Project delays accumulate

The Research Question

"Can LLMs predict workflow failures before execution and explain why they will fail?"

---

STEP 3: The Two AI Approaches Explained

Approach 1: Embedding-Based Supervised Contrastive Learning

What is an embedding?

Workflow Graph → [Embedding Model] → Vector [0.2, 0.8, -0.3, ...]
                                          ↓
                                   Numerical representation
                                   that captures meaning

What is contrastive learning?

Working Workflow  ──────────────────────────────→ [CLUSTER A] ✅
Broken Workflow   ──────────────────────────────→ [CLUSTER B] ❌

The model learns to PUSH these apart in vector space

Step-by-step process:

1. Convert workflow graph → numerical embedding
2. Train model to cluster valid vs invalid workflows
3. New workflow submitted → model classifies it
4. Output: "This workflow will likely fail" (Yes/No)

Performance: 59% precision across diverse error types

---

Approach 2: Fine-Tuned LLM for Error Explanation

What is fine-tuning?

Pretrained LLM                    Fine-Tuned LLM
(General Knowledge)    →→→→→→→    (Seismic Workflow Expert)
     GPT-like model    Training        Domain-specific
                       on domain
                         data

Step-by-step process:

1. Take a pretrained LLM (already knows language/reasoning)
2. Further train it on seismic workflow examples
3. Feed it a workflow graph representation
4. Output: "This workflow fails because Filter 3 receives incompatible data format from Filter 2"

---

STEP 4: Comparing the Two Approaches

Performance Comparison Table

Visual Trade-off Analysis

HIGH  │  Embedding ●
      │              
Prec- │                    Fine-Tuned ●
ision │                    
      │                                    Pretrained ●
LOW   └─────────────────────────────────────────────────
      No Explanation    Some Explanation    No Explanation
      
                    EXPLANATION ABILITY →

💡 Key Insight: Fine-tuned LLM trades some detection precision for explanation capability — going from 0% to 32.61% explanation accuracy is a massive practical gain

---

STEP 5: Understanding Why This Matters

The Value Proposition

WITHOUT AI Validation:
Cost = Computation Time + Expert Diagnosis Time + Delay Cost
     = HIGH 💸

WITH AI Validation:
Cost = Quick AI Check (seconds) + Targeted Fix
     = LOWER 💰

Real-World Impact Chain

Better Error Detection
        ↓
Fewer Failed Workflow Executions
        ↓
Less Wasted Computing Resources
        ↓
Faster Geophysical Data Processing
        ↓
Accelerated Geological Discoveries

---

STEP 6: Critical Evaluation of Results

What the Numbers Tell Us

59% precision — Is this good?

Baseline (random guess for binary): ~50%
Embedding approach:                  59%
Improvement:                        +9 percentage points

• ✅ Better than random
• ⚠️ Still room for improvement
• ✅ Valuable for pre-screening before expensive execution

32.61% explanation accuracy — Is this meaningful?

Pretrained model:    0%  ████░░░░░░░░░░░░░░░░
Fine-tuned model: 32.61% ████████████░░░░░░░░

• ✅ Infinitely better than baseline (0% → 32.61%)
• ✅ Proves fine-tuning adds domain-specific value
• ⚠️ Still needs improvement for production reliability

---

STEP 7: Synthesizing the Concepts

Complete System Architecture

Seismic Workflow (Graph)
         │
         ▼
┌─────────────────────┐
│  Graph Representation│  ← Convert workflow to LLM-readable format
│  (Text/Structured)  │
└─────────────────────┘
         │
    ┌────┴────┐
    ▼         ▼
[Embedding  [Fine-Tuned
 Approach]   LLM]
    │         │
    ▼         ▼
"Will Fail" "Will Fail BECAUSE..."
    ↓         ↓
    └────┬────┘
         ▼
  Expert Reviews AI Prediction
         ▼
  Fix BEFORE Execution ✅

---

🧠 Concept Check Questions

Test your understanding:

1. Why is pre-execution error detection valuable in seismic workflows?
2. What is the key difference between what the embedding approach and fine-tuned LLM provide?
3. Why does the fine-tuned LLM outperform the pretrained model in explanation accuracy?
4. How does contrastive learning help classify valid vs. invalid workflows?
5. What does 32.61% explanation accuracy vs. 0% tell us about the value of domain fine-tuning?

---

📝 Summary Card

┌─────────────────────────────────────────────────────┐
│           KEY TAKEAWAYS                             │
├─────────────────────────────────────────────────────┤
│ 1. Seismic workflows are complex graph structures   │
│    prone to hard-to-detect errors                   │
│                                                     │
│ 2. LLMs can analyze these graphs BEFORE execution   │
│                                                     │
│ 3. Embedding approach = better at DETECTION (59%)   │
│                                                     │
│ 4. Fine-tuned LLM = better at EXPLANATION (32.61%)  │
│                                                     │
│ 5. Both approaches outperform baseline models       │
│                                                     │
│ 6. AI validation = less waste + faster processing   │
└─────────────────────────────────────────────────────┘