Step-by-Step Teaching Guide

Step 1: Understanding the Problem First

Before any AI solution makes sense, you must clearly understand what problem existed.

The Manual Bug Triage Problem

Imagine a company receiving thousands of customer bug reports. Each one required an engineer to:

Customer Bug Report
        ↓
Engineer reads investigation
        ↓
Engineer determines root cause
        ↓
Engineer classifies the issue
        ↓
Engineer documents reasoning
        ↓
Work can finally begin

Why This Was Painful

Key Insight: The problem wasn't just speed. It was inconsistency and lost institutional knowledge.

---

Step 2: Understanding Why Simple AI Wasn't Enough

Many teams make this mistake: they assume the newest, largest AI model solves everything.

The LLM Trap

The team initially tried Large Language Models (LLMs). Here's why that failed for this specific task:

What LLMs are GOOD at:          What this task NEEDED:
✓ Reasoning across text         ✓ Deterministic predictions
✓ Generating summaries          ✓ Confidence scores
✓ Flexible responses            ✓ Explainability
✗ Consistent classification     ✓ Consistent classification
✗ Confidence scoring            ✗ Flexible/creative answers
✗ Deterministic behavior

Core Lesson: A deterministic prediction means the system gives the same answer for the same input every time. Bug classification requires this. Creative writing does not.

Quick Knowledge Check

Ask yourself: Why would a customer support system need deterministic behavior rather than creative AI responses?

Answer: Because engineers need to trust and verify predictions. If the AI gives different answers each time, engineers cannot build confidence in the system.

---

Step 3: The Hybrid AI Architecture Solution

The breakthrough was recognizing that different tasks need different tools.

The Two-Model Architecture

Customer Bug Report
         ↓
┌─────────────────────────────────────────┐
│         CUSTOM ML MODEL                 │
│  • Trained on Salesforce's OWN history  │
│  • Fast, deterministic classification   │
│  • Provides confidence score            │
│  • Explains its reasoning               │
└─────────────────────────────────────────┘
         ↓
┌─────────────────────────────────────────┐
│              LLM (Opus)                 │
│  • Synthesizes engineering context      │
│  • Generates richer summaries           │
│  • Handles unstructured information     │
└─────────────────────────────────────────┘
         ↓
    Engineer receives:
    Classification + Confidence + Summary

Why This Division Makes Sense

Think of it like a hospital:

• The specialist doctor (Custom ML) gives a precise diagnosis based on years of specific medical records
• The medical writer (LLM) translates that diagnosis into a clear report for the patient

Neither replaces the other. Each does what it does best.

---

Step 4: The "Trust by Design" Principle

This is the most important concept in the article. Building AI that people actually use requires deliberate trust engineering.

The Three Pillars of Trust by Design

Pillar 1: Transparency

Every prediction includes:
├── The classification itself
├── A confidence score (How certain is the AI?)
└── Evidence explanation (Why did it decide this?)

Pillar 2: Human Control

High confidence prediction → Can be auto-accepted
Low confidence prediction  → Requires human review
Engineer disagrees         → Human overrides AI

Pillar 3: Continuous Learning

Engineer correction
        ↓
Becomes new training data
        ↓
Model improves
        ↓
Future predictions get better
        ↓
(Cycle repeats)

Critical Distinction: The goal was NOT to replace engineers. It was to teach the system to think the way engineers already think.

---

Step 5: How Institutional Knowledge Was Preserved

This step addresses a subtle but powerful concept: encoding human expertise into AI.

The Traditional Knowledge Problem

Expert Engineer retires/leaves
         ↓
Years of judgment disappears
         ↓
New engineers make inconsistent decisions
         ↓
Quality suffers

The BugWiser Solution

Years of historical engineering decisions
         ↓
Used as training data for custom ML model
         ↓
Model learns HOW experts evaluated bugs
         ↓
Expertise is preserved and applied consistently
         ↓
Even when individual experts leave

Analogy to Understand This

Think of it like a master chef's recipe book:

• The chef's judgment is encoded into written recipes
• Any trained cook can now replicate the dish consistently
• The chef's expertise lives beyond any single person

BugWiser did this with engineering judgment at scale.

---

Step 6: Measuring Real Success

Results matter. Here's how to evaluate whether an AI system actually worked:

The Metrics That Mattered

What 90% Acceptance Rate Really Means

100 bug predictions
├── ~90 accepted as-is (AI was correct)
└── ~10 corrected by engineers
         ↓
Those 10 corrections become training data
         ↓
Next batch: potentially 91-92% acceptance
         ↓
System continuously improves

---

Step 7: The Universal Lessons to Apply Anywhere

These principles extend far beyond bug triage.

Lesson 1: Match the Model to the Task

Need creativity and synthesis?     → Use LLM
Need consistency and precision?    → Use specialized ML model
Need both?                         → Use both strategically

Lesson 2: Bigger ≠ Better

"The best AI model is not always the largest model. It is the one best suited for the problem you're trying to solve."

Lesson 3: Design for Trust, Not Just Accuracy

An AI system with 95% accuracy that engineers don't trust will not be used. An AI system with 90% accuracy that engineers trust and can verify will transform workflows.

Lesson 4: Human Feedback is a Feature, Not a Fallback

Traditional thinking: Human review = AI failure
BugWiser thinking:    Human review = AI improvement data

---

Summary: The Complete Mental Model

PROBLEM: Manual, inconsistent, slow bug triage
         ↓
INSIGHT: Need deterministic + explainable AI
         ↓
SOLUTION: Hybrid (Custom ML + LLM)
         ↓
PRINCIPLE: Trust by Design
    ├── Confidence scores
    ├── Explainability
    └── Human-in-the-loop
         ↓
RESULT: 300 days → Less than 1 week
        90% prediction acceptance
        Preserved institutional knowledge

---

Self-Assessment Questions

Test your understanding:

1. Why couldn't a general-purpose LLM alone solve this problem?
2. What makes a prediction "trustworthy" in an engineering context?
3. How does the feedback loop prevent the model from becoming outdated?
4. When would you choose a specialized ML model over an LLM in your own work?
5. What is the difference between automating a task and encoding expertise into a system?

If you can answer all five confidently, you have mastered the core concepts of this article.