Databricks integrates OpenTelemetry tracing in Unity Catalog for enhanced AI agent observability.

Concept 1: What Are AI Traces and Why Do They Matter?

What is a trace?

A trace is a detailed record of everything that happens when an AI agent processes a request. Think of it like a flight recorder for your AI system.

What does a trace capture?

Why do traces matter?

Without traces, you are essentially flying blind. You cannot answer:

• Why did the agent give that answer?
• Which step caused the slowdown?
• Did the agent behave correctly?

Simple analogy: Traces are like a detailed receipt for every AI interaction — showing every ingredient used, every step taken, and how long each step took.

---

Concept 2: The Problem With Traditional Trace Storage

Where traces usually live

Traditionally, traces are stored inside SaaS observability tools (specialized monitoring platforms).

Why this creates problems

Traditional Approach:
Agent → Observability Tool → (stuck here)
                           ↓
              Hard to query with SQL
              Hard to join with business data
              Governance is fragmented
              Requires extra pipelines to move data
              Sensitive prompt data is hard to protect

The core limitation

When traces live only in observability systems:

• You cannot easily run SQL analytics on them
• You cannot join them with your business data
• You need duplicate pipelines to move them elsewhere
• Governance (who can see what) becomes complicated

---

Concept 3: OpenTelemetry (OTel) — The Open Standard

What is OpenTelemetry?

OpenTelemetry (OTel) is an open-source standard for collecting and exporting telemetry data from software systems.

What does it collect?

OpenTelemetry collects three types of signals:

1. TRACES  → execution paths and spans
2. LOGS    → structured event records
3. METRICS → numerical measurements (latency, counts, etc.)

Why does the standard matter?

• It separates instrumentation from storage
• Any OTel-compatible tool can send data to any OTel-compatible destination
• You are not locked into one vendor's format

Simple analogy: OTel is like a universal power adapter. Your device (agent) uses one standard plug, and it works with any compatible outlet (storage system).

---

Concept 4: Unity Catalog — Governed Data Storage on Databricks

What is Unity Catalog?

Unity Catalog is Databricks' centralized governance layer for all data assets. It provides:

Why store traces in Unity Catalog?

Traces stored in Unity Catalog become first-class data — meaning they can be:

• Queried with SQL
• Joined with business data
• Governed like any other sensitive dataset
• Used in analytics pipelines

---

Concept 5: Zerobus Ingest — The Managed Pipeline

What problem does it solve?

Normally, getting data from an application into a data lake requires:

App → Message Bus (Kafka) → Processing Layer → Storage

This is complex, expensive, and requires infrastructure management.

What is Zerobus Ingest?

Zerobus Ingest is Databricks' fully managed, serverless ingestion engine that:

App (OTel client)
      ↓
  Zerobus Ingest  ← handles throughput, durability, zero infrastructure
      ↓
  Unity Catalog (Delta tables)

Key capabilities

• Supports OTLP via gRPC (standard OTel protocol)
• Supports REST API for frameworks like MLflow
• Acts as a single-sink — data goes directly to the lakehouse
• Bypasses Kafka entirely
• Handles up to 200 QPS (queries per second) by default

Simple analogy: Zerobus Ingest is like a managed postal service. You drop your package (trace data) at the door, and it handles all the routing, sorting, and delivery — no warehouse management required.

---

Concept 6: The Table Structure in Unity Catalog

What tables are created?

When you set up OTel tracing in Unity Catalog, six table types are provisioned:

<table_prefix>_otel_spans       → detailed execution data per request
<table_prefix>_otel_logs        → structured log/event data
<table_prefix>_otel_metrics     → numerical telemetry (latency, counts)
<table_prefix>_otel_annotations → MLflow-specific metadata, tags, feedback
<table_prefix>_trace_unified    → one record per trace (full picture)
<table_prefix>_trace_metadata   → MLflow tags grouped by trace ID (faster)

How to think about these tables

---

Concept 7: Instrumenting an Agent With MLflow

What is instrumentation?

Instrumentation means adding code (or using automatic tools) to capture trace data from your agent.

Two approaches

Automatic tracing:

mlflow.langchain.autolog()
# Automatically captures all LangGraph model calls and tool calls

Manual tracing:

@mlflow.trace
def my_agent_entrypoint(input):
    # Creates a root span for the entire request
    ...

What is a span?

A span is one unit of work within a trace. A full trace is made up of many spans:

Trace (full request)
  └── Root Span: handle_user_request
        ├── Span: call_llm
        ├── Span: call_tool_genie (1st call)
        ├── Span: call_tool_genie (2nd call)
        ├── Span: call_tool_genie (3rd call)
        └── Span: generate_response

---

Concept 8: Analytics on Trace Data

What becomes possible once traces are in Unity Catalog?

1. Ad-hoc SQL queries

SELECT request, response, latency_ms
FROM mlflow_experiment_trace_unified
WHERE latency_ms > 5000

2. Native dashboards (built into MLflow Experiment UI)

• Trace volume over time
• Error rates
• Latency (P50/P99)
• Token usage and cost

3. Custom dashboards (AI/BI Dashboards)

• Custom cost analysis with your negotiated pricing
• Per-tool latency breakdown
• Outlier detection

4. Natural language queries via Genie

• Non-technical users can ask questions in plain English
• Example: "Which tool had the highest error rate last week?"

---

Concept 9: Governance and PII Handling

Why is governance critical for traces?

Traces contain prompts and responses, which often include:

• Personal information (names, emails)
• Sensitive business data
• Confidential queries

How Unity Catalog handles this

Trace Data in Unity Catalog
         ↓
┌─────────────────────────────────┐
│  Fine-grained access controls   │ ← Who can access which tables
│  Column masking                 │ ← Hide sensitive columns
│  Row-level filtering            │ ← Restrict which rows are visible
└─────────────────────────────────┘

Important note: The system does not automatically detect or redact PII. You must configure the governance rules yourself using Unity Catalog's tools.

---

Concept 10: ETL Pipelines and Change Data Feed (CDF)

What is Change Data Feed?

Change Data Feed (CDF) is a Delta table feature that tracks only the new or changed rows since the last time you processed the data.

Why does this matter for traces?

Without CDF:

Every pipeline run → Scan entire trace table → Slow and expensive

With CDF:

Every pipeline run → Read only new traces since last run → Fast and efficient

What can ETL pipelines do with traces?

• Monitor for latency spikes → trigger alerts
• Detect tool failure patterns → notify teams
• Track token usage anomalies → flag cost overruns
• Feed dashboards and notification systems

---

Concept 11: The Evaluation Workflow

What is evaluation in AI?

Evaluation means systematically scoring your agent's outputs to measure quality.

How traces enable evaluation

Step 1: Bootstrap an evaluation dataset from real traces

Real user interactions → Captured as traces → Extracted as eval dataset

This is better than synthetic data because it reflects actual user behavior.

Step 2: Define judges

• Built-in judges: MLflow provides standard quality metrics
• Custom judges: You define rules specific to your agent's expected behavior

Step 3: Score and review

• Results appear in the MLflow Experiment UI
• You can see which responses passed or failed each judge

Development vs. Production evaluation

---

Concept 12: The Continuous Improvement Flywheel

What is the flywheel?

All the concepts above connect into a self-reinforcing cycle:

        Production Agent
              ↓
         Traces captured
              ↓
    Stored in Unity Catalog
         ↙         ↘
  Analytics        Evaluation
  Dashboards       & Monitoring
  Alerts           ↓
         ↘         ↙
      Insights & Improvements
              ↓
         Better Agent
              ↓
        (cycle repeats)

Why is this powerful?

Each loop through the cycle:

• Reveals new failure patterns
• Improves evaluation datasets
• Refines agent behavior
• Reduces costs
• Improves user experience

---

Summary: How All Concepts Connect

1. AI Agent runs anywhere
        ↓
2. OTel standard instruments the agent (spans, logs, metrics)
        ↓
3. Zerobus Ingest receives the telemetry (no Kafka needed)
        ↓
4. Unity Catalog stores it in Delta tables (governed, scalable)
        ↓
5. Teams query with SQL, build dashboards, use Genie
        ↓
6. Governance protects sensitive prompt data
        ↓
7. ETL pipelines enable streaming analytics and alerts
        ↓
8. Evaluation workflows score agent quality
        ↓
9. Insights feed back into agent improvement (the flywheel)

Every concept builds on the previous one, creating a complete, production-ready observability system for AI agents.