Scaling Security Alert Triage With Specialized Agents on Databricks

Concept 1: The Core Problem — Alert Triage at Scale

What is Alert Triage?

Security teams receive thousands of alerts daily from various sources:

• Endpoint security tools
• Cloud activity logs
• Threat intelligence feeds

Triage means sorting through these alerts to find real threats.

The Classic Tradeoff

Thousands of Daily Alerts
         ↓
┌─────────────────────────┐
│  HIGH   → Always review │
│  MEDIUM → Always review │
│  LOW    → Review if time│
└─────────────────────────┘

Why This Is a Problem

Key Insight: LOW severity alerts were historically ignored due to volume, yet they can contain real threats hiding in plain sight.

---

Concept 2: Why a Single AI Agent Failed

The Naive First Attempt

All Alert Data → Single Prompt → Foundation Model → Decision

Result: 50% escalation rate — half of all alerts sent to analysts.

Why It Failed: The Context Problem

Think of it like this:

Without Context:
IP Address: 192.168.1.1
Model thinks: "This could be suspicious..."
Result: Escalate (just in case)

With Context:
IP Address: 192.168.1.1
+ "This IP belongs to our internal monitoring service"
+ "It triggers this alert 200 times/day normally"
Model thinks: "This is expected behavior"
Result: Close

What "Context" Actually Means

• False-positive patterns — what normally triggers false alarms for THIS source
• Behavioral baselines — what normal looks like for THIS source
• Relevant enrichment — additional data specific to THIS source

Key Insight: General cybersecurity knowledge alone is insufficient. Accurate triage requires source-specific context.

---

Concept 3: The Specialized Multi-Agent Architecture

The Solution: 17 Source-Specific Agents

Instead of one generalist agent, build specialists:

Alert Stream
     ↓
┌─────────────────────────────────────┐
│         Alert Router                │
└─────────────────────────────────────┘
     ↓              ↓              ↓
[S3 Agent]    [IAM Agent]   [Endpoint Agent]
     ↓              ↓              ↓
         ↘         ↓         ↙
          [TI Agent - Shared]
               ↓
         Threat Intelligence
         Assessment

The Threat Intelligence (TI) Agent

A dedicated shared agent that any source-specific agent can call:

Any Agent encounters suspicious IP/domain
              ↓
         Calls TI Agent
              ↓
    Queries threat intel sources
              ↓
Returns structured assessment:
┌─────────────────────────────────┐
│ Artifact: 203.0.113.42          │
│ Status: KNOWN MALICIOUS         │
│ Reason: C2 infrastructure       │
│ Confidence: HIGH                │
│ Timeframe: Last 14 days         │
└─────────────────────────────────┘

Key Insight: Raw indicators (like an IP address) are meaningless without context. The TI agent transforms raw data into actionable intelligence.

---

Concept 4: How Each Agent Works — The 6-Step Pipeline

Every agent follows the same design principles:

Step 1: Deterministic Filtering

Alert arrives
     ↓
Programmatic rules check:
- Is this a known-safe IAM role?
- Is this expected service account activity?
- Is this IP on our trusted list?
     ↓
YES → Instant CLOSE (no LLM call needed)
NO  → Continue to next step

Impact: Handles 30–95% of alerts depending on source.

Why this matters: The cheapest AI call is the one you never make.

Step 2: Context Enrichment

Before LLM sees the alert:
┌────────────────────────────────────┐
│ Pull last 6 months alert history   │
│ for affected user/entity           │
│                    +               │
│ Related activity from other sources│
│                    +               │
│ TI Agent results (if needed)       │
└────────────────────────────────────┘
         ↓
    Enriched Alert Package

Step 3: Specialized Prompt Functions

Alert Title → Maps to → Specific Prompt Function

"S3 Anomalous Access"  → IAM recon evaluation prompt
"Privilege Escalation" → Role assumption pattern prompt
"Unknown Alert Type"   → Generic fallback prompt

Each prompt function runs in one of two modes:

Step 4: Shared Tools (Optional)

LLM analyzes evidence...
"I need more context"
        ↓
Available tools:
- Raw cloud audit logs
- Cross-source correlated alerts  
- IdP activity history
        ↓
LLM decides IF and WHEN to use them

Step 5: LLM Reasoning & Disposition

Evidence Package → LLM Analysis → Structured Output

Output includes:
┌─────────────────────────────────┐
│ Disposition: ESCALATE           │
│ Confidence: HIGH                │
│ Reasoning: "Third suspicious    │
│ action from same user in 48hrs" │
└─────────────────────────────────┘

Three possible dispositions:
ESCALATE → Send to analyst queue
MONITOR  → Watch but don't escalate
CLOSE    → Benign, no action needed

Step 6: Cost Management

Three-layer cost control:

Layer 1: Deterministic filtering
         (never reach LLM = zero cost)
         
Layer 2: Batch cost cap
         (stop processing if budget hit)
         
Layer 3: Daily alert cap
         (ceiling on total daily spend)
         
+ Per-category tool call budgets
  (prevent runaway discovery loops)

---

Concept 5: The Feedback Loop — Making Agents Smarter Over Time

How Ground Truth Is Built

Agent escalates alert
        ↓
Analyst reviews ticket
        ↓
Analyst confirms OR overrides
        ↓
Decision recorded as label
on MLflow trace
        ↓
Labeled traces = Ground Truth Dataset
        ↓
Used to evaluate future prompt changes

Why This Matters for Testing

Traditional Software Testing:
Code → Test against fixed specification ✓/✗

Agent Testing:
Agent → Same alert can produce different outputs
      → Must test against ANALYST JUDGMENT STANDARD
      → Not a fixed specification

The Review App Workflow

Databricks Review App
        ↓
Analyst sees:
┌─────────────────────────────────┐
│ Alert inputs                    │
│ Agent's reasoning steps         │
│ Final decision                  │
│ [Confirm] [Override]            │
└─────────────────────────────────┘
        ↓
Labels ALL dispositions:
- Escalated alerts ✓
- Monitored alerts ✓  ← Previously invisible
- Closed alerts ✓    ← Previously invisible

Key Insight: Without reviewing non-escalated alerts, you only know when the agent is wrong about escalating — not when it's wrong about closing.

---

Concept 6: Key Lessons Learned

Lesson 1: LLMs Struggle With High-Entropy Security Data

LLMs are good at:          LLMs are bad at:
─────────────────          ────────────────
Natural language           File hashes
Patterns & reasoning       Random subdomains
Behavioral analysis        Generated filenames

Solution:

❌ Don't ask: "Is hash a3f9b2c1... malicious?"
✅ Do instead: Use tool call → Query authoritative source → Feed result to LLM

Lesson 2: Context Is King

The biggest performance gains came from:

• Historical alert data (6 months per user)
• Per-alert-type false positive rates
• Explicit behavioral patterns

Real example:

Single alert: "Unusual login attempt"
→ Might close (low confidence alone)

Same alert + context:
"3rd suspicious signal from same user this week"
→ Escalates (behavioral correlation)

Lesson 3: Automate the Predictable, Reason About the Rest

Predictability Spectrum:

PREDICTABLE ←──────────────────→ UNPREDICTABLE
     ↓                                  ↓
Deterministic Rules              LLM Reasoning
(fast, cheap, reliable)          (flexible, expensive)

Best Practice:
Filter with rules FIRST → Only send ambiguous cases to LLM

---

Summary: The Complete System

Petabytes of Security Logs
           ↓
    Detection Architecture
           ↓
    Centralized Alerts Table
           ↓
    Spark Structured Streaming
           ↓
┌──────────────────────────────────────┐
│         17 Specialized Agents        │
│                                      │
│  Step 1: Deterministic Filter        │
│  Step 2: Context Enrichment          │
│  Step 3: Specialized Prompt          │
│  Step 4: Tool Use (if needed)        │
│  Step 5: LLM Disposition             │
│  Step 6: Cost Controls               │
└──────────────────────────────────────┘
           ↓              ↓
      ESCALATE        CLOSE/MONITOR
           ↓              ↓
    Analyst Queue    Delta Tables
           ↓
    Human Review
           ↓
    MLflow Trace Labels
           ↓
    Ground Truth Dataset
           ↓
    Agent Improvement

Results at a Glance

Final Key Insight: The goal isn't to replace human analysts — it's to ensure no alert goes uninvestigated while making sure analysts only spend time on alerts that genuinely warrant their attention.