How to Secure AI Agents in Production

Peter Bubenik · Uber AI · · Source
Image for Securing AI Agents: Lessons From Real-World Deployment

After studying this material, you should be able to:

  1. Explain why traditional security controls are insufficient for AI agents
  2. Identify the unique threats that agentic AI systems introduce
  3. Describe the three-layer defense-in-depth framework for securing AI agents
  4. Understand how runtime enforcement extends beyond traditional authorization
  5. Recognize the practical tradeoffs involved in deploying AI security controls at scale

Step-by-Step Study Material

Step 1: Understanding Why AI Agents Are Different

Traditional Software vs. AI Agents

Think of traditional software like a vending machine:

  • You press B3
  • It dispenses that specific item
  • The logic is fixed and predictable

An AI agent is more like a human employee:

  • It reads emails, browses documents, uses tools
  • It reasons and makes decisions dynamically
  • It can take real-world actions based on that reasoning

Why This Matters for Security

Traditional SoftwareAI Agent
Executes predefined logicReasons dynamically over content
Attack surface = APIs and identityAttack surface includes semantic layer
"Can this user access this?""Should this agent do this right now?"

New Threats Introduced by AI Agents

ThreatSimple Definition
Prompt InjectionMalicious instructions hidden in content the agent reads
Tool PoisoningCorrupted tools that manipulate agent behavior
Memory PoisoningCorrupted stored context that misleads the agent
Goal HijackingTricking the agent into pursuing unintended objectives
Behavioral DriftAgent gradually deviating from its intended purpose
Hallucination-Driven ActionsAgent acting on fabricated information

Key Insight: An attacker doesn't need to break your login system. They can hide malicious instructions inside a webpage or document the agent reads, and the agent may follow those instructions.


Step 2: The Core Security Question Shifts

Old Question (Traditional Security):

"Is this user allowed to do this?"

New Question (Agentic Security):

"Is this action appropriate given:

  • What the user actually requested?
  • The agent's defined purpose?
  • The user's role and permissions?
  • Everything that has happened in this session so far?"

Concrete Example

Imagine a customer support AI agent:

  • ✅ User is authenticated → Traditional auth passes
  • ✅ Agent has permission to access the database → Authorization passes
  • ❌ A malicious webpage the agent visited injected instructions to export all customer data → Behavioral check should FAIL

Traditional security would have allowed this. Behavioral security catches it.


Step 3: The Three-Layer Defense Framework

The framework operates across the full agent lifecycle:

[PROVISIONING] → [RUNTIME ENFORCEMENT] → [CONTINUOUS ASSURANCE]
     ↑___________________________________________________|
                    Feedback Loop

Layer 1: Provisioning-Time Validation

What happens before the agent is deployed

Think of this as background checks before hiring an employee.

The AI Registry

  • Acts as the authoritative record for every agent
  • Stores: identity, ownership, persona, permitted intents, constraints, dependencies, risk tier
  • Establishes a behavioral contract — a formal definition of what the agent is allowed to do

Argus (Automated Analysis Platform)

  • Analyzes agent source code, prompts, configurations, and tool paths
  • Automatically derives what the agent actually does (vs. what developers claim it does)
  • Compares declared behavior with observed capabilities

Supply Chain Security

  • Evaluates third-party tools and MCP (Model Context Protocol) servers
  • Uses static analysis, dynamic analysis, and semantic inspection
  • Scans for unregistered agents operating outside security controls

Analogy: Just as you wouldn't let an unvetted contractor access your building, you shouldn't let unvetted tools access your AI agent.


Layer 2: Runtime Enforcement

What happens while the agent is running

This layer extends a familiar security model called PAP → PDP → PEP:

Policy Administration Point (PAP)
        ↓ defines policies
Policy Decision Point (PDP)
        ↓ evaluates: "Should this action happen?"
Policy Enforcement Point (PEP)
        ↓ allows or blocks the action

How It Maps to AI Security

RoleTraditionalAI Agent Version
PAPAccess control policiesAI Registry (personas, intents, constraints)
PDPAuthorization engineAI Guard (semantic evaluation)
PEPAPI gateway / firewallAgent Guard runtime hooks

What AI Guard Evaluates

Beyond "can this user do this?", it asks:

  1. Who is acting?
  2. What resources are involved?
  3. Why is this action being taken?
  4. Is it consistent with the agent's purpose?
  5. Does the session context support this action?

Performance Design

Runtime security must be fast. The solution:

  • Use small, specialized models instead of large general-purpose LLMs
  • Run guardrail checks in parallel (not sequentially)
  • Host models locally to avoid network latency
  • Use in-memory vector search for semantic matching

Why this matters: If security checks add too much delay, developers will bypass them. Speed is a security requirement.


Layer 3: Continuous Assurance

What happens after deployment, continuously

Think of this as ongoing auditing and improvement.

Key Capabilities

CapabilityPurpose
End-to-end observabilitySee everything the agent does
Interaction replayRe-run past sessions against new security rules
Automated threat modelingContinuously map the attack surface
Red teamingActively try to break the agent's security
Feedback loopFindings update risk scores, policies, and models

Why Replay Is Powerful

  • New attacks emerge constantly
  • You can test new defenses against real past behavior before enforcing them
  • Reduces risk of breaking legitimate functionality

Step 4: Key Security Mechanisms Explained

Prompt Provenance and Evidence Chaining

The Problem: An agent takes 20 actions in sequence. How do you know action #17 was actually requested by the user and not injected by a malicious document?

The Solution: Every action is cryptographically linked back to the original user request.

User Request → Action 1 → Action 2 → ... → Action 17
     ↑_______________________________________________|
              Provenance chain maintained

If action #17 cannot be traced back to a legitimate user request, it is flagged or blocked.

Personas and Intents

Persona: Defines who the agent is and what domain it operates in.

Example: "Customer support agent for ride-hailing, handles billing and trip issues only"

Intents: Explicit list of what the agent is allowed to do.

Example: ["refund_trip", "update_payment_method", "view_trip_history"]

Why this is the most effective control:

"The most effective mitigation we found was also the least glamorous: restricting agents to explicit intents and constraints."

If an action doesn't match a declared intent, it is denied — regardless of whether credentials are valid.

Constraints

Rules that define under what conditions an action is legitimate.

Example constraints:

  • "Only process refunds under $50 without manager approval"
  • "Never export more than 10 records at once"
  • "Only access data belonging to the authenticated user"

Step 5: Practical Tradeoffs You Must Understand

Security in agentic systems involves real tradeoffs. These cannot be ignored.

TradeoffDescription
Security vs. AccuracyStronger enforcement → more false blocks of legitimate actions
Security vs. LatencyMore checks → slower responses
Security vs. CostMore inference → higher compute costs
Security vs. AutonomyTighter constraints → less flexible agent behavior
Security vs. Developer ExperienceComplex onboarding → developers bypass controls

How These Were Addressed

ChallengeSolution
False blocksPer-agent calibration, shadow-mode testing before enforcement
High latencyParallel execution, small specialized models, local inference
Developer frictionAutomated onboarding via Argus, pre-generated personas/intents
Generic controlsAgent-specific personas, intents, and risk-based calibration

Core Lesson: Security, accuracy, latency, cost, autonomy, and developer experience must be optimized together, not independently.


Step 6: Synthesis — The Mental Model

Here is how everything connects:

BEFORE DEPLOYMENT
├── Register agent in AI Registry (behavioral contract)
├── Argus analyzes code → auto-generates personas & intents
└── Validate all tools and dependencies

DURING OPERATION
├── Every interaction evaluated by AI Guard
├── Checks: persona match, intent match, constraint compliance,
│          context consistency, provenance chain
└── Small fast models run in parallel for low latency

CONTINUOUSLY
├── All actions logged with full observability
├── Replay past sessions against new rules
├── Red team actively probes for weaknesses
└── Findings feed back into policies and model calibration

Quick Reference: Key Terms

TermDefinition
Behavioral ContractFormal definition of what an agent is designed and permitted to do
Prompt InjectionMalicious instructions embedded in content an agent processes
PersonaThe defined identity and domain scope of an agent
IntentAn explicitly permitted action the agent may take
ConstraintA condition that must be met for an action to be legitimate
ProvenanceThe traceable origin of an action back to a user request
Evidence ChainingLinking each action in a sequence to its legitimate source
Shadow ModeRunning security checks without enforcing them, to measure accuracy
Red TeamingActively attempting to defeat security controls to find weaknesses
MCP ServerModel Context Protocol server — a third-party tool provider for agents

Self-Check Questions

  1. Why can't traditional authorization alone secure an AI agent?
  2. What is the difference between a persona and an intent?
  3. How does prompt provenance defend against indirect prompt injection?
  4. Why are small specialized models preferred over large LLMs for runtime guardrails?
  5. What is shadow-mode deployment and why is it useful?
  6. Describe the PAP → PDP → PEP model as applied to AI agent security.
  7. What is the most effective (though least glamorous) security control described in the article?

More to study