Before building anything, understand what environment you are operating in.
Admiral Insurance operates in a high-stakes context:
High stakes + High volume + Multi-jurisdiction =
You cannot afford to "figure it out as you go"
| Too Narrow | Too Broad |
|---|---|
| Proves nothing | Never ships |
| No real complexity | Impossible timeline |
| Cannot test real systems | Stalls in development |
Admiral chose settlement quotes in UK lending as their first use case because it was:
✅ Bounded — Limited number of conversation paths
✅ Complex enough — Touched telephony AND back-end integrations
✅ High volume — Enough traffic to generate real feedback quickly
✅ Visible impact — Faster resolution was measurable immediately
Pre-launch testing catches 80% of problems.
The remaining 20% only appear when REAL customers talk to the system.
Therefore: Get to production FAST to discover what testing cannot show you.
Ask these questions:
If you answer yes to all five, you have a viable first use case.
Many teams think:
"We'll start with lower standards for the AI pilot, then tighten things up later."
An AI agent making an error is different in kind from a human agent making an error:
| Human Error | AI Error |
|---|---|
| One instance | Potentially thousands of instances simultaneously |
| Judgment call | Systematic pattern |
| Explainable | Harder to audit after the fact |
| Correctable in the moment | May repeat before detection |
Commitment 1: Build for production from Day 1
Commitment 2: Governance that moves at technology speed
Problem: Traditional governance takes 6 months to approve something.
Result: You approve technology that is already outdated.
Solution: Rebuild the review cadence to match how fast
models and tooling are actually changing.
Commitment 3: Legal and compliance from Day Zero
Day Zero Compliance vs. Late Compliance
─────────────────────────────────────────────────────
Shapes the design Blocks the launch
Finds problems early (cheap) Finds problems late (expensive)
Builds trust incrementally Creates last-minute panic
Compliance owns the outcome Compliance feels like an obstacle
Think of compliance as a building with multiple floors:
┌─────────────────────────────────────┐
│ ADMIRAL'S INTERNAL CULTURE │ ← Highest standard
│ (sometimes stricter than law) │
├─────────────────────────────────────┤
│ ADMIRAL'S INTERNAL RULES │ ← Company policy
│ (business-specific requirements) │
├─────────────────────────────────────┤
│ REGULATION │ ← Legal minimum
│ (the floor, not the ceiling) │
└─────────────────────────────────────┘
Critical mindset shift:
Regulation tells you the minimum you must do.
Your design should reflect who you want to be, not just what you must do.
Admiral operates in UK, Italy, France, and Spain. Each has different rules, for example:
Solution: Build agents that handle differences without letting customers exploit escalation paths to avoid resolvable conversations.
This is one of the most technically important concepts in the article:
| Use Case | Logic Type | Why |
|---|---|---|
| Understanding caller intent | Non-deterministic | Requires flexibility and inference |
| Detecting customer distress | Non-deterministic | Nuanced, contextual signals |
| Required regulatory statement | Deterministic | Must be provable, zero improvisation |
| Authentication gate | Deterministic | Must be consistent and auditable |
| Mandatory conversation path | Deterministic | Cannot be skipped or altered |
MAIN AGENT
│
├── [Deterministic Gate: Authentication] ──→ PASS/FAIL
│ ↓ (if pass)
├── [Sub-Agent: Intent Detection] (non-deterministic)
│
├── [Sub-Agent: Distress Detection] (non-deterministic)
│ ↓ (if distress detected)
│ [Route to Human Agent]
│
├── [Sub-Agent: Settlement Quote] (mixed)
│
└── [Deterministic Gate: Regulatory Disclosure] ──→ ALWAYS RUNS
Admiral keeps certain conversations entirely off the AI agent:
This is not just a compliance requirement. It is an ethical design choice that sits above the regulatory floor.
Business writes requirements
↓
Hands off to engineering
↓
Engineering builds something
↓
Business says "that's not what we meant"
↓
Repeat for months
Bring into one room, at the same time:
Result: A working v0 agent, connected to backend and telephony, in 4 to 5 hours.
Technology introduced TO people → Resistance, confusion, low adoption
Technology built WITH people → Ownership, understanding, real adoption
Every change, large or small, follows the same process:
Write change (prompt, workflow, logic)
↓
Run against FULL simulation test suite
(hundreds to thousands of simulated conversations)
↓
Deploy to 1% of live traffic
↓
Monitor: CSAT + Resolution Rate in real time
↓
Numbers hold? → Expand to 10% → 50% → 100%
Numbers drop? → Roll back immediately
Timeline improvement: This cycle went from weeks to hours.
Easy: Write a prompt that passes 5 simulated tests.
Hard: Build test suites that GENUINELY STRESS the agent
AND define clear success criteria BEFORE you start.
Admiral runs hundreds to thousands of simulated
conversations against EVERY change before it ships.
Treat simulation testing as real work, not a checkbox.
Admiral made a costly assumption early on:
Wrong approach:
Write all prompts in English → Translate to French, Spanish, Italian
Result: Performance in non-English markets was significantly worse
Right approach:
Write prompts NATIVELY in each market's language
Result: Responses matched local customer expectations and culture
Lesson: Localization is about culture and expectation, not just vocabulary.
| Metric | What It Measures | Admiral's Finding |
|---|---|---|
| Handling Time | Speed to resolution | AI often faster than humans (no dead air while systems load) |
| CSAT | Customer satisfaction | Climbed incrementally, market by market |
| Resolution Rate | Problems actually solved | Improved through repeated testing cycles |
Important: Improvements came through repeated rounds of testing and adjustment, not a single launch.
PHASE 1: SELECTION
└── Choose one use case: bounded, complex enough, high volume, measurable
PHASE 2: FOUNDATION
├── Bring legal/compliance in on Day Zero
├── Set governance cadence to match technology speed
└── Commit to building for production, not proof of concept
PHASE 3: DESIGN
├── Layer compliance: regulation → internal rules → culture
├── Assign deterministic logic to anything that must be provable
├── Assign non-deterministic logic to intent and nuance
└── Design human escalation paths for vulnerable customers
PHASE 4: BUILD
├── Run co-build workshops with all stakeholders in one room
├── Produce working v0 within hours, not weeks
└── Transfer business process knowledge, not just technical specs
PHASE 5: DEPLOY
├── Run full simulation test suite against every change
├── Roll out at 1% → monitor → expand to 100%
└── Write prompts natively in each market's language
PHASE 6: ITERATE
├── Use real customer conversations to find the last 20%
├── Track handling time, CSAT, and resolution rate
└── Expand to new use cases using proven integrations
Choosing a narrow, real use case is not a compromise. It is the mechanism that gets you to production, which is the only place real learning happens.
Admiral moved faster because it set higher standards, not despite them. Governance that moves at technology speed, compliance from Day Zero, and deterministic logic for critical paths all made the system more deployable, not less.
Building with auditability, consistency, and human escalation from Day One made Admiral's agents more disciplined. Regulation became the foundation of a trustworthy system, not an obstacle to building one.
You are starting an AI agent project at a financial services company. How would you select your first use case? What five criteria would you apply?
A colleague suggests bringing legal and compliance in "once we have something to show them." How would you respond, and what would you propose instead?
Explain the difference between deterministic and non-deterministic logic in AI agents. Give two examples of each from the insurance context.
Your AI agent performs well in English but poorly after translation to German. What is the likely cause and what is the correct fix?
Describe the production rollout process Admiral uses. Why does starting at 1% of traffic matter?
What is the "last 20 percent" problem, and why can pre-launch testing not solve it?