Processing Images and Documents in ElevenAgents

Concept 1: Why Multimodal Input Matters

The core problem: Many real-world situations cannot be fully described in words alone.

Think about these scenarios:

• A site supervisor photographing a materials shortage
• A customer showing a damaged product
• Someone uploading a policy PDF for review

The traditional limitation:

Customer describes problem in words
→ Agent asks clarifying questions
→ Back-and-forth conversation
→ Slower resolution

With multimodal input:

Customer sends photo/document directly
→ Agent sees exactly what the customer sees
→ Faster, more accurate resolution

Key insight: When a customer can show rather than describe, the agent resolves issues faster without asking them to switch channels.

---

Concept 2: How ElevenAgents Is Structured Around Channels

What is a "channel"? A channel is the communication pathway a customer uses to reach the agent.

The critical design principle:

One Agent Configuration
        ↓
┌─────────────────────────────┐
│  Prompt  │  Model  │ Tools  │
│  Knowledge Base  │  Voice  │
└─────────────────────────────┘
        ↓
Deployed across ALL channels simultaneously

What varies per channel:

1. The transport layer (how data physically moves)
2. The input types it supports

What stays the same:

• The agent's behavior, personality, and capabilities

---

Concept 3: The Two Types of Input Representation

This is the most technically important concept. Every input — regardless of channel — gets normalized into one of two internal representations before reaching the model.

Path 1: File-Backed Inputs

What qualifies: Images and PDFs

Customer uploads image/PDF
        ↓
Platform stores the file
        ↓
Assigns a unique file_id
        ↓
Raw file passed directly to model's context window
        ↓
Model sees actual visual/document structure

Why this matters: The model receives the actual file, not a text description of it. This preserves:

• Spatial relationships
• Visual layout
• Document formatting

Path 2: Inline Inputs

What qualifies: Everything else

Customer sends voice note
        ↓
Platform transcribes it
        ↓
Plain text enters the transcript
        ↓
Model reads text (no file reference)

Why the distinction matters for developers

---

Concept 4: Setting Up Multimodal Input

Two requirements must both be met before file input works:

Requirement 1: Enable the file input flag

conversation_config.conversation.file_input.enabled = True

Set this either:

• Via API at agent creation
• In dashboard: Settings → Advanced Settings → File Input

Requirement 2: Use a vision and document-capable model

The flag alone does nothing without a compatible model. Both must be configured together.

❌ Flag enabled + standard model = file input does NOT work
❌ Flag disabled + vision model = file input does NOT work
✅ Flag enabled + vision/document model = file input works

---

Concept 5: How Files Move Through Each Channel

Web and Mobile Flow

Critical rule: Sequencing is mandatory

Step 1: Upload the file first
        ↓
Receive file_id from upload endpoint
        ↓
Step 2: Send message referencing that file_id
        ↓
Model receives file in context window

Why sequencing matters:

If the message is sent without the file_id, the model has no reference to the file — regardless of whether the upload succeeded.

The SDKs simplify this by handling the upload and reference steps internally in a single call.

WhatsApp Flow

WhatsApp works completely differently — your application plays no role in the upload:

Customer sends image/document in WhatsApp chat
        ↓
File goes to Meta's infrastructure first
        ↓
Meta notifies ElevenLabs via webhook
        ↓
ElevenLabs downloads file server-to-server
        ↓
ElevenLabs stores its own copy + assigns file_id
        ↓
Agent receives it as multimodal input

Key implication: Since your application never handles the upload, you never hold the file directly during the conversation. You can only retrieve it after the conversation ends.

Side-by-Side Comparison

---

Concept 6: Carrying Context Across Sessions

The fundamental architectural rule:

ElevenAgents processes each conversation independently. Nothing carries into the next conversation automatically.

This creates a real problem for multimodal use cases:

Session 1: Customer sends photo of broken part
           Agent processes it, conversation ends
                    ↓
Session 2: Customer calls back to follow up
           Agent starts from ZERO — no memory of the photo
           Customer must repeat themselves ❌

The Solution Pattern: Two Halves

Half 1: When a conversation ends (State OUT)

Conversation ends
        ↓
Post-call webhook fires
        ↓
Delivers to your backend:
  - Full transcript
  - Analysis results
  - Data collection fields
  - File URLs for any files in the session
        ↓
Your backend stores relevant data against
a durable customer identifier
(phone number, user ID, account key)

Half 2: When a customer returns (State IN)

Customer starts new conversation
        ↓
Your application looks up stored context
        ↓
Injects context via dynamic variables at session start
        ↓
Agent begins conversation already knowing
what it learned previously ✅

The simple formula:

Webhook carries state OUT → Your system stores it → Dynamic variables carry it back IN

---

Concept 7: Context Injection Varies by Channel

The pattern is consistent, but the mechanism differs:

Important limitation:

ElevenAgents does NOT merge sessions across channels into a single thread.

A WhatsApp conversation and a web conversation are separate sessions even for the same customer. However, because the webhook output and dynamic variable injection work identically across all channels, one persistence layer handles all of them.

---

Concept 8: Carrying Files Forward Across Sessions

Files are scoped to one conversation. What you carry forward depends on what the next session actually needs.

Case 1: You only need the information from the file (most common)

Customer sends photo of cracked door seal (Session 1)
        ↓
Agent interprets the photo
        ↓
Post-call webhook delivers transcript + data collection results
        ↓
You extract: "claim concerns a cracked door seal"
        ↓
Store against customer ID
        ↓
Session 2: Inject as dynamic variable
"Customer previously reported a cracked door seal"
        ↓
Agent knows the context without needing the photo again ✅

Case 2: You need the actual file (compliance, records, downstream systems)

Post-call webhook arrives
        ↓
Each uploaded file appears as a file_input event
with a signed file URL
        ↓
⚠️ URL is valid for only 15 MINUTES
        ↓
Download and store immediately when webhook arrives
        ↓
Do NOT defer this step

Fallback if you miss the window: While the conversation still exists, the GET conversation API reissues fresh URLs.

Edge case to plan for: In zero-retention mode, file_input events may be absent. Don't assume every file-backed turn carries a URL.

---

Summary: The Complete Mental Model

SETUP
└── Enable file_input flag + vision-capable model

INPUT PATHS
├── File-backed (images, PDFs)
│   └── Stored with file_id → raw file to model → preserves structure
└── Inline (voice, text, location, contacts)
    └── Normalized to text → lives in transcript

CHANNELS
├── Web/Mobile: You upload → get file_id → reference in message
└── WhatsApp: Meta → ElevenLabs downloads → file_id assigned automatically

CROSS-SESSION CONTEXT
├── State OUT: Post-call webhook → your backend stores it
├── State IN: Dynamic variables injected at session start
└── Files: Download from webhook within 15 minutes or use GET API fallback

The core philosophy: ElevenAgents handles the complexity of normalizing diverse inputs. Your integration responsibility is managing what persists between conversations — that's where the real continuity lives.