By the end of this material, students should be able to:

Integrate a voice-enabled AI agent into a Unity game using the ElevenLabs Unity SDK, enabling real-time spoken conversation between players and NPCs.

Specifically, students will:

• Set up the ElevenLabs SDK in Unity
• Start and end voice agent sessions
• Customize agent behavior per character
• Send game context to the agent mid-conversation
• Trigger game actions from agent responses
• Handle platform-specific requirements

---

Step-by-Step Study Material

Step 1: Understand the Core Concept

What Problem Are We Solving?

Traditional NPCs have pre-written dialogue. This SDK allows NPCs to:

How It Works — The Big Picture

Player speaks → Microphone → ElevenLabs Agent → Voice response → AudioSource → Player hears NPC

Three core components work together:

1. Agent — Created in ElevenLabs dashboard (holds the AI prompt + voice)
2. Conversation API — Manages the live session
3. Unity SDK — Bridges your game to the API

---

Step 2: Set Up Your Environment

Prerequisites Checklist

• Unity project (desktop, WebGL, or mobile)
• ElevenLabs account
• Newtonsoft Json package (SDK dependency)

2.1 Create Your Agent

1. Go to the ElevenLabs dashboard
2. Create a new agent
3. Set the prompt (defines personality/role)
4. Assign a voice
5. For initial testing → go to Security settings → turn Authentication OFF

⚠️ Why turn off Authentication for testing? Authentication requires a server to generate signed URLs. Disabling it lets you connect with just the Agent ID during development. Always re-enable before shipping.

2.2 Install the SDK in Unity

Window → Package Manager → + → Install package from git URL

Paste this URL:

https://github.com/elevenlabs/unity.git#v0.1.0

2.3 Import the Sample

Package Manager → ElevenAgents → Samples → GettingStarted → Import

Paste your Agent ID into the sample → Press Play

Expected result: A cube greets you when you walk toward it (~10 minutes total)

---

Step 3: Start and End a Conversation Session

The Core Pattern

A session opens when the player enters a trigger zone and closes when they leave:

// Conceptual structure
void OnTriggerEnter() {
    await conversation.StartSessionAsync(new SessionConfig {
        AgentId = "your-agent-id"
    });
}

void OnTriggerExit() {
    conversation.EndSession();
}

Key Behaviors to Understand

💡 Mental Model: Think of StartSessionAsync like opening a phone call — it waits until both sides are connected before returning.

---

Step 4: Make the Voice Come From the Character

Default vs. Spatial Audio

Default behavior:    Voice plays in 2D (everywhere equally)
With AudioSource:    Voice plays FROM the character's position

Implementation

// Attach your character's AudioSource component
new SessionConfig {
    AgentId = "your-agent-id",
    OutputAudioSource = characterAudioSource  // ← add this
}

Driving Animations with Volume

// GetOutputVolume() returns 0.0 to 1.0
float volume = conversation.GetOutputVolume();

// Use it to animate a mouth or bob a character
mouthOpenAmount = volume;
characterTransform.localScale = baseScale + (Vector3.up * volume * bobAmount);

💡 Use Case: The sample makes a cube bob up and down while talking. You can use the same value to blend a mouth-open animation shape key.

---

Step 5: One Agent, Many Characters

The Problem

You have 10 NPCs. Creating 10 separate agents is wasteful.

The Solution: Dynamic Variables

Define placeholders in your agent's prompt on the dashboard:

You are a guard. Your current mood is {{mood}}. Your name is {{name}}.

Then each character fills in their own values at session start:

new SessionConfig {
    AgentId = "shared-agent-id",
    DynamicVariables = new Dictionary<string, object> {
        { "mood", "angry" },        // string
        { "name", "Captain Rex" },  // string
        { "alertLevel", 3 },        // number
        { "isOnDuty", true }        // boolean
    }
}

Going Further: Overrides

For bigger changes (different voice, different first message):

new SessionConfig {
    AgentId = "shared-agent-id",
    Overrides = new AgentOverrides {
        FirstMessage = "Halt! Who goes there?",
        // Voice override also possible
    }
}

⚠️ Each override must be enabled in the agent's Security settings first.

---

Step 6: Keep the Agent Informed Mid-Conversation

The Problem

The player picks up a sword during a conversation. The NPC doesn't know.

The Solution: Contextual Updates

// Send game state without interrupting the conversation
conversation.SendContextualUpdate("The player just picked up the ancient sword.");
conversation.SendContextualUpdate("A dragon has appeared nearby. Threat level: high.");

Important Distinction

💡 Mental Model: SendContextualUpdate is like whispering stage directions to an actor mid-scene — the audience (player) doesn't hear it, but the actor (agent) adjusts.

---

Step 7: Let the Agent Control Your Game

The Concept: Client Tools

The agent can call functions in your game. This is two-way communication:

Game → Agent:  "The player entered the dungeon"  (contextual update)
Agent → Game:  "OpenDoor(doorId: 'main-gate')"   (client tool)

Setup Process

Step A: Define the tool in the ElevenLabs dashboard

Tool name: OpenDoor
Parameters: { "doorId": string }

Step B: Register a handler in Unity under the same name

conversation.RegisterClientTool("OpenDoor", async (parameters) => {
    string doorId = parameters["doorId"].ToString();
    
    // Your game logic
    bool success = await GameManager.OpenDoor(doorId);
    
    if (!success) {
        throw new ClientToolException("Door not found: " + doorId);
    }
    
    return "Door opened successfully";  // Agent receives this
});

Key Details

---

Step 8: Handle Events and UI Updates

Available Events

conversation.UserTranscriptReceived += (text) => {
    subtitleUI.text = "Player: " + text;
};

conversation.AgentResponded += (text) => {
    subtitleUI.text = "NPC: " + text;
};

conversation.ModeChanged += (mode) => {
    listeningIndicator.SetActive(mode == ConversationMode.Listening);
};

conversation.Interrupted += () => {
    Debug.Log("Player interrupted the NPC");
};

✅ All events fire on Unity's main thread — safe to update UI directly, no Dispatcher needed.

Useful Controls

conversation.SetMicMuted(true);          // Mute player mic
conversation.SetVolume(0.8f);            // Adjust agent volume
conversation.SendUserMessage("Hello");   // Text instead of voice
float[] freq = conversation.GetFrequencyData(); // For audio visualizers

---

Step 9: Platform-Specific Requirements

Quick Reference

WebGL Limitations to Remember

❌ Play Mode doesn't work when WebGL is active build target
   → Switch to Desktop target for testing in Editor

❌ AudioSource mixer routing doesn't work
❌ Doppler effect doesn't work  
❌ OutputAudioSource ignored in WebRTC sessions
✅ Voice plays through Web Audio API instead

---

Step 10: Security — Before You Ship

Development vs. Production

DEVELOPMENT:
Player → Agent ID → ElevenLabs
(Simple, but exposes your agent)

PRODUCTION:
Player → Your Server → Signed URL (short-lived) → ElevenLabs
(Secure, API key never in game build)

Production Code Pattern

// ❌ NEVER in a shipped build
new SessionConfig { AgentId = "your-agent-id" }

// ✅ Correct for production
string signedUrl = await YourServer.GetSignedUrl();
new SessionConfig { SignedUrl = signedUrl }

🔴 Critical Rule: Your ElevenLabs API key must never be inside a game build. Anyone can extract it and use your credits.

---

Summary: Complete Mental Model

┌─────────────────────────────────────────────────────┐
│                    YOUR UNITY GAME                   │
│                                                      │
│  Player walks in    →    StartSessionAsync()         │
│  Player speaks      →    Microphone captured         │
│  Game event occurs  →    SendContextualUpdate()      │
│  Agent speaks       →    AudioSource (spatial!)      │
│  Agent wants action →    ClientTool handler fires    │
│  Player walks out   →    EndSession()                │
└─────────────────────────────────────────────────────┘
                           ↕ Conversation API
┌─────────────────────────────────────────────────────┐
│              ELEVENLABS DASHBOARD                    │
│                                                      │
│  Agent Prompt + Voice + Tools + Security Settings    │
└─────────────────────────────────────────────────────┘

Key Concepts Checklist

• Agent created in dashboard with prompt and voice
• SDK installed via Package Manager
• Session started/ended with trigger colliders
• OutputAudioSource set for spatial voice
• GetOutputVolume() driving mouth animation
• Dynamic variables personalizing shared agents
• SendContextualUpdate for mid-conversation game events
• Client tools letting agent trigger game code
• Events updating UI on main thread
• Microphone permissions handled on mobile
• Authentication enabled and signed URLs used before shipping