Step-by-Step Teaching

Step 1: Understand the Big Picture

What is AI Gateway?

Think of AI Gateway as a single front door to many AI models. Instead of managing separate API keys and billing for each provider, you use one unified API key.

The Partnership:

• Vercel (AI Gateway) + ElevenLabs = 14 audio models in one place
• Same dashboard for usage, cost, and request tracing as your other AI models
• No markup on provider pricing

Your App → One AI Gateway API Key → Any of 14 ElevenLabs Models

---

Step 2: Learn the Six Model Types

Think of them in three groups:

Group A: Voice Communication

Group B: Audio Generation

Group C: Multi-Speaker

---

Step 3: Understand the Speech Pipeline (Most Complex Pattern)

This is the most important architecture to grasp. It chains three separate models:

Microphone Audio
      ↓
[Model 1: Scribe v2 Realtime]  ← Speech to Text
      ↓
  Text Transcript
      ↓
[Model 2: Claude / Any LLM]    ← Reasoning
      ↓
  Text Response
      ↓
[Model 3: Eleven Flash v2.5]   ← Text to Speech
      ↓
  Audio Output to User

Key benefit: Each step is swappable independently. Change the voice model without touching the transcription model.

The Code:

import {
  experimental_streamSpeech as streamSpeech,
  experimental_streamTranscribe as streamTranscribe,
  streamText,
} from 'ai';

// STEP 1: Transcribe microphone input → text
const heard = streamTranscribe({
  model: 'elevenlabs/scribe-v2-realtime',
  audio: microphoneStream,
  inputAudioFormat: { type: 'audio/pcm', rate: 16000 }, // 16kHz audio
});

// STEP 2: Send transcript to a language model → text response
const { textStream } = streamText({
  model: 'anthropic/claude-haiku-5.5',
  prompt: await heard.text,
});

// STEP 3: Convert LLM text response → spoken audio
const { audioStream } = streamSpeech({
  model: 'elevenlabs/eleven-flash-v2.5',
  text: textStream,
  voice: 'hpp4J3VqNfWAUOO0d1Us', // ElevenLabs voice ID
});

Why streaming matters: The user hears the reply start before the LLM finishes writing it. No waiting.

---

Step 4: Understand Realtime Voice Agents (Simpler Pattern)

Unlike the pipeline above, realtime agents handle everything in one model over a single WebSocket connection.

User speaks → [WebSocket] → Eleven Realtime v1 → [WebSocket] → User hears reply

No separate transcription or speech steps needed.

Two model options:

• eleven-realtime-v1-max → Higher quality
• eleven-realtime-v1-mini → Faster/lighter

The Code (Server Side):

import { gateway } from 'ai';

// Server only mints a short-lived token
// Your real API key never reaches the browser
const { token, url } = await gateway.experimental_realtime.getToken({
  model: 'elevenlabs/eleven-realtime-v1-max',
});

Browser Side (Key Settings):

// Use useRealtime hook from @ai-sdk/react
sessionConfig: {
  voice: 'your-elevenlabs-voice-id',
  turnDetection: { type: 'semantic-vad' } // ✅ Supported
  // turnDetection: { type: 'server-vad' } // ❌ NOT supported
}

Important: semantic-vad = the model understands meaning to detect when you've finished speaking, not just silence.

---

Step 5: Generate Music and Sound Effects

Both use the same generateSpeech function. The text parameter describes the sound instead of words to speak.

Music Example:

import { generateSpeech } from 'ai';

const { audio } = await generateSpeech({
  model: 'elevenlabs/music-v2.5',
  text: 'A mellow lo-fi beat with soft piano', // describe the music
  providerOptions: {
    elevenlabs: {
      musicLengthMs: 30000 // 30 seconds
    }
  },
});

Music model options:

• music-v2.5 → Latest
• music-v2 → Previous version

What you can specify in text:

• Genre ("lo-fi", "jazz", "cinematic")
• Mood ("mellow", "intense", "uplifting")
• Instruments ("soft piano", "electric guitar")
• Vocals or no vocals

Sound Effects work identically, just use elevenlabs/sound-effects-v2 and describe a short sound.

---

Step 6: Generate Multi-Voice Dialogue

One API call → one continuous audio file with multiple distinct voices.

import { experimental_streamSpeech as streamSpeech } from 'ai';

// Define your voice IDs
const host = 'hpp4J3VqNfWAUOO0d1Us';
const guest = 'CwhRBWXzGAHq8TQ4Fs17';

const { audioStream } = streamSpeech({
  model: 'elevenlabs/eleven-v4',
  text: [
    { type: 'text', text: 'Did the deploy go out?', voice: host },  // Line 1
    { type: 'text', text: 'It shipped an hour ago.', voice: guest }, // Line 2
  ],
  providerOptions: {
    elevenlabs: {
      dialogue: { voices: [host, guest] } // Register all voices used
    }
  },
});

Multi-voice model options:

• eleven-v4 → Standard
• eleven-v4-turbo → Faster
• eleven-v3 → Previous generation

Key advantage: No audio stitching. One request returns one seamless recording.

---

Step 7: Summary Mental Model

┌─────────────────────────────────────────────────────┐
│              ElevenLabs on AI Gateway               │
├─────────────────┬───────────────────────────────────┤
│ USE CASE        │ MODEL / APPROACH                  │
├─────────────────┼───────────────────────────────────┤
│ Transcribe audio│ scribe-v2-realtime (streamTranscribe) │
│ Speak text      │ eleven-flash-v2.5 (streamSpeech)  │
│ Full voice chat │ eleven-realtime-v1-max (WebSocket) │
│ Generate music  │ music-v2.5 (generateSpeech)       │
│ Sound effects   │ sound-effects-v2 (generateSpeech) │
│ Multi-speaker   │ eleven-v4 (streamSpeech + array)  │
└─────────────────┴───────────────────────────────────┘

One API key. One usage dashboard. All six capabilities.

---

Quick Knowledge Check

Ask yourself:

1. Why would you choose the pipeline approach over a realtime agent?
2. What does semantic-vad do that silence detection cannot?
3. How does generateSpeech serve double duty for both music and sound effects?
4. What is the security reason for minting a token server-side for realtime agents?