← Home

Voice conversion technology at ElevenLabs transforms voices for immersive content across languages.

Peter Bubenik · Elevenlabs Research · · Source
Voice conversion technology at ElevenLabs transforms voices for immersive content across languages.

TL;DR

ElevenLabs is building identity-preserving automatic dubbing that can translate speech into other languages while keeping the same person’s voice and emotional delivery—making global content feel native, not subtitled.

Key Discoveries (12 total)

  1. ElevenLabs introduced “Agent Skills,” a collection of reusable capabilities designed to help developers build faster with ElevenLabs.

  2. Voice conversion can transform one person’s voice into another’s by generating the same message in a new voice while keeping the original speaker’s intonation (how the speech rises/falls and flows).

  3. Voice cloning works by encoding a target voice’s identity (the recognizable “sound” of a person) so the system can generate new speech that matches that identity.

  4. High-quality voice conversion and cloning could significantly reduce production time and costs across content industries.

  5. People who contribute their voices for training could earn passive fees, creating a new way for voice owners to be compensated over time.

  6. ElevenLabs’ voice research is primarily aimed at a flagship product: an identity-preserving automatic dubbing tool planned for release early next year.

  7. The core goal is one-click multilingual accessibility in the original speaker’s voice, so audiences can hear content in their language without losing the creator’s “presence.”

  8. The system aims for “native-grade” speech in the target language, meaning it should sound natural to native listeners—not robotic or obviously translated.

  9. Voice cloning is used to preserve speaker identity across languages, generating new utterances that still sound like the same person.

  10. Voice conversion is used to preserve emotion, intent, and speaking style, improving immersion by keeping the performance—not just the words.

  11. The article highlights a fundamental technical tradeoff: emphasizing the target voice too much can lose the source emotion/timing, while emphasizing the source too much can weaken the illusion of the target identity.

  12. The paper explicitly raises ethical and legal risks and the need for safeguards, citing real-world harm (a 2020 scam using audio deepfakes to authorize a $35 million transfer) and concerns like fraud, disinformation, defamation, and copyright/consent issues.

Why This Matters

Most translation today is either text (captions) or “dubbed” audio that loses the original speaker’s identity and performance. If ElevenLabs can reliably keep both the voice (who it sounds like) and the delivery (how it’s said) while switching languages, global education, entertainment, and communication could feel far more human and engaging—like the creator is speaking directly to you.

At the same time, the same realism that makes this powerful also makes it risky. The article makes clear that convincing voice replication can enable fraud and manipulation, so building safeguards alongside the technology isn’t optional—it’s central to whether society can benefit from these tools without being harmed by them.

The Research Approach

ElevenLabs’ “how” is essentially a two-part pipeline: (1) learn what a person’s voice sounds like (identity) and (2) learn how they are saying it (delivery), then recombine those pieces to produce speech in a new language that still feels like the same person.

Here is the process in simple steps, using the article’s own framing:

  1. Collect many examples of the target voice They train the system on a dataset with “many examples” of the target speaker’s speech. This is like giving the model a large photo album of someone’s face from different angles so it can recognize what stays consistent.

  2. Encode the target speaker’s “voice identity” (voice cloning) The model learns a compact representation of what makes that voice recognizable. The article calls this “encoding the target voice’s identity,” meaning the stable traits that make you say “that sounds like the same person,” even across different sentences.

  3. Break speech down into small building blocks (phonemes) The article explains that speech can be decomposed from sentences → words → phonemes.
    A phoneme is the smallest sound unit that can change meaning in a language (for example, the difference between the “b” and “p” sound). The model operates at this “fundamental” level so it can rebuild speech precisely.

  4. Parse the source utterance to capture content + performance The system takes the original recording (source speech) and extracts:

    • Content: what was said (the message)
    • Delivery: intonation, emotion, intent, timing (how it was said)

  5. Map the source content into the target language They describe training “robust multi-language models” that can “parse utterances in the source language and map them onto the target language with the right intonation.”
    In plain terms: the model learns how to express the same idea in another language while trying to keep the same rise/fall, emphasis, and pacing.

  6. Render the new speech using the target voice’s characteristics (voice conversion) This is the “recombination” step: generate the translated speech in the target voice, while trying to preserve the original performance. The article summarizes the goal as expressing “source speech content with target speech characteristics.”

  7. Tune the balance between identity and emotion (the key tradeoff) They explicitly describe a balancing act:

    • Push too hard toward the target voice and you can lose sync with the source performance.
    • Push too hard toward the source performance and the target identity becomes less believable.

A helpful analogy from the article is face-swapping: you place “markers” on a face to constrain how the swap happens. In voice conversion, the “markers” are like the constraints imposed by the target voice’s learned phoneme-level characteristics. More constraints can improve identity, but can also restrict faithful transfer of the original delivery.

How It Works in Practice

The article uses a few concrete scenarios to show the method “in action” (even if it does not provide experimental tables or metrics).

Educational YouTube video example (English → Spanish)

  • Scenario: A creator records an educational video in English. A Spanish-only viewer would normally rely on captions.
  • Method in action: ElevenLabs aims to generate “that same person speaking the same message naturally in native-grade Spanish.”
  • What this illustrates: the pipeline is not just translation; it is translation + voice identity preservation + performance preservation, so the viewer experiences it as if the creator is directly speaking to them.

Emotion + identity stress test (angry shouting → “Morgan Freeman” voice)

  • The article gives a vivid example: rendering “somebody shouting angrily in Morgan Freeman’s voice.”
  • What this illustrates: the core technical challenge is not producing words, but keeping two things true at once:
    • It should still sound like Morgan Freeman (identity)
    • It should still feel angry and shouted (emotion and delivery)
  • This example is used to explain the tradeoff: emphasize the target voice too much and you may flatten or distort the anger; emphasize the anger too much and it may stop sounding like the target speaker.

Real-world misuse case (2020 $35M deepfake scam)

  • They cite scammers using an audio deepfake to impersonate a CEO and authorize a $35 million transfer.
  • What this illustrates methodologically: the same “identity encoding + realistic generation” approach that enables dubbing also enables impersonation, which is why safeguards must be designed alongside the core system.

What Makes This Different

Traditional dubbing and many older speech systems typically separate tasks in a way that loses “human presence.”

1) It targets “identity-preserving” dubbing, not just translation Traditional dubbing usually replaces the original actor with a different voice actor. Even if the translation is accurate, the speaker identity is gone. Here, the novelty is the explicit goal: same speaker, new language.

2) It treats performance as a first-class object (not an afterthought) Many systems focus on intelligibility (clear words) and correctness (right translation). ElevenLabs emphasizes preserving emotion, intent, and style of delivery for immersion. That is a different optimization target: not only “is it correct,” but “does it feel like the same moment, performed by the same person.”

3) Multilingual modeling is built into the conversion step They highlight “robust multi-language models” that can parse source utterances and map them to target language “with the right intonation.” This is important because intonation patterns differ across languages; you cannot simply swap words and expect natural speech.

4) The “face-swap marker” framing clarifies controllability By comparing voice conversion constraints to face-swap markers, they are describing a controllable system: you can adjust how strongly the target identity constrains the output versus how much the source performance drives it. That explicit tradeoff is central to making the output both believable and expressive.

Overcoming Challenges

The article is candid about two main categories of obstacles: technical tradeoffs and misuse risks.

Challenge 1: The identity vs. emotion tradeoff

  • Problem: If you force the output to match the target voice too strongly, you can lose timing and emotional alignment with the source. If you preserve the source too strongly, the target identity becomes less convincing.
  • How they address it (as described): they frame it as a tuning problem in the conversion process—how much “preference” to give to target speech characteristics versus source delivery. The multilingual models are trained to “map” utterances while keeping intonation, which is essentially an attempt to reduce the tradeoff by improving the mapping quality.

Challenge 2: Cross-language naturalness (“native-grade” speech)

  • Problem: Even if a voice sounds right, translated speech can sound unnatural to native listeners if rhythm, stress, and phrasing are off.
  • How they address it: training multilingual models that learn to produce target-language speech patterns while still carrying over the source speaker’s delivery cues.

Challenge 3: Ethical and legal misuse (fraud, disinformation, consent, copyright)

  • Problem: The same capability that preserves identity can be used to impersonate people or generate content without consent.
  • How they address it (at the approach level): they argue safeguards should be introduced “at the time of development” to minimize harm. The article does not list specific technical safeguards (like watermarking or verification), but it clearly positions safety as part of the development methodology, not a later add-on.

If you want, I can turn this into a simple diagram-style explanation (inputs → models → outputs) and label exactly where “voice cloning” ends and “voice conversion” begins, since the article uses both terms and they can blur together for readers.

More to study