Agent Skills: ElevenLabs Performance Tuning

ElevenLabs Performance Tuning

Overview

Optimize ElevenLabs TTS latency and throughput through model selection, streaming strategies, audio format tuning, and caching. Latency ranges from ~75ms (Flash) to ~500ms (v3) depending on configuration.

The two highest-leverage, lowest-effort levers — model choice (Step 1) and output format (Step 2) — are documented inline below. The four deeper integrations (HTTP streaming, WebSocket streaming, caching, parallel generation) are summarized here with copy-ready code in the full implementation walkthrough.

Prerequisites

• ElevenLabs SDK installed (@elevenlabs/elevenlabs-js)
• An ElevenLabs API key exported as ELEVENLABS_API_KEY (used by the SDK and passed as xi_api_key on the WebSocket handshake)
• Understanding of your latency requirements
• Audio playback infrastructure (browser, mobile, server-side)

Instructions

Step 1: Model Selection for Latency

The single biggest performance lever is model choice:

| Model | Avg Latency | Quality | Languages | Use Case | |-------|-------------|---------|-----------|----------| | eleven_flash_v2_5 | ~75ms | Good | 32 | Real-time chat, IVR, gaming | | eleven_turbo_v2_5 | ~150ms | Good | 32 | Balanced speed/quality | | eleven_multilingual_v2 | ~300ms | High | 29 | Narration, content creation | | eleven_v3 | ~500ms | Highest | 70+ | Maximum expressiveness |

// Select model based on use case
function selectModel(useCase: "realtime" | "balanced" | "quality" | "max_quality"): string {
  const models = {
    realtime:    "eleven_flash_v2_5",
    balanced:    "eleven_turbo_v2_5",
    quality:     "eleven_multilingual_v2",
    max_quality: "eleven_v3",
  };
  return models[useCase];
}

Step 2: Output Format Optimization

Smaller formats = faster transfer:

| Format | Size/Second | Quality | Best For | |--------|-------------|---------|----------| | mp3_44100_128 | ~16 KB/s | High | Downloads, archival | | mp3_22050_32 | ~4 KB/s | Medium | Streaming, mobile | | pcm_16000 | ~32 KB/s | Raw | Server-side processing | | pcm_44100 | ~88 KB/s | Raw | High-quality processing | | ulaw_8000 | ~8 KB/s | Phone | Telephony/IVR |

// Use smaller format for streaming, higher quality for downloads
const streamingConfig = {
  output_format: "mp3_22050_32",  // 4 KB/s — fast streaming
  model_id: "eleven_flash_v2_5",   // ~75ms first byte
};

const downloadConfig = {
  output_format: "mp3_44100_128", // 16 KB/s — high quality
  model_id: "eleven_multilingual_v2",
};

Step 3: HTTP Streaming for Time-to-First-Byte

Call client.textToSpeech.stream() instead of .convert() and write each chunk to the response as it arrives, so playback starts before generation finishes — roughly halving time-to-first-byte. Set style: 0.0 in voice_settings to shave another 10–20%. Full server handler: implementation.md § Step 3.

Step 4: WebSocket Streaming for Lowest Latency

For interactive apps where text arrives incrementally (e.g., an LLM token stream), open a stream-input WebSocket, sendText() chunks as they arrive, and tune chunk_length_schedule — fewer characters per chunk means lower latency but less prosody context. Full bidirectional client: implementation.md § Step 4.

Step 5: Audio Caching

Cache generated audio for repeated content (greetings, prompts, errors) in an LRU cache keyed by a SHA-256 of voiceId:modelId:text, so a changed voice or model never serves stale audio. This eliminates ~99% of latency for repeated phrases. Full cachedTTS helper: implementation.md § Step 5.

Step 6: Parallel Generation

Generate multiple segments concurrently with a p-queue whose concurrency matches your plan's request limit (going higher returns 429s, not more throughput). Full chapter-generator: implementation.md § Step 6.

Output

Applying these levers produces:

• A model + output-format choice matched to the use case (Steps 1–2).
• A streaming code path (HTTP or WebSocket) that logs measured time-to-first-byte, e.g. Time to first byte: 78ms / WebSocket TTFB: 91ms.
• An LRU audio cache emitting [Cache HIT] / [Cache MISS] telemetry for repeated content.
• A concurrency-bounded batch path that logs per-segment generation time.

Expected latency after tuning: ~75–150ms first byte on Flash/Turbo with streaming, versus ~300–500ms for a blocking convert() call on a higher-quality model.

Performance Optimization Checklist

| Optimization | Latency Impact | Implementation | |-------------|----------------|----------------| | Flash model | -60% vs v2, -85% vs v3 | Change model_id | | Streaming endpoint | -50% time-to-first-byte | Use .stream() instead of .convert() | | WebSocket streaming | Best for LLM integration | See Step 4 | | Smaller output format | -30% transfer time | mp3_22050_32 vs mp3_44100_128 | | Audio caching | -99% for repeated content | LRU cache with SHA-256 keys | | style: 0 | -10-20% latency | Remove style exaggeration | | Concurrency queue | Maximize throughput | p-queue matching plan limit |

Error Handling

| Issue | Cause | Solution | |-------|-------|----------| | High TTFB | Wrong model | Switch to eleven_flash_v2_5 | | Choppy streaming | Network buffering | Use pcm_16000 for direct playback | | Cache miss storm | TTL expired for popular content | Use stale-while-revalidate pattern | | WebSocket drops | Network instability | Reconnect with buffered text | | Memory pressure | Audio cache too large | Set maxSize limit on LRU cache | | HTTP 429 | Concurrency above plan limit | Lower p-queue concurrency |

Examples

Real-time IVR (lowest latency). Pick eleven_flash_v2_5 + ulaw_8000 via selectModel("realtime"), then stream over HTTP:

await streamToResponse(greeting, voiceId, res); // logs "Time to first byte: 78ms"

LLM voice agent (incremental text). Open a WebSocket and forward tokens as they stream from the model, ending with finish():

const stream = await createTTSStream({ voiceId, chunkLengthSchedule: [50, 100, 150] });
stream.sendText("Hello, "); stream.sendText("how are you?");
const audio = await stream.finish();

Audiobook batch (throughput). Cache repeated phrases and generate chapters concurrently:

const buffers = await generateChapters(chapters, voiceId); // 5-wide, cache-backed

Full, runnable versions of every snippet above are in the implementation walkthrough.

Resources

• ElevenLabs Streaming API
• WebSocket API Reference
• ElevenLabs Models
• LRU Cache
• Full implementation walkthrough — copy-ready HTTP streaming, WebSocket, caching, and parallel-generation code

Next Steps

For cost optimization once latency is tuned, see the elevenlabs-cost-tuning skill, which covers character-usage budgeting, model-tier cost tradeoffs, and cache-hit-rate targets.