Speech-to-Text for AI Agents: How to Build Voice-Enabled Agent Pipelines
I've built voice-enabled agent pipelines for customer service automation, real-time meeting assistants, and hands-free operational tools. What I've learned is that connecting speech-to-text to an AI agent isn't a single integration — it's a pipeline with distinct stages, each requiring deliberate architectural choices. Get it right, and you have a voice interface that handles millions of interactions reliably. Get it wrong, and you'll spend months debugging latency issues, accuracy degradation, and cost overruns that compound at scale.
In this guide, I'll walk you through building a production voice-enabled agent pipeline: the architecture patterns, the STT API selection criteria that actually matter for agents, the output mode tradeoff that directly impacts your LLM costs, and the integration points with popular agent frameworks.
The Voice-Enabled Agent Pipeline: Architecture Overview
A voice-enabled agent pipeline has three core stages:
Stage 1: Audio Capture — Raw audio from a microphone, phone call, or stream. This stage includes VAD (voice activity detection) to determine when someone is speaking and silence suppression to remove dead air.
Stage 2: Speech-to-Text (STT) — The transcription layer that converts audio to text. This is where your pipeline succeeds or fails — accuracy, latency, and cost are all determined here.
Stage 3: Agent Processing — The LLM interprets the transcript, reasons about intent, and generates a response or action.
Stage 4 (Optional): Text-to-Speech (TTS) — If your agent speaks back, you need TTS to convert the response to audio.
The STT layer is the keystone. It determines what your agent sees, how fast it sees it, and how much it costs to process each conversation. Everything upstream (audio quality, VAD tuning) and downstream (prompt engineering, response latency) is optimized around your STT performance.
Choosing the Right STT API for AI Agent Workloads
Not all speech-to-text APIs are built for agent pipelines. When I evaluate STT providers for agent integrations, I look at four dimensions that matter in production:
Latency — Agent applications are latency-sensitive. A customer service agent waiting 3 seconds for transcription adds 3 seconds to every interaction. Real-time agent pipelines need streaming transcription with sub-second latency, not batch processing. For streaming use cases, look for WebSocket-based APIs that support partial results. AssemblyAI and Deepgram both offer streaming APIs with sub-500ms latency targets. OpenAI's Whisper API is optimized for batch workloads and isn't designed for real-time streaming.
Accuracy on Agent Audio — Benchmark accuracy numbers are quoted on clean, curated audio. Agent audio is rarely clean — background noise, overlapping speakers, domain-specific terminology, and varying audio quality all affect accuracy differently. Before committing, run a test batch with 50-100 audio files from your actual production workload. A provider that scores 95% on generic benchmarks might drop to 78% on your specific audio domain.
Output Modes for LLM Ingestion — This decision most directly affects your pipeline cost and agent quality. Raw transcripts include everything — filler words, false starts, repeated phrases. Your LLM prompt has to process all of this noise. Clean transcripts remove disfluencies but preserve full semantic content. Agent-optimized transcripts go further — they structure output specifically for LLM consumption. Privocio's three output modes — Raw, Clean, and Agent — let you balance fidelity against token cost. For high-volume agent pipelines, Agent mode can reduce token counts by 40% or more compared to Raw.
Pricing Model — Per-minute pricing creates unpredictable costs at scale. A pipeline processing 1,000 hours of audio per month at $0.36/minute costs $21,600/month. Fixed-rate pricing like Privocio's scales linearly without per-minute surprises.
STT API Comparison for AI Agent Pipelines
| Provider | Latency | Streaming | Output Modes | Pricing | Best For |
|---|---|---|---|---|---|
| AssemblyAI | ~300ms | Yes (WebSocket) | Raw, Clean, Sentiment | Per-minute | Real-time agents |
| Deepgram | ~250ms | Yes (WebSocket) | Raw, interim, formatted | Per-minute | Low-latency agents |
| OpenAI Whisper API | 1-5 min batch | No | Raw only | Per-minute | Async transcription |
| Privocio | ~500ms | Yes | Raw, Clean, Agent | Fixed ($19-$79/4 wks) | High-volume, cost-sensitive |
| LiveKit | ~200ms | Yes (built-in) | Raw with VAD | Per-minute + infra | Real-time conferencing |
Framework Integration: LiveKit, Pipecat, and Custom Stacks
Once you've selected your STT provider, you need to integrate it with your agent framework. Three patterns dominate:
LiveKit is a real-time streaming platform with built-in STT integration. It's the go-to choice for video conferencing and voice agents where sub-300ms latency is critical. LiveKit's agent framework handles audio routing, VAD, and transcription in a single platform, and integrates with Pipecat for higher-level agent orchestration.
Pipecat is a framework specifically designed for building voice and video AI agents. It handles coordination between STT, LLM, and TTS services, letting you focus on agent logic rather than infrastructure plumbing. Pipecat supports multiple STT providers and provides a clean abstraction for multi-modal agents.
Custom Stack — For production pipelines with specific compliance requirements or specialized output formatting, a custom integration gives you full control. Build your own audio capture layer, connect to your STT provider via WebSocket, and handle the transcript stream in your agent code. This requires more engineering but eliminates framework constraints.
Output Modes: The Token Cost Lever
I've seen teams spend weeks optimizing their LLM prompts, then ignore the transcript format that feeds those prompts. The output mode directly determines your token costs.
Raw transcripts preserve everything — filler words, repeated phrases, false starts, speaker labels, timestamps. A 60-second utterance might produce a 400-token Raw transcript with significant noise.
Clean transcripts remove the most egregious filler but preserve speaker turns and full semantic content. Token count typically drops 15-25% vs. Raw.
Agent-optimized transcripts restructure output for LLM consumption — removing filler, standardizing formatting, stripping content that adds tokens without meaning. For conversational agents, Agent mode typically reduces token counts by 40% or more compared to Raw.
At high volume, this matters. Processing 1,000 hours of audio per month with 300 tokens/Raw vs. 180 tokens/Agent at $0.01/1K tokens with GPT-4o — that's meaningful savings at scale without changing your model, prompts, or agent logic.
Real-Time vs. Batch: When Each Matters
Most agent pipelines need real-time transcription — the agent responds while the user is still speaking. But batch has its place:
Real-time (streaming WebSocket APIs) — Required for interactive agents where response latency affects user experience. Use interim results to start agent processing before the user finishes speaking.
Batch (async APIs with webhooks) — Appropriate for post-call analysis, meeting summaries, content indexing, and quality assurance. Batch typically offers higher accuracy (providers can spend more compute per request) and lower per-minute pricing.
For voice agents that speak back, you need the full STT-LLM-TTS stack. For TTS, ElevenLabs is the dominant choice for agent applications — low latency, natural-sounding voices, and fine-tuning for brand voice. Azure TTS and Amazon Polly are viable alternatives if you're already invested in those clouds.
Related Guides
- How Clean Transcripts Cut Your LLM Token Costs by 40% or More — The data behind transcript optimization and token reduction
- Voice Pipeline Architecture: Building the STT-LLM-TTS Stack — Deep dive into full-stack voice agent architecture
- How to Add Voice Input to Your AI Chatbot — Step-by-step integration for adding speech to existing text agents
- Real-Time vs Batch Transcription — Decision framework for choosing streaming vs. async
- Async Transcription with Webhooks — Implementation guide for webhook-based batch processing
- Transcription Output Modes Explained — Raw vs. Clean vs. Agent mode technical details
Frequently Asked Questions
What's the minimum latency achievable with a production voice agent pipeline?
With optimized streaming STT (Deepgram, AssemblyAI, or LiveKit) and a fast LLM, end-to-end latency of 800ms-1.5s is achievable in production. Real-time conversational quality requires under 1 second round-trip — anything slower feels laggy.
How do I handle multiple speakers in a voice agent?
Most streaming STT APIs don't provide real-time speaker diarization. For multi-party conversations, use separate audio channels per speaker (best for controlled environments) or defer to batch diarization after the fact. Design your agent prompts to handle single-speaker input and handle multi-party asynchronously.
Can I use Privocio for real-time agent pipelines?
Yes — Privocio supports streaming transcription via WebSocket with sub-1-second latency on the Go and Enterprise plans. The Agent output mode delivers token-optimized transcripts that reduce LLM costs by 40% or more. Enterprise plans include dedicated infrastructure for predictable performance at scale.
What's the most common failure point in voice agent pipelines?
Audio quality. Teams optimize their LLM prompts and agent logic, then deploy to production audio that varies wildly — mobile phone recordings, Bluetooth headsets, conference room echo. The single highest-impact improvement is investing in audio preprocessing: noise reduction, normalization, and VAD tuning.
How do I keep costs predictable at high volume?
Switch from per-minute to fixed-rate pricing. At 500+ hours per month, per-minute STT costs become a significant budget variable. Fixed-rate plans like Privocio's $19/4 weeks for 400 hours give you predictable costs that scale linearly.
Conclusion: Start With Production Audio, Not Benchmarks
The voice-enabled agent pipeline you build today will handle thousands or millions of conversations. Start with a provider that gives you control over output modes, pricing predictability, and deployment flexibility. Run your pilot with production audio, optimize for your actual latency and accuracy requirements, and build failure handling into every stage.
If you're evaluating speech-to-text for your agent pipeline, compare Privocio's pricing and features — or start with the free tier to test Agent output mode with your own audio.
Image Credits:
Images sourced from Unsplash (Unsplash License).