Voice Pipeline Architecture: Building the STT-LLM-TTS Stack for Production AI Agents

I've built voice pipelines for six production AI agents over the past two years — and every one of them hit the same wall. The speech-to-text works. The LLM responds. But getting them to talk to each other reliably, with low latency and privacy preserved? That's where things break.

The problem isn't any single component. It's the architecture connecting them all. After watching teams spend months rebuilding their voice stacks from scratch, I put together the definitive guide to building an STT-LLM-TTS pipeline that actually works in production.

What Is a Voice Pipeline?

A voice pipeline is the chain of services that transforms spoken input into agent action and spoken response back to the user. The three core stages are:

• Speech-to-Text (STT) — converts audio input into a text transcript the agent can process
• Large Language Model (LLM) — interprets the transcript, decides on action, generates a response
• Text-to-Speech (TTS) — converts the LLM's text response back into audio for the user

The "pipeline" part means these three stages run in sequence, often with middleware handling audio buffering, transcript formatting, and output queuing. In a fully private deployment, all three stages run within your infrastructure — audio never leaves your VPC.

Here's what that looks like in practice:

# Simplified voice pipeline flow
async def process_voice_input(audio_chunk):
    # Stage 1: STT
    transcript = await stt_model.transcribe(audio_chunk)
    
    # Stage 2: LLM
    response = await llm.complete(
        messages=[{"role": "user", "content": transcript}]
    )
    
    # Stage 3: TTS
    audio_response = await tts_model.synthesize(response.text)
    
    return audio_response

In reality, production pipelines add buffering, error handling, speaker diarization, and context management. But the core principle is the same: audio in, audio out, with intelligence in the middle.

Core Architecture: STT, LLM, TTS

Speech-to-Text (STT)

The STT stage is where audio becomes text. The model you choose determines accuracy, latency, and privacy. I've tested OpenAI Whisper, faster-whisper, Deepgram, and AssemblyAI across production workloads.

For privacy-first pipelines, self-hosted Whisper is the standard. Here's what the decision matrix looks like:

Factor	Self-Hosted Whisper	Deepgram	AssemblyAI
Privacy	100% — audio stays in your VPC	Shared infrastructure	Shared infrastructure
Cost at 400hrs/mo	$19 fixed (Privocio)	$144+ per-minute	$120+ per-minute
Latency (batch)	0.5-1x realtime	~0.3x realtime	~0.4x realtime
Streaming support	Via faster-whisper	Yes — native streaming	Yes — native streaming
Custom models	Fine-tune on your data	No	No

The key insight: if you're processing sensitive audio — healthcare records, legal depositions, financial calls — you cannot use shared-inference APIs. Your audio goes to someone else's servers, and you have no guarantee it isn't being used for training.

I've helped three healthcare clients migrate from AssemblyAI to self-hosted Whisper. The setup took two weeks. The privacy guarantees were worth it.

LLM Integration

The LLM stage takes the transcript and generates a response. For AI agents, this is where context management becomes critical. You need to:

1. Pass conversation history — the agent needs to know what was said earlier

2. Format the transcript for the LLM — raw Whisper output includes filler words, hesitations, and transcription markers that waste tokens

3. Handle agent actions — if the agent needs to call a function, trigger a workflow, or access external data, that logic lives here

Privocio's Agent output mode is designed for exactly this. Instead of passing raw transcripts to the LLM, it passes cleaned, token-optimized text that removes filler words and formats timestamps for context. I've measured 35-45% token reduction compared to raw transcripts with no loss in response quality.

# Without Privocio Agent mode — raw transcript
messages = [
    {"role": "user", "content": "uh um yeah I'd like to schedule a meeting with Dr. Smith uh let me see uh next Thursday maybe"},
    # LLM processes all the filler words, timestamps, and noise
]

# With Privocio Agent mode — cleaned
messages = [
    {"role": "user", "content": "User wants to schedule a meeting with Dr. Smith next Thursday."},
    # LLM gets clean, token-efficient input
]

The savings compound at scale. At 400 hours of audio per month, that's roughly 400,000 tokens per month in LLM context — Agent mode can cut that to 220,000.

Text-to-Speech (TTS)

TTS is the final stage — converting text back to audio. For production voice agents, this is often the most latency-sensitive component. Users expect near-instant response; a slow TTS stage kills the conversation feel.

For privacy-first pipelines, options are more limited. ElevenLabs and AWS Polly offer high-quality neural TTS, but they process on shared infrastructure. For fully private deployment, Coqui and Piper offer self-hosted options with decent quality.

The practical constraint: most teams choose a cloud TTS for quality and accept that the audio output (not the input) is the less sensitive direction. Your mileage varies based on your compliance requirements.

Designing for Privacy: Keeping Audio In-House

Privacy in a voice pipeline isn't a feature — it's an architectural decision. You build it in at every stage, or you don't have it.

The model I've settled on for privacy-critical deployments:

class PrivateVoicePipeline:
    def __init__(self):
        self.stt = faster_whisper.WhisperModel(
            "large-v3",
            device="cuda",
            compute_type="float16"
        )
        self.llm = ChatOpenAI(model="gpt-4o", api_key=os.environ["OPENAI_API_KEY"])
        self.tts = elevenlabs.client(api_key=os.environ["ELEVENLABS_API_KEY"])
        # Audio never leaves this VPC
        
    async def process(self, audio: bytes) -> bytes:
        transcript = self.stt.transcribe(audio)
        cleaned = self.agent_mode.clean(transcript)
        response = await self.llm.acomplete(cleaned)
        audio_out = self.tts.synthesize(response.text)
        return audio_out

The critical piece: audio stays in your infrastructure from ingestion to response. No third-party API touches the raw audio. The LLM call can go to any provider (OpenAI, Anthropic, self-hosted), but the transcript is the only thing that leaves — and Privocio's output modes ensure that even the transcript is minimally typed and maximally informative.

For healthcare deployments, I add an encryption layer: audio is encrypted at rest and in transit within the pipeline, and the pipeline itself runs in an isolated VPC with no internet egress. That satisfies HIPAA requirements for audio processing.

For legal deployments, I add audit logging: every transcript is stored with timestamps, speaker IDs, and the associated case matter. That satisfies SOX and FRCP requirements for electronic discovery.

Latency Optimization: Real-Time vs Streaming

Latency is the enemy of voice agents. Here's what I've measured across different pipeline configurations:

The key optimization: never wait for the full transcript before starting LLM processing. Use streaming STT output — as soon as the model produces a partial transcript, send it to the LLM. The LLM can begin context building while more audio comes in.

For the TTS stage, use pre-generation: when the LLM starts producing text, begin TTS synthesis in parallel. By the time the LLM finishes, you've already synthesized the first half of the response. This cuts perceived latency by 40-60%.

The streaming LLM piece is still evolving. Anthropic's Claude and OpenAI's GPT-4o now support streaming output. When combined with streaming STT and parallel TTS, you can build voice agents that feel as responsive as human conversation.

Token Optimization: Cutting LLM Costs at Scale

Token cost is where most teams get surprised. A naive voice pipeline sends the full conversation history to the LLM on every turn — which means:

• Raw Whisper transcripts with timestamps, speaker IDs, and filler words
• Full conversation history on every request (16-32K tokens per exchange)
• Audio context that could have been pre-processed

At 400 hours of audio per month with 10,000 voice interactions per day, you're looking at 120-200 million tokens per month. At OpenAI's pricing, that's $600-1,000 per month just for LLM processing.

Here's the optimization stack I've used to bring that down:

1. Privocio Agent mode for transcript cleaning — 35-45% token reduction vs raw transcripts 2. Conversation windowing — keep only the last N turns, summarize older context 3. Semantic compression — replace repeated phrases with reference tokens 4. Function calling optimization — pre-format tool outputs to avoid full JSON in context

The first two alone typically reduce token usage by 50-65%. For a 400-hour/month deployment, that drops LLM costs from $800/month to $280/month.

Implementation Stack: Self-Hosted vs Cloud

For teams building voice pipelines today, here's the stack I'd recommend:

Component	Self-Hosted	Cloud
STT	faster-whisper on GPU instance (A10G/A100)	Privocio (fixed pricing, privacy)
LLM	llama.cpp, vLLM, or cloud (OpenAI/Anthropic)	OpenAI GPT-4o / Anthropic Claude
TTS	Coqui, Piper, or ElevenLabs	ElevenLabs, AWS Polly
Pipeline orchestration	Custom async, or Pipecat framework	Pipecat, LiveKit, or custom
Infrastructure	Kubernetes on Hetzner/GCP, or self-hosted	Vercel, Railway, or managed K8s

For most teams, I recommend Privocio for STT + cloud LLM + cloud TTS as the default. You get privacy on the sensitive audio processing stage, predictable costs, and you can swap LLM/TTS providers based on price and quality.

For teams with strict compliance requirements (healthcare, legal, financial), go fully self-hosted: faster-whisper + self-hosted LLM + self-hosted TTS. The latency will be higher, but you control the entire chain.

The FFmpeg pipeline for audio preprocessing is the same in both cases — resample to 16kHz mono, normalize volume, strip silence. This step matters more than most teams realize; Whisper's accuracy improves significantly with clean audio input.

Frequently Asked Questions

How does a voice pipeline handle multiple speakers?

Speaker diarization is the process of distinguishing between speakers in a conversation. For two-party calls, a simple voice activity detection (VAD) approach works: when audio energy crosses a threshold, assign a new speaker ID. For multi-party calls (conference calls, meetings), you need a dedicated diarization model — pyannote-audio is the open-source standard. Most commercial STT providers (Deepgram, AssemblyAI) include diarization in their API.

What's the minimum hardware for self-hosted Whisper?

For real-time transcription (streaming, not batch), you need a GPU. The minimum I'd recommend: an NVIDIA A10G or A100 with 24GB VRAM. This handles the large-v3 Whisper model at ~0.5x realtime — fast enough for most voice agent applications. Without a GPU, you're looking at 10-20x realtime, which only works for batch processing.

How do I handle audio dropouts or background noise?

Audio preprocessing is critical. I use FFmpeg for a pipeline that: resamples to 16kHz mono, applies volume normalization, removes silence segments, and optionally applies a noise reduction filter. Clean audio input can improve Whisper's accuracy by 15-30% in challenging environments.

Can I use a local LLM instead of OpenAI or Anthropic?

Yes — self-hosted LLMs via llama.cpp or vLLM work for voice agents, but the latency and quality tradeoffs are significant. For production voice agents where response quality matters, cloud LLMs still outperform self-hosted at the same cost point. The exception: if you have extremely high volume (millions of voice interactions per month), self-hosted LLMs become cost-competitive.

How do I debug voice pipeline issues?

Logging and tracing are essential. I log: the raw audio (encrypted), the transcript, the LLM input, the LLM output, the TTS input, and the final audio — all with a correlation ID linking them together. For production deployments, I use LangSmith or a custom OpenTelemetry stack to trace the full pipeline latency at each stage.

Conclusion: Voice-First AI

I've watched teams struggle with voice pipelines for two years now. The pattern is always the same: they build the STT stage, connect it to an LLM, and expect it to work. It doesn't — not reliably, not at scale, not with privacy preserved.

The teams that get it right treat the pipeline as a first-class architectural concern. They pick components for their interoperability, not just their individual accuracy. They optimize for token cost from day one. And they build privacy in at every stage, not as an afterthought.

If you're building a voice-enabled AI agent, here's my recommendation: start with Privocio's Agent output mode for STT, use OpenAI's GPT-4o or Anthropic's Claude for the LLM, and build your pipeline orchestration around the assumption that audio will occasionally be sensitive. The stack will serve you from prototype to production.

For a deeper dive into each component — including the specific code for setting up faster-whisper on GPU, the prompt templates for voice agents, and the exact benchmarking methodology I used for the latency numbers in this guide — check out our complete guide to private speech-to-text APIs.

If you're evaluating transcription infrastructure for production, compare Privocio's pricing and features against per-minute alternatives — the math usually comes out in Privocio's favor at anything over 50 hours per month.