Real-time voice pipeline: from raw audio to AI

Key points: Building a real-time voice pipeline means chaining Deepgram STT (streaming with VAD), an LLM via OpenRouter (function calling), and OpenAI TTS — all connected by an authenticated JWT WebSocket. This article details each step, latencies, fallbacks, and the bugs that cost me sleepless nights.

The heart of TAMSIV is voice. Not a gadget, not a hidden microphone button. Voice IS the primary interface. You press, you speak, the AI understands and executes. But building a real-time voice pipeline solo means entering a world where every millisecond counts and everything can break at any moment.

After three weeks of intensive development and an unreasonable amount of coffee, I have a pipeline that responds in 1.5 to 3 seconds. Here's how it works, chunk by chunk.

How does the voice pipeline architecture work?

The complete pipeline in one line:

16kHz mono PCM Audio → WebSocket (JWT) → Deepgram Live STT (VAD) → OpenRouter LLM → Function calling → OpenAI TTS → Voice response

Six steps, six potential points of failure. Each has its constraints, latencies, and pitfalls. Everything must chain together in less than 3 seconds for the experience to be fluid. Beyond that, the user thinks the app has crashed.

Let's break down each step.

Why is WebSocket essential for real-time audio?

Classic HTTP doesn't work for audio streaming. You would need to send the entire recording in one block, wait for processing, then receive the response. Latency would be unacceptable.

The WebSocket allows a continuous bidirectional flow: the phone sends audio chunks while the user speaks, and the backend can start processing even before the user has finished.

JWT Authentication

Each WebSocket connection is authenticated via a Supabase JWT token:

ws://backend:3001?token=eyJhbGciOiJIUzI1NiIs...

The token is validated upon connection. If the token expires mid-conversation (Supabase tokens expire after 1 hour), the client detects the disconnection and automatically reconnects with a fresh token. I had to handle this case explicitly — initially, long conversations would mysteriously crash.

WebSocket security is detailed in the article on security audit and rate limiting.

How does Deepgram handle streaming Speech-to-Text?

The audio must be in 16-bit PCM, 16kHz, mono. The phone captures audio in this format and sends raw binary chunks via the WebSocket. No compression, no encoding — raw PCM is the fastest format to process.

Deepgram receives these chunks and transcribes them in streaming. But the real magic is VAD (Voice Activity Detection).

Why VAD changes everything?

Without VAD, you have to implement a client-side silence timeout: if the user doesn't speak for X seconds, it's assumed they've finished. The problem:

• Too short (1s): you cut off the user who is thinking between two sentences.
• Too long (3s): the app lags, the user waits.
• Variable depending on the user: some speak fast, others take their time.

Deepgram's VAD detects when the user has finished speaking with remarkable accuracy. It analyzes the audio signal in real-time and sends a speech_final event when it is certain the user has finished. This takes about 200ms after speech ends.

Intermediate vs. Final Results

Deepgram sends two types of results:

• is_final: false — Intermediate, unstable results. The detected word may change as the context is enriched.
• is_final: true — Confirmed results. The text will no longer change.

The trick: accumulate intermediate results to display a real-time preview (good for UX) while only transmitting final results to the LLM (good for quality). I display intermediate results in gray and final results in white — the user sees their voice transform into text live.

For a comparison between native STT (free, on-device) and Deepgram cloud (more accurate), read my detailed article on native STT vs. Deepgram.

How does the LLM orchestrate actions via function calling?

The complete transcription goes to OpenRouter with function calling. OpenRouter is a router that provides access to 400+ LLM models with automatic fallback — if the primary model is down, a fallback takes over in a few seconds.

The LLM receives the transcription and must understand the user's intent. It has 7 functions:

• create_task — Create a task
• update_task — Modify an existing task
• create_memo — Create a memo
• update_memo — Modify an existing memo
• create_calendar_event — Create a calendar event
• ask_clarification — Ask the user for clarification
• end_conversation — End the conversation

The LLM analyzes the sentence ("remind me to buy bread tomorrow at 10 AM"), identifies the action (create_task), extracts the parameters (title, date, time), and returns a structured function call. The backend executes the action and sends the result to the frontend.

The PendingCreation pattern is crucial here: the backend creates a preview of the item, and the user can validate, edit, or cancel before final saving to the database. Zero unpleasant surprises.

LLM Latency

The LLM accounts for the bulk of the latency: between 800ms and 2 seconds depending on the model and query complexity. This is where the choice of model via OpenRouter makes a difference — a fast but less accurate model vs. a slow but more reliable model. TAMSIV uses a configurable model with automatic fallback if the primary model is too slow or unavailable.

How does OpenAI TTS generate the voice response?

Once the action is executed, the backend generates a text response ("Got it! I've created the task 'Buy bread' for tomorrow at 10 AM"). This text goes to OpenAI TTS with the nova voice.

The audio is streamed back via the same WebSocket. The frontend starts playing as soon as the first audio chunks arrive, without waiting for the complete response. This reduces perceived latency by about 500ms — the user hears the beginning of the response while the end is still being generated.

For voice customization and TTS voice selection, I discuss it in the article on voice customization.

What are TAMSIV's three WebSocket modes?

Over the course of development, three WebSocket modes have emerged:

1. LiveWebSocketServer (default) — Native device STT + Deepgram fallback, LLM orchestration, OpenAI TTS. This is the standard mode, the most economical.
2. RealtimeWebSocketServer — OpenAI Realtime API, bidirectional, low latency. More expensive but smoother for long conversations.
3. WebSocketServer — Batch STT/TTS, legacy mode. Used for cases where streaming is not necessary.

The mode is selectable on the admin side via the app_config table in Supabase. This allows switching between modes without deploying a new version of the app.

How to handle errors in a real-time pipeline?

The golden rule: anything can fail at any time. Deepgram can be down. OpenRouter can timeout. OpenAI TTS can return a 429 error. The WebSocket connection can drop mid-conversation.

Here are the resilience mechanisms:

• Smart retries: Each step has a configured number of retries with exponential backoff. No infinite retry loops.
• Circuit breakers: If a service fails too often, we stop calling it for X seconds to avoid overloading an already struggling service.
• Fallbacks at each step: Native STT if Deepgram is down, alternative LLM model via OpenRouter, text response if TTS fails.
• AlertService: Each fallback triggers an email alert (via Resend) + Supabase log. I know in real-time when something degrades.

Each error handling line represents a bug encountered in production. The pipeline is robust today, but it took dozens of debugging sessions to get here.

What is the latency budget for each step?

Breakdown of a complete voice interaction:

• Audio capture + WebSocket send: ~50ms (negligible)
• Deepgram STT + VAD: ~200-400ms after speech ends
• OpenRouter LLM (function calling): ~800ms-2000ms (variable)
• OpenAI TTS (first chunk): ~300-500ms

Total: 1.3 to 3 seconds. The goal is to stay under 2 seconds for 90% of interactions. Beyond that, the experience becomes frustrating.

Streaming TTS is the best lever: the user hears the beginning of the response after ~1.5s on average, even if the complete generation takes 3 seconds. Perceived latency is much lower than actual latency.

What lessons can be learned for building your own voice pipeline?

After three weeks of intensive development, here's what I would advise:

1. Start with WebSocket: It's the backbone. If the WebSocket is solid, the rest falls into place.
2. Use the STT provider's VAD: Don't implement your own silence detection — it's a complexity sinkhole for an inferior result.
3. Stream everything: Streaming STT, streaming TTS. Every millisecond saved improves UX.
4. Plan for fallbacks from day 1: Not in "I'll see later" mode. Each step must have a Plan B.
5. Measure latency in production: Local benchmarks are misleading. Real latency depends on network, server load, and geographical location. The admin dashboard was indispensable for this.

FAQ

Why Deepgram rather than Google Speech-to-Text or AWS Transcribe?

Deepgram offers the best quality/latency ratio for streaming. Google STT is excellent but more expensive and slower in streaming mode. AWS Transcribe is robust but WebSocket integration is more complex. Deepgram also has integrated VAD, which greatly simplifies the code.

Does the pipeline work offline?

Native device STT works offline. But the LLM and TTS require an internet connection. In offline mode, TAMSIV allows classic text input and queues voice requests for when the connection returns.

How much does a complete voice interaction cost?

With native STT (free), an economical LLM via OpenRouter (~$0.001-0.01), and OpenAI TTS (~$0.015/1000 chars), an interaction costs between $0.01 and $0.03. Cost details are in the retrospective article on 650 commits.

Can OpenAI TTS be replaced by an open-source solution?

Technically yes. Projects like Coqui TTS or Bark produce decent results. But the quality of OpenAI's "nova" voice remains superior, and streaming is better supported. For a production project, the additional cost of OpenAI TTS ($15/million characters) is justified by the quality.

How to handle multiple languages in the voice pipeline?

Deepgram supports automatic language detection. The LLM via OpenRouter is naturally multilingual. OpenAI TTS generates audio in the language of the provided text. TAMSIV supports 6 languages (FR, EN, DE, ES, IT, PT) without specific language configuration in the pipeline. Details of internationalization are in the article on i18n in 6 languages.