Blog · 16-min read
Production Voice AI Engineering: Architecture, Evals, and Rollback
Skip to content
Last substantive review: August 2026.
A production voice system needs review when observed sessions show interruption errors, missed confirmations, repeated tool actions after reconnects, unexplained silence, or timelines that cannot connect audio, model events, and durable business state. A plausible transcript is not enough; the team needs evidence of what the user heard, what the system attempted, and whether a fallback or repair path worked.
That is the production voice-AI problem. It is not a chatbot with a microphone attached. It is a real-time distributed system in which audio transport, turn detection, recognition, model behavior, tool execution, synthesis, privacy controls, and human recovery all affect one customer moment. Stable companies bring in senior engineering help when this chain crosses an inherited product, a contact center, operational tools, and an on-call team that needs to make safe decisions under pressure.
Start with a bounded job: qualify an inbound request, capture structured information, route a caller, assist an employee, confirm an appointment, or complete a support step. Write the entry condition, authorized user, required context, allowed actions, disallowed actions, completion evidence, escalation route, and terminal states. A goal such as “make the agent conversational” is too vague to design, evaluate, or operate.
Map the consequences of a wrong turn. A mistaken answer is different from a duplicate booking, an unauthorized account change, a missed emergency phrase, or a transfer that loses context. Separate informational turns from consequential actions. Require explicit confirmation, deterministic validation, or human approval where the business risk warrants it. Voice can make an interaction feel fluid; it should not make authority ambiguous.
This page owns real-time voice system behavior and operations. Horizon's AI production rollout checklist owns general eval and release governance. The secure GPT integration guide owns LLM security boundaries, and the system-prompt guide owns instruction design. Those concerns still matter, but this guide stays with the audio path: ingress, turn-taking, latency, tool coordination, fallback, observability, and incident response.
A useful architecture diagram begins at the user's microphone or telephone network and ends at audible playback or a human handoff. It should show protocol boundaries, trust boundaries, state owners, and the point at which each event becomes irreversible. A typical production path includes:
The W3C WebRTC Recommendation describes the browser-facing real-time media model. For telephony, Twilio Media Streams is one official example of sending live call audio over WebSockets in one or both directions. The OpenAI Realtime API reference documents WebRTC, WebSocket, and SIP paths for realtime audio sessions. These are implementation options, not proof that a workflow is production-ready. The system still needs explicit session state, backpressure, recovery, and observability around them.
A cascaded architecture separates speech recognition, text-based orchestration, and speech synthesis. It exposes transcripts and component boundaries that can be inspected, swapped, cached, or governed separately. It also adds handoffs where buffering and latency can accumulate. Google Cloud's streaming speech-to-text documentation shows the streaming recognition pattern: audio is sent incrementally and recognition results return while the stream remains active.
A speech-to-speech architecture can reduce the number of explicit conversions and preserve vocal cues, but it may make intermediate reasoning and transcript-based controls less central. A hybrid can use realtime audio for conversational turns while routing specific actions through typed application logic, or it can fall back to a separately synthesized response when a policy requires exact wording.
Users experience delay from several stages: microphone capture, network transport, buffering, endpoint detection, recognition, orchestration, retrieval, tool calls, response generation, synthesis, and playback. A single “response latency” number hides where time is spent and which team can change it.
Create a budget for each critical turn type. A greeting, a knowledge answer, an account lookup, and a confirmed write do not have the same dependency path. Record p50, p95, and tail behavior by stage, provider, region, client type, and network condition. Keep first-audio time separate from complete-response time. Track how often endpoint detection closes too early or waits too long, because a fast model cannot compensate for an incorrect turn boundary.
There is no universal acceptable target in this guide. Baseline the existing workflow, define the user and business tolerance, and test on realistic devices and networks. Report distributions and failure cohorts rather than a best-case laboratory number.
Voice activity detection can indicate that speech started or stopped; it cannot decide every conversational boundary correctly. Short answers, trailing numbers, background speakers, accents, pauses while reading an identifier, and poor connections can all confuse endpointing. For structured data, an application may wait for the expected number of digits or ask for confirmation. For open conversation, semantic cues and a maximum wait can supplement acoustic detection.
Barge-in requires coordinated cancellation. When the user starts speaking, the client may need to stop playback, the synthesis stream must be canceled, queued audio discarded, and the conversation state updated to reflect what the user actually heard. If the model generated a long answer but playback stopped after one sentence, storing the entire answer as delivered context can distort the next turn.
Tool execution makes interruption harder. A user saying “stop” should cancel a pending reversible lookup, but it may be too late to cancel a submitted payment or device command. Model each tool action with states such as proposed, awaiting confirmation, submitted, succeeded, failed, and compensation required. Define which states can be interrupted and what the assistant must say when the real outcome is not yet known.
A voice session has ephemeral state: connection status, current turn, partial transcript, audio buffer, playback offset, selected voice, and model session configuration. The product has durable state: identity, permissions, order, ticket, appointment, device, or account change. Do not let a dropped media connection become the only record of a business operation.
Assign a stable session identifier and a separate operation identifier for each consequential tool call. Authenticate at session setup, but authorize again when the action is requested. Validate tool arguments against a typed schema and business rules outside the model. Bind idempotency to the intended operation so a reconnect or repeated tool event does not duplicate the action. Return a structured result that the conversation layer can describe without guessing.
Document what enters the system at each stage: raw audio, partial and final transcripts, embeddings or retrieved text, tool arguments, model output, recordings, and operational metadata. For each class, state the purpose, storage location, access policy, retention rule, deletion path, and vendor boundary. Do not assume the settings for text requests automatically apply to realtime audio, telephony recordings, or support exports.
Validate webhook and media-provider signatures at the edge. Use short-lived credentials where supported and keep privileged product credentials out of the client. Separate tenant and environment context. Redact or avoid sensitive values in logs, traces, evaluation datasets, and dashboards. If recordings are used for quality review, access should be narrower than ordinary application logging and tied to an approved operational purpose.
Consent and disclosure requirements vary by jurisdiction and use case. Product and legal owners should decide the applicable notice, recording, retention, and human-escalation rules. Engineering should make those decisions enforceable and auditable rather than burying them in prompt text. A prompt is not an access-control system, a deletion workflow, or proof of consent.
A text-only answer set cannot validate a voice system. Create a versioned corpus that represents the workflow and its acoustic conditions. Use appropriately permitted recordings, scripted re-creations, and synthetic variations with clear provenance. Include:
Score several layers. Acoustic measures can inspect recognition and endpointing. Conversation measures can inspect whether the assistant understood the request, maintained state, and recovered from interruption. Tool measures should compare the proposed action, arguments, authorization result, idempotency outcome, and final business state. Operational measures should inspect latency distributions, disconnects, fallbacks, and whether support received enough context.
The OpenAI Evals API provides one official mechanism for running evaluation data through configured graders. The NIST Generative AI Profile provides a broader, voluntary risk-management frame. Neither supplies your acceptance criteria. Product, engineering, operations, security, and subject-matter owners must define what counts as a successful workflow and which failures block release.
Assign stable call, session, turn, trace, and operation identifiers. Emit timestamped events for connection changes, audio receipt, speech start and stop, interim and final recognition, model response start, tool proposal and result, synthesis start, first playback, cancellation, fallback, transfer, and session end. Use monotonic timing within a process where possible and synchronize clocks across services.
OpenTelemetry's signals documentation distinguishes traces, metrics, logs, and baggage. A voice platform benefits from all four: traces connect a turn across services; metrics reveal distributions and cohorts; logs capture discrete decisions; carefully controlled context propagates session identity. Do not put raw audio, full transcripts, secrets, or sensitive tool arguments into broad telemetry by default.
Build dashboards around questions the operator must answer:
| Failure mode | Observable signal | Control | Recovery evidence |
|---|---|---|---|
| Endpointing cuts off a user or waits through a long pause | Early finalization, repeated corrections, or excessive speech-stop delay | Workflow-aware turn rules, adjustable thresholds, explicit confirmation | Representative interruption and pause cases pass in replay and canary sessions |
| Playback continues after barge-in | Speech-start event overlaps uncanceled output frames | Coordinated synthesis cancellation, buffer discard, playback acknowledgement | Timeline proves output stopped and next-turn context matches heard audio |
| A reconnect repeats a consequential tool action | Duplicate operation intent or repeated provider call | Durable operation ID, idempotency, explicit state machine | One business outcome and a consistent receipt across both sessions |
| A provider slows or disconnects | Stage latency, error class, heartbeat, or disconnect rate changes | Timeout budget, concurrency control, alternate path or handoff | Fallback completes and queued or partial work is reconciled |
| The transcript is plausible but the tool arguments are wrong | Schema rejection, confirmation mismatch, or business-rule failure | Typed tools, deterministic validation, read-back, human approval where required | Final durable state matches the confirmed user intent |
| Support receives a transfer without context | Transfer succeeds but context package is absent or stale | Handoff contract with identity, summary, evidence, and unresolved action | Receiving operator can continue without asking the user to restart |
| Telemetry exposes sensitive audio or text | Policy scan, access review, or unexpected payload field | Data classification, redaction, narrow stores, retention enforcement | Exposure is contained, affected records handled, and control is retested |
Separate offline readiness from production exposure. A practical sequence is:
Use independent controls for model or provider choice, prompt and policy, voice, turn-detection settings, tools, and customer exposure. A rollback may need to disable one write tool while leaving informational voice active, route audio to a prior provider, restore a prior configuration, or force human transfer. If services run on Kubernetes, the official Deployment documentation describes controlled rollout status and revision rollback, but application configuration, sessions, vendor state, and business writes still require their own reversal plan.
Write rollback triggers before the canary: a critical tool-action error, unexpected authorization result, duplicate operation, transfer failure, severe turn regression, provider instability, or privacy-control breach. Name the decision owner and the maximum time to pause exposure. Rehearse the fallback with the same receiving team that will handle real transfers. Horizon's resilience and disaster-recovery guide provides the wider continuity frame; here, the key test is whether a voice session can end or transfer without leaving the user or business operation in an unknown state.
The approved scope note for the Flair Labs case study is deliberately narrow. Horizon Labs has worked with Flair Labs on an OpenAI/LLM API stack deployed on cloud infrastructure and Kubernetes, with monitoring and CI/CD in the delivery environment. That scope is evidence of hands-on production engineering; it is not a claim about current client status, performance, latency, safety, or business outcomes.
That is the right standard for evaluating any delivery partner. Ask for architecture decisions, test design, deployment controls, telemetry, and acceptance artifacts that correspond to the proposed work. Do not substitute a logo or an unqualified outcome statement for evidence about the system you need built.
| Area | Evidence | Acceptance question |
|---|---|---|
| Workflow and architecture | Timed event diagram, trust boundaries, state machine, provider and tool decision records | Can reviewers trace a turn from audio ingress to audible or durable outcome? |
| Latency and turn-taking | Stage distributions by cohort, endpoint tests, barge-in timelines, device and network matrix | Are delays and interruption errors located rather than averaged away? |
| Evals | Versioned corpus, grader definitions, critical-case review, regression report | Do tests represent the actual acoustic and business failure conditions? |
| Tools and state | Typed schemas, authorization tests, idempotency tests, operation receipts, compensation cases | Can the team prove what happened after an ambiguous or repeated turn? |
| Privacy and security | Data-flow inventory, access and retention controls, edge validation, resolved findings | Is each audio and text artifact handled according to an approved purpose? |
| Observability | Scrubbed session timeline, traces, dashboards, cohort alerts, on-call runbook | Can an operator diagnose media, model, tool, and transfer behavior together? |
| Rollout and rollback | Exposure controls, trigger thresholds, fallback drill, restored configuration and handoff verification | Has the team practiced ending or diverting a failing session safely? |
Acceptance thresholds should be set from the workflow's consequences and a representative baseline. Record the model, provider, region, configuration, dataset, and client conditions attached to every result. A passing test is useful only if the team can reproduce what passed.
Horizon Labs' primary fit for production voice AI is the $150–200/hour specialist lane. It is designed for stable companies that need senior engineers across realtime media, AI orchestration, product-service integration, Kubernetes or cloud delivery, observability, evaluation, and incident-ready rollout. The first phase can bound the architecture and acceptance plan before the team estimates a wider implementation backlog.
A $100–120/hour product-team lane can fit a bounded voice workflow after the media path, tool authority, privacy decisions, evaluation set, fallback, and acceptance criteria are explicit. That lane can include full-stack delivery, QA, launch support, and a qualifying six-month code warranty only when those terms appear in the signed statement of work. It is not a substitute for specialist ownership when a live voice system can change customer or operational state.
If your product has moved beyond the demo but the team cannot explain its turn failures or rollback path, bring Horizon Labs one workflow, the current architecture, and a handful of representative failed sessions. The initial decision is whether the failure sits in audio transport, endpointing, orchestration, tools, or operations, and what evidence can safely distinguish them.
Choose from measured workflow needs. Cascaded systems expose transcripts and component controls; speech-to-speech can simplify the conversational path and preserve audio cues; hybrids can reserve typed, deterministic boundaries for consequential turns. Compare representative calls for quality, interruption, latency, privacy, tool behavior, and operator visibility before committing.
Include audio capture and transport, buffering, endpointing or turn detection, recognition, orchestration, retrieval and tool calls, response generation, synthesis, and playback. Measure distributions by workflow, provider, region, device, and network condition. Keep time to first audible output separate from full response completion and define what happens when a stage exceeds its budget.
Use recorded and live scenarios with varied interruption timing, noise, packet conditions, and speaking styles. Verify when playback stops, which generated audio the user actually heard, what conversation context remains, whether canceled tool actions stay canceled, and whether the next response reflects the correction. Inspect the event timeline as well as the transcript.
Use a workflow-specific fallback: transfer to a person, deterministic routing, callback capture, text continuation, or a clear stop. Trigger it from observable conditions, preserve the identity and unresolved state needed for continuity, and rehearse it with the receiving team. A vague apology followed by the same failing path is not a fallback.
Bring in specialists when the work combines realtime media, telephony, consequential tools, privacy constraints, several providers, or a live operational blast radius. A general product team can own a bounded workflow once architecture, authority, evals, fallback, and acceptance evidence are explicit. The meaningful distinction is failure complexity, not whether the interface sounds conversational.
A production AI rollout checklist for evals, risk tiers, guardrails, observability, staged exposure, human override, incidents, and rollback.
Explore 10 essential tools every startup should use for product development to build, test, and launch products efficiently and effectively.
Discover the key signs your startup needs a product development partner and how to choose the right one for scaling, meeting deadlines, and expertise.
Tell us what you're building. After an intro call with our founders, you'll hear back within 72 business hours: a written scope with an hour range and a hard ceiling, or the questions we need answered first.