Production Voice AI Engineering: Architecture, Evals, and Rollback

Last substantive review: August 2026.

A production voice system needs review when observed sessions show interruption errors, missed confirmations, repeated tool actions after reconnects, unexplained silence, or timelines that cannot connect audio, model events, and durable business state. A plausible transcript is not enough; the team needs evidence of what the user heard, what the system attempted, and whether a fallback or repair path worked.

That is the production voice-AI problem. It is not a chatbot with a microphone attached. It is a real-time distributed system in which audio transport, turn detection, recognition, model behavior, tool execution, synthesis, privacy controls, and human recovery all affect one customer moment. Stable companies bring in senior engineering help when this chain crosses an inherited product, a contact center, operational tools, and an on-call team that needs to make safe decisions under pressure.

Define the workflow before choosing the voice stack

Start with a bounded job: qualify an inbound request, capture structured information, route a caller, assist an employee, confirm an appointment, or complete a support step. Write the entry condition, authorized user, required context, allowed actions, disallowed actions, completion evidence, escalation route, and terminal states. A goal such as “make the agent conversational” is too vague to design, evaluate, or operate.

Map the consequences of a wrong turn. A mistaken answer is different from a duplicate booking, an unauthorized account change, a missed emergency phrase, or a transfer that loses context. Separate informational turns from consequential actions. Require explicit confirmation, deterministic validation, or human approval where the business risk warrants it. Voice can make an interaction feel fluid; it should not make authority ambiguous.

This page owns real-time voice system behavior and operations. Horizon's AI production rollout checklist owns general eval and release governance. The secure GPT integration guide owns LLM security boundaries, and the system-prompt guide owns instruction design. Those concerns still matter, but this guide stays with the audio path: ingress, turn-taking, latency, tool coordination, fallback, observability, and incident response.

Draw the architecture as a timed event path

A useful architecture diagram begins at the user's microphone or telephone network and ends at audible playback or a human handoff. It should show protocol boundaries, trust boundaries, state owners, and the point at which each event becomes irreversible. A typical production path includes:

1. Audio ingress: browser, mobile client, device, or telephony provider captures and transports media.
2. Session edge: the system authenticates the session, validates provider signatures, applies routing policy, and assigns stable call and trace identifiers.
3. Media processing: audio is decoded, resampled, buffered, and optionally cleaned or segmented without losing timing information.
4. Turn layer: voice activity detection, endpointing, push-to-talk, or an application rule decides when a user turn begins and ends.
5. Recognition or realtime model: streaming speech recognition produces interim and final text, or a speech-to-speech model consumes audio directly.
6. Orchestration: instructions, conversation state, retrieval, business rules, and tool permissions shape the next action.
7. Tool boundary: typed, validated calls reach product services; consequential operations acquire an idempotency key and an auditable result.
8. Output path: text-to-speech or realtime audio is generated, buffered, and played with cancellation support.
9. Operations path: events, traces, metrics, logs, approved recordings, and escalation context reach support and engineering systems.

The W3C WebRTC Recommendation describes the browser-facing real-time media model. For telephony, Twilio Media Streams is one official example of sending live call audio over WebSockets in one or both directions. The OpenAI Realtime API reference documents WebRTC, WebSocket, and SIP paths for realtime audio sessions. These are implementation options, not proof that a workflow is production-ready. The system still needs explicit session state, backpressure, recovery, and observability around them.

Choose cascaded, speech-to-speech, or hybrid from the evidence

A cascaded architecture separates speech recognition, text-based orchestration, and speech synthesis. It exposes transcripts and component boundaries that can be inspected, swapped, cached, or governed separately. It also adds handoffs where buffering and latency can accumulate. Google Cloud's streaming speech-to-text documentation shows the streaming recognition pattern: audio is sent incrementally and recognition results return while the stream remains active.

A speech-to-speech architecture can reduce the number of explicit conversions and preserve vocal cues, but it may make intermediate reasoning and transcript-based controls less central. A hybrid can use realtime audio for conversational turns while routing specific actions through typed application logic, or it can fall back to a separately synthesized response when a policy requires exact wording.

Budget latency by stage, not as a single promise

Users experience delay from several stages: microphone capture, network transport, buffering, endpoint detection, recognition, orchestration, retrieval, tool calls, response generation, synthesis, and playback. A single “response latency” number hides where time is spent and which team can change it.

Create a budget for each critical turn type. A greeting, a knowledge answer, an account lookup, and a confirmed write do not have the same dependency path. Record p50, p95, and tail behavior by stage, provider, region, client type, and network condition. Keep first-audio time separate from complete-response time. Track how often endpoint detection closes too early or waits too long, because a fast model cannot compensate for an incorrect turn boundary.

There is no universal acceptable target in this guide. Baseline the existing workflow, define the user and business tolerance, and test on realistic devices and networks. Report distributions and failure cohorts rather than a best-case laboratory number.

Turn-taking is an application behavior

Voice activity detection can indicate that speech started or stopped; it cannot decide every conversational boundary correctly. Short answers, trailing numbers, background speakers, accents, pauses while reading an identifier, and poor connections can all confuse endpointing. For structured data, an application may wait for the expected number of digits or ask for confirmation. For open conversation, semantic cues and a maximum wait can supplement acoustic detection.

Barge-in requires coordinated cancellation. When the user starts speaking, the client may need to stop playback, the synthesis stream must be canceled, queued audio discarded, and the conversation state updated to reflect what the user actually heard. If the model generated a long answer but playback stopped after one sentence, storing the entire answer as delivered context can distort the next turn.

Tool execution makes interruption harder. A user saying “stop” should cancel a pending reversible lookup, but it may be too late to cancel a submitted payment or device command. Model each tool action with states such as proposed, awaiting confirmation, submitted, succeeded, failed, and compensation required. Define which states can be interrupted and what the assistant must say when the real outcome is not yet known.

Keep session state and business state separate

A voice session has ephemeral state: connection status, current turn, partial transcript, audio buffer, playback offset, selected voice, and model session configuration. The product has durable state: identity, permissions, order, ticket, appointment, device, or account change. Do not let a dropped media connection become the only record of a business operation.

Assign a stable session identifier and a separate operation identifier for each consequential tool call. Authenticate at session setup, but authorize again when the action is requested. Validate tool arguments against a typed schema and business rules outside the model. Bind idempotency to the intended operation so a reconnect or repeated tool event does not duplicate the action. Return a structured result that the conversation layer can describe without guessing.

Design privacy and security around the audio lifecycle

Document what enters the system at each stage: raw audio, partial and final transcripts, embeddings or retrieved text, tool arguments, model output, recordings, and operational metadata. For each class, state the purpose, storage location, access policy, retention rule, deletion path, and vendor boundary. Do not assume the settings for text requests automatically apply to realtime audio, telephony recordings, or support exports.

Validate webhook and media-provider signatures at the edge. Use short-lived credentials where supported and keep privileged product credentials out of the client. Separate tenant and environment context. Redact or avoid sensitive values in logs, traces, evaluation datasets, and dashboards. If recordings are used for quality review, access should be narrower than ordinary application logging and tied to an approved operational purpose.

Consent and disclosure requirements vary by jurisdiction and use case. Product and legal owners should decide the applicable notice, recording, retention, and human-escalation rules. Engineering should make those decisions enforceable and auditable rather than burying them in prompt text. A prompt is not an access-control system, a deletion workflow, or proof of consent.

Build an evaluation corpus from real failure conditions

A text-only answer set cannot validate a voice system. Create a versioned corpus that represents the workflow and its acoustic conditions. Use appropriately permitted recordings, scripted re-creations, and synthetic variations with clear provenance. Include:

• quiet and noisy environments, speakerphone, headset, mobile, and constrained bandwidth;
• business names, product terms, identifiers, dates, amounts, and ambiguous homophones;
• accents, speaking rates, code-switching, and supported languages relevant to the actual audience;
• silence, hesitation, self-correction, interruption, double talk, and background speech;
• vendor timeout, partial transcript, tool rejection, reconnect, and degraded synthesis;
• requests the system should refuse, clarify, or escalate;
• consequential actions that require confirmation and duplicate-prevention;
• successful handoff with identity, summary, transcript excerpt, and unresolved state.

Score several layers. Acoustic measures can inspect recognition and endpointing. Conversation measures can inspect whether the assistant understood the request, maintained state, and recovered from interruption. Tool measures should compare the proposed action, arguments, authorization result, idempotency outcome, and final business state. Operational measures should inspect latency distributions, disconnects, fallbacks, and whether support received enough context.

The OpenAI Evals API provides one official mechanism for running evaluation data through configured graders. The NIST Generative AI Profile provides a broader, voluntary risk-management frame. Neither supplies your acceptance criteria. Product, engineering, operations, security, and subject-matter owners must define what counts as a successful workflow and which failures block release.

Observability needs one timeline across media, model, and tools

Assign stable call, session, turn, trace, and operation identifiers. Emit timestamped events for connection changes, audio receipt, speech start and stop, interim and final recognition, model response start, tool proposal and result, synthesis start, first playback, cancellation, fallback, transfer, and session end. Use monotonic timing within a process where possible and synchronize clocks across services.

OpenTelemetry's signals documentation distinguishes traces, metrics, logs, and baggage. A voice platform benefits from all four: traces connect a turn across services; metrics reveal distributions and cohorts; logs capture discrete decisions; carefully controlled context propagates session identity. Do not put raw audio, full transcripts, secrets, or sensitive tool arguments into broad telemetry by default.

Build dashboards around questions the operator must answer:

• Which stage is adding delay for this cohort?
• Did the user interrupt, did playback stop, and what context was retained?
• Was a tool proposed, authorized, submitted, duplicated, or rejected?
• Did the session fall back or transfer, and did the destination receive context?
• Is a provider, region, client version, language, or workflow driving the change?

Failure modes and the recovery each one needs

Failure mode	Observable signal	Control	Recovery evidence
Endpointing cuts off a user or waits through a long pause	Early finalization, repeated corrections, or excessive speech-stop delay	Workflow-aware turn rules, adjustable thresholds, explicit confirmation	Representative interruption and pause cases pass in replay and canary sessions
Playback continues after barge-in	Speech-start event overlaps uncanceled output frames	Coordinated synthesis cancellation, buffer discard, playback acknowledgement	Timeline proves output stopped and next-turn context matches heard audio
A reconnect repeats a consequential tool action	Duplicate operation intent or repeated provider call	Durable operation ID, idempotency, explicit state machine	One business outcome and a consistent receipt across both sessions
A provider slows or disconnects	Stage latency, error class, heartbeat, or disconnect rate changes	Timeout budget, concurrency control, alternate path or handoff	Fallback completes and queued or partial work is reconciled
The transcript is plausible but the tool arguments are wrong	Schema rejection, confirmation mismatch, or business-rule failure	Typed tools, deterministic validation, read-back, human approval where required	Final durable state matches the confirmed user intent
Support receives a transfer without context	Transfer succeeds but context package is absent or stale	Handoff contract with identity, summary, evidence, and unresolved action	Receiving operator can continue without asking the user to restart
Telemetry exposes sensitive audio or text	Policy scan, access review, or unexpected payload field	Data classification, redaction, narrow stores, retention enforcement	Exposure is contained, affected records handled, and control is retested

Roll out one workflow and cohort at a time

Separate offline readiness from production exposure. A practical sequence is:

1. Recorded evaluation: run the pinned corpus against the candidate architecture and inspect critical failures.
2. Failure injection: introduce slow tools, dropped connections, partial audio, provider errors, duplicate events, and unavailable transfer destinations.
3. Internal sessions: use trained testers who can annotate timing, confusion, and recovery while operators exercise the session timeline.
4. Shadow or assistive mode: where privacy and architecture permit, generate suggestions without giving the system action authority or direct customer audio output.
5. Named canary: expose one bounded workflow, language, region, or customer cohort with staffed observation and a ready fallback.
6. Controlled expansion: increase exposure only after the hold period covers expected call types and acceptance evidence remains within thresholds.

Use independent controls for model or provider choice, prompt and policy, voice, turn-detection settings, tools, and customer exposure. A rollback may need to disable one write tool while leaving informational voice active, route audio to a prior provider, restore a prior configuration, or force human transfer. If services run on Kubernetes, the official Deployment documentation describes controlled rollout status and revision rollback, but application configuration, sessions, vendor state, and business writes still require their own reversal plan.

Write rollback triggers before the canary: a critical tool-action error, unexpected authorization result, duplicate operation, transfer failure, severe turn regression, provider instability, or privacy-control breach. Name the decision owner and the maximum time to pause exposure. Rehearse the fallback with the same receiving team that will handle real transfers. Horizon's resilience and disaster-recovery guide provides the wider continuity frame; here, the key test is whether a voice session can end or transfer without leaving the user or business operation in an unknown state.

Production proof should stay inside the evidence

The approved scope note for the Flair Labs case study is deliberately narrow. Horizon Labs has worked with Flair Labs on an OpenAI/LLM API stack deployed on cloud infrastructure and Kubernetes, with monitoring and CI/CD in the delivery environment. That scope is evidence of hands-on production engineering; it is not a claim about current client status, performance, latency, safety, or business outcomes.

That is the right standard for evaluating any delivery partner. Ask for architecture decisions, test design, deployment controls, telemetry, and acceptance artifacts that correspond to the proposed work. Do not substitute a logo or an unqualified outcome statement for evidence about the system you need built.

Acceptance evidence for a production voice-AI system

Area	Evidence	Acceptance question
Workflow and architecture	Timed event diagram, trust boundaries, state machine, provider and tool decision records	Can reviewers trace a turn from audio ingress to audible or durable outcome?
Latency and turn-taking	Stage distributions by cohort, endpoint tests, barge-in timelines, device and network matrix	Are delays and interruption errors located rather than averaged away?
Evals	Versioned corpus, grader definitions, critical-case review, regression report	Do tests represent the actual acoustic and business failure conditions?
Tools and state	Typed schemas, authorization tests, idempotency tests, operation receipts, compensation cases	Can the team prove what happened after an ambiguous or repeated turn?
Privacy and security	Data-flow inventory, access and retention controls, edge validation, resolved findings	Is each audio and text artifact handled according to an approved purpose?
Observability	Scrubbed session timeline, traces, dashboards, cohort alerts, on-call runbook	Can an operator diagnose media, model, tool, and transfer behavior together?
Rollout and rollback	Exposure controls, trigger thresholds, fallback drill, restored configuration and handoff verification	Has the team practiced ending or diverting a failing session safely?

Acceptance thresholds should be set from the workflow's consequences and a representative baseline. Record the model, provider, region, configuration, dataset, and client conditions attached to every result. A passing test is useful only if the team can reproduce what passed.

Choose the team by real-time and operational risk

Horizon Labs' primary fit for production voice AI is the $150–200/hour specialist lane. It is designed for stable companies that need senior engineers across realtime media, AI orchestration, product-service integration, Kubernetes or cloud delivery, observability, evaluation, and incident-ready rollout. The first phase can bound the architecture and acceptance plan before the team estimates a wider implementation backlog.

A $100–120/hour product-team lane can fit a bounded voice workflow after the media path, tool authority, privacy decisions, evaluation set, fallback, and acceptance criteria are explicit. That lane can include full-stack delivery, QA, launch support, and a qualifying six-month code warranty only when those terms appear in the signed statement of work. It is not a substitute for specialist ownership when a live voice system can change customer or operational state.

If your product has moved beyond the demo but the team cannot explain its turn failures or rollback path, bring Horizon Labs one workflow, the current architecture, and a handful of representative failed sessions. The initial decision is whether the failure sits in audio transport, endpointing, orchestration, tools, or operations—and what evidence can safely distinguish them.

Frequently asked questions

Should a production voice AI system use a cascaded or speech-to-speech architecture?

Choose from measured workflow needs. Cascaded systems expose transcripts and component controls; speech-to-speech can simplify the conversational path and preserve audio cues; hybrids can reserve typed, deterministic boundaries for consequential turns. Compare representative calls for quality, interruption, latency, privacy, tool behavior, and operator visibility before committing.

What belongs in a voice AI latency budget?

Include audio capture and transport, buffering, endpointing or turn detection, recognition, orchestration, retrieval and tool calls, response generation, synthesis, and playback. Measure distributions by workflow, provider, region, device, and network condition. Keep time to first audible output separate from full response completion and define what happens when a stage exceeds its budget.

How should teams test barge-in and interruption?

Use recorded and live scenarios with varied interruption timing, noise, packet conditions, and speaking styles. Verify when playback stops, which generated audio the user actually heard, what conversation context remains, whether canceled tool actions stay canceled, and whether the next response reflects the correction. Inspect the event timeline as well as the transcript.

What is a safe fallback when the voice AI degrades?

Use a workflow-specific fallback: transfer to a person, deterministic routing, callback capture, text continuation, or a clear stop. Trigger it from observable conditions, preserve the identity and unresolved state needed for continuity, and rehearse it with the receiving team. A vague apology followed by the same failing path is not a fallback.

When should a company hire specialist voice AI engineers?

Bring in specialists when the work combines realtime media, telephony, consequential tools, privacy constraints, several providers, or a live operational blast radius. A general product team can own a bounded workflow once architecture, authority, evals, fallback, and acceptance evidence are explicit. The meaningful distinction is failure complexity, not whether the interface sounds conversational.

Primary technical sources

• OpenAI Realtime API reference
• W3C WebRTC Recommendation
• Twilio Media Streams
• Google Cloud streaming speech-to-text
• OpenTelemetry signals
• Kubernetes Deployments
• NIST AI RMF Generative AI Profile
• OpenAI Evals API