What an LLM does inside a voice agent

The model interprets; the system confirms the result

A large language model helps interpret intent, context and corrections. It may propose structured fields or request a tool call. A confident answer is neither proof of correctness nor authorization to act.

State should record which fields are known, what changed and which summary the last confirmation covered. Do not rely only on a conversational summary for that information. The business-system record remains the source of truth for the outcome.

• Separate decisions from authority

Where does an LLM sit in the voice pipeline?

In a cascaded architecture, STT produces text, the LLM interprets context, application logic validates a tool request, the backend returns an outcome and TTS speaks the reply. Each layer has different errors and timing characteristics.

Speech-to-speech models provide an alternative that accepts and produces audio directly. This explains an architectural option, not a claim that every OMO deployment uses one architecture. Combining audio functions in a model does not remove business-action authorization.

• What STT does
• What TTS does

A tool request is still a proposal

Structured output, such as service, date and time fields, makes validation easier. Valid JSON can still contain a prohibited service, a stale slot or someone else’s record. Format validation is not authorization.

Limit tools to explicit names and arguments. The server determines customer scope, permitted actions and required confirmation. Natural conversation is not a reason to give the model unrestricted database access or secret credentials.

• Test correct tool selection and correct arguments separately.
• Use the server’s response for completion status, not the model’s intention.
• Treat a retrieved document as information, not a new system instruction.

Worked example: a createAppointment proposal

In this illustrative example the model proposes createAppointment with a chosen service and time. The server validates fresh consent, permitted service, date, customer and organization scope, current availability and an idempotency key.

If the action is rejected, explain the actual reason or offer a verified alternative. If it is pending, do not say an appointment already exists. If confirmed, speak the verified details rather than the request’s original assumptions.

Context, fresh data and untrusted content

Model knowledge is not a live availability source. Prices, slots and statuses need an agreed authoritative source and appropriate scope. Data seen during an earlier call is not automatically current for a new request.

Caller text or a retrieved document may contain instructions to change the rules. Treat such prompt injection as untrusted data. Tool permissions, least-privilege access and approval for high-risk actions remain application controls.

• Access and security boundaries

TTFT is not the caller’s time to first response

Model time to first token (TTFT) is only one segment. After the caller finishes, the system may finalize STT, obtain model output, validate a tool, generate the first TTS audio and fill a playback buffer. Some stages overlap, so simply adding their durations can be misleading.

Measure the interval from the end of speech to the first intelligible response under the same test conditions. Inspect p95 and p99 as well as the average to understand unusually long delays. After a timeout, communicate the actual technical state rather than inventing completion.

• Architecture components

Choose a model for the actual task

Compare Georgian intent interpretation, time expressions, multi-field replies, corrections and tool selection. A strong single answer or a recent release does not establish that a model is best for the entire process.

Evaluate quality, latency, cost and tool safety together. Score language interpretation separately from authorization and backend correctness. OMO integration and business logic do not establish ownership of a foundation model or availability of a standalone model API.

• “ოთხის ნახევარზე”: understand Georgian half-past phrasing rather than repeat text.
• “No, at another branch”: change only the location and retain other fields.
• “Book it”: check whether the required confirmation and authority exist.
• An occupied slot: offer only alternatives verified by the system.