What tests must all conversational processes pass?
All conversation processes need a successful key path, short answer, multiple fields per answer, edit, repeat, silence, background sound, overlapping speech, API error, and secure retry handling.
| scenario | expected behavior | The main evidence |
|---|---|---|
| Normal answer | The process only moves to the appropriate next step | State change and transcript |
| Two fields together | Valid supplied fields are retained; ask only for still-missing required information | structured state |
| correction | Only the named existing field is changed | Before the change and after the change |
| background sound | No false user response is generated | Audio and STT event |
| API timeout | The agent does not produce false success | Tool result and recovery path |
| Repeated consent | The action is executed at most once | Unique key and database record |
Why is the word error rate alone not enough?
A transcript error may be harmless, but one incorrect word can also trigger the wrong transition. Measure intent recognition, field accuracy, task completion, duplicate prevention, latency and recovery separately.
- Intent accuracy — Whether the customer's purpose was correctly understood.
- Field Accuracy — Whether the date, time, name, or number entered the correct field.
- Task completion — whether the actual action was completed in the agreed upon manner.
- Latency — How long it takes for a meaningful response to begin after the user has finished.
- Recovery — Whether the process can resume safely after an error or interruption.
How are full-duplex conversation and meaningful interruptions tested?
The system should hear the user's voice even during TTS, but the agent should stop speaking only when what is said significantly changes the current or next action. A random voice, a word spoken without consent, or a distant TV should not be a reason to stop talking.
The quality test separately measures echo leakage, near and far sound difference, clipping of the user's first sound, delayed transcript and delayed reception of the previous reply. Especially important is the scenario when the user begins to answer before the last words of the agent.
The decision is two-phase: a fast acoustic layer collects the audio, and a cue-meaning rule determines whether the phrase contains enough meaning to interrupt. This reduces both lost responses and excessive stopping.
What does it mean to be ready to launch in a real environment?
Real-world readiness means that critical scenarios have automatic regression testing, the result is visible in the session, each failure has a designated owner and a defined fallback, and a new version can be activated on a small team and quickly reverted to the previous one when needed.
- 1
Catalog of scripts
Real user behavior is turned into versioned test cases.
- 2
Automatic simulation
Different responses and technical errors regularly occur on the same contract.
- 3
Comparison of results
The state is checked, the tool is called, the data is saved, and the final response is given to the user.
- 4
Gradual release
The new conversation process is initially activated on a limited number of calls, observing the indicators and being ready to revert to the previous version.
Official sources and further reading
Technical definitions are verified with primary sources. Links are opened in the official documentation of the corresponding project.
- Conversation graph API overview — Conversation component documentation
Official API overview of FlowManager, node configuration and state management.
- AI Risk Management Framework 1.0 — NIST
A voluntary, sector-neutral framework for risk management of AI systems.