Voice Agent Barge-In: Stop Stale Audio After an Interruption

When a user interrupts a voice agent, stopping the model request is only part of the job. Audio may already be queued, a speech service may still return chunks and the conversation state may contain words the user never heard. Treat barge-in as a coordinated state transition across generation, synthesis, playback and dialogue history.

The browser’s AudioScheduledSourceNode stop method illustrates one local playback control. It does not cancel an upstream model, empty every application queue or decide what belongs in conversation history. Those responsibilities need explicit application logic.

Name the state of each turn

Assign a unique generation identifier to every agent response. Carry it through text generation, speech synthesis, queued chunks and playback callbacks. Keep the active turn identifier in one authoritative state store.

Useful application states include generating, speaking, interrupted, completed and failed. Define the allowed transitions. An interrupted generation must not become active again merely because a delayed audio callback arrives.

Do not use the visible message index as the only identity if messages can be inserted, retried or edited. A stable identifier prevents a late result from attaching to whichever turn happens to occupy that index now.

Separate detection from interruption policy

The system may detect voice activity, receive a push-to-talk action or observe an explicit stop command. Each is evidence of an event, not proof that every sound should cancel the agent.

Background noise, echo and a brief acknowledgment can produce different product expectations. Define whether the agent pauses immediately, waits for a minimum speech signal or uses an explicit control. Test that policy with consented or synthetic audio representing the actual environment.

Do not claim a universal threshold from a few samples. A quiet room and a shared office have different acoustic conditions. Keep accessibility and an obvious manual stop control in the design rather than making interruption depend entirely on one detector.

Cancel every relevant layer

Once the application decides to interrupt, mark the generation inactive before waiting for network cancellation. Then stop local playback using the supported audio mechanism, clear queued chunks for that generation and request cancellation from upstream services where supported.

The order matters because cancellation is not instantaneous. If you wait for the speech service to acknowledge cancellation before marking local state, additional chunks can enter the playback queue during that gap.

Reject late results whose generation identifier is no longer active. Releasing a request object or hiding a user-interface element is not enough if asynchronous callbacks still reference the old response.

Keep heard text distinct from generated text

A model may have generated three paragraphs while the speaker played only the first sentence. If the user interrupts, storing all three paragraphs as spoken dialogue can confuse the next turn.

Track what was generated, synthesized, queued and actually played at a useful granularity. The exact precision depends on your implementation; do not promise word-perfect playback attribution if you only have chunk-level completion events.

When building the next prompt, represent interruption honestly. For example, retain the completed user turn and a clearly marked partial assistant response, rather than pretending the user heard a recommendation that never left the queue.

Test the race conditions deliberately

Create a deterministic local fixture that delays chunks and cancellation acknowledgments. Interrupt before the first audio, during a chunk, between chunks and immediately before normal completion.

For each case, verify that no obsolete chunk plays after a new response begins. Also check that an old completion event cannot mark the new generation complete. These are state-machine assertions, not subjective listening impressions.

Repeat rapid stop-and-restart actions. A single slow manual test can miss bugs that appear only when turn A is interrupted, turn B starts and turn A’s final callback arrives afterward.

Preserve action boundaries during conversation

If the agent can perform external actions, keep those operations separate from speech playback. Interrupting audio does not automatically undo an action already accepted by another service.

The agent should report observed action state accurately after interruption. “I stopped speaking” and “the order was cancelled” are different claims. Use the service’s supported action and readback flow for the latter, with whatever authorization the action requires.

Avoid sending another mutation merely because the user interrupted the explanation. The next user request may be a clarification rather than permission to repeat or reverse the earlier operation.

Measure the user-visible result

Record interruption detection time, local playback stop time, late chunks rejected and whether the next turn remains coherent. Use appropriately protected telemetry and avoid retaining raw voice recordings unnecessarily.

Listen to representative tests as well as checking counters. A queue can be logically empty while a device buffer still produces a noticeable tail. Describe the measured behavior of the actual audio stack rather than promising instant silence from an upstream cancellation event.

Conclusion

Barge-in requires more than an abort request. Invalidate the old generation, stop its playback, reject late audio and keep conversation history aligned with what the user actually heard.