AI must not only be able to speak, but also refrain from interrupting at will, and Omni AI is revolutionizing real-time interaction.

AI can already listen, see, and respond to queries.

Yet in the face of continuously evolving conversations, how does it maintain consistent understanding and reasoning while generating responses?

When you ask AI to analyze a chart on the screen and suddenly add "Wait a minute, look at the curve on the right", the system needs to stop its ongoing response, update its target, and verify if the background calculations are still valid.

The core difficulty of real-time interaction lies in these exact moments.

The latest review proposed by researchers from institutions including the University of Illinois Urbana-Champaign, Carnegie Mellon University, and Columbia University puts forward a unified conceptual framework that connects omni-modal intelligence and interactive capabilities, systematically sorts out the full technical chain from streaming representation, model architecture, training methods to real-time deployment and evaluation, and further summarizes the key trade-offs between intelligence and interactivity as well as future research directions.

Paper Link: https://doi.org/10.13140/RG.2.2.20245.69602

Project Link: https://github.com/ShenYifanS/Awesome-Interactive-Omni

Figure 1 | The review summarizes 349 related works in total, proposes a unified conceptual framework that takes omni-modal intelligence and interactive capabilities as two interrelated components, linking representation, architecture, training, deployment and evaluation. Source: Figure 1 of the paper.

Real-Time Interactive Omni-Modal Intelligence

Omni AI (Omni-modal intelligence) pursues unified understanding, reasoning and generation across multiple modalities such as text, image, audio and video. However, having multi-modal capabilities does not equate to having natural interactive capabilities.

The review divides real-time interactive omni-modal intelligence into two interrelated components:

Omni Intelligence: Focuses on cross-modal understanding, reasoning and generation.

Omni Interactivity: Focuses on response timing, turn management, interruption handling and dynamic adjustment.

The paper further summarizes three key features: streaming input, continuously evolving internal state, and online response control. This means the model needs to continuously receive information, update judgments, and dynamically decide when to respond, wait or revise outputs.

Figure 2 | Full-duplex interaction allows simultaneous speaking and listening, and requires handling backchannels, speech overlap and interruptions. "Native" further emphasizes that time alignment, incremental processing and interaction control are embedded into the model architecture and training itself. Simply reducing response latency is not enough to meet all these requirements. Source: Figure 2 of the paper.

Two Technical Routes

Under the unified conceptual framework, the review summarizes two main technical paths.

The first path is adaptation based on existing models. It adds real-time interactive capabilities to pre-trained language models or omni-modal models through streaming processing, modality modules, post-training and external orchestration. Its advantage is reusing existing capabilities, while its challenges lie in latency, state synchronization and interruption control.

The second path is building a native interactive model. It integrates continuous multi-modal processing, temporal structure and interaction control into the model architecture and training to achieve tighter synergy between perception and interaction, but puts higher requirements on training data and capability preservation.

When to Speak, and How to Handle Interruptions?

Real-time interaction requires the model to accurately judge when to respond, when to wait, and how to handle mid-process interruptions.

Centering on these issues, the review systematically sorts out response timing control, turn management and interruption recovery mechanisms, and analyzes how existing methods combine speech, vision, semantics and task status to realize dynamic interactive decision-making.

At the training level, the article summarizes how to use interactive signals such as pauses, backchannels, speech overlap, corrections and silences to enable the model to learn not only what to answer, but also when to answer.

Aiming at interaction recovery after interruption, the review further summarizes six key links:

Detect: Identify whether the user has actually initiated an interruption, and distinguish valid instructions from background sounds.

Yield: Stop or reduce the current voice output in a timely manner.

Rollback: Align with the content that the user actually heard.

Cancel: Abort outdated tasks.

Revise: Update intent and tasks based on new instructions.

Resume: Continue the interaction based on the updated state.

Balancing Intelligence and Interactivity

Facing complex reasoning and tool calls, the model responding too early may lead to insufficient information, while waiting for the calculation to complete will damage the interaction experience.

The review generalizes this problem as the Intelligence–Interactivity Trade-off: the training phase focuses on the impact of interaction adaptation on original capabilities, while the operation phase needs to balance response speed and answer quality.

Figure 3 | The trade-off between intelligence and interactivity: the training phase focuses on capability preservation, while the operation phase focuses on response timing and task completion quality. Source: Figure 14 of the paper.

To address this trade-off, the review further discusses Multi-Rate Interactive Intelligence, which balances real-time interaction performance and complex task processing capabilities by coordinating fast responses with slower reasoning, retrieval and tool calls.

One implementation idea is front-back end collaboration: the front end is responsible for continuous perception and fast response, while the back end asynchronously executes complex reasoning and tool tasks, and avoids using outdated calculation results through state synchronization.

Figure 4 | The front end maintains low-latency interaction, while the back end asynchronously executes retrieval, tool calls, reasoning and memory operations. Source: Figure 6 of the paper.

Evaluation and Outlook

Aiming at the problem that existing evaluations cannot measure both answer quality and interactive behavior at the same time, the review systematically sorts out existing evaluation methods from six dimensions: latency, timing quality, content quality, robustness, long conversation quality and recovery capability, emphasizing that it is necessary to measure both intelligence level and interaction performance under comparable interactive scenarios and resource budgets.

The review further summarizes four noteworthy future directions:

Multi-time-scale collaboration: Coordinate perception, reasoning and interaction at different time scales.

Persistent state and long-term memory: Maintain task state and memory in long-duration interactions.

Active and social interaction: Judge the timing of active responses, and understand roles and permissions in multi-person interactions.

Safety and efficiency in open environments: Balance safety, computational cost and system reliability.

Evolving from "being able to listen, see and speak" to continuous understanding, responding at the right time and dynamically adjusting actions is an important research direction for the next stage of real-time interactive omni-modal intelligence.

References:

https://doi.org/10.13140/RG.2.2.20245.69602

This article is from the WeChat official account "AI Era", author: LRST, published by 36Kr with authorization.