JD.com and Mira Murati, former CTO of OpenAI, have bet on the same AI track.

Imagine this scenario:

An elderly person living alone slips accidentally in the living room, and the pain leaves them unable to call for help. At this moment, the smart device they are wearing or the camera in their home "notices" the anomaly. Without waiting for any voice command, the AI actively issues an early warning and quickly contacts family members or the emergency center.

Or, you are watching an intense football match. The moment a critical goal is scored, before you have time to rewind or ask questions, your AI glasses automatically provide you with slow-motion analysis and tactical interpretation.

These scenarios are no longer fantasies about the future, but real problems that JoyAI-VL-Interaction, the world's first full-stack open-source vision-language interaction model just open-sourced by JD, is trying to solve.

In the past two years, the capability boundaries of large models have been continuously expanded, but the mainstream interaction method still remains stuck in the "turn-based" logic of "user asks, model answers". It is efficient, but unreasonable in many scenarios. Many important events happen too fast for users to ask questions; many scenarios do not have voice commands at all.

This year, a judgment is becoming a consensus in the industry: AI is evolving from "predicting the next Token" to "predicting the next physical state". This also means that AI must evolve from a passive information processor to an active participant.

Right at this juncture, JD open-sourced JoyAI-VL-Interaction, the world's first full-stack open-source real-time vision-language interaction model, which can autonomously judge when to respond, when to remain silent, and when to assign complex tasks to the background model in continuous video streams.

What JoyAI-VL-Interaction intends to prove is: AI that truly enters the physical world should not keep waiting to be asked. It should learn to perceive, make active judgments, and offer help at the right moment.

This is also a bigger signal released by JD AI: from model capabilities to industrial scenarios, AI competition is shifting from question-and-answer interactions on screens to the real world.

Why Vision-Language Interaction?

In the real physical world, a large amount of key information occurs at moments when users have no time to ask questions. The feeling of "no time to react" is sometimes an experience issue, but more often a capability boundary issue caused by the paradigm.

The industry is not unaware of this limitation.

In the first half of 2026, real-time interaction has become the most popular keyword in multimodal AI. The industry is generally advancing along two paths: one is to make turn-based dialogue faster, and the other is to make voice calls more natural.

The former emphasizes low latency or arbitrary input and output, but its core is still "it only responds when you ask"; the latter allows the model to listen and speak at the same time and be interrupted at any time, making the experience closer to a real human call, but the focus still remains on voice scenarios.

The problem is that a large number of changes in the real world do not turn into a sentence first. Fire accidents, falls, approaching vehicles, screen content changes, and production line anomalies all appear in visuals before they are expressed in language. If AI can only wait for people to speak, it is difficult for it to truly be "present".

Mira Murati's Thinking Machines Lab is the other party that independently reached the exact same judgment as JD almost simultaneously. On May 11, the company proposed the concept of interaction models and released several research preview demos, pointing out that the autonomous response paradigm of interaction models has far greater potential for human-AI collaborative work than the traditional one-question-one-answer paradigm.

The fact that the two teams converged on the same idea at almost the same time is a signal in itself: scaling interactivity as a native capability of the model is an unavoidable direction for the industry in the coming years.

The difference is that JD places vision-language at a more core position, separates voice as a pluggable I/O, and makes vision-language the "first-class driving modality" for the model's autonomous decision-making.

In other words, from the moment the camera is turned on, JoyAI-VL-Interaction will continuously "watch" the visual changes in the physical world, and autonomously judge whether it should speak, what to say, and whether to assign the task to others.

This is where the potential of visual interaction lies: it can be used in scenarios such as elderly and child care, assistance for the visually impaired, AI glasses, event commentary, store inspection, warehousing and logistics, and robot collaboration. Users do not need to organize their problems into a sentence first, and the AI can capture demands from environmental changes.

Therefore, vision is not just another input method, but an irreplaceable perception channel for AI to move towards "predicting the next physical state".

This point is also emphasized in the technical report of JD JoyAI-VL-Interaction. The report shows that in six real streaming scenarios, JoyAI-VL-Interaction achieves a winning rate of 77.6% against leading domestic models and 87.9% against foreign models; in the monitoring and early warning scenario that best tests event capture capabilities, the winning rate reaches 100%. The report argues that the gap is not only in response quality, but in the ability to act at the right moment.

However, achieving active visual interaction is indeed more difficult.

Data acquisition for voice interaction is relatively straightforward, and a large number of voice instruction datasets allow models to learn when humans speak, how to interrupt, and how to respond. The data required for visual interaction is completely different. What the model needs to learn is which signals are worth responding to and which signals should be ignored in continuously changing frames.

The deeper barrier is the scenario definition capability. In scenarios, voice interaction has a natural trigger boundary: the user speaking is the start of the interaction. Visual interaction has no clear start and end points, and the model must judge the boundary by itself in the unbounded information flow.

This is where JD's uniqueness lies: the company does not look for scenarios in abstract laboratories, but operates natively on real business networks covering retail, logistics, healthcare, industry and other fields.

This means that JD AI is not facing a single chat entry, but a huge number of real tasks: how goods circulate, how devices collaborate, how robots cooperate with humans, and how anomalies are detected in advance. The model can learn from real demands and iterate based on real feedback.

Although there are tradeoffs in technical routes, the interaction form of future general AGI will definitely be active intelligence, and agents must have a complete loop of environmental perception, autonomous decision-making and real-time response. Therefore, many companies do not want to develop visual interaction large models, but currently lack the soil for visual interaction to grow. This is also why capital and computing resources first poured into the voice interaction track.

Therefore, JD's choice to start from vision is not just a technical route choice, but a decision determined by its strategic position. Compared with many large model players, JD is closer to the operation sites of the physical world, and has a greater demand for AI that can actively perceive and respond in real time.

To make this day come faster, someone needs to set off earlier.

Lightweight, Open-Source, Deployable

What does the world's first full-stack open-source mean?

Redefining the interaction paradigm sounds grand, but when it comes to real applications, the first threshold is very simple: AI should not always disturb people, nor should it remain silent when it is supposed to give reminders.

People usually expect AI to be as talkative as possible, but in real-time visual interaction scenarios, a model that keeps interrupting is not smart. The truly valuable capability is to actively appear at critical moments and stay quiet at irrelevant moments.

Therefore, JoyAI-VL-Interaction has also trained "remaining silent" as a capability. The model needs to master three layers of judgment: in which scenarios it should actively respond, in which scenarios it should remain silent, and in which scenarios it should distribute tasks to other models.

The value of this set of capabilities is limited if it can only stay in papers. JD's emphasis on "full-stack open-source" this time lies in opening up the model, inference system and application construction path together, so that developers can truly run, modify and use it.

JD chose a more easily spread engineering route: the 8B parameter model can be deployed on a single 3090 graphics card. With this parameter scale, individual developers can run it, consumer-grade hardware can support it, and end-side devices can implement it.

For real-time visual interaction, this lightweight design does not mean reduced capabilities, but a clearer division of labor.

JoyAI-VL-Interaction is more like a front-end interaction layer, responsible for perceiving the environment, judging the timing, and completing short communications. When encountering complex tasks that require deep reasoning, it will automatically distribute them to background agents selected by users such as OpenClaw, Codex, and Claude Code, so the 8B model is more than sufficient.

For example, the model can first say to the user "Let me think about it", then hand over the difficult problem to the background and continue to stay present; after the background returns the result, it synchronizes the answer to the user. During this process, it can also continue to complete other instant interactions for the user.

JD has also made lightweight designs on the underlying system: through video encoding, long-term memory and context compression, the model can continuously watch long video streams at low cost, and control end-to-end latency at the sub-second level. For ordinary readers, the focus is not on these technical terms, but the result: AI can stay in real scenarios for longer periods of time with lower barriers to use.

This cost-effective and deployable choice directly leads to JD's open-source strategy. Only when the model is lightweight enough, the system is complete enough, and the deployment threshold is low enough, can real-time visual interaction evolve from experiments by a small number of teams into an application ecosystem jointly explored by more developers and enterprises.

JD has already open-sourced this inference system, with a clear goal: anyone with a 3090 or higher graphics card and a camera can quickly build their own real-time visual interaction application.

JoyAI-VL-Interaction has obtained day-0 support from vLLM-Omni and has been natively merged into the mainline of vLLM-Omni.

Bring AI Back to the Physical World

The purpose of open sourcing is to hand over the application imagination to a larger market. Because the value of technological breakthroughs must ultimately be tested by the real world.

The first batch of application scenarios for JoyAI-VL-Interaction is already very intuitive: during live sports events, AI can automatically comment at the moment of a critical goal or game-winning shot; when monitoring stock trends, it can continuously observe screen changes and remind users of anomalies; in home care, it can actively give early warnings when the elderly fall or children approach dangerous areas; paired with AI glasses, it can help users identify roads, goods, screens and the surrounding environment; when serving the visually impaired, it can convert visual information into real-time assistance.

For JD, what is more anticipated is that it can be deployed on robots: a model that knows when to speak, when to remain silent, and when to ask for help from background systems can make robots more efficient, and closer to the "measured" intelligent assistant that people expect.

The fundamental reason why JD dares to "stir up" this field at this juncture is that it holds physical world data assets that other large model players do not have.

In the industry context of 2026, the weight of physical world data assets is particularly prominent.

2026 is known in the industry as the "first year of embodied intelligence data". Under this grand background, there is a sharp contradiction: high-quality physical interaction data is extremely scarce, far from meeting the needs of large-scale training, and the bottleneck of algorithm iteration is fully shifting from the model side to the data side.

At this point in time, JD announced that it will accumulate 10 million hours of high-quality real-scenario video data within two years, and mobilize 600,000 people to participate in collection.

JD has more than 3,000 real business scenarios covering retail, logistics, healthcare, industry and other fields. This year, it has also innovated a community grid collection model in Suqian, deploying self-developed JoyEgoCam head-mounted terminals in batches to mobilize surrounding small and medium-sized enterprises and residents to collect data in real operation scenarios.

The deployment speed is very fast. In March, JD announced the completion of the world's first embodied intelligence data collection center in Suqian; in April, it released the industry's first embodied data infrastructure covering the full link of collection, storage, labeling, training, evaluation, simulation and testing; in May, JoyEgoCam achieved mass production to continuously collect first-person perspective data.

These data are the scarcest fuel for training embodied models and visual interaction models. With the addition of embodied data to training, the value of JoyAI-VL-Interaction will further extend from "a model that can actively see" to more specific physical spaces such as robots, autonomous vehicles, warehousing, stores and homes.

Between the model and applications, JoyAI-Echo, which JD open-sourced on June 3, also plays a key role. Echo excels at real-time generation of long videos, while Interaction excels at real-time understanding and interaction. Open-sourcing two models in a row within one month means that JD has connected both the input and output ends of video multimodality, and placed AI's march into the physical world in a more long-term position.

At this year's 618 launch conference, JD stated that it aims to become "the world's largest physical world operation center".

In the era of human-computer interaction, the industry is paying more and more attention to how AI understands the physical world, and JD's problem-solving logic is different from most large model players: the company itself operates natively in the physical world.

Warehousing, delivery, retail, healthcare, and industry are all training grounds and test sites for AI and embodied intelligence. JD Logistics alone plans to invest 3 million robots, 1 million autonomous vehicles, and 100,000 drones within five years, and these hardware will also become places where JoyAI-VL-Interaction can be put into use.

Whether