Doubao In-Vehicle 2.0: Use a unified AI brain to interconnect the entire vehicle

Last year, ByteDance sent shockwaves through the mobile phone industry with its deeply customized "Doubao Phone". Now, the product paradigm of the Doubao mobile assistant has also begun to spread to the automotive sector. 

On the opening day of the Beijing Auto Show, ByteDance's Volcano Engine released a new generation of automotive AI solutions based on the Agentic AI architecture, covering two major offerings: the AI cabin suite solution and the Doubao cabin assistant solution.

The former full-stack AI cabin suite is a full-stack capability solution that automakers can flexibly adopt, while the latter Doubao cabin assistant solution is exactly the killer cabin interaction product that ByteDance is making every effort to build. It delivers a more complete product-level delivery, is interconnected and synchronously evolves in capabilities with the Doubao APP, and will be mass-produced and deployed in vehicles within this year.

In terms of voice interaction experience, the Doubao cabin assistant will free the in-vehicle voice assistant from the old command-based interaction of question-and-answer mode, realize the understanding of vague natural language, break it down into corresponding instructions, and execute vehicle control operations.

According to introductions, the Doubao cabin assistant will also achieve full-duplex dialogue capability, which means the vehicle and users can communicate in real time at the same frequency, speak whenever they want, interrupt at any time, and no wake-up words are required.

That is not all. In terms of functions, the voice assistant will no longer simply perform vehicle control operations such as adjusting temperature and playing music, but cover the entire travel services including driving, route planning, entertainment and interaction. For example, directly say to the Doubao cabin assistant "park in the parking space closest to the entrance", the system will mobilize the assisted driving system to automatically park the car in the corresponding space.

In addition, the Doubao cabin assistant can also act as a "joyride tour guide", automatically identify beautiful scenery along the way, actively recommend sightseeing routes, automatically slow down the car and open the windows. For example, it will prompt during driving "turn right to pass the cherry blossom avenue, detour for 4 more minutes", and after the passenger agrees, it will automatically slow down and open the windows.

A number of industry insiders disclosed to 36Kr that Volcano Engine has promoted the project implementation by forming a cross-departmental special team internally, "it is expected to launch the complete capabilities of the Doubao cabin assistant in the second half of this year".

It needs to be clarified that ByteDance's move is not to engage in vehicle manufacturing, but to co-create deeply with automakers to deliver an intergenerational leap in cabin interaction experience and realize a true full-link end-to-end AI.

The trend of integrating large AI model capabilities into automotive cabins first originated from Tesla. After Elon Musk deployed the Grok large model developed by xAI on vehicles, it first achieved an intergenerational improvement in natural language interaction experience, and its intent understanding of language, personalization and memory capabilities were also significantly enhanced. It can also automatically plan routes and select POIs (points of interest) in combination with natural language instructions.

According to 36Kr, during this year's CES, the chairman of a leading automaker was deeply touched after seeing a video of employees experiencing Tesla's Grok on site, and immediately made a decision to deploy large models on vehicles this year. The internal team was quickly set up, and the first version was completed in only two months.

The "Grok + FSD combination", or the integration of large models into the cabin, is strongly stimulating the nerves of automakers. After years of fierce competition in intelligent driving, the experience has been close to maturity and even homogenization, while cabin interaction has been relatively quiet. The landing of large models in the cabin has reawakened the imagination space connecting interaction, vehicle control and services.

"Almost all automakers are deploying cabin agents." Industry insiders revealed that on the first day of this year's Beijing Auto Show, a large number of automakers including Geely and Li Auto unveiled corresponding products.

The technical acceleration point of automotive cabin agents appeared at the end of 2024. With the launch of S2S (Speech-to-Speech, end-to-end speech) promoted by ChatGPT, the delay of voice interaction has been greatly reduced, which provides a foundation for real-time natural dialogue in the cabin, and also creates favorable conditions for the automotive industry to explore super agents for cabin interaction.

According to exclusive information from 36Kr, Volcano Engine is one of the enterprises that has made large investments, and it has already cooperated deeply with a leading star automaker. According to 36Kr, this automaker has also invested hundreds of millions of yuan in development fees for Volcano Engine, while ByteDance has transferred personnel from Volcano Engine and Doubao respectively to form a project team for this project.

In addition to Doubao, Alibaba's Tongyi Qianwen is also rapidly entering the cabin Agent market. It has in-depth cooperation with Qualcomm 8797 to promote the deployment of end-side models on vehicles. Obviously, in this new battlefield of automotive AI cabins, tech giants and automakers have already been fully prepared.

Volcano Engine sets up a special team to focus on full-link AI assistant

Deploying large models on vehicles is not a new thing, but in just one year, the product form has been completely different.

When DeepSeek became a hit last year, there was a boom of deploying large models on vehicles, but at that time, most automakers accessed large models through cloud engines. For example, most of the deployments of the Doubao large model on vehicles were done through Volcano Engine, which opened APIs for automakers to access and complete adaptation. Many automakers including BYD, Mercedes-Benz and SAIC have adopted this method.

But the effect is not ideal. "We tried it after accessing it, and it can't even pronounce the basic wake-up words well." A R&D staff of an automaker told 36Kr Auto. This also leads to the fact that after the model is accessed, it can only improve the Q&A capability, but cannot realize Agent capability.

The problem does not only lie in technology, but also in the cooperation mechanism. Volcano Engine and automakers have adopted a brand-new cooperation model.

36Kr learned that Volcano Engine and its automaker partner set up a special team to promote the Doubao cabin assistant project, in which the automaker provides the complete vehicle platform and implementation capabilities, invests hundreds of millions of yuan in financial support, and gives Volcano Engine the leading role in the development of new cabin interaction.

In addition, in this project, Volcano Engine has also deployed a model of about 30B scale on the vehicle side to undertake full-domain perception capabilities: continuous input of vision, voice and environmental information to achieve "always online". On the cloud side, 3 to 4 core Agents are running, responsible for tasks such as cabin-driving collaboration, riding experience, comfort control and emotional interaction.

On this basis, the system can complete full-duplex voice dialogue — the communication between users and the system is no longer turn-based, but can be interrupted, inserted and continued at any time, which is close to the natural dialogue between people.

"The advantage of deploying large models on the end side is that they can access all local files and have operation permissions for local Apps." A cabin R&D staff said.

It is worth noting that in the past, the "intelligent computing power" in the automotive industry was mainly concentrated in the field of intelligent driving, and there is almost no precedent for deploying a 30B-level large model on the vehicle side. Even for intelligent driving models, the upper limit known in the industry is currently around 4B.

A source disclosed to 36Kr that in order to run this super-large model on the vehicle side, Volcano Engine approached NVIDIA to customize the Thor z chip, "which is customized and optimized for memory and bandwidth", and deployed it on the vehicle in the form of an AI Box "external" computing power box. At the auto show, Volcano Engine also disclosed this hardware solution.

Public information shows that NVIDIA released the Thor, a central computing chip for integrated cabin and driving, in Q4 2024, including product lines such as Super/X/S/U/Z. Thor Z is the entry version, with a single-chip computing power of 360Tops.

"This is more like running a real-time video stream system on the car, an attempt that ignores cost, but it is difficult to be implemented at the commercial level." An industry insider commented. According to his calculation, if users frequently use voice interaction, Agent scheduling and visual perception capabilities, the monthly model and computing power cost per vehicle may exceed 10 yuan, which is much higher than the existing in-vehicle service cost.

In addition, the two sides also need to overcome more problems at the engineering level. "The two sides are cooperating on a new vehicle platform, and the automaker itself does not have OS capabilities, which is equivalent to redoing a complete set of infotainment platforms. Almost all Apps need to be re-adapted, and even the map needs deep customization. This is a very complex and long-cycle thing." An industry insider said.

Volcano Engine has also made great efforts for this project. "A team of hundreds of people is devoted to it, and at the same time it is cooperating closely with leading Internet application service providers such as Meituan and AutoNavi, and each App takes several months to repackage." A person familiar with the matter said.

One car, one AI brain

The interaction at the whole vehicle level has always been complex, including functional interaction related to high safety, entertainment interaction unrelated to safety, and cross-interaction related to intelligent driving.

The segmentation of such usage scenarios and the resistance across departments usually make it difficult for automakers to break through limitations and explore the form of a full-vehicle AI brain. Volcano Engine believes that in a closed cabin scene, interaction led by a unified AI brain is a more extreme solution.

According to the relevant person in charge, in the new generation of automotive AI solutions, Volcano Engine integrates three engines: the dialogue reasoning engine, the goal-driven engine, and the learning and growth engine into a unified "automotive brain".

Moreover, the architectural paradigm of AI is also changing. Openclaw became popular quickly after the Spring Festival, and automakers generally believe that AI should no longer enter the car in the form of a simple chatbot, but serve as a "comprehensive key" for automakers with a new task paradigm.

Volcano Engine has put forward a more specific solution idea. Different from the "turn-based" interaction mode of Chatbot, under the new AI architecture paradigm, Agentic AI has autonomous driving capability, can perceive the environment in real time, receive feedback, and continue to learn and iterate autonomously. Relying on a powerful large model base, it can link full-domain knowledge and multiple tools to independently promote tasks with clear goals; it can also conduct self-review on execution results to achieve continuous evolution.

To break it down, the "dialogue reasoning engine" can realize natural communication like a real person, bidding farewell to the cold and mechanical "turn-based" Q&A. Through the end-side large model rejection, VLM recognition and ASR capabilities same as those of the Doubao input method, it realizes the capabilities of no wake-up words all the time and multi-person dialogue, and each communication will naturally join the dialogue when it should appear. Based on the industry-leading dual-stream full-duplex capability, it can realize real-time synchronous dialogue between people and the car, allowing users to interrupt and interject at any time, as natural as communicating with real people.

The "goal-driven engine" can independently call full-domain in-vehicle tools based on task goals and environmental feedback, and really do things like a human. It can help you complete complex, multi-step and cross-scenario tasks from beginning to end without your repeated explanations. For example, according to the state of the child in the back row and the memory of the car, it can choose a variety of appropriate ways: singing, playing cartoons, telling stories, playing games, lulling to sleep, etc., to accompany the child throughout the journey.

The "learning and growth engine" can continuously sum up experience in the process and improve itself like a human. It is not limited to basic memory capabilities such as preferences, topics and scenarios, but can also precipitate experience during task execution to form reusable Skills.

Through the deep integration of AI and vehicles, Volcano Engine will work with automakers to create a vivid, smart and universal intelligent cabin user experience, making the car more like a human, with free and emotional communication, high IQ and growing service capabilities, and operations as simple as human instinct.

Of course, facing the engineering complexity of automaker interaction, the Volcano Engine team also has a rational cognition. The relevant person in charge of the team said in an interview that the project priority of the Doubao AI cabin assistant is to do a good job in basic capabilities such as vehicle control.

"In the process of doing vehicle control, we have iterated several versions, and gradually found that we need to access more vehicle control capabilities, from more than 100 at the beginning to hundreds, and then to thousands. We will find how to converge, how to avoid hallucinations, and how to achieve a result that meets user expectations."

This requires strengthening the capabilities of the end side. In the past, the Doubao model was almost all deployed on the cloud, but according to introductions, the Doubao large model has now been deployed on the automotive end side.

In addition to strengthening the end-side capabilities, it also depends on the self-learning and evolution capabilities of the model. According to the business person in charge, a complex scenario may be composed of many tool calls across different fields. "In essence, self-learning means that in the process of user usage, the model itself extracts the KnowHow (knowledge points) for this scenario, and then stores the KnowHow in reverse to guide the process of tool calls, or the sequence and timing changes of tool calls in such scenarios. That is the real self-learning. So I think most of the self-learning claims on the market now are just small iterations in a certain field, not a real complete evolution."

Volcano Engine's years of accumulation in the automotive industry has also helped the team establish a faster understanding of full-vehicle interaction. According to the data disclosed by Volcano Engine, there are currently more than 7 million intelligent vehicles equipped with the Doubao large model, and the installation volume ranks among the top in the industry. During this auto show, a number of heavy new cars equipped with the Doubao large model made their debut, including the Mercedes-Benz all-electric GLC, SAIC Audi E7X, SAIC Volkswagen ID. ERA 9X, Chery Exeed EX7, FAW Hongqi HS6 PHEV, Buick Electra E7, Roewe's new series "Jiayue" and other models.

With the full-link AI capabilities gradually implemented in the cabin, the automotive industry will obviously set off a new upsurge of AI technology.

Automakers are competing for cabin AI, and tech giants are entering the market one after another

The "demonstration project" of Volcano Engine and automakers is still under construction. On the other side, Alibaba's Tongyi Qianwen is also deeply binding the Qualcomm 8797 platform to promote the large-scale deployment of end-side large models in the new generation of cabins. Qualcomm 8797/8397 is the 5th generation cabin-driving integrated automotive-grade chip launched in 2024, which targets NVIDIA's Thor series, with a single-chip maximum computing power of 640TOPS.

36Kr learned that the end-side model size that Qianwen mainly promotes is around 4B, and automakers including BYD, GAC, Li Auto and Xpeng are in contact with it. This means that in the cabin battlefield, Doubao and Qianwen are facing off again. Qianwen is mainly tied to Qualcomm 8797, while Doubao mainly promotes NVIDIA's AI BOX form, through which NVIDIA smoothly cuts into the cabin market from the intelligent driving sector.

The "soul issue" is placed in front of automakers again. If they rely entirely on external models, they may get a better experience in a short time, but in the long run, the entry and data may be taken away. If they choose self-development, they will face another layer of constraints — high investment, uncertain returns, and the capability gap