Fudan University teachers and students have raised more than 1 billion yuan in Pre-A round financing, aiming to enable AI to understand human emotions and situations.

By | SUN Xiaowen, HAI Ruojing
Edited by | HAI Ruojing

Do human emotions, including joy, anger, sorrow, and fear, matter to AI?

When talking to others, we pay attention to their tone, pauses, and expressions, and think about what they really mean while listening. But when the communication is organized into prompt text, a lot of emotional information will be filtered out.

Mosi Intelligence, a model company founded by Professor QI Xipeng (Distinguished Professor of Fudan University, Vice Dean of Shanghai Institute of Creative Intelligence), has always attached great importance to the emotional perception capability of AI.

In the view of LI Shimin, CEO of Mosi Intelligence: For AI to understand the user's intention, it needs to connect what the user said, what is happening around, and previous interactions. Emotion is also one of the situational signals. For example, if the user suddenly ends the conversation with AI, it may be because the task is completed, or because AI gave a wrong answer and made the user impatient. This kind of feedback is hidden in the voice, tone, and broader context.

The "situational intelligence" they want to achieve is to enable AI to continuously understand the scene: who is present, what happened before, what the current environment is, what people are doing, and what may happen next. However, there are still many unsolved problems to fully realize this intelligent perception and prediction. Therefore, Mosi Intelligence chooses to take the interactive model as the engineering entry point — enabling the model to understand and generate multi-modal content such as voice and video, and support real-time interaction. They named this system MOSS Live.

Exclusive information from *Waves*: Mosi Intelligence has recently completed a Pre-A round of financing of over RMB 1 billion, jointly led by Huakong Fund, the main investors include China Development Bank Science and Technology Innovation, Shanghai Science and Technology Innovation Fund, New Micro Capital, Xuhui Capital, Pursue Light Hard Technology Venture Capital, Lingang Ceyuan Fund, etc. Old shareholders such as Yuanhe Holdings, Shanghai Science and Technology Innovation Group under Shanghai State-owned Investment, and Xinglian Capital continued to invest. The funds from this round will be used for the R&D of full-modal real-time interactive systems and global commercial implementation.

As early as the beginning of 2023, QI Xipeng led the team to launch MOSS, a ChatGPT-like language model, which quickly became a trending topic and received widespread attention. At that time, some investment institutions invited them to start a business, but the team did not leave immediately.

In the same year, the team developed SpeechGPT, an end-to-end large speech model, trying to make the model understand and generate speech, and retain information such as emotion, speech rate and timbre in the sound. In 2024, they developed AnyGPT, exploring to enable the model to process more different types of modal information.

Along this research path, the goal of situational intelligence has gradually become clear. To achieve this goal, the model must enter the real situation, obtain real feedback from customers on site, and feed back to optimize the model. At the end of 2024, the team finally incubated the startup and established Mosi Intelligence, with LI Shimin, the core author of SpeechGPT and a young Fudan PhD, serving as CEO.

LI Shimin told us that in the past six months, the Mosi Intelligence team has released and iterated more than 20 model versions, including speech generation, audio understanding, and visual understanding. He talked about his experience of polishing interactive models with B-end customers, the demands in real situations are different from academic indicators, customers require accuracy, speed, and low cost. These constraints are also changing the model architecture and reasoning framework.

As for the currently popular AI for AI, he believes that it is not a new thing nor a moat, many teams have used AI to accelerate model R&D in the past. Nowadays, the leading edge of a single-point model may be caught up quickly, but if the model can be quickly put into products, and real-scene feedback can be turned into the next round of training materials, the system iteration speed may widen the gap.

What is the difference between the real-time interactive model they are developing and the voice assistants we use in daily life? What capabilities are still missing from situational intelligence? What kind of AI models do end customers such as smart cockpit providers really want? Why can this two-year-old company raise RMB 1 billion in financing?

The following is the conversation between *Waves* and LI Shimin, CEO of Mosi Intelligence, edited and organized:

Let AI understand the "scene"

*Waves*: Mosi has always emphasized "situational intelligence", how to understand the "situation" here exactly?

LI Shimin: The situation we understand, the corresponding English is also context, but it is more generalized than the "context" often mentioned in large models.

Information in the real physical environment actually flows in continuously. Who is here, what things are here, what just happened, what is the spatio-temporal relationship between people and objects, all these we consider as part of the situation.

There is a very simple example, everyone has had the experience of looking for keys. If you suddenly ask AI: "Where did I put my keys?" If AI can only see the current frame of you, it naturally does not know the answer; but if it has been continuously receiving the video stream of the environment, it may see where you put the keys after entering the door, whether people left later, and whether the things moved.

AI with situational intelligence answers questions relying on the situational perception capabilities accumulated continuously before, rather than the information obtained at the moment of asking the question.

*Waves*: It sounds like a accompanying AI assistant. Now people expect more than anthropomorphic conversations, they also hope that AI, like humans, can be more autonomous in the physical world.

LI Shimin: The underlying logic for us to choose to provide interactive capabilities for smart terminals is to enable AI to interact with the physical world like humans. In the real environment, people will pause and correct their words; the surrounding sounds, pictures and environmental states will also keep changing.

Eventually AI will definitely enter terminals such as glasses, cars, toys, and robots. In these scenarios, people will not sort out a Prompt every time to fully tell AI what they want.

Real situational intelligence requires duplex interaction, where input and output run in parallel, the model can watch and listen at the same time, and give feedback while receiving information. Therefore, the next generation of AI will move from "answering questions" to "staying present continuously", being able to continuously understand people and the environment, and respond and act at the right time. What we are doing now, full-modal real-time interaction, is to solve this problem.

*Waves*: Can you give examples to explain the differences between full-modal real-time interaction and the familiar AI voice assistants?

LI Shimin: A real interactive system needs to solve the problems of how information comes in at the same time, how they influence each other, and how to update continuously when the scene changes.

For example, if a person in the car says "I feel a little cold", if you only do speech recognition, you finish the work after recognizing these three words. But real interactive intelligence needs to know the current temperature in the car, who is speaking, what the state of the vehicle is, what the speaker's past preferences are, and finally judge whether this sentence is just describing a feeling, or hoping AI to adjust the air conditioner.

What we really care about is whether AI can understand the scene.

*Waves*: Professor QI Xipeng's team first became popular on the Internet with the language model MOSS in 2023, then turned to "speech understanding and generation models", and later added video. Are all these steps to approach the goal of "situational intelligence"?

LI Shimin: At the beginning of founding Mosi, we described situational intelligence as: AI can fully perceive and understand complex physical situations, and continuously improve model capabilities during interactions.

At first we emphasized the first half of the sentence more. The most natural way for AI to interact with the physical world is voice, so we started with voice first, to make the model understand the user's tone and emotion. If users type and express themselves in text, information such as tone and emotion will be lost.

In 2025, our speech model capabilities have reached a relatively leading level, so we began to integrate "visual understanding", enabling the model to both understand what is being said and see the physical world clearly.

Now, we will emphasize the second half of the sentence more, "continuously improve model capabilities during interactions". As the basic large language models and multi-modal models gradually mature, we are also moving towards more comprehensive multi-modal interaction.

Interactive intelligence is an important engineering entry point. If AI cannot understand the situation, it will expose errors during interaction. Building interactive models is to integrate situational intelligence into the system.

*Waves*: So what progress have the current multi-modal interactive models made? You recently proposed MOSS Live, how to understand it specifically?

LI Shimin: In the past six months or so, we have successively released and iterated more than 20 model versions, covering speech generation, audio understanding, speech recognition, visual understanding and content generation, and we are also continuously open-sourcing them.

Some models have achieved relatively good results on specific tasks. For example, MOSS-Audio can not only understand what the user said, but also analyze the emotion, speech rate and state when the user speaks, even the tiny tremor in the voice;

MOSS-Transcribe-Diarize-0.9B, a multi-speaker speech recognition model, can outperform models with larger parameters on some specific tasks.

We are currently in the stage of connecting many previous single-point capabilities into a system. We prefer to regard MOSS Live as a set of real-time interactive systems. This system includes interactive models and agentic models. The interactive model continuously watches, listens, and speaks to understand the current people and environment; the agentic model is responsible for more complex reasoning, tool invocation and task execution.

*Waves*: Why does Mosi still develop video generation models? This track is also very crowded.

LI Shimin: It should be viewed together with video understanding. In our technical system, understanding and generation promote each other.

For example, training a video generation model requires accurately describing and labeling the content in the video first, which relies on the capabilities of the understanding model. The understanding model can also help evaluate the quality of the generated content; in turn, the effect of the generation model can help us verify the capabilities of the understanding model. These two things have a lot of common accumulation at the underlying level, including audio and video data, data processing pipelines, as well as training and reasoning infrastructure. When we develop video generation, we also reuse the capabilities accumulated in speech and audio-video understanding before.

Therefore, we will iterate understanding and generation together, and ultimately serve the goal of full-modal interaction.

*Waves*: What are the main difficulties in realizing the ideal AI situational intelligence?

LI Shimin: There are several core points. First, the continuous understanding capability for audio and video streams and duplex interaction. Enable the model to run in parallel on the information input side and output side, being able to watch, listen and speak at the same time.

Second, memory capability, which not only includes historical pictures, but also can encode long-term information during interaction and update it in the interaction process.

After obtaining this information, it is also necessary to understand the user's intention, and do planning and goal decomposition before outputting actions. Current language models and agents have done a good job in this part.

Finally, we also need to consider how to improve from the feedback of real interactions. This is different from coding tasks. Coding tasks can directly judge the result by whether the code runs correctly, but in real life, the feedback is not so clear. Users may not directly say "you are wrong", but get angry, or even fall silent directly. At this time, the model needs to combine more situational information to distinguish.

These problems have not been well solved in the current interactive models.

User demands are reshaping AI models

*Waves*: As early as 2023, you developed the end-to-end speech model SpeechGPT, what was the external feedback like at that time?

LI Shimin: People thought the paradigm was very new, but there were also doubts.

At that time, the most obvious advantage of end-to-end speech was to reduce latency and retain more non-text information in the speech. But when it came to real user scenarios, its disadvantages were also obvious: the model had a large number of parameters, and the deployment cost was high; in order to reduce latency, the effect on some specific tasks was not ideal. Therefore, some customers instead hoped that we would make a more intelligent three-stage solution to solve their practical problems first.

It was not until GPT-4o was launched in 2024 that people understood the value of end-to-end speech, which is to make interaction more natural. This incident inspired me a lot. We can rely on technology and cognition to create a new model paradigm, but ultimately it must be combined with real user scenarios to create greater user value. So we decided to start our business at the end of 2024.

In 2025, we developed many single-point models, including multi-speaker speech recognition, audio generation, and independent audio understanding, and many technical selections will be adjusted according to actual market feedback.

*Waves*: At this stage, you are also doing joint development with some B-end users. What kind of users mainly need full-modal capabilities?

LI Shimin: Our main current contacts are smart terminal customers, including smart cockpits, AI smart hardware, etc. Their usage scenarios are different, but they have some common requirements for models:

Understand the content of the user's speech, and also combine the tone, surrounding environment and previous communication to judge the user's intention; perform better in anthropomorphism and fluency; and have an Always-on experience, so that users can wake up AI and interact with it at any time.

These demands behave differently on different terminals, but the underlying understanding and interaction problems to be solved are interlinked, so model capabilities can be reused in different scenarios.

*Waves*: Most of the current smart cockpits have voice interaction, which can already respond to some demands immediately. But will multi-modal interaction in the cockpit become a rigid demand?

LI Shimin: It depends on what level of AI the customer wants in the cockpit. For clear instructions such as turning on the air conditioner and adjusting the volume, the existing voice interaction has solved many problems. But some customer demands we have seen already involve more continuous and complex communication. For example, when multiple people in the car are talking, the user is talking to other people and does not want to trigger the voice assistant; when interacting with the voice assistant in real time, the user may be thinking halfway through the sentence and does not want to be interrupted by AI; after opening the window, how to avoid mis-triggering by background sounds such as wind noise and car horn sounds. These are the problems we are currently working on that some existing cockpit voice assistants have not solved.

Polishing these specific details, let the model know when to reply to the user and when to be silent, after achieving these, users will feel that AI has suddenly become much smarter.

On this basis, some customers also hope that the model can understand visual and audio signals synchronously. Because relying only on sound, the AI still has limited information about the scene; adding vision can supplement the understanding of the surrounding environment and help the model judge the user's intention. For cockpits that want to provide a continuous interactive experience, multi-modal has specific practical uses.

*Waves*: Can the computing power of terminals support this?

LI Shimin: We will design models of different sizes according to different scenarios.

For example, the end-to-end speech transcription model, if used in embodied robots, small AI hardware, mobile phones and other scenarios with high latency requirements and limited computing power of end-side chips, we can use a 0.9B lightweight model to provide local services.

We also have a flagship version of the model in the same series, which can complete long audio multi-person transcription, speaker separation and timestamp annotation, and is mainly provided through the cloud. This version is more suitable for scenarios that do not have very high latency requirements but hope to obtain stronger model capabilities. For example, content production, online modal interaction, or content and quality analysis for customer service and doctor-patient communication.

*Waves*: There are so many demands from the user side, how do you judge which ones are worth investing in R&D?

LI Shimin: Now to judge whether a technology needs in-depth investment, the first principle is demand-oriented; the second standard is reusability, whether the demand only exists