Tang Jie's signed paper reveals the in-house multimodal model, is the next-generation GLM on the horizon?

After the stunning debut of GLM-5.3 and GLM-5.3-Flash (the two models belong to the same series in name but are trained with different base models), Zhipu AI has continuously improved the model's capabilities in coding, long-term agents and other fields. However, how the next-generation GLM will be upgraded remains a key direction that draws widespread public attention.

Apart from stronger reasoning performance and lower cost, another noteworthy direction is: native multimodality.

Recently, researchers from Tsinghua University and Zhipu AI jointly published a paper on arXiv titled 《Can Visual Generation Expand What Unified Multimodal Models Know?》, with the corresponding author being Tang Jie, Chief Scientist of Zhipu AI and Professor of Tsinghua University.

The paper introduces GLM-Omni, an undisclosed internal model of Zhipu AI, and attempts to answer a somewhat counterintuitive question: Can the knowledge the model learns through image generation be transformed into language reasoning capabilities?

In the experiment, GLM-Omni achieved a 72% success rate in correctly drawing the specified jersey number on digital tasks, but if no image was provided and the same number was only asked in text, the accuracy rate was only 2%.

The researchers then adjusted the training method, allowing the model to put image generation, image observation and text answers into the same training sequence. The average text question-answering accuracy of the model increased from 12.57% to 26.86%. On this basis, when the model was allowed to generate an image first and then observe the image to answer the question before responding, the accuracy further increased to 41.14%.

This study is not direct evidence that the next-generation GLM will adopt a certain architecture, but it provides a research and development clue worthy of tracking: After Zhipu AI has realized native multimodal understanding on GLM-5.3-Flash, researchers are still exploring how to establish a closer connection between visual generation and language reasoning.

Zhipu AI is just missing you

To understand the significance of this paper, we must first clarify a concept: Native multimodality does not mean that the model can generate images on its own.

In August this year, Zhipu AI launched GLM-5.3-Flash, calling it the first native multimodal model in the GLM-5 series. It adopts a brand-new pre-trained base, supports multimodal inputs such as images and videos, can understand the interface and observe rendering results in the Coding Agent workflow, and continue to modify the code according to interactive feedback.

However, from the perspective of the public model interface, the main output of GLM-5.3-Flash is still text. Its ability to understand visual information does not mean that it can natively generate images in the same model.

This is also an important technical demarcation point for current multimodal models.

One type of model mainly focuses on the unified understanding of multiple types of information, enabling the model to simultaneously process inputs including text, images, videos and other content;

The other type of model further attempts to unify understanding and generation, so that the same model can not only understand visual content, but also directly generate images, and explore knowledge sharing between the two capabilities.

The problems faced by the latter are not only how to draw images well, but also whether the knowledge the model learns from generation tasks can be called by other tasks.

When Tang Jie previously talked about native multimodality, his focus was not only on adding an additional input or output capability. In his view, the unified perception of information such as vision, sound and even touch is an important foundation for the model to further understand the environment and perform long-term tasks.

For Agents that need to continuously observe the environment and adjust actions based on feedback, multimodality is not just a function expansion, it also determines whether the model can truly be applied to complex real-world tasks.

In June this year, a netizen asked Musk on X: When will Chinese models reach the level of Fable? Musk replied that it might be in the first quarter. Tang Jie commented: It won't take that long.

On the evening of October 21, the official operation team sent a compensation notice to users of the programming package. At the end of the notice, it was also written: "An additional reset card will be given away after the new model is released", which is equivalent to an announcement that the update is coming soon.

The Hidden Capabilities of Multimodal Output

Tang Jie, the corresponding author of the paper, is the Chief Scientist of Zhipu AI. The three co-first authors, Li Wenkai, Hong Wenyi and Yang Zhuoyi, completed this research during their internship at Zhipu AI.

The model GLM-Omni is based on Zhipu AI's open-source GLM-4.1V-Thinking with a parameter size of 9B, which is a relatively small test model. The research team proposed a test method called Knowledge Transfer Probing (KTP), whose core is to distinguish two things:

Whether the model has learned a certain piece of knowledge during image generation;

Whether it can express this knowledge through language.

This is also a potential value worthy of exploration for unified multimodal models. It can not only allow the model to view and draw images at the same time, but also enable knowledge learned from different modalities to complement each other, rather than working independently.

To verify this, the researchers constructed a fictional world called MagicWorld, where entities such as teams, characters and badges do not exist in reality. For example, a fictional team has the jersey number 10. During training, the model was only taught to generate images of jerseys printed with the number 10, and this number was not provided in the text prompts.

The dataset contains 7 types of knowledge including color, quantity, number, shape, spatial location, with a total of 350 entries, which is expanded into 8552 sets of image training samples. This can largely rule out the possibility that the model answers questions relying on existing text memory.

The participants in the test include not only the internal Zhipu AI GLM-Omni, but also BAGEL from ByteDance and SenseNova-U1 from SenseTime. The three models adopt different visual generation architectures. The researchers respectively checked whether they can correctly draw the target attributes and answer questions in text without the assistance of images.

The first round of experiments immediately revealed the obvious phenomenon of "being able to draw but not being able to describe".

GLM-Omni's average image generation accuracy reached 54%, but the accuracy of directly answering questions about the same knowledge in text was only 14.57%. Among them, the image accuracy rate of digital tasks was 72%, while the text question-answering accuracy rate was only 2%. The gap in digital tasks of SenseNova-U1 was even larger, at 88% and 0% respectively.

This means that the model can already reproduce certain knowledge in the process of image generation, but it is difficult to directly call it through language. In other words, putting visual generation and language understanding into the same model does not mean that the knowledge mastered by the two capabilities can automatically communicate with each other.

So, where exactly is the problem? The researchers then tried two different training methods.

The first method is text alignment: add text question-answering pairs for some fictional entities, let the model learn how to answer related questions, and then test whether it can transfer the learned knowledge to entities that have never seen text answers.

But the effect was not stable. GLM-Omni's average text accuracy even dropped from 14.57% to 12.57%. This shows that simply adding text question-answering pairs is not enough to stably convert visual knowledge into language capabilities.

The second method is more interesting, called Interleaved Image–Text Alignment.

The researchers no longer treat image generation and question answering as two independent tasks, but put them into the same training sequence. For example, when the model faces a question about a jersey number, it first generates the related image, then observes the image, and finally gives the text answer.

As a result, in the text test that did not allow temporary drawing, GLM-Omni's average accuracy increased from 12.57% to 26.86%.

To confirm whether the improvement came from image generation, the researchers also conducted a set of controlled experiments: retaining the same training samples and text supervision, only removing the loss signal of the image generation part in the interleaved training. As a result, the accuracy dropped back to 13.71%.

This reveals the first important conclusion of the paper:

1) The knowledge sharing of the unified model not only depends on whether the architecture is unified, but also depends on whether the training signals of different tasks can be effectively connected.

2) Image generation provides more than just an output function, it itself may help the model build knowledge associations that can be utilized by other tasks.

However, even after joint training, a large amount of visual knowledge of the model still cannot be directly expressed in text. The researchers then changed their thinking: since the model can draw the answer, why not let it draw the relevant content first when answering the question?

This is the Visual Chain-of-Thought adopted in the third round of experiments.

Different from the previous improvement of the training method, this time the change is the reasoning process when the model actually answers the question: generate the image first, then observe the image, and finally organize the text answer.

The effect is even more obvious. GLM-Omni's average text question-answering accuracy further increased from 26.86% to 41.14%, an increase of 14.28 percentage points. Among them, the accuracy of digital tasks increased from 0% to 36%.

This also leads to the second important finding of the paper: Image generation can not only help the model learn knowledge during the training phase, but also become an intermediate representation for invoking knowledge during the reasoning phase. For some questions that are difficult to answer directly in language, the model can first present relevant information by generating images, and then extract the answer from them.

Overall, this paper does not prove that the unified multimodal model is necessarily better than the combination of independent models, but it provides a technical value worthy of further verification: Visual generation is not just about drawing images for users, it may also become a new way for models to learn, organize and invoke knowledge.

This may also be an important direction to observe Zhipu AI's training of the next-generation model.

Multimodal Route Selection

Unified multimodality, what choice will the next-generation GLM make? Since letting the model draw images by itself may also improve reasoning ability, why have leading manufacturers not fully adopted this architecture?

In March 2025, OpenAI integrated native image generation into GPT-4o, emphasizing that image generation should become the core capability of large language models. But in subsequent products, OpenAI turned to professional models such as G