The first self-developed architecture brain-inspired large model in China, "Luxi Technology", has passed the "Record of Generative Artificial Intelligence Service" | Early-stage Project
Author | Huang Nan
Editor | Yuan Silai
Exploring the origin of large-scale models begins with Google's "Attention Is All You Need" paper in 2017. It creatively proposed the Transformer architecture (hereinafter referred to as: T architecture) that has the ability to handle local and long-range dependencies and is in line with the concurrent processing of large-scale data by the GPU, which has promoted the advent of GPT.
However, under its great reputation, many people have overlooked the two major flaws of the T architecture: First, the computing mechanism overly relies on "matrix multiplication", resulting in excessive computing power consumption; second, a single operation requires the invocation of all computing units, resulting in excessive computing power and storage consumption.
This means that whether it is training or inference, when facing a large amount of data sets and long input sequences, the T architecture needs to repeatedly retrieve the existing information in the large-scale model weights (memory) countless times, and compare the relative distances between the new data and the existing data one by one. This processing mechanism of "reading the book word by word, from the cover to the back cover" not only gives it the ability to "accurately capture long-range dependencies", but also brings huge computing power and storage costs.
In the context of the increasingly aging performance of the T architecture, some AI large-scale model companies have begun to explore the research and development of non-T architectures and achieve breakthroughs in technology, industry, application, and compliance. "Luxi Technology", which Hardcore recently came into contact with, is precisely a large-scale model company that focuses on building a self-developed brain-inspired architecture from scratch.
Zhou Peng, CTO of "Luxi Technology", told Hardcore that the brain-inspired large-scale model adopts the "brain-inspired algorithm" route that mimics the structure and function of the human brain, also known as the "brain-inspired algorithm" route. Compared with the Transformer architecture and the pan-GPT-like models, the computing and storage mechanism of the human brain is significantly more efficient.
During computing, the human brain adopts a "sparse computing" mechanism, only needing to retrieve the neurons directly related to the current calculation, and the remaining idle neurons are in a dormant state. When storing, the human brain does not need to repeatedly "flip through the book". When comparing new and old data, only the memory after high-fidelity compression needs to be invoked, and the information itself is in an "implicit state".
Based on this thinking, "Luxi Technology" has proposed the "NEURARK Brain-inspired Architecture", which replicates the high-fidelity, lightweight, low-power consumption, and zero-delay computing and storage mechanism of the human brain. By simplifying the underlying computing mechanism and using low-energy-consumption operations such as "dot product + addition" to replace the high-energy-consumption budget mode of "matrix multiplication".
At the same time, "Luxi Technology" has also established an "implicit state management" mechanism similar to the human brain storage mode, which can reduce the data storage occupation to an extremely low level by only needing to invoke the memory during the inference process of the large-scale model, without the need to invoke the existing data set.
Currently, the "NEURARK Brain-inspired Architecture" of "Luxi Technology" has reached a cooperation and signing agreement with the leading state-owned enterprises in the information industry, and is carrying out joint research and development and product promotion in the fields of the general medical large-scale model, the diabetes + thyroid and breast disease-specific large-scale model, etc. Among them, the brain-inspired medical large-scale model has been pre-deployed in several top tertiary hospitals, medical alliance network hospitals, and key departments in a central province in China.
In addition, the company has also reached cooperation intentions with government units, state-owned enterprises, and leading manufacturers in infrastructure industries such as finance, education, meteorology, water conservancy, government affairs, and network security. Some have already completed the signing. Many intelligent device mainframe manufacturers and chip manufacturers are actively contacting "Luxi Technology", hoping to apply the brain-inspired architecture and brain-inspired large-scale model in the fields of large-scale model chip adaptation, end-side intelligent operating system research and development, and intelligent device customized design.
Hardcore has learned that the brain-inspired large-scale model independently developed by "Luxi Technology" obtained the "Generative Artificial Intelligence Service Record" from the Cyberspace Administration of China in August this year. This is also the first self-developed architecture large-scale model, non-Transformer architecture large-scale model, and large-scale model using the "brain-inspired algorithm" technical route to obtain this record in China, and its non-T architecture large-scale model is about to be officially launched.