36Kr Research Institute | Insights into Natural Language Processing (NLP) Technology in China's Artificial Intelligence in 2024
1. Industry Definition and Development Process
Natural Language Processing (NLP) technology is a branch field of artificial intelligence, focusing on the interaction research between computers and human natural language, aiming to enable computers to have the ability to understand, generate, and process human language (covering text and speech forms). As an interdisciplinary technology that integrates computer science, artificial intelligence, and linguistics, NLP has the characteristics of diversity, interdisciplinarity, complexity, interactivity, and constant change.
The development process of Natural Language Processing (NLP) can be divided into four main stages:
(1) The Germination and Initial Stage (1950s - 1960s). NLP research began with machine translation research. During World War II, computers achieved great success in codebreaking, and people carried out machine translation research based on this. However, due to insufficient understanding of human language, artificial intelligence, and machine learning structures, as well as limited computing power and data volume, the initial systems could only perform word-level translation queries and simple rule processing, such as the early rule-based machine translation systems.
(2) The Rule-Dominated Stage (1970s - 1980s). A series of NLP systems manually constructed based on rules appeared, and their complexity and depth gradually increased, beginning to involve grammar and reference processing. Some systems could be applied to tasks such as database queries. With the development of linguistics and knowledge-based artificial intelligence, the later generation of systems benefited from modern language theories, clearly distinguishing declarative language knowledge and its processing procedures. This stage is characterized by manually constructed complex rule systems, promoting the progress of NLP in the complexity of language understanding.
(3) The Statistical Learning Stage (1990s - 2012). Digital text is increasingly abundant, and algorithm research has become a promising direction. Initially, models were extracted by obtaining a certain amount of online text, but word counting had limited improvement on language understanding. Later, the field shifted to building annotated language resources and using supervised machine learning techniques to construct models, such as building resources to mark the meaning of words, instances of named entities, or grammatical structures. This period repositioned the research direction of NLP, making language processing more dependent on statistical models and algorithms, and accumulating data and algorithm foundations for the subsequent era of deep learning.
(4) The Deep Learning Stage (2013 to Present). The introduction of deep learning methods has completely changed the working mode of NLP. From 2013 to 2018, the models constructed by deep learning could better handle context and similar semantics, such as achieving semantic understanding by representing words and sentences through vector space. Since 2018, NLP has become a successful example of large-scale self-supervised neural network learning. The Transformer model and pre-trained language models (such as BERT and GPT) have further improved the performance of NLP, promoting its wide application in various fields and entering a new stage.
2. Industry Development Driving Factors
National Policy Support and Regulation
NLP is booming under the strong support, active guidance, and strict regulation of national policies. The government has issued a series of policies to support the artificial intelligence industry, providing a solid policy guarantee for the R & D and innovative application of NLP technology. For example, the "Overall Layout Plan for the Construction of Digital China" emphasizes the vigorous promotion of the innovative application of digital technologies, including the deep integration of artificial intelligence-related technologies in various fields, providing a macro strategic guideline for the application of NLP technology in multiple industries. It encourages enterprises and scientific research institutions to actively explore the innovative practice of NLP technology in improving the level of digital service and optimizing business processes. At the same time, in recent years, the management measures issued by the Cyberspace Administration of China for AIGC have detailed regulations on the application of NLP technology in the field of content generation from multiple aspects such as content review, data security, and ethical norms, effectively promoting the industry to achieve large-scale development on a standardized track.
The Increasing Demand for Intelligentization in Traditional Industries
With the acceleration of the digitalization process, traditional industries such as finance, healthcare, and law are facing the dual challenges of massive data processing and business process optimization, and the requirements for the intelligent level of business processing are continuously rising. In the financial field, NLP technology has become an important tool to improve the efficiency of investment research and the level of risk management. When investment researchers are faced with a massive amount of financial information, company financial reports, market dynamics, and other information, natural language processing products with functions such as information classification, sentiment analysis, automatic abstracting, and personalized information recommendation can quickly screen out valuable information, accurately insight into market trends and investment opportunities, and significantly improve decision-making efficiency and accuracy. In the healthcare industry, NLP helps to achieve the automation and structuring of medical record entry, greatly reducing the workload of doctors. In the legal field, NLP is used to achieve functions such as the rapid generation of legal documents, the intelligent review of contract terms, case retrieval and analysis, effectively improving the efficiency and accuracy of legal work, reducing labor costs and the risk of errors. These intelligent needs of traditional industries provide a broad application scenario and market space for NLP technology, becoming a strong driving force for the continuous development of the NLP industry.
3. Industry Development Status
Industrial Chain Structure
The NLP industrial chain is jointly composed of the upstream basic layer, the midstream technology layer, and the downstream application layer.
The upstream basic layer is the foundation of the entire NLP industry, mainly covering hardware equipment, data services, open-source models, and cloud services. In terms of hardware equipment, to meet the needs of large-scale data computing, it is necessary to be equipped with high-performance servers, GPUs, TPUs, and other professional chips. These hardware facilities provide strong computing power support for the training of complex NLP models. In terms of data services, the data collection sources are diverse, such as web crawlers grabbing text from a massive number of web pages, and sensors collecting voice data. At the same time, it also involves rigorous data cleaning work to remove duplicate, incorrect, and irrelevant data to ensure data accuracy, as well as a professional data annotation process. According to the needs of different NLP tasks, the text is annotated for parts of speech, semantics, entities, etc., providing high-quality materials for model training and laying the foundation for model learning and optimization. Open-source models provide a convenient technical starting point for the industry's development. Many open-source NLP models contributed by numerous scientific research institutions and developers, such as BERT, allow enterprises and researchers to conduct secondary development and optimization based on these open-source achievements, accelerating technological innovation and iteration. Cloud services, with their advantages of elastic computing, storage, and network resources, reduce the threshold for the R & D and application of NLP technology.
The midstream of the industrial chain is the R & D and service of NLP technology and products. Here, many advanced natural language processing technologies are gathered, such as neural network models based on deep learning, including Recurrent Neural Network (RNN), Long Short-Term Memory Network (LSTM), Attention Mechanism (Attention), and the popular Transformer architecture in recent years. The main competitors can be divided into Internet enterprises and AI enterprises. Internet enterprises have a relatively complete product ecosystem, rich product experience and data, and a huge customer resource, and can use the advantages of the C-end to promote product innovation and application. AI enterprises have a strong technical accumulation, taking the vertical field and sub-scenario as a breakthrough point to layout multiple industries for customized product development.
The downstream of the industrial chain is the application field of NLP products, which can be divided into two dimensions: application scenarios and application industries. The main application scenarios include intelligent voice, intelligent customer service, intelligent risk control, intelligent supervision, etc.; the main application industries include finance, e-commerce, travel, government affairs, etc. In the intelligent voice scenario, NLP technology realizes the functions of speech recognition, speech synthesis, and speech interaction. For example, an intelligent voice assistant can accurately recognize the user's voice instructions and give a voice response, and is widely used in smart phones, smart home devices, and other equipment. In the intelligent customer service scenario, by understanding the customer's consultation intention, quickly answering questions and handling complaints, it not only improves customer satisfaction but also reduces the labor cost of enterprises, and is widely used in e-commerce, finance, and other industries. In the intelligent risk control scenario, NLP is used to analyze a massive amount of financial data, including news public opinion, company financial reports, social remarks, etc., to early warn financial risks and assist financial institutions in formulating risk control strategies; in the intelligent supervision scenario, NLP is used to analyze and interpret regulatory policy documents, enterprise compliance reports, and other texts to improve the efficiency and accuracy of supervision, playing an important role in financial supervision, market supervision, and other fields.
Market Size
In recent years, with the vigorous development of artificial intelligence technology as a whole and the increasingly urgent demand for digital transformation in various industries, NLP technology, with its unique advantages in text understanding, generation, and interaction, has rapidly penetrated in many fields. From the wide application of intelligent customer service in e-commerce, finance, and other industries, to the intelligent writing assistant helping with content creation in the media, advertising, and other fields, the commercial value of NLP technology is demonstrated. According to CCID Consulting data, the NLP market size reached 30.85 billion yuan in 2024, and it is expected to reach 210.50 billion yuan in 2030, with an average annual compound growth rate of 36.5%.
4. Industry Development Trends
Trend 1: Multi-modal Fusion Leading the Interactive Revolution
With the continuous evolution of technology, NLP will no longer be limited to simple text processing but will be deeply integrated with other modalities such as images and audio. In the field of intelligent devices, the future smart home system can accurately understand the user's scene and needs by combining voice instructions (NLP) with camera image recognition (CV), achieving more intelligent home control. For example, when the user says "Turn off the light in the living room where someone is," the system can quickly locate the person and the corresponding lamp in the living room scene and perform the operation. In the aspect of educational technology, multi-modal NLP can help create an immersive learning environment. The text in the teaching materials is combined with image and audio explanations, and NLP technology interacts and feeds back in various forms such as voice and text according to the student's learning progress and questions, greatly improving the learning effect and experience.
Trend 2: Parallel Development of Model Lightweight and Personalized Customization
On the one hand, to meet the needs of mobile terminals and edge computing devices, NLP models will continue to be lightweight. Through model compression technology, new algorithm architecture optimization, and other means, the requirements of the model for computing resources and storage are reduced, so that the intelligent voice assistant can also run efficiently on resource-constrained terminals such as mobile phones and wearable devices, with faster response speed and lower energy consumption. On the other hand, personalized customization for different industries and different user groups has become a trend. Enterprises can train exclusive NLP models based on their own business data. For example, medical enterprises can build a professional medical term understanding and analysis model for medical record processing and medical research; financial institutions can create a language model that fits their own risk control and investment strategies for market analysis and decision-making, achieving the precision and specialization of NLP services, and deeply enabling the digital transformation and innovative development of various industries.