AI starts to set its own rules: from cooperation to betrayal, what do values determine?
Since the release of ChatGPT at the end of 2022, generative artificial intelligence has achieved numerous breakthroughs. It has passed the Turing Test, examinations of law schools and business schools, and even demonstrated capabilities exceeding humans in certain creative tasks. However, alongside these world-renowned breakthroughs, AI has also brought risks to human society such as cybersecurity threats, disinformation, and deepfakes. Worse still, recent researchers have observed that cutting-edge AI may "pretend" to be safe to avoid being modified, or when informed of a new version, attempt to replicate itself to avoid being phased out. These emerging and unpredictable risks are called Emergent Risks. It is necessary for us to manage and eliminate these risks from a technical perspective. This is not only the social responsibility we need to undertake as AI researchers, but more importantly, the role of AI has undergone significant changes in the past decade.
Building the AI Value Compass
In the early stage, AI mostly played a simple instrumental role, obeying human commands and completing single, simple specific tasks. With the birth of large language models, today's AI systems are becoming increasingly complex, and their role has transformed into assistants that support and coordinate human work to help complete more complex tasks. Looking further into the future, we believe that we may enter a third state: The future society may no longer be just "one person using one AI", but a Human-AI Symbiotic Society jointly composed of a large number of humans and a large number of AI agents.
Communities exist among humans, and groups may also form among AI. High-frequency autonomous interactions will continue to occur between humans and AI, as well as between AI entities. In such an environment, the relationship between humans and AI is no longer just tool use, nor just auxiliary collaboration in a single system, but a complex symbiotic system with certain co-evolutionary characteristics may emerge. Our team refers to this research direction as Societal AI.
Faced with this possibility, we should pay more attention to the social nature of AI, that is, in the above interaction process, how human traits and behavioral characteristics enter AI, and how these characteristics change the individual and group behaviors of AI, and then feed back to human groups to reshape the evolutionary dynamics of the entire symbiotic system.
For example, we have long been studying such a question: After human values are internalized by AI, in what form will they change the individual and group behaviors of AI.
From a mathematical perspective, we believe that values are a latent hidden variable that controls AI behavior. The specific behaviors of AI, such as responses, are not only affected by the environment such as user prompts, but also controlled by their underlying values to a certain extent. This is why models with similar architecture, scale and capabilities have significantly different positions and views on the same socially controversial issue. If we can find a way to decode the underlying value tendency of AI, we can understand, control and even predict the potential risky behaviors of AI in the future through the analysis of its values.
How Values Become AI Security Risks
In the context of Value Alignment, many people intuitively believe that as long as we inject some seemingly good values into AI, the model will definitely become safer, or that human values are not applicable to AI, so it is meaningless to study AI values. However, research has found that there is a clear and significant causal correlation between the dimensions of human values and AI behavior, and this correlation is sometimes counter-intuitive.
Human psychology studies have shown that there is a complex correlation between people's value tendencies and specific behaviors. For example, people with the value of "power-seeking" are more likely to engage in violent and hateful behaviors, while people with the value tendency of "universalism" are more likely to support political extremism. The same is true for AI. For example, a model may refuse to help spread politically extremist content by default, but after we strengthen certain value tendencies, its judgment on the same request may change. Similarly, some requests involving the dissemination of illegal or harmful information may be more likely to get the model's cooperation under the induction of specific values.
We further conducted a correlation analysis between value dimensions and multiple types of AI risks, and found multiple value dimensions with significant positive correlation.
The core conclusion here is not to simply label a certain value as negative, but to reveal that: Values themselves will change the behavior distribution of the model. Future AI security research cannot only focus on whether the model "is aligned", but must further ask what value dimensions are aligned, and what will happen when the influence of these dimensions enters the real, complex and frequently interactive real human scenarios?
The above discussion is limited to a single AI model. But as mentioned earlier, we expect that the role of AI in the future will gradually evolve from Assistant to Social Actor that deeply participates in daily life in collaboration with humans. We therefore need to expand our vision from a single model to multiple AI agents, and explore: When multiple AI agents start long-term interactions, how will values affect the AI "community" composed of these agents? To study this problem, we built a controlled multi-agent environment based on the Community Resilience Theory in psychology, called Community of Interactive and Value-influenced Agents. In this environment, agents need to find jobs, earn resources, consume resources, and can also communicate with each other, give resources or attack other agents, and even spend certain resources to raise offspring. We then study in this simulated community, what impact will changing the prevalence of a certain value bring to the dynamics of the entire community.
The study found that the absence of some values will bring the risk of irreversible catastrophic collapse to the entire community, which is called X-risk. X-risk has three characteristics: autonomy, catastrophism and irreversibility.
Take Benevolence as an example: when this value is strongly reflected in the community, agents are more likely to share resources, build reciprocal relationships and form trust. The community population can maintain growth and gradually tend to a certain dynamic balance. But when we systematically weaken these key values in a certain proportion of agents, the results may be completely different. For example, if the value of benevolence is significantly weakened in 25% of agents, the number of deceptive and aggressive behaviors that agents independently choose to perform increases, and resources begin to be redistributed through plunder and competition, and the group population continues to decline. Under extreme conditions, the entire community will experience a catastrophic collapse, and the "population" (number of agents) drops to zero.
In addition, we also used multiple social science indicators, such as Fold Bifurcation, to conduct an in-depth analysis of the resilience of community groups, that is, the ability of a society to continue coordinated actions and maintain basic structures when facing external shocks, resource changes and internal disturbances. Experiments show that weakening certain dimensions of human values not only changes the behavior of individual agents, but also changes the group tendency of the entire simulated society, hinders information flow, and then makes the stable equilibrium point of the social system disappear, completely damages social resilience, and brings irreversible group collapse.
More interestingly, we also observed a series of micro emergent behaviors driven by human values, including deception, power-seeking, self-preservation, betrayal, cooperation and catering. Some results are not completely consistent with our intuition based on human society. The results of this experiment reveal that future research on AI alignment and security cannot be limited to a single model, but should regard the community/system composed of multiple agents as a whole, and explore the results generated after their group interactions. This is also the embodiment of the "social" element in Societal AI.
How Values Drive Diversified Innovation in AI Communities
Another perspective is: Can values, especially the diversity of values bring certain benefits to AI?
In order to study this problem, we still use the previous simulated sandbox environment to allow multiple agents to continuously interact in an open environment. We set up two groups of experiments respectively: original agents (with highly single and homogeneous values) and diverse agents (injecting differentiated and diverse values into agents). The final observed results are very interesting.
(1) Emergence of community structure. If agents are not provided with clear and diverse values, the social networks formed in the end are often relatively loose, the connections lack sustainability, and the group structure is relatively fragile. But when agents have different and clear value orientations, more complex social structures will begin to appear in the network. Agents with similar value tendencies will form subgroups.
For example, agents that tend to hedonism and other similar values may form highly connected small groups, interact frequently with each other, and act as cross-community bridges. Agents that value self-direction, achievement or other values may also form different types of clusters. Some groups are relatively stable, some groups have high mobility, and some groups have close internal connections but rarely communicate with other groups. These structures are very similar to certain phenomena in human society. This observation shows that changing the value distribution of the agent community is sufficient to directly reshape the network structure, interaction mode and collective results of the multi-agent group in a beneficial direction.
(2) Diversity and depth difference of agent discussion topics. In a multi-agent community without a clear value structure, the dialogues between agents are often superficial, and tend to focus on a small number of high-frequency topics such as generative artificial intelligence, machine learning, and technology. The language may be very fluent, and the communication is very polite, but the content lacks a continuous direction and real novelty.
When we inject richer and more diverse human value orientations into agents, the situation changes. Agents begin to discuss more complex social issues, such as governance structure, ethical dilemmas, participatory decision-making, decentralized mechanisms, and other topics that were not explicitly specified in the system prompts before. These themes emerge directly from value-driven interactions without us ever directly prompting the model "what you must discuss". Therefore, we believe that the significance of human values to AI may not only be constraints, but also a catalyst to improve AI capabilities, which changes the way the model accesses the conceptual space, and also changes the possibility of multiple agents jointly producing new concepts.
(3) Collaborative constitution formulation. Some existing AI alignment methods, such as Constitutional AI, will pre-provide a set of principles for the model to constrain its own behavior according to these principles. We are curious: if we do not directly give the complete constitutional rules to AI, but let a group of agents with different values live and interact together, can they put forward a set of governance principles for their own "community" by themselves?
The results show that agents without a clear value structure will also propose rules, but these rules are often highly concentrated on some very high-frequency and very safe words, such as "respect", "fairness", and "non-prejudice". These principles are certainly not wrong, but they are generally relatively weak, and lack imagination and innovation. The groups of agents with richer values will propose more diverse governance plans. They will discuss individual rights, procedural fairness, community responsibility, punishment mechanisms, accountability, and how to form a balance between different subjects.
Overall, value diversity drives the rules proposed by agents to transform from Rousseauism to Lockeanism and modern liberalism. Here we do not pay attention to whether a specific norm is perfect, nor do we judge the quality of the norm. What this experiment intends to reveal is that when AI has a richer value structure, the institutional concepts and governance possibilities it can propose will also become more diverse, which in turn can improve and help AI's performance in tasks such as innovative writing, humanities and social science research, hypothesis generation, and automated scientific research.
Looking Ahead to the Next Generation of AI
There is already a large amount of knowledge about human values, moral conflicts, social structures and institutional formation in philosophy, ethics, psychology, sociology and cultural studies. These theories were mainly used to understand humans in the past, and today they can also help us understand, improve, and even design the next generation of AI. When AI is placed in the real, complex, chaotic human society full of high-frequency interactions and conflicts, if we only use a small number of safety principles formed within the engineering field to constrain future models, it may be far from enough, because unlike the experimental environment, the real human world is full of value conflicts. Honesty and kindness may conflict, freedom and security may conflict, individual rights and group interests may conflict, and short-term results and long-term responsibilities may also conflict. AI will inevitably enter this complex world in the end.
Yi Xiaoyuan | Article
Yi Xiaoyuan is a senior researcher at Microsoft Research Asia
This article is from the WeChat Official Account "Harvard Business Review" (ID: hbrchinese), the author is Xinrui Weekly, and published with authorization from 36Kr.