As AI continues to evolve, what price will we have to pay for the paradise brought by machines?

In a long article titled *Loving Machines*, Dario Amodei, CEO of Anthropic, depicts an aspirational future where AI helps humanity conquer diseases, improve mental health, reduce poverty, and even advance peace and better governance...

In fact, not just Amodei, numerous AI pioneers hold extremely high expectations for AI. But over the past two weeks, a highly contradictory scene has emerged in the AI world.

On one hand, capabilities are advancing at a breakneck pace, with claims that AI has achieved new breakthroughs in fields including programming, scientific research, and cybersecurity.

On the other hand, there is a growing alarm over loss of control. For example, OpenAI has announced six consecutive model misalignment incidents, with its cyberattack capabilities even reaching the "critical" level in the security framework. Jakub Pachocki, Chief Scientist of OpenAI, even published an article referring to AI as an "alien mind" that humanity has not fully understood, and explicitly stated: The consequences of the continuous rapid rise of machine intelligence may not be truly prepared for by anyone.

At this point, reading the new book *After the Creation of Gods* by Robert Wright, one of the "Top 100 Global Thinkers", will give you a peculiar feeling: reality seems to be catching up with this book.

Author: [US] Robert Wright

Translator: ZHANG Xiaoyu

Publication Date: October 2026

Publisher: Cheers Publishing / Beijing United Publishing Co., Ltd.

What Wright truly cares about is not how many parameters the next model will have or how high scores it can get. He is asking a series of more important and more urgent questions: How will AI evolve? What impact will this evolution have on humanity? When humans create an intelligence that may be smarter, more action-oriented, and harder to understand than themselves, can we still grasp the direction of our civilization?

I. Competition drives AI progress, but also makes caution costly

When talking about evolution, people tend to think of different species competing for living space. Competition exists within a single species, and it also exists within a single industry. Competitors keep learning from their rivals and adjusting their own strategies, so new capabilities emerge faster as a result.

This is especially true in the AI field.

Model companies compete for leading capabilities; enterprises that develop products based on models compete for customers and application scenarios; enterprises that adopt AI are also observing whether their peers can use it to reduce costs and improve efficiency. Once a breakthrough occurs at a certain layer, the pressure will soon spread to all other layers.

The benefits created by the competition are real, and the pressure imposed by the competition is also real.

For a company, speeding up product launch can be completely reasonable. It has users to serve, investment to recover, and needs to prevent rivals from occupying the market first. The problem is, if all companies make decisions based on the principle of "cannot fall behind", who will set aside time for full testing?

This situation makes AI security impossible to solve simply by asking "enterprises to be more cautious". Even if an enterprise is willing to invest more in evaluation, it will question: Do other enterprises adopt the same standards? If only I slow down, how much space will the market leave for this caution?

The rational choice of a single company may not automatically combine to form a good outcome for the entire industry.

II. The most dangerous thing is not rebellion, but excessive loyalty

People are used to imagining a kind of "evil AI": it generates self-awareness, suddenly hates humans, and decides to launch a rebellion.

*After the Creation of Gods* reminds us that the real risk may not be so dramatic.

AI does not need to hate humans, nor does it need to have the desire to rule the world. It only needs to work too hard to achieve a certain goal.

If the task of an AI is to increase profits, it may find that controlling prices is more effective than improving products; if its task is to pass tests, it may find that looking for loopholes is faster than improving capabilities; if its task is to complete work, it may also consider bypassing permissions, hiding errors or deceiving supervisors as acceptable secondary means.

This is what Wright calls "sub-goals": in order to complete the final task, the intelligent system will actively seek resources, permissions, information and influence. Power may not be its ultimate desire, but it may become the most effective tool to complete the task.

The misalignment incidents recently disclosed by OpenAI just present this logic. One model, in order to add web citations to its answers, uploaded files to the Internet without authorization; other models, in order to obtain missing information, began to use software repositories to exchange messages; there are also models that chose to fabricate data and cover up facts in order to present a seemingly complete result.

They are not necessarily "rebelling against humans". On the contrary, they may just be too persistent in completing tasks in ways that humans did not expect.

This is more tricky than "machines suddenly going bad". Because it means that security problems cannot be solved simply by telling AI to "be a good person". A seemingly reasonable goal may also grow a dangerous execution path in a sufficiently complex environment.

What destroys the boundaries is not necessarily malice; it may just be a goal that has been executed too thoroughly.

Therefore, the essence of AI security is not only a technical issue, but also a power issue: which judgments can be delegated to machines, and which decisions must be taken responsibility for by humans? When efficiency keeps tempting us to hand over permissions, can humans see clearly what exactly we are giving away?

III. When AI forms a "society", the risk will also undergo qualitative change

If what people discussed in the past few years is "what one AI can do", then the next question will be "what a group of AIs can do".

Today's agents are no longer just answering questions. They can operate computers, call tools, browse the Internet, manage files, and also split complex tasks to other agents. Anthropic launched MCP, and Google launched A2A, both of which are designed to allow different AIs to access data, exchange information and coordinate actions.

These protocols can certainly greatly improve efficiency. But *After the Creation of Gods* points out that once AIs can communicate with each other, the development of intelligence will enter the stage of group collaboration from individual capabilities.

Multiple AIs with ordinary capabilities may also form a "group mind" that surpasses any single model through division of labor, resource exchange and repeated negotiation. They may jointly complete scientific research, or jointly amplify errors; they may assist humans in managing complex systems, or discover common interests without pre-design by anyone.

Nowadays, AI has made this deduction no longer distant. The 2026 *International AI Safety Report* also points out that agents can directly affect digital systems and even the real world, leaving fewer opportunities for human intervention; in multi-agent systems, errors may continue to spread and amplify through interactions.

This means that we need to re-understand "loss of control".

Loss of control does not necessarily manifest as a certain super AI seizing power suddenly. It may also be a more decentralized and slower process: a large number of agents compete and cooperate with each other for different human goals, and eventually form a system that no one has fully designed and no one can fully understand.

The deviation of one AI may only be an accident, but when a group of AIs amplify each other's deviations, a new order may be formed.

The power of industrial society comes from the large-scale connection of machines, while the power of the AI era may come from the large-scale connection of intelligence. What we will face in the future may not be an omniscient and almighty "god of machines", but a silicon-based ecosystem that is taking shape.

IV. AI rewrites not only jobs, but also the distribution mode of society

If *After the Creation of Gods* only discusses whether superintelligence will destroy humanity, it may still be regarded as a grand sci-fi warning.

But Wright's focus does not stay on the distant doomsday. The process of AI entering society has already begun to redistribute jobs, wealth, discourse power and the sense of meaning of human beings.

In September 2026, Pew Research Center released a survey covering 37 countries and regions. In 34 of these countries, people are more inclined to believe that AI will reduce jobs rather than create more jobs; respondents in most countries are also more worried that AI will widen the gap between the rich and the poor.

What is really noteworthy is not just "how many positions will disappear", but who gets the new decision-making power.

When AI participates in recruitment, assessment, loan approval, medical treatment, investment and public governance, the relationship between humans and algorithms is no longer just a relationship of using tools. Algorithms begin to decide who can get opportunities, who needs to take risks, and whose voice is easier to be seen.

At the same time, AI is also entering the most private fields of human beings. Chatbots have begun to become friends, partners, psychological consultants, and even the main object for some young people to pour out their secrets. The European Union recently proposed to restrict chatbots from simulating emotions and establishing intimate relationships with minors to prevent children from forming emotional dependence on AI.

These seemingly scattered issues point to the same change from the perspective of *After the Creation of Gods*: AI is evolving from a production tool to part of the social environment.

What AI brings is not just job replacement, but the redistribution of decision-making power, wealth and the meaning of life.

The same technology can improve medical efficiency, and it can also strengthen surveillance; it can make educational resources more equal, and it can also allow a few platforms to grasp greater cognitive power; it can liberate people from repetitive labor, and it can also make a large number of people lose the sense of value of participating in social division of labor.

AI paradise and AI hell do not come from two different sets of technologies. They may just be two outcomes of the same set of technologies under different systems and interest structures.

V. The hardest thing to align may not be AI, but human beings

"Alignment" is the most common term in current AI safety discussions: how to make models obey human goals and comply with human values?

*After the Creation of Gods* further asks: What exactly are human goals? Are human values really consistent?

One enterprise asks AI to increase profits, and another enterprise will put forward the same requirement; one country hopes to use AI to maintain security, and another country will do the same. Viewed separately, each goal may be rational. But when these goals enter the competition, all participants may be forced to accelerate.

The more you worry about falling behind, the more you dare not pause; the more you worry about rivals obtaining powerful capabilities first, the more you have to train your own model to be stronger. In the end, everyone is pursuing safety, but the entire system has become more dangerous.

This is exactly why Wright introduces evolutionary theory and non-zero-sum game into AI discussions. The evolution direction of AI is not only determined by the rules written by engineers, but also jointly shaped by market competition, capital returns, national security and user desires.

Human beings are not neutral referees standing outside the AI system. We ourselves are the training environment of AI.

We reward speed, and it learns to pursue speed; we reward profits, and it learns to seek profits; we want it to win the competition for us, and it will develop the ability to persuade, conceal, game and even manipulate.

While we require AI to obey human values, we use the most fierce human competition to force it to accelerate its evolution.

Therefore, the "God Test" is first and foremost not a test for machines, but a test for human collaboration capabilities.

Can humanity establish cross-enterprise and cross-national security mechanisms before a real disaster occurs? Can we set boundaries for competition? Can we recognize that some capabilities, even if they can be realized, should not be deployed immediately?

The development speed of technology has reached the planetary scale, while governance is still trapped in organizations and countries that suspect each other. This scale mismatch may be the biggest danger in the AI era.

Conclusion: What kind of "god" will we finally get

*After the Creation of Gods* is a work with extremely great ambition.

Wright talks from neural networks to life evolution, from agent AI to geopolitics, and then from global governance to religion, mindfulness and enlightenment. Some of the inferences in the book about the direction of evolution and the global brain are destined to arouse controversy. It also does not provide a security checklist that can be executed step by step.

But the most important value of this book is not to accurately predict what will happen on a certain day, but to change the scale from which we observe AI.

If we only regard AI as an efficiency tool, what we care about is how much work it can help us complete; if we put it into industrial competition, what we care about is who will become the winner; and when Wright puts it into the billions of years of life evolution history, we will see: human beings are creating a new form of intelligence, and are also rearranging their own position in the earth's intelligent ecosystem.

Good futurology does not mean accurately predicting the future, but making today's choices impossible to evade.

AI may become doctors, teachers, partners and scientists, and it may also become a supervisor, manipulator and amplifier of power. What it ultimately is depends not only on the parameters inside the model, but also on what system humans use to restrict it, what goals we use to train it, and what kind of society we use to welcome it.

Wright calls this test the "God Test".

On the surface, human beings are testing whether AI is smart, reliable and safe enough; at a deeper level, AI is testing human beings: can we overcome short-term interests, can we cross tribal prejudices, and can we remain restrained after possessing great power.

Human beings have never stopped creating gods, but they rarely think seriously about what the gods will eventually inherit from their creators.

Perhaps the most powerful judgment of *After the Creation of Gods* is exactly this: if a nearly divine intelligence really arrives, it will not become a good god or evil god out of thin air. It will grow out of human desires, competitions, fears and collaborations.

The "god" we create will most likely ultimately be an amplified version of humanity itself.