What are the philosophical underpinnings behind AI safety, and how does effective altruism influence Silicon Valley?

In its recently submitted prospectus, Anthropic disclosed nearly 100 pages of risk factors. In addition to common financial and market risks, it also includes statements about the catastrophic, even existential risks to humanity brought by cutting-edge AI. Anthropic stated that it cannot guarantee that its own safety measures will necessarily be effective.

The prospectus also reveals that after the listing, the seven co-founders will hold 50.1% of the voting rights for general voting matters through special shares; at the same time, an institution called the "Long-Term Interest Trust" has the right to elect a majority of the board members. The goal of this arrangement is to allow the company to take humanity's long-term interests into account while pursuing commercial benefits. However, some media have noticed a detail: the founders can designate in advance who will succeed their voting rights in the event that they lose the ability to perform their duties. These candidates require internal approval, but the list is kept confidential. That is to say, the public currently has no way of knowing who may participate in Anthropic's decision-making in the future.

This prospectus has once again demonstrated to the public the importance of AI security for Anthropic as a company. But at the same time, if their warnings come true, as members of the public, shouldn't we have the right to know who is authorized to make decisions on behalf of humanity?

To understand Anthropic's emphasis on security, as well as the trust and power disputes surrounding it, we have to mention a concept that has emerged repeatedly recently, namely Effective Altruism, referred to as EA for short.

Just this month, *The Economist* put EA on its cover, calling it "the most important idea of the century". Some reports point out that around AI security policies, EA supporters have clashed with the tech right mentioned in our podcast, and even the Trump administration has joined the debate — the U.S. Department of Defense directly shouted a slogan on social media: "Americanism, not Effective Altruism". A concept that originally sounded like a philosophical discussion has thus become the object that the U.S. government publicly draws a clear line from.

In this episode of *Silicon Valley 101*, we have invited a guest who has close ties to the EA circle: Vitalik Buterin, founder of Ethereum, and his friend, tech journalist Afra. The following is the first part of the interview, where Afra introduces the four stages of the development of Effective Altruism; the life and deeds of its spiritual leader Will MacAskill; the differences in community temperament and action capacity between EA, rationalism, and e/acc (Effective Accelerationism); and how EA dominated the AI security narrative and gradually recruited the smartest people in Silicon Valley.

The following are selected excerpts from this conversation:

Four Stages of EA: From the Mosquito Net Era to the AI Security Era

Yiwen: Generally speaking, what topics do you think EA pays more attention to?

Afra: Actually, the more than ten years during which EA has influenced the global landscape can be roughly divided into four stages, and the focus of each stage is quite different.

The first stage is the so-called "Mosquito Net Era". In this period, you use very detailed, quantifiable evidence to improve the efficiency of charity, so as to influence, for example, global health indicators. This is a kind of measurable goodness, supported by evidence and data. Because in the early days when EA was just established, many people wanted to do good deeds and donate money through charity, but the efficiency of charity was very low, and the money they donated could not reach the intended recipients.

But the group of EA people wanted to use a quantifiable, scientific way to get the money to the right places and maximize its value. That's where the name of the mosquito net era comes from — the mosquito net refers to the most effective way that "the more money you donate, the more lives you can save".

Then came the next stage of EA, the so-called "Long-Term Planning Era". This long-term planning period took place around 2015, when this group of people began to disproportionately research existential risks and issues related to future generations, and funds and institutions also began to expand. They no longer cared about how many lives a mosquito net could save, but about the quality of life and living standards of generations of human descendants thousands or tens of thousands of years in the future.

So a large amount of capital — including funds from SBF and many other wealthy people — entered the community at this stage. I think SBF is probably the most extreme symbol of this era, because he himself is a person who pays extremely close attention to long-termism issues.

I read a *New Yorker* profile about Will MacAskill, and after reading it, I came to a conclusion: SBF may be one of the decisive factors for the entire EA's shift to long-termism.

Then came the era when the reputation of the entire community collapsed. A huge trust crisis broke out, and everyone began to reflect on what went wrong with the EA group. At the same time, ChatGPT was born, which also brought a turning point for AI — the AI risks and existential issues that the EA and rationalist groups had discussed before suddenly became a concern for more people.

So I think this stage is full of setbacks, but also full of exposure. First of all, it exposed a lot of problems, for example, Will MacAskill's failure in judging people, he has been criticized by the community all the time, because he still does not reflect enough on SBF and kept sheltering SBF back then.

But this period is also the era that truly made EA one of the most influential schools of thought for the 21st century, as *The Economist* put it.

Then came what we now know as the AI Security Era. I think it is in this AI security era that the general public has heard of the term "Effective Altruism", because AI security has become one of their core issues.

The entire EA network, on the one hand, has many members joining cutting-edge labs as employees, and on the other hand, a large amount of funds are invested in AI security, which makes the entire EA circle intersect with the policy circle, the national security circle, and the circles in Washington. This has even led to, for example, President Trump and the U.S. Department of Defense tweeting that "the United States will certainly not become EA" and so on.

I think the current stage is a new stage for EA, with new challenges, but it is also the era when it has most entered public consciousness.

Entering the Stronghold of EA and Rationalism

Hongjun: I don't know if you've noticed the relationship between Anthropic, some security audit institutions, and EA recently? After Jacob Coxon resigned from Anthropic in September, an independent researcher Kevin Bass sorted out the capital chain on X, he looked at METR, the institution arranged by Anthropic to conduct independent security audits: the funders of METR and some other security institutions are also early investors of Anthropic — Dustin Moskovitz, founder of Coefficient Giving, who holds a large number of Anthropic shares. So this so-called "independent assessment" may actually be tied to Anthropic's commercial interests. There are many voices denouncing EA on the Internet, and many people have begun to question the source of funds for all security institutions. What do you think of these doubts?

Afra: After Jacob's tweet went viral, people pulled out the entire EA's capital injection into AI security, and then started making all kinds of conspiracy theories. For me, it is a conspiracy theory in itself.

Hongjun: Why do you think it is a conspiracy theory?

Afra: Many traditional media will question: what is the purpose and hidden agenda behind the flow of this money to these organizations? I think there will definitely be some kind of projected agenda. But what is the agenda of this group of EA people? It is to prevent AI from becoming uncontrollable and causing consequences that humanity cannot bear. That's their agenda, there is no hidden agenda.

Kevin Roose, former *New York Times* reporter and host of the podcast Hard Fork, recently wrote a book called *The AGI Chronicles*, and he received some funding with an EA background while writing it. So many people questioned: is the source of this money unclear? If you take EA's money, do you have to write their agenda into the book?

But I think the biggest problem may be: When an ideological movement has so much money, is it discovering the most important problems, or is it also participating in and deciding what problems will be considered important.

We are actually living with the consequences of this now. They said very early on that AI security is important and that a lot of money should be poured into it, and we are now living under this influence. I have seen some traditional think tanks in Washington that can hardly recruit qualified AI policy analysts, because young, capable talents are all flowing into EA-sponsored think tanks and organizations.

If you are a young researcher in your early 20s, just graduated from school with a PhD from Stanford, and want to study AI economics, you will find that there are a large number of researcher positions, a lot of research grants, many research positions, and very good annual salaries in the field of AI alignment and AI security; these institutions just give you money, don't interfere much with your research direction, and no one asks you to change your beliefs. Other research institutions do not have so many resources and so much freedom. So I have seen more and more such young people being absorbed by these so-called "EA institutions" related to AI security; I have also seen more and more very smart people being safety-pilled (persuaded by the concept of AI security) because they meet and discuss issues with these people every day.

But in my opinion, it is very normal to be safety-pilled at this stage. Even if you don't care that much about AI security issues, and you read all kinds of ordinary news and watch all kinds of TV reports every day, you will still be safety-pilled.

Hongjun: I know you often go to Berkeley, and some EA organizations in San Francisco, or what we call rationalist organizations. I also heard that in San Francisco, for example, the METR institution we just mentioned — if you want to go in and participate in some activities, such as activities held by Anthropic or researchers, and you say "I am not an EA", they will ask — if you are not EA, you may not even be able to enter many activities in San Francisco.

Afra: I think EA is very interesting. If you ask some people "Are you an EA?", they will say "Oh, I'm not an EA, but I'm very close to their people". No one who is part of EA keeps saying that they are EA all the time. In fact, many of my EA friends are very EA, but they don't consider themselves EA members, which I think is also very interesting.

Hongjun: What's the atmosphere like in Berkeley? I think Berkeley is more biased towards the rationalist community, right? I know there is an organization called Lighthaven, where people often gather and hold activities, and its atmosphere is different from that of San Francisco.

Afra: Yes, I think Berkeley has formed its own system now. There are a large number of third-party AI security audit institutions in Berkeley, or organizations like METR, to be honest, there are already dozens of them. Lighthaven is actually an event venue for rationalists and some EA-sponsored activities, it is an activity park in Berkeley composed of about six to seven single-family houses.

You can attend lectures there, socialize freely, and roast marshmallows by the campfire at night. Lighthaven is regarded as a stronghold for AI security, EA and rationalism.

I myself occasionally go to Lighthaven to hold some AI-related meetings and meet people. I just came back from two conferences two weeks ago: one is called The Curve, and the other is called the Progress Conference.

One is a conference related to cutting-edge AI research, focusing on the explosive, curve-level development of AI; the second one, the Progress Conference, is more about promoting social progress, very similar to Ezra Klein's book *Abundance*, discussing how to use existing technology to create a more abundant, more futuristic society. It actually discusses all kinds of cutting-edge scientific research, such as how to educate the next generation, be more confident in the future, and discuss such topics.

Hongjun: Do you think these topics, when people discuss them, such as promoting the progress of events, promoting whether AI should accelerate development or slow down, will have strong influence on the actual impact on things?

Afra: I think so. I remember that at last year's The Curve, the conference about cutting-edge AI, there were many people from Washington and Congress, and many government lawmakers from California. The group effect brought by The Curve is quite obvious, it is the best Venn diagram intersection between Washington and Silicon Valley. So I understand their influence.

Hongjun: What about the rationalist community?

Afra: I think the rationalist community is more laid-back. Rationalist people don't actually want to seek influence that much. So the people in the rationalist community are still the same as before, everyone writes blogs and researches things they are interested in. They don't have so much of the 4D Chess that people in Silicon Valley talk about now, they don't have so many calculations in their heads like "I do A today, which can achieve B, so I can finally reach goal Z, but how do I reverse-engineer this thing".

Rationalist people don't actually have so many grand plans, they are very lovely. Just like Vitalik said, Rationalists like to do what they love to do, many of them write novels, write blogs, and discuss very nerdy math and AI issues.

EA Spiritual Leader: Will MacAskill

Hongjun: Then I saw that you also went to London to participate in a program called GovAI (Center for AI Governance) Fellowship, and you met the spiritual leader of EA we just mentioned, Will MacAskill. Did you communicate with him?

Afra: The first time I met Will MacAskill was at The Curve. I talked to him for more than ten minutes, and I think he almost condensed many of the ideological changes of EA over the past ten years. Gideon Lewis-Kraus from *The New Yorker* wrote an excellent profile of him in 2022. In the article, Will himself, especially when he first started founding these EA organizations, had a bit of the air of a prophet and ascetic. Gideon, as the author, wrote that when you are around Will, you will feel the moral pressure that you should do good, because Will lives a very simple life, and he keeps donating his income.

Philosopher, co-founder of the Effective Altruism movement William MacAskill. Image source: Sam Deere / CC BY-SA 4.0 (Wikimedia Commons)

Hongjun: What do you think is his core influence on the entire EA circle?

Afra: He is a spiritual leader of EA, and at the same time he is the person who founded some core EA organizations in Oxford. When he was 22 years old, he met an Australian professor Toby Ord, who also attaches great importance to EA and long-termism.

The two hit it off immediately, and Will promised on the spot that he would donate his wealth permanently as much as he could. His willingness to bind his future self is also a point that many people will criticize later: EA has a concept that I can sign a contract with my future richer self, and let the latter do what my current self wants to do.

I remember Vitalik's criticism of this EA concept is: Your future self will always betray your current self, because 10 or 20 years later, your thoughts will change, your behavior will change, your friends and your circle around you will change, and the things you want to do will change. How can you guarantee that your future richer self, who became rich to do the good deeds you want to do today, will stick to the promises you made today? But that was Will at that time, he said he would donate all his income.

I think there is another background: a few years after EA was established, the Occupy Wall Street movement broke out. When Occupy Wall Street was in full swing, the moral claim at that time was: these people on Wall Street are vampires, they are evil people who have made the lives of many middle-class and lower