May 31 JDN 2461192

They call themselves effective accelerationists, co-opting the acronym EA from Effective Altruism despite being about as diametrically opposed as it is possible to be.

They rally behind the Techno-Optimist Manifesto, which I admit makes a lot of good points and has a very seductive quality to it; but when you get to the end and reach the conclusions they draw from many reasonable-sounding premises, the result is absolute horror.

What do they want?

Totally unrestricted artificial intelligence produced as fast as possible.

Let’s be clear about this: They not only want to develop artificial intelligence (many people want that). They not only want to replace humanity with artificial intelligence (serious philosophers have suggested that this might be a long-term evolution for our civilization). They want to do it right now, without restrictions.

Their reasoning seems to go something like this:

1. Artificial intelligence has tremendous potential.
2. Improved computing has already been a tremendous boon to humanity, Applications of artificial intelligence to many fields such as scientific research and biotechnology could continue to be so.
3. Free-market capitalism is the most efficient economic system yet devised.

Therefore,

4. We should allow (or even incentivize) corporations to make artificial intelligence as powerful as possible as quickly as possible.

If we were to apply their same reasoning to other technologies, it would be obvious what’s wrong here. Consider the following argument:

1. Nuclear energy is tremendously powerful and very environmentally-friendly.
2. Cleaner, more powerful energy has been a boon to humanity, and would continue to be so if improved further.
3. Free-market capitalism is the most efficient economic system yet devised.

Therefore,

4. We should grant unrestricted access to nuclear material to anyone who is rich enough to afford it.

From three entirely reasonable premises (that honestly should be uncontroversial, even though they aren’t), we have made some kind of leap of logic to derive a conclusion that is utterly insane, and could literally result in the destruction of our entire civilization.

(And, if you’re not horrified enough yet, this isn’t even hypothetical after all: The US government is seriously considering giving weapons-grade plutonium to tech startups, apparently based on this exact reasoning.)

I agree that artificial intelligence has tremendous potential. That is why it must be kept on a tight leash.

In the near term, it is already poised to severely disrupt our economy and education system. But sufficiently-powerful AI genuinely could result in harms that could kill billions of people or even destroy human civilization. (I don’t think this will happen; but if the effective accelerationists have their way, that outcome becomes a lot more likely.)

Part of their argument seems to be that delaying new medical treatments and solutions to global problems will cost lives. This is true. But it’s far fewer lives than we would be putting at risk by pulling out all the stops and letting corporations do whatever they feel like doing.

I, too, want to see a glorious future where humanity transcends our current limits through biotechnology or cybernetics or truly-sentient artificial general intelligence. But the benefits of doing that a few years—or even decades, or even centuries—sooner are simply not worth the risk of destroying our entire civilization.

When dealing with a technology this powerful and a change this radical, the most important thing is to do it correctly—not to do it quickly. We need to make sure that the artificial intelligence we create is wise and benevolent so that its great power benefits humanity—and with this much at stake, it’s worth taking a long time to get it right if we have to.

A world run by copies of Lieutenant Commander Data would be a paradise. A world run by copies of Grok would be a hellscape. It’s worth spending some time to make sure we get the former and not the latter.