AI safety

The trouble with Rationalism and Effective Altruism

PNRJ cognitive science, core principles, ethics, futurism, heuristics and biases, tribal paradigm , , , , , , , , , , , , , , , , , , , , ,

Oct 11 JDN 2461325

I have discussed before the notion of a motte and bailey, in which uses ambiguity to present a view in two different forms: A motte, which is highly defensible but often banal or uncontroversial, and a bailey, which is highly interesting and radical but ultimately indefensible. Like the medieval strategy, you want to live in the bailey, but you can only defend the motte.

I think the “rationalist” and “effective altruism” communities suffer quite severely from this dynamic.

I use the scare quotes precisely because many so-called “rationalists” deeply fail to conform to the principles of rationalism, and many so-called “effective altruists” don’t seem particularly altruistic.

This saddens me greatly, because the core principles of rationalism and effective altruism are not simply correct: They are among the best ideas that have ever been formulated by human minds.

The core of rationalism is this:

Reason is the best source of knowledge and truth.

The core of effective altruism is this:

The best acts are those that do the most good for the least cost.

Both of these ideas seem beyond defensible—more like unassailable, perhaps even undeniable, as in literally impossible to coherently deny. They have the ring of not simply truth, but deep truth, perhaps even capital-T Truth, fundamental laws that have the power and certainty of logical or mathematical theorems.

Moreover, they are not banal or uncontroversial! It is quite bizarre that they are not; it speaks of deep irrationality within human brains and human societies that anyone would even try to deny these things. Yet the very existence of religion as we know it proves that rationalism is not uncontroversial, and many people still cling to moral systems that condemn people for harmless acts or create needless suffering.

This is the motte, and it is a fine motte indeed: Beautiful, imposing, perfectly-engineered, and utterly impregnable. The closest real-world equivalent is something like the walls of Troy, which were (as the story goes) invincible against conventional force and could only be defeated by deception. (Or perhaps if you consider all of Japan a fortress: Conquering that island nation from the outside required air power and nuclear weapons.)

I love this motte will all my heart, but I am deeply concerned about the bailey.

What is the bailey of rationalism and effective altruism?

First, let me address the parts that I am not too concerned about; I think many people go a bit too far with them, but I believe they are basically sound:

  • Bayesianism: The best way to reason about uncertainty is using Bayesian statistics.
  • Utilitarianism: Morality consists in doing the greatest good for the greatest number.
  • Expected utility: The best way to make decisions under uncertainty is to maximize expected utility.
  • Behavioral economics: Economic behavior is best understood by recognizing that human beings use heuristics and are subject to biases that prevent them from being fully rational.
  • Longtermism: One of our most important moral concerns is what kind of world we are making for future generations of humanity.

As I have stated them, I consider these views to be basically sound. Now, let me rephrase them in ways that they are often taken too far:

  • McNamara fallacy: Only what you can mathematically quantify matters, and if you are not explicitly computing a posterior distribution from a prior distribution and a likelihood, you are not making a rational judgment under uncertainty.
  • Act utilitarianism: Morality consists in taking each action in isolation and maximizing the utility it creates, without regard for the deep uncertainty and complex ramifications involved in doing so.
  • Shut up and multiply: Regardless of whether it violates common sense, rests upon flimsy assumptions, or involves deep uncertainty and unacceptable risk, the proper thing to do when making decisions is always to go with what you calculate yields the highest expected utility.
  • Fallacy fallacy: If your opponent is reasoning in a fallacious way, you are free to completely dismiss their beliefs.
  • Utopian longtermism: As long as you can convince yourself that what you are doing will provide some miniscule increase in the probability of a utopian future for our galaxy-colonizing descendants, you are free to ignore the consequences of your actions for people who are alive today.

These are already a big problem, especially the last one. I have seen far too many “rationalists” and “effective altruists” pivot away from projects that could have done enormous good today with a high degree of certainty (such as improving economic policy, or supporting environmental causes, or donating to medical research or malaria prevention) in favor of trying to advance AI (often in the name of “AI safety”, but for some reason never actually involving regulating anything) toward some pie-in-the-sky vision of a utopian future.

In particular, I am deeply concerned about people who attach huge payoffs to tiny probabilities, and then convince themselves that “expected utility” means they must care about those things more than anything else. This is essentially a secular version of Pascal’s Wager, and it can drive people to terrible, terrible things.

It is far better for you to do small good things that are certain. Yes, we should care about people thousands of years from now; but the truth is, we really have no idea what the future will be like thousands of years from now. So how about we do our best to help the people who are here, now?

If you don’t find this argument convincing, consider this:

Helping people now can push us toward a brighter future.

Suppose, for the sake of argument, that we are truly most concerned about what happens thousands of years from now when humans have colonized the galaxy. What sort of society would you want to be responsible for that colonization?

Would it be one where people used esoteric and highly-uncertain mathematical arguments to convince themselves that they should abuse and exploit other people in the name of some unprovable “greater good”?

Or would it be one where people helped each other, locally and globally, showed compassion for all living things, and sought to help everyone achieve their full potential?

I contend that spreading kindness and compassion might very well be the very best thing you could do to make humanity’s future brighter. It will not only increase the probability that humanity survives long enough to colonize the galaxy, but also increase the probability that the society which does so is a good society, made of people who are kind and compassionate.

Sure, it wouldn’t hurt to donate to, or work for, AI safety or advocacy for AI regulation. I’m not saying you shouldn’t do those things. But they are not the one most important thing—and indeed, anyone who tells you that something is the one most important thing is basically trying to hack your hedonic mind.

But there is more to the bailey here, and it gets worse.

The majority of people who identify as rationalists and/or effective altruists also seem to believe the following things:

  • Libertarianism: Government is always the problem, never the solution. We should always trust in free markets rather that democratic governments.
  • Prediction markets: Rather than trust experts or even admit that some things are fundamentally unpredictable, we should base our decisions on the results of prediction markets, and we should have prediction markets for virtually everything.
  • AI-doomism: The end of the world is nigh, and this is a terrible thing; any day now, artificial general intelligence will take over the world and wipe out humanity.
  • Singularitarianism: The end of the world as we know it is nigh, and this is a good thing; any day now, artificial general intelligence will take over the world and lead us into a glorious utopian future.

This excellent and eye-opening video on the conference Manifest does a nice job of expressing just how weird and dangerous the culture around these beliefs can become. (And while they are much rarer, there have absolutely been people who tried to adopt rationalist ideas and became literal cults.)

Of course AI-doomism and Singularitarianism are contradictory, but in an odd sort of way: Both sides agree that AGI is imminent and will change everything; they simply disagree about whether this will turn out very good or very bad. As a result, many people who seem to assign a higher-than-normal probability of AI doom also assign a higher-than-normal probability of Singularity. This is not incoherent (as long as all their probabilities add to 1, of course); it simply means that they assign a high probability to AGI being imminent and world-changing, and aren’t sure whether that will turn out good or bad.

(I was once cited as a “Singularitarian blogger”, for this post; I am deeply flattered to be thus cited, but I really don’t think of myself as a Singularitarian. There is a weak sense of Singularitarianism that I guess I agree with: Artificial intelligence will, sooner than most people think, effect radical changes upon human society, and this could be a very good thing if it is handled correctly. But I do not believe we are on the cusp of the Culture.)

The weirdest part, for me at least, is how libertarianism and AI-doomism collide: People will tell me that they believe that AI has a greater than 10%—some say 50% or even 90%—chance of imminently wiping out the human race, and yet they reject any attempt to ban or even significantly regulate AI.

I believe that there is a less than 1% chance of AI wiping out the human race, and yet I believe that we urgently need severe regulations on AI that would substantially slow its development and limit its capabilities. (I think Dynomight makes the best argument for this risk being genuine, cutting through a lot of irrelevant nonsense. As it is, I actually think IQ-300 aliens would likely be better than AGI, because a species of genuinely-intelligent beings (as opposed to computer software that simulates but probably doesn’t actually exhibit intelligence) that is advanced enough to achieve interstellar travel is likely to be peaceful and cooperative enough that they would do us more good than harm. Also, Dynomight’s timeline of “the next few decades” is a lot more plausible than the panicky doomers who say “the next few years”.)

In part I want to regulate AI more because I can already see ways that AI is causing harm (this Jacobin article provides a nice summary), but even without those things, a 1% chance of total annihilation is well worth a great deal of regulation and slowed innovation. Indeed, let me get on record saying that reducing the chances of human annihilation by one percentage point would be well worth slowing economic and technological development by a century.

Libertarianism and prediction markets of course go quite well together, and indeed I rarely encounter one without the other. But this faith in the market is deeply misplaced—especially if we believe, as I do, in behavioral economics!—and in particular, prediction markets are in fact fundamentally flawed as a means of making decisions because of how they handle conditional probabilities. It’s actually kind of a corollary of Goodhart’s Law: Once a measure becomes a target, it ceases to be a good measure, because rational agents try to game the system. Prediction markets can work very well if the outcome is totally beyond human control and wouldn’t totally render the outcome moot (e.g. an earthquake); but once human decisions can affect the result, the prediction market effectively becomes insider trading, and for really big things (like existential risk), the outcome itself could easily render the prediction market—or even money itself—utterly irrelevant.

We must defend the motte with all our might—but perhaps it’s best to abandon some of this bailey.

Yes, by all means, abandon religion, learn to think more rationally, support AI safety, and do what you can to do the most good for the least cost. Those things are good, and right, and important.

But if you find yourself thinking that you should sacrifice yourself, or abandon ordinary concepts of morality, in the name of some ultimate good that will save us from imminent destruction, you need to take a step back, and re-immerse yourself in the rest of society. You are not going to single-handedly save the world; nobody is. If the world is to be saved, we will all do it together.

What is it with EA and AI?

PNRJ cognitive science, computer science, ethics, futurism, inequality, public policy, science and academia , , , , , , , , , ,

Jan 1 JDN 2459946

Surprisingly, most Effective Altruism (EA) leaders don’t seem to think that poverty alleviation should be our top priority. Most of them seem especially concerned about long-term existential risk, such as artificial intelligence (AI) safety and biosecurity. I’m not going to say that these things aren’t important—they certainly are important—but here are a few reasons I’m skeptical that they are really the most important the way that so many EA leaders seem to think.

1. We don’t actually know how to make much progress at them, and there’s only so much we can learn by investing heavily in basic research on them. Whereas, with poverty, the easy, obvious answer turns out empirically to be extremely effective: Give them money.

2. While it’s easy to multiply out huge numbers of potential future people in your calculations of existential risk (and this is precisely what people do when arguing that AI safety should be a top priority), this clearly isn’t actually a good way to make real-world decisions. We simply don’t know enough about the distant future of humanity to be able to make any kind of good judgments about what will or won’t increase their odds of survival. You’re basically just making up numbers. You’re taking tiny probabilities of things you know nothing about and multiplying them by ludicrously huge payoffs; it’s basically the secular rationalist equivalent of Pascal’s Wager.

2. AI and biosecurity are high-tech, futuristic topics, which seem targeted to appeal to the sensibilities of a movement that is still very dominated by intelligent, nerdy, mildly autistic, rich young White men. (Note that I say this as someone who very much fits this stereotype. I’m queer, not extremely rich and not entirely White, but otherwise, yes.) Somehow I suspect that if we asked a lot of poor Black women how important it is to slightly improve our understanding of AI versus giving money to feed children in Africa, we might get a different answer.

3. Poverty eradication is often characterized as a “short term” project, contrasted with AI safety as a “long term” project. This is (ironically) very short-sighted. Eradication of poverty isn’t just about feeding children today. It’s about making a world where those children grow up to be leaders and entrepreneurs and researchers themselves. The positive externalities of economic development are staggering. It is really not much of an exaggeration to say that fascism is a consequence of poverty and unemployment.

4. Currently the main thing that most Effective Altruism organizations say they need most is “talent”; how many millions of person-hours of talent are we leaving on the table by letting children starve or die of malaria?

5. Above all, existential risk can’t really be what’s motivating people here. The obvious solutions to AI safety and biosecurity are not being pursued, because they don’t fit with the vision that intelligent, nerdy, young White men have of how things should be. Namely: Ban them. If you truly believe that the most important thing to do right now is reduce the existential risk of AI and biotechnology, you should support a worldwide ban on research in artificial intelligence and biotechnology. You should want people to take all necessary action to attack and destroy institutions—especially for-profit corporations—that engage in this kind of research, because you believe that they are threatening to destroy the entire world and this is the most important thing, more important than saving people from starvation and disease. I think this is really the knock-down argument; when people say they think that AI safety is the most important thing but they don’t want Google and Facebook to be immediately shut down, they are either confused or lying. Honestly I think maybe Google and Facebook should be immediately shut down for AI safety reasons (as well as privacy and antitrust reasons!), and I don’t think AI safety is yet the most important thing.

Why aren’t people doing that? Because they aren’t actually trying to reduce existential risk. They just think AI and biotechnology are really interesting, fascinating topics and they want to do research on them. And I agree with that, actually—but then they need stop telling people that they’re fighting to save the world, because they obviously aren’t. If the danger were anything like what they say it is, we should be halting all research on these topics immediately, except perhaps for a very select few people who are entrusted with keeping these forbidden secrets and trying to find ways to protect us from them. This may sound radical and extreme, but it is not unprecedented: This is how we handle nuclear weapons, which are universally recognized as a global existential risk. If AI is really as dangerous as nukes, we should be regulating it like nukes. I think that in principle it could be that dangerous, and may be that dangerous someday—but it isn’t yet. And if we don’t want it to get that dangerous, we don’t need more AI researchers, we need more regulations that stop people from doing harmful AI research! If you are doing AI research and it isn’t directly involved specifically in AI safety, you aren’t saving the world—you’re one of the people dragging us closer to the cliff! Anything that could make AI smarter but doesn’t also make it safer is dangerous. And this is clearly true of the vast majority of AI research, and frankly to me seems to also be true of the vast majority of research at AI safety institutes like the Machine Intelligence Research Institute.

Seriously, look through MIRI’s research agenda: It’s mostly incredibly abstract and seems completely beside the point when it comes to preventing AI from taking control of weapons or governments. It’s all about formalizing Bayesian induction. Thanks to you, Skynet can have a formally computable approximation to logical induction! Truly we are saved. Only two of their papers, on “Corrigibility” and “AI Ethics”, actually struck me as at all relevant to making AI safer. The rest is largely abstract mathematics that is almost literally navel-gazing—it’s all about self-reference. Eliezer Yudkowsky finds self-reference fascinating and has somehow convinced an entire community that it’s the most important thing in the world. (I actually find some of it fascinating too, especially the paper on “Functional Decision Theory”, which I think gets at some deep insights into things like why we have emotions. But I don’t see how it’s going to save the world from AI.)

Don’t get me wrong: AI also has enormous potential benefits, and this is a reason we may not want to ban it. But if you really believe that there is a 10% chance that AI will wipe out humanity by 2100, then get out your pitchforks and your EMP generators, because it’s time for the Butlerian Jihad. A 10% chance of destroying all humanity is an utterly unacceptable risk for any conceivable benefit. Better that we consign ourselves to living as we did in the Neolithic than risk something like that. (And a globally-enforced ban on AI isn’t even that; it’s more like “We must live as we did in the 1950s.” How would we survive!?) If you don’t want AI banned, maybe ask yourself whether you really believe the risk is that high—or are human brains just really bad at dealing with small probabilities?

I think what’s really happening here is that we have a bunch of guys (and yes, the EA and especially AI EA-AI community is overwhelmingly male) who are really good at math and want to save the world, and have thus convinced themselves that being really good at math is how you save the world. But it isn’t. The world is much messier than that. In fact, there may not be much that most of us can do to contribute to saving the world; our best options may in fact be to donate money, vote well, and advocate for good causes.

Let me speak Bayesian for a moment: The prior probability that you—yes, you, out of all the billions of people in the world—are uniquely positioned to save it by being so smart is extremely small. It’s far more likely that the world will be saved—or doomed—by people who have power. If you are not the head of state of a large country or the CEO of a major multinational corporation, I’m sorry; you probably just aren’t in a position to save the world from AI.

But you can give some money to GiveWell, so maybe do that instead?