behavioral economics

The trouble with Rationalism and Effective Altruism

PNRJ cognitive science, core principles, ethics, futurism, heuristics and biases, tribal paradigm , , , , , , , , , , , , , , , , , , , , ,

Oct 11 JDN 2461325

I have discussed before the notion of a motte and bailey, in which uses ambiguity to present a view in two different forms: A motte, which is highly defensible but often banal or uncontroversial, and a bailey, which is highly interesting and radical but ultimately indefensible. Like the medieval strategy, you want to live in the bailey, but you can only defend the motte.

I think the “rationalist” and “effective altruism” communities suffer quite severely from this dynamic.

I use the scare quotes precisely because many so-called “rationalists” deeply fail to conform to the principles of rationalism, and many so-called “effective altruists” don’t seem particularly altruistic.

This saddens me greatly, because the core principles of rationalism and effective altruism are not simply correct: They are among the best ideas that have ever been formulated by human minds.

The core of rationalism is this:

Reason is the best source of knowledge and truth.

The core of effective altruism is this:

The best acts are those that do the most good for the least cost.

Both of these ideas seem beyond defensible—more like unassailable, perhaps even undeniable, as in literally impossible to coherently deny. They have the ring of not simply truth, but deep truth, perhaps even capital-T Truth, fundamental laws that have the power and certainty of logical or mathematical theorems.

Moreover, they are not banal or uncontroversial! It is quite bizarre that they are not; it speaks of deep irrationality within human brains and human societies that anyone would even try to deny these things. Yet the very existence of religion as we know it proves that rationalism is not uncontroversial, and many people still cling to moral systems that condemn people for harmless acts or create needless suffering.

This is the motte, and it is a fine motte indeed: Beautiful, imposing, perfectly-engineered, and utterly impregnable. The closest real-world equivalent is something like the walls of Troy, which were (as the story goes) invincible against conventional force and could only be defeated by deception. (Or perhaps if you consider all of Japan a fortress: Conquering that island nation from the outside required air power and nuclear weapons.)

I love this motte will all my heart, but I am deeply concerned about the bailey.

What is the bailey of rationalism and effective altruism?

First, let me address the parts that I am not too concerned about; I think many people go a bit too far with them, but I believe they are basically sound:

  • Bayesianism: The best way to reason about uncertainty is using Bayesian statistics.
  • Utilitarianism: Morality consists in doing the greatest good for the greatest number.
  • Expected utility: The best way to make decisions under uncertainty is to maximize expected utility.
  • Behavioral economics: Economic behavior is best understood by recognizing that human beings use heuristics and are subject to biases that prevent them from being fully rational.
  • Longtermism: One of our most important moral concerns is what kind of world we are making for future generations of humanity.

As I have stated them, I consider these views to be basically sound. Now, let me rephrase them in ways that they are often taken too far:

  • McNamara fallacy: Only what you can mathematically quantify matters, and if you are not explicitly computing a posterior distribution from a prior distribution and a likelihood, you are not making a rational judgment under uncertainty.
  • Act utilitarianism: Morality consists in taking each action in isolation and maximizing the utility it creates, without regard for the deep uncertainty and complex ramifications involved in doing so.
  • Shut up and multiply: Regardless of whether it violates common sense, rests upon flimsy assumptions, or involves deep uncertainty and unacceptable risk, the proper thing to do when making decisions is always to go with what you calculate yields the highest expected utility.
  • Fallacy fallacy: If your opponent is reasoning in a fallacious way, you are free to completely dismiss their beliefs.
  • Utopian longtermism: As long as you can convince yourself that what you are doing will provide some miniscule increase in the probability of a utopian future for our galaxy-colonizing descendants, you are free to ignore the consequences of your actions for people who are alive today.

These are already a big problem, especially the last one. I have seen far too many “rationalists” and “effective altruists” pivot away from projects that could have done enormous good today with a high degree of certainty (such as improving economic policy, or supporting environmental causes, or donating to medical research or malaria prevention) in favor of trying to advance AI (often in the name of “AI safety”, but for some reason never actually involving regulating anything) toward some pie-in-the-sky vision of a utopian future.

In particular, I am deeply concerned about people who attach huge payoffs to tiny probabilities, and then convince themselves that “expected utility” means they must care about those things more than anything else. This is essentially a secular version of Pascal’s Wager, and it can drive people to terrible, terrible things.

It is far better for you to do small good things that are certain. Yes, we should care about people thousands of years from now; but the truth is, we really have no idea what the future will be like thousands of years from now. So how about we do our best to help the people who are here, now?

If you don’t find this argument convincing, consider this:

Helping people now can push us toward a brighter future.

Suppose, for the sake of argument, that we are truly most concerned about what happens thousands of years from now when humans have colonized the galaxy. What sort of society would you want to be responsible for that colonization?

Would it be one where people used esoteric and highly-uncertain mathematical arguments to convince themselves that they should abuse and exploit other people in the name of some unprovable “greater good”?

Or would it be one where people helped each other, locally and globally, showed compassion for all living things, and sought to help everyone achieve their full potential?

I contend that spreading kindness and compassion might very well be the very best thing you could do to make humanity’s future brighter. It will not only increase the probability that humanity survives long enough to colonize the galaxy, but also increase the probability that the society which does so is a good society, made of people who are kind and compassionate.

Sure, it wouldn’t hurt to donate to, or work for, AI safety or advocacy for AI regulation. I’m not saying you shouldn’t do those things. But they are not the one most important thing—and indeed, anyone who tells you that something is the one most important thing is basically trying to hack your hedonic mind.

But there is more to the bailey here, and it gets worse.

The majority of people who identify as rationalists and/or effective altruists also seem to believe the following things:

  • Libertarianism: Government is always the problem, never the solution. We should always trust in free markets rather that democratic governments.
  • Prediction markets: Rather than trust experts or even admit that some things are fundamentally unpredictable, we should base our decisions on the results of prediction markets, and we should have prediction markets for virtually everything.
  • AI-doomism: The end of the world is nigh, and this is a terrible thing; any day now, artificial general intelligence will take over the world and wipe out humanity.
  • Singularitarianism: The end of the world as we know it is nigh, and this is a good thing; any day now, artificial general intelligence will take over the world and lead us into a glorious utopian future.

This excellent and eye-opening video on the conference Manifest does a nice job of expressing just how weird and dangerous the culture around these beliefs can become. (And while they are much rarer, there have absolutely been people who tried to adopt rationalist ideas and became literal cults.)

Of course AI-doomism and Singularitarianism are contradictory, but in an odd sort of way: Both sides agree that AGI is imminent and will change everything; they simply disagree about whether this will turn out very good or very bad. As a result, many people who seem to assign a higher-than-normal probability of AI doom also assign a higher-than-normal probability of Singularity. This is not incoherent (as long as all their probabilities add to 1, of course); it simply means that they assign a high probability to AGI being imminent and world-changing, and aren’t sure whether that will turn out good or bad.

(I was once cited as a “Singularitarian blogger”, for this post; I am deeply flattered to be thus cited, but I really don’t think of myself as a Singularitarian. There is a weak sense of Singularitarianism that I guess I agree with: Artificial intelligence will, sooner than most people think, effect radical changes upon human society, and this could be a very good thing if it is handled correctly. But I do not believe we are on the cusp of the Culture.)

The weirdest part, for me at least, is how libertarianism and AI-doomism collide: People will tell me that they believe that AI has a greater than 10%—some say 50% or even 90%—chance of imminently wiping out the human race, and yet they reject any attempt to ban or even significantly regulate AI.

I believe that there is a less than 1% chance of AI wiping out the human race, and yet I believe that we urgently need severe regulations on AI that would substantially slow its development and limit its capabilities. (I think Dynomight makes the best argument for this risk being genuine, cutting through a lot of irrelevant nonsense. As it is, I actually think IQ-300 aliens would likely be better than AGI, because a species of genuinely-intelligent beings (as opposed to computer software that simulates but probably doesn’t actually exhibit intelligence) that is advanced enough to achieve interstellar travel is likely to be peaceful and cooperative enough that they would do us more good than harm. Also, Dynomight’s timeline of “the next few decades” is a lot more plausible than the panicky doomers who say “the next few years”.)

In part I want to regulate AI more because I can already see ways that AI is causing harm (this Jacobin article provides a nice summary), but even without those things, a 1% chance of total annihilation is well worth a great deal of regulation and slowed innovation. Indeed, let me get on record saying that reducing the chances of human annihilation by one percentage point would be well worth slowing economic and technological development by a century.

Libertarianism and prediction markets of course go quite well together, and indeed I rarely encounter one without the other. But this faith in the market is deeply misplaced—especially if we believe, as I do, in behavioral economics!—and in particular, prediction markets are in fact fundamentally flawed as a means of making decisions because of how they handle conditional probabilities. It’s actually kind of a corollary of Goodhart’s Law: Once a measure becomes a target, it ceases to be a good measure, because rational agents try to game the system. Prediction markets can work very well if the outcome is totally beyond human control and wouldn’t totally render the outcome moot (e.g. an earthquake); but once human decisions can affect the result, the prediction market effectively becomes insider trading, and for really big things (like existential risk), the outcome itself could easily render the prediction market—or even money itself—utterly irrelevant.

We must defend the motte with all our might—but perhaps it’s best to abandon some of this bailey.

Yes, by all means, abandon religion, learn to think more rationally, support AI safety, and do what you can to do the most good for the least cost. Those things are good, and right, and important.

But if you find yourself thinking that you should sacrifice yourself, or abandon ordinary concepts of morality, in the name of some ultimate good that will save us from imminent destruction, you need to take a step back, and re-immerse yourself in the rest of society. You are not going to single-handedly save the world; nobody is. If the world is to be saved, we will all do it together.

What behavioral economics needs

PNRJ cognitive science, core principles, critique of neoclassical economics, heuristics and biases, mathematical models, science and academia , , , , , , , , , , , , , ,

Apr 16 JDN 2460049

The transition from neoclassical to behavioral economics has been a vital step forward in science. But lately we seem to have reached a plateau, with no major advances in the paradigm in quite some time.

It could be that there is work already being done which will, in hindsight, turn out to be significant enough to make that next step forward. But my fear is that we are getting bogged down by our own methodological limitations.

Neoclassical economics shared with us its obsession with mathematical sophistication. To some extent this was inevitable; in order to impress neoclassical economists enough to convert some of them, we had to use fancy math. We had to show that we could do it their way in order to convince them why we shouldn’t—otherwise, they’d just have dismissed us the way they had dismissed psychologists for decades, as too “fuzzy-headed” to do the “hard work” of putting everything into equations.

But the truth is, putting everything into equations was never the right approach. Because human beings clearly don’t think in equations. Once we write down a utility function and get ready to take its derivative and set it equal to zero, we have already distanced ourselves from how human thought actually works.

When dealing with a simple physical system, like an atom, equations make sense. Nobody thinks that the electron knows the equation and is following it intentionally. That equation simply describes how the forces of the universe operate, and the electron is subject to those forces.

But human beings do actually know things and do things intentionally. And while an equation could be useful for analyzing human behavior in the aggregate—I’m certainly not objecting to statistical analysis—it really never made sense to say that people make their decisions by optimizing the value of some function. Most people barely even know what a function is, much less remember calculus well enough to optimize one.

Yet right now, behavioral economics is still all based in that utility-maximization paradigm. We don’t use the same simplistic utility functions as neoclassical economists; we make them more sophisticated and realistic. Yet in that very sophistication we make things more complicated, more difficult—and thus in at least that respect, even further removed from how actual human thought must operate.

The worst offender here is surely Prospect Theory. I recognize that Prospect Theory predicts human behavior better than conventional expected utility theory; nevertheless, it makes absolutely no sense to suppose that human beings actually do some kind of probability-weighting calculation in their heads when they make judgments. Most of my students—who are well-trained in mathematics and economics—can’t even do that probability-weighting calculation on paper, with a calculator, on an exam. (There’s also absolutely no reason to do it! All it does it make your decisions worse!) This is a totally unrealistic model of human thought.

This is not to say that human beings are stupid. We are still smarter than any other entity in the known universe—computers are rapidly catching up, but they haven’t caught up yet. It is just that whatever makes us smart must not be easily expressible as an equation that maximizes a function. Our thoughts are bundles of heuristics, each of which may be individually quite simple, but all of which together make us capable of not only intelligence, but something computers still sorely, pathetically lack: wisdom. Computers optimize functions better than we ever will, but we still make better decisions than they do.

I think that what behavioral economics needs now is a new unifying theory of these heuristics, which accounts for not only how they work, but how we select which one to use in a given situation, and perhaps even where they come from in the first place. This new theory will of course be complex; there’s a lot of things to explain, and human behavior is a very complex phenomenon. But it shouldn’t be—mustn’t be—reliant on sophisticated advanced mathematics, because most people can’t do advanced mathematics (almost by construction—we would call it something different otherwise). If your model assumes that people are taking derivatives in their heads, your model is already broken. 90% of the world’s people can’t take a derivative.

I guess it could be that our cognitive processes in some sense operate as if they are optimizing some function. This is commonly posited for the human motor system, for instance; clearly baseball players aren’t actually solving differential equations when they throw and catch balls, but the trajectories that balls follow do in fact obey such equations, and the reliability with which baseball players can catch and throw suggests that they are in some sense acting as if they can solve them.

But I think that a careful analysis of even this classic example reveals some deeper insights that should call this whole notion into question. How do baseball players actually do what they do? They don’t seem to be calculating at all—in fact, if you asked them to try to calculate while they were playing, it would destroy their ability to play. They learn. They engage in practiced motions, acquire skills, and notice patterns. I don’t think there is anywhere in their brains that is actually doing anything like solving a differential equation. It’s all a process of throwing and catching, throwing and catching, over and over again, watching and remembering and subtly adjusting.

One thing that is particularly interesting to me about that process is that is astonishingly flexible. It doesn’t really seem to matter what physical process you are interacting with; as long as it is sufficiently orderly, such a method will allow you to predict and ultimately control that process. You don’t need to know anything about differential equations in order to learn in this way—and, indeed, I really can’t emphasize this enough, baseball players typically don’t.

In fact, learning is so flexible that it can even perform better than calculation. The usual differential equations most people would think to use to predict the throw of a ball would assume ballistic motion in a vacuum, which absolutely not what a curveball is. In order to throw a curveball, the ball must interact with the air, and it must be launched with spin; curving a baseball relies very heavily on the Magnus Effect. I think it’s probably possible to construct an equation that would fully predict the motion of a curveball, but it would be a tremendously complicated one, and might not even have an exact closed-form solution. In fact, I think it would require solving the Navier-Stokes equations, for which there is an outstanding Millennium Prize. Since the viscosity of air is very low, maybe you could get away with approximating using the Euler fluid equations.

To be fair, a learning process that is adapting to a system that obeys an equation will yield results that become an ever-closer approximation of that equation. And it is in that sense that a baseball player can be said to be acting as if solving a differential equation. But this relies heavily on the system in question being one that obeys an equation—and when it comes to economic systems, is that even true?

What if the reason we can’t find a simple set of equations that accurately describe the economy (as opposed to equations of ever-escalating complexity that still utterly fail to describe the economy) is that there isn’t one? What if the reason we can’t find the utility function people are maximizing is that they aren’t maximizing anything?

What behavioral economics needs now is a new approach, something less constrained by the norms of neoclassical economics and more aligned with psychology and cognitive science. We should be modeling human beings based on how they actually think, not some weird mathematical construct that bears no resemblance to human reasoning but is designed to impress people who are obsessed with math.

I’m of course not the first person to have suggested this. I probably won’t be the last, or even the one who most gets listened to. But I hope that I might get at least a few more people to listen to it, because I have gone through the mathematical gauntlet and earned my bona fides. It is too easy to dismiss this kind of reasoning from people who don’t actually understand advanced mathematics. But I do understand differential equations—and I’m telling you, that’s not how people think.