Pascal’s Wager

The trouble with Rationalism and Effective Altruism

PNRJ cognitive science, core principles, ethics, futurism, heuristics and biases, tribal paradigm , , , , , , , , , , , , , , , , , , , , ,

Oct 11 JDN 2461325

I have discussed before the notion of a motte and bailey, in which uses ambiguity to present a view in two different forms: A motte, which is highly defensible but often banal or uncontroversial, and a bailey, which is highly interesting and radical but ultimately indefensible. Like the medieval strategy, you want to live in the bailey, but you can only defend the motte.

I think the “rationalist” and “effective altruism” communities suffer quite severely from this dynamic.

I use the scare quotes precisely because many so-called “rationalists” deeply fail to conform to the principles of rationalism, and many so-called “effective altruists” don’t seem particularly altruistic.

This saddens me greatly, because the core principles of rationalism and effective altruism are not simply correct: They are among the best ideas that have ever been formulated by human minds.

The core of rationalism is this:

Reason is the best source of knowledge and truth.

The core of effective altruism is this:

The best acts are those that do the most good for the least cost.

Both of these ideas seem beyond defensible—more like unassailable, perhaps even undeniable, as in literally impossible to coherently deny. They have the ring of not simply truth, but deep truth, perhaps even capital-T Truth, fundamental laws that have the power and certainty of logical or mathematical theorems.

Moreover, they are not banal or uncontroversial! It is quite bizarre that they are not; it speaks of deep irrationality within human brains and human societies that anyone would even try to deny these things. Yet the very existence of religion as we know it proves that rationalism is not uncontroversial, and many people still cling to moral systems that condemn people for harmless acts or create needless suffering.

This is the motte, and it is a fine motte indeed: Beautiful, imposing, perfectly-engineered, and utterly impregnable. The closest real-world equivalent is something like the walls of Troy, which were (as the story goes) invincible against conventional force and could only be defeated by deception. (Or perhaps if you consider all of Japan a fortress: Conquering that island nation from the outside required air power and nuclear weapons.)

I love this motte will all my heart, but I am deeply concerned about the bailey.

What is the bailey of rationalism and effective altruism?

First, let me address the parts that I am not too concerned about; I think many people go a bit too far with them, but I believe they are basically sound:

  • Bayesianism: The best way to reason about uncertainty is using Bayesian statistics.
  • Utilitarianism: Morality consists in doing the greatest good for the greatest number.
  • Expected utility: The best way to make decisions under uncertainty is to maximize expected utility.
  • Behavioral economics: Economic behavior is best understood by recognizing that human beings use heuristics and are subject to biases that prevent them from being fully rational.
  • Longtermism: One of our most important moral concerns is what kind of world we are making for future generations of humanity.

As I have stated them, I consider these views to be basically sound. Now, let me rephrase them in ways that they are often taken too far:

  • McNamara fallacy: Only what you can mathematically quantify matters, and if you are not explicitly computing a posterior distribution from a prior distribution and a likelihood, you are not making a rational judgment under uncertainty.
  • Act utilitarianism: Morality consists in taking each action in isolation and maximizing the utility it creates, without regard for the deep uncertainty and complex ramifications involved in doing so.
  • Shut up and multiply: Regardless of whether it violates common sense, rests upon flimsy assumptions, or involves deep uncertainty and unacceptable risk, the proper thing to do when making decisions is always to go with what you calculate yields the highest expected utility.
  • Fallacy fallacy: If your opponent is reasoning in a fallacious way, you are free to completely dismiss their beliefs.
  • Utopian longtermism: As long as you can convince yourself that what you are doing will provide some miniscule increase in the probability of a utopian future for our galaxy-colonizing descendants, you are free to ignore the consequences of your actions for people who are alive today.

These are already a big problem, especially the last one. I have seen far too many “rationalists” and “effective altruists” pivot away from projects that could have done enormous good today with a high degree of certainty (such as improving economic policy, or supporting environmental causes, or donating to medical research or malaria prevention) in favor of trying to advance AI (often in the name of “AI safety”, but for some reason never actually involving regulating anything) toward some pie-in-the-sky vision of a utopian future.

In particular, I am deeply concerned about people who attach huge payoffs to tiny probabilities, and then convince themselves that “expected utility” means they must care about those things more than anything else. This is essentially a secular version of Pascal’s Wager, and it can drive people to terrible, terrible things.

It is far better for you to do small good things that are certain. Yes, we should care about people thousands of years from now; but the truth is, we really have no idea what the future will be like thousands of years from now. So how about we do our best to help the people who are here, now?

If you don’t find this argument convincing, consider this:

Helping people now can push us toward a brighter future.

Suppose, for the sake of argument, that we are truly most concerned about what happens thousands of years from now when humans have colonized the galaxy. What sort of society would you want to be responsible for that colonization?

Would it be one where people used esoteric and highly-uncertain mathematical arguments to convince themselves that they should abuse and exploit other people in the name of some unprovable “greater good”?

Or would it be one where people helped each other, locally and globally, showed compassion for all living things, and sought to help everyone achieve their full potential?

I contend that spreading kindness and compassion might very well be the very best thing you could do to make humanity’s future brighter. It will not only increase the probability that humanity survives long enough to colonize the galaxy, but also increase the probability that the society which does so is a good society, made of people who are kind and compassionate.

Sure, it wouldn’t hurt to donate to, or work for, AI safety or advocacy for AI regulation. I’m not saying you shouldn’t do those things. But they are not the one most important thing—and indeed, anyone who tells you that something is the one most important thing is basically trying to hack your hedonic mind.

But there is more to the bailey here, and it gets worse.

The majority of people who identify as rationalists and/or effective altruists also seem to believe the following things:

  • Libertarianism: Government is always the problem, never the solution. We should always trust in free markets rather that democratic governments.
  • Prediction markets: Rather than trust experts or even admit that some things are fundamentally unpredictable, we should base our decisions on the results of prediction markets, and we should have prediction markets for virtually everything.
  • AI-doomism: The end of the world is nigh, and this is a terrible thing; any day now, artificial general intelligence will take over the world and wipe out humanity.
  • Singularitarianism: The end of the world as we know it is nigh, and this is a good thing; any day now, artificial general intelligence will take over the world and lead us into a glorious utopian future.

This excellent and eye-opening video on the conference Manifest does a nice job of expressing just how weird and dangerous the culture around these beliefs can become. (And while they are much rarer, there have absolutely been people who tried to adopt rationalist ideas and became literal cults.)

Of course AI-doomism and Singularitarianism are contradictory, but in an odd sort of way: Both sides agree that AGI is imminent and will change everything; they simply disagree about whether this will turn out very good or very bad. As a result, many people who seem to assign a higher-than-normal probability of AI doom also assign a higher-than-normal probability of Singularity. This is not incoherent (as long as all their probabilities add to 1, of course); it simply means that they assign a high probability to AGI being imminent and world-changing, and aren’t sure whether that will turn out good or bad.

(I was once cited as a “Singularitarian blogger”, for this post; I am deeply flattered to be thus cited, but I really don’t think of myself as a Singularitarian. There is a weak sense of Singularitarianism that I guess I agree with: Artificial intelligence will, sooner than most people think, effect radical changes upon human society, and this could be a very good thing if it is handled correctly. But I do not believe we are on the cusp of the Culture.)

The weirdest part, for me at least, is how libertarianism and AI-doomism collide: People will tell me that they believe that AI has a greater than 10%—some say 50% or even 90%—chance of imminently wiping out the human race, and yet they reject any attempt to ban or even significantly regulate AI.

I believe that there is a less than 1% chance of AI wiping out the human race, and yet I believe that we urgently need severe regulations on AI that would substantially slow its development and limit its capabilities. (I think Dynomight makes the best argument for this risk being genuine, cutting through a lot of irrelevant nonsense. As it is, I actually think IQ-300 aliens would likely be better than AGI, because a species of genuinely-intelligent beings (as opposed to computer software that simulates but probably doesn’t actually exhibit intelligence) that is advanced enough to achieve interstellar travel is likely to be peaceful and cooperative enough that they would do us more good than harm. Also, Dynomight’s timeline of “the next few decades” is a lot more plausible than the panicky doomers who say “the next few years”.)

In part I want to regulate AI more because I can already see ways that AI is causing harm (this Jacobin article provides a nice summary), but even without those things, a 1% chance of total annihilation is well worth a great deal of regulation and slowed innovation. Indeed, let me get on record saying that reducing the chances of human annihilation by one percentage point would be well worth slowing economic and technological development by a century.

Libertarianism and prediction markets of course go quite well together, and indeed I rarely encounter one without the other. But this faith in the market is deeply misplaced—especially if we believe, as I do, in behavioral economics!—and in particular, prediction markets are in fact fundamentally flawed as a means of making decisions because of how they handle conditional probabilities. It’s actually kind of a corollary of Goodhart’s Law: Once a measure becomes a target, it ceases to be a good measure, because rational agents try to game the system. Prediction markets can work very well if the outcome is totally beyond human control and wouldn’t totally render the outcome moot (e.g. an earthquake); but once human decisions can affect the result, the prediction market effectively becomes insider trading, and for really big things (like existential risk), the outcome itself could easily render the prediction market—or even money itself—utterly irrelevant.

We must defend the motte with all our might—but perhaps it’s best to abandon some of this bailey.

Yes, by all means, abandon religion, learn to think more rationally, support AI safety, and do what you can to do the most good for the least cost. Those things are good, and right, and important.

But if you find yourself thinking that you should sacrifice yourself, or abandon ordinary concepts of morality, in the name of some ultimate good that will save us from imminent destruction, you need to take a step back, and re-immerse yourself in the rest of society. You are not going to single-handedly save the world; nobody is. If the world is to be saved, we will all do it together.

Heaven and Hell, or how to hack a hedonic mind

PNRJ cognitive science, core principles, current events, ethics, heuristics and biases, tribal paradigm , , , , , , , , ,

Sep 13 JDN 2461297

I am writing this on the 25th anniversary of 9/11, perhaps the most vivid moment in recent history in which religious beliefs directly caused horrific violence. Yes, there were political motivations for the attacks as well; but the way that the conspirators spoke of Islam and the afterlife and the fact that they were willing—eager, even—to sacrifice their own lives in the service of their cause shows that religion was also a very important motivating factor.

I’m not going to spend this post re-litigating the American and international response to the attacks, except to say what I already said over a decade ago: we sacrificed vital civil liberties for more or less nothing. (I would also like to note how prescient this line reads today, written in 2015: “If we’re worried about millions of people dying, it is far more likely that there would be a repeat of the 1918 influenza pandemic”)

Instead, I’d like to talk about how religion made 9/11 a rational behavior.

If you allow the premise that there is a Heaven in which we will receive eternal bliss if we do the right thing, and a Hell in which we will receive eternal suffering if we do the wrong thing, and then posit that the right thing God wants you to do is to crash an airplane into a skyscraper, it is rational for you to crash an airplane into a skyscraper.

If Heaven and Hell are at stake, any action can be made rational.

In order to avoid this conclusion, you either need to deny the rationality of religion itself, or you need to somehow ensure that whatever actions are prescribed by religion are good ones. But people don’t always agree on what actions are good—especially if they come from different religions—and if you already know which actions are good… what did you need religion for?

Is the idea perhaps that most people are too stupid to understand what is really good, and so instead we have to motivate them with lies? The people who are smart enough to understand ethics come up with the rules, and then they pass down these rules to the priests so that the priests can convince everyone else that following those rules leads to Heaven and breaking them leads to Hell? Is that why religious people seem so afraid of atheists undermining the authority of religion? You think most people can’t handle the truth? That if you stopped telling people that they are ruled by an almighty totalitarian who is willing and able to inflict eternal suffering on the disobedient, they wouldn’t understand why murder is wrong?

The better answer is to deny the rationality of religion itself.

This is the problem with Pascal’s Wager: If you are motivated by expected utility (which, most philosophers agree, is the rational way to behave), and you allow that there are outcomes which result in infinite utility, then only those outcomes matter, and you can be convinced to do literally anything in order to achieve those outcomes, as long as their probability of occurring is not exactly zero.

And if there are two different possible outcomes with infinite positive and negative utility attached (e.g. maybe Christianity is true, or maybe Islam is true)? Then your decision becomes undefined. Every possible action results in infinity minus infinity expected utility.

But it actually makes perfect sense that people would try to motivate you this way.

Given that you are a hedonic mind, that is, one which is motivated to seek pleasure and avoid pain, and I want to give you some belief that will make you do whatever I want, how can I have the best possible chance to convince you to do whatever I want you to do?

The answer, of course, is to convince you that doing what I want will result in infinite positive utility—eternal bliss—while doing whatever I don’t want will result in infinite negative utility—eternal suffering. I should tell you that doing what I want sends you to Heaven and not doing what I want sends you to Hell.

If I can convince you of that, I can make you do whatever I want. Once you hold that belief, obeying my every command is the only rational action for you to take, for whatever harms it may cause, they could never be as bad as eternal suffering, and whatever benefits you must forgo to do them, they could never be as good as eternal bliss.

Heaven and Hell are an optimal hack for a hedonic mind.

They are like the sudo command in Linux: Promising Heaven and threatening Hell grants full root access and allows you to make any possible command with absolute authority. In Linux operating systems, you need to already have administrative privileges to use sudo; in religion, I suppose what you need is the spiritual authority of a religious leader (or enough charisma and skill at deception to found your own cult).

Or to make it a little more concrete: They are like a gun held to your head, a gun that is so powerful it can make you suffer for all eternity.

And hey, maybe you’ll get lucky, and what whoever is in control of your beliefs (your priest? your holy book? your church community as a whole?) will actually motivate you to do good things! Maybe it won’t be so bad to be a mindlessly obedient automaton as long as you are programmed for good!

But what if you don’t get lucky? What if the person holding an infinite gun to your head orders you to do something bad? You’ll do that instead. The moment you accepted the belief that certain actions result in infinite happiness or infinite suffering, you gave up all your rational autonomy. As long as that belief remains in your head, the only rational thing to do will always be whatever sends you to Heaven and keeps you from going to Hell.

This is what Voltaire was talking about when he said:
“Those who can make you believe absurdities can make you commit atrocities.”

The answer, of course, is to never accept that belief.

There are various tricks you could use to do that: One would be to temporally discount the future so that bliss millennia from now means almost nothing to you. Another would be to assign some sort of arbitrary cap on how high or low the utility from a given outcome can be.

Alternatively, you could insist upon assigning a probability of zero to any belief that would result in infinite utility—just categorically say that you will never believe such a claim. This could actually be accomplished with a nice, smooth prior that is a normal distribution over outcomes, for while any particular positive utility has a positive probability, there is no probability of an infinite outcome. And the larger the utility that is promised, the lower the probability you’ll accept it—as I previously suggested as a reply to Pascal’s Wager (and Pascal’s Mugging).

In practice, I think people sort of do this.

I think that, deep in their minds, most people have some sort of “reasonableness checker” heuristic that tries to ensure that nobody is able to convince them to behave in horrible ways by making them adopt crazy beliefs. I believe that the reason most people do not commit atrocities is that they cannot really be convinced to believe absurdities—not completely. They can believe in believing them, but they can’t really believe them.

But then, there are the people who don’t have that heuristic, or for whom it has malfunctioned in some way, and those people actually believe the crazy beliefs they were told to believe. And those people can be motivated to commit atrocities—and so several of them did, 25 years ago.


Want to support this blog? Join my Patreon, and get early access to all draft posts before they go live! https://www.patreon.com/c/pnrjulius

The afterlife

PNRJ cognitive science, core principles, ethics, Logic of Kindness , , , , , , , , , , , , , ,

Dec 1 JDN 2460646

Super-human beings aren’t that strange a thing to posit, but they are the sort of thing we’d expect to see clear evidence of if they existed. Without them, prayer is a muddled concept that is difficult to distinguish from simply “things that don’t work”. That leaves the afterlife. Could there be an existence for human consciousness after death?

No. There isn’t. Once you’re dead, you’re dead. It’s really that unequivocal. It is customary in most discussions of this matter to hedge and fret and be “agnostic” about what might lie beyond the grave—but in fact the evidence is absolutely overwhelming.

Everything we know about neuroscience—literally everything—would have to be abandoned in order for an afterlife to make sense. The core of neuroscience, the foundation from which the entire field is built, is what I call the Basic Fact of Cognitive Science: you are your brain. It is your brain that feels, your brain that thinks, your brain that dreams, your brain that remembers. We do not yet understand most of these processes in detail—though some we actually do, such as the processing of visual images. But it doesn’t take an expert mechanic to know that removing the engine makes the car stop running. It doesn’t take a brilliant electrical engineer to know that smashing the CPU makes the computer stop working. Saying that your mind continues to work without your brain is like saying that you can continue to digest without having a stomach or intestines.

This fundamental truth underlies everything we know about the science of consciousness. It can even be directly verified in a piecemeal form: There are specific areas of your brain that, when damaged, will cause you to become blind, or unable to understand language, or unable to speak grammatically (those are two distinct areas), or destroy your ability to form new memories or recall old ones, or even eliminate your ability to recognize faces. Most terrifying of all—yet by no means surprising to anyone who really appreciates the Basic Fact—is the fact that damage to certain parts of your brain will even change your personality, often making you impulsive, paranoid or cruel, literally making you a worse person. More surprising and baffling is the fact that cutting your brain down the middle into left and right halves can split you into two people, each of whom operates half of your body (the opposite half, oddly enough), who mostly agree on things and work together but occasionally don’t. All of these are people we can actually interact with in laboratories, and (except for language deficits of course) talk to them about their experiences. It’s true that we can’t ask people what it’s like when their whole brain is dead, but of course not; there’s nobody left to ask.

This means that if you take away all the functions that experiments have shown require certain brain parts to function, whatever “soul” is left that survives brain death cannot do any of the following: See, hear, speak, understand, remember, recognize faces, or make moral decisions. In what sense is that worth calling a “soul”? In what sense is that you? Those are just the ones we know for sure; as our repertoire expands, more and more cognitive functions will be mapped to specific brain regions. And of course there’s no evidence that anything survives whatsoever.

Nor are near-death experiences any kind of evidence of an afterlife. Yes, some people who were close to dying or briefly technically dead (“He’s only mostly dead!”) have had very strange experiences during that time. Of course they did! Of course you’d have weird experiences as your brain is shutting down or struggling to keep itself online. Think about a computer that has had a magnet run over its hard drive; all sorts of weird glitches and errors are going to occur. (In fact, powerful magnets can have an effect on humans not all that dissimilar from what weaker magnets can do to computers! Certain sections of the brain can be disrupted or triggered in this way; it’s called transcranial magnetic stimulation and it’s actually a promising therapy for some neurological and psychological disorders.) People also have a tendency to over-interpret these experiences as supporting their particular religion, when in fact it’s usually something no more complicated than “a bright light” or “a long tunnel” (another popular item is “positive feelings”). If you stop and think about all the different ways you might come to see “a bright light” and have “positive feelings”, it should be pretty obvious that this isn’t evidence of St. Paul and the Pearly Gates.

The evidence against an afterlife is totally overwhelming. The fact that when we die, we are gone, is among the most certain facts in science. So why do people cling to this belief? Probably because it’s comforting—or rather because the truth that death is permanent and irrevocable is terrifying. You’re damn right it is; it’s basically the source of all other terror, in fact. But guess what? “Terrifying” does not mean “false”. The idea of an afterlife may be comforting, but it’s still obviously not true.

While I was in the process of writing this book, my father died of a ruptured intracranial aneurysm. The event was sudden and unexpected, and by the time I was able to fly from California to Michigan to see him, he had already lost consciousness—for what would turn out to be forever. This event caused me enormous grief, grief from which I may never fully recover. Nothing would make me happier than knowing that he was not truly gone, that he lives on somewhere watching over me. But alas, I know it is not true. He is gone. Forever.

However, I do have a couple of things to say that might offer some degree of consolation:

First, because human minds are software, pieces of our loved ones do go on—in us. Our memories of those we have lost are tiny shards of their souls. When we tell stories about them to others, we make copies of those shards; or to use a more modern metaphor, we back up their data in the cloud. Were we to somehow reassemble all these shards together, we could not rebuild the whole person—there are always missing pieces. But it is also not true that nothing remains. What we have left is how they touched our lives. And when we die, we will remain in how we touch the lives of others. And so on, and so on, as the ramifications of our deeds in life and the generations after us ripple out through the universe at the speed of light, until the end of time.

Moreover, if there’s no afterlife there can be no Hell, and Hell is literally the worst thing imaginable. To subject even a single person—even the most horrible person who ever lived, Hitler, Stalin, Mao, whomever—to the experience of maximum possible suffering forever is an atrocity of incomparable magnitude. Hitler may have deserved a million years of suffering for what he did—but I’m not so sure about maximum suffering, and forever is an awful lot longer than a million years. Indeed, forever is so much longer than a million years that if your sentence is forever, then after serving a million years you still have as much left to go as when you began. But the Bible doesn’t even just say that the most horrible mass murderers will go to Hell; no, it says everyone will go to Hell by default, and deserve it, and can only be forgiven if we believe. No amount of good works will save us from this fate, only God’s grace.

If you believe this—or even suspect it—religion has caused you deep psychological damage. This is the theology of an abusive father—“You must do exactly as I say, or you are worthless and undeserving of love and I will hurt you and it will be all your fault.” No human being, no matter what they have done or failed to do, could ever possibly deserve a punishment as terrible as maximum possible suffering forever. Even if you’re a serial rapist and murderer—and odds are, you’re not—you still don’t deserve to suffer forever. You have lived upon this planet for only a finite time; you can therefore only have committed finitely many crimes and you can only deserve at most finite suffering. In fact, the vast majority of the world’s population is comprised of good, decent people who deserve joy, not suffering.

Indeed, many ethicists would say that nobody deserves suffering, it is simply a necessary evil that we use as a deterrent from greater harms. I’m actually not sure I buy this—if you say that punishment is all about deterrence and not about desert, then you end up with the result that anything which deters someone could count as a fair punishment, even if it’s inflicted upon someone else who did nothing wrong. But no ethicist worthy of the name believes that anybody deserves eternal punishment—yet this is what Jesus says we all deserve in the Bible. And Muhammad says similar things in the Qur’an, about lakes of eternal burning (4:56) and eternal boiling water to drink (47:15) and so on. It’s entirely understandable that such things would motivate you—indeed, they should motivate you completely to do just about anything—if you believed they were true. What I don’t get is why anybody would believe they are true. And I certainly don’t get why anyone would be willing to traumatize their children with these horrific lies.

Then there is Pascal’s Wager: An infinite punishment can motivate you if it has any finite probability, right? Theoretically, yes… but here’s the problem with that line of reasoning: Anybody can just threaten you with infinite punishment to make you do anything. Clearly something is wrong with your decision theory if any psychopath can just make you do whatever he wants because you’re afraid of what might happen just in case what he says might possibly be true. Beware of plausible-seeming theories that lead to such absurd conclusions; it may not be obvious what’s wrong with the argument, but it should be obvious that something is.

Pascal’s Mugging

PNRJ cognitive science, heuristics and biases , , , , , , , , , ,

Nov 10 JDN 2458798

In the Singularitarian community there is a paradox known as “Pascal’s Mugging”. The name is an intentional reference to Pascal’s Wager (and the link is quite apt, for reasons I’ll discuss in a later post.)

There are a few different versions of the argument; Yudkowsky’s original argument in which he came up with the name “Pascal’s Mugging” relies upon the concept of the universe as a simulation and an understanding of esoteric mathematical notation. So here is a more intuitive version:

A strange man in a dark hood comes up to you on the street. “Give me five dollars,” he says, “or I will destroy an entire planet filled with ten billion innocent people. I cannot prove to you that I have this power, but how much is an innocent life worth to you? Even if it is as little as $5,000, are you really willing to bet on ten trillion to one odds that I am lying?”

Do you give him the five dollars? I suspect that you do not. Indeed, I suspect that you’d be less likely to give him the five dollars than if he had merely said he was homeless and asked for five dollars to help pay for food. (Also, you may have objected that you value innocent lives, even faraway strangers you’ll never meet, at more than $5,000 each—but if that’s the case, you should probably be donating more, because the world’s best charities can save a live for about $3,000.)

But therein lies the paradox: Are you really willing to bet on ten trillion to one odds?

This argument gives me much the same feeling as the Ontological Argument; as Russell said of the latter, “it is much easier to be persuaded that ontological arguments are no good than it is to say exactly what is wrong with them.” It wasn’t until I read this post on GiveWell that I could really formulate the answer clearly enough to explain it.

The apparent force of Pascal’s Mugging comes from the idea of expected utility: Even if the probability of an event is very small, if it has a sufficiently great impact, the expected utility can still be large.

The problem with this argument is that extraordinary claims require extraordinary evidence. If a man held a gun to your head and said he’d shoot you if you didn’t give him five dollars, you’d give him five dollars. This is a plausible claim and he has provided ample evidence. If he were instead wearing a bomb vest (or even just really puffy clothing that could conceal a bomb vest), and he threatened to blow up a building unless you gave him five dollars, you’d probably do the same. This is less plausible (what kind of terrorist only demands five dollars?), but it’s not worth taking the chance.

But when he claims to have a Death Star parked in orbit of some distant planet, primed to make another Alderaan, you are right to be extremely skeptical. And if he claims to be a being from beyond our universe, primed to destroy so many lives that we couldn’t even write the number down with all the atoms in our universe (which was actually Yudkowsky’s original argument), to say that you are extremely skeptical seems a grievous understatement.

That GiveWell post provides a way to make this intuition mathematically precise in terms of Bayesian logic. If you have a normal prior with mean 0 and standard deviation 1, and you are presented with a likelihood with mean X and standard deviation X, what should you make your posterior distribution?

Normal priors are quite convenient; they conjugate nicely. The precision (inverse variance) of the posterior distribution is the sum of the two precisions, and the mean is a weighted average of the two means, weighted by their precision.

So the posterior variance is 1/(1 + 1/X^2).

The posterior mean is 1/(1+1/X^2)*(0) + (1/X^2)/(1+1/X^2)*(X) = X/(X^2+1).

That is, the mean of the posterior distribution is just barely higher than zero—and in fact, it is decreasing in X, if X > 1.

For those who don’t speak Bayesian: If someone says he’s going to have an effect of magnitude X, you should be less likely to believe him the larger that X is. And indeed this is precisely what our intuition said before: If he says he’s going to kill one person, believe him. If he says he’s going to destroy a planet, don’t believe him, unless he provides some really extraordinary evidence.

What sort of extraordinary evidence? To his credit, Yudkowsky imagined the sort of evidence that might actually be convincing:

If a poorly-dressed street person offers to save 10(10^100) lives (googolplex lives) for $5 using their Matrix Lord powers, and you claim to assign this scenario less than 10-(10^100) probability, then apparently you should continue to believe absolutely that their offer is bogus even after they snap their fingers and cause a giant silhouette of themselves to appear in the sky.

This post he called “Pascal’s Muggle”, after the term from the Harry Potter series, since some of the solutions that had been proposed for dealing with Pascal’s Mugging had resulted in a situation almost as absurd, in which the mugger could exhibit powers beyond our imagining and yet nevertheless we’d never have sufficient evidence to believe him.

So, let me go on record as saying this: Yes, if someone snaps his fingers and causes the sky to rip open and reveal a silhouette of himself, I’ll do whatever that person says. The odds are still higher that I’m dreaming or hallucinating than that this is really a being from beyond our universe, but if I’m dreaming, it makes no difference, and if someone can make me hallucinate that vividly he can probably cajole the money out of me in other ways. And there might be just enough chance that this could be real that I’m willing to give up that five bucks.

These seem like really strange thought experiments, because they are. But like many good thought experiments, they can provide us with some important insights. In this case, I think they are telling us something about the way human reasoning can fail when faced with impacts beyond our normal experience: We are in danger of both over-estimating and under-estimating their effects, because our brains aren’t equipped to deal with magnitudes and probabilities on that scale. This has made me realize something rather important about both Singularitarianism and religion, but I’ll save that for next week’s post.