rationalism

The trouble with Rationalism and Effective Altruism

PNRJ cognitive science, core principles, ethics, futurism, heuristics and biases, tribal paradigm , , , , , , , , , , , , , , , , , , , , ,

Oct 11 JDN 2461325

I have discussed before the notion of a motte and bailey, in which uses ambiguity to present a view in two different forms: A motte, which is highly defensible but often banal or uncontroversial, and a bailey, which is highly interesting and radical but ultimately indefensible. Like the medieval strategy, you want to live in the bailey, but you can only defend the motte.

I think the “rationalist” and “effective altruism” communities suffer quite severely from this dynamic.

I use the scare quotes precisely because many so-called “rationalists” deeply fail to conform to the principles of rationalism, and many so-called “effective altruists” don’t seem particularly altruistic.

This saddens me greatly, because the core principles of rationalism and effective altruism are not simply correct: They are among the best ideas that have ever been formulated by human minds.

The core of rationalism is this:

Reason is the best source of knowledge and truth.

The core of effective altruism is this:

The best acts are those that do the most good for the least cost.

Both of these ideas seem beyond defensible—more like unassailable, perhaps even undeniable, as in literally impossible to coherently deny. They have the ring of not simply truth, but deep truth, perhaps even capital-T Truth, fundamental laws that have the power and certainty of logical or mathematical theorems.

Moreover, they are not banal or uncontroversial! It is quite bizarre that they are not; it speaks of deep irrationality within human brains and human societies that anyone would even try to deny these things. Yet the very existence of religion as we know it proves that rationalism is not uncontroversial, and many people still cling to moral systems that condemn people for harmless acts or create needless suffering.

This is the motte, and it is a fine motte indeed: Beautiful, imposing, perfectly-engineered, and utterly impregnable. The closest real-world equivalent is something like the walls of Troy, which were (as the story goes) invincible against conventional force and could only be defeated by deception. (Or perhaps if you consider all of Japan a fortress: Conquering that island nation from the outside required air power and nuclear weapons.)

I love this motte will all my heart, but I am deeply concerned about the bailey.

What is the bailey of rationalism and effective altruism?

First, let me address the parts that I am not too concerned about; I think many people go a bit too far with them, but I believe they are basically sound:

  • Bayesianism: The best way to reason about uncertainty is using Bayesian statistics.
  • Utilitarianism: Morality consists in doing the greatest good for the greatest number.
  • Expected utility: The best way to make decisions under uncertainty is to maximize expected utility.
  • Behavioral economics: Economic behavior is best understood by recognizing that human beings use heuristics and are subject to biases that prevent them from being fully rational.
  • Longtermism: One of our most important moral concerns is what kind of world we are making for future generations of humanity.

As I have stated them, I consider these views to be basically sound. Now, let me rephrase them in ways that they are often taken too far:

  • McNamara fallacy: Only what you can mathematically quantify matters, and if you are not explicitly computing a posterior distribution from a prior distribution and a likelihood, you are not making a rational judgment under uncertainty.
  • Act utilitarianism: Morality consists in taking each action in isolation and maximizing the utility it creates, without regard for the deep uncertainty and complex ramifications involved in doing so.
  • Shut up and multiply: Regardless of whether it violates common sense, rests upon flimsy assumptions, or involves deep uncertainty and unacceptable risk, the proper thing to do when making decisions is always to go with what you calculate yields the highest expected utility.
  • Fallacy fallacy: If your opponent is reasoning in a fallacious way, you are free to completely dismiss their beliefs.
  • Utopian longtermism: As long as you can convince yourself that what you are doing will provide some miniscule increase in the probability of a utopian future for our galaxy-colonizing descendants, you are free to ignore the consequences of your actions for people who are alive today.

These are already a big problem, especially the last one. I have seen far too many “rationalists” and “effective altruists” pivot away from projects that could have done enormous good today with a high degree of certainty (such as improving economic policy, or supporting environmental causes, or donating to medical research or malaria prevention) in favor of trying to advance AI (often in the name of “AI safety”, but for some reason never actually involving regulating anything) toward some pie-in-the-sky vision of a utopian future.

In particular, I am deeply concerned about people who attach huge payoffs to tiny probabilities, and then convince themselves that “expected utility” means they must care about those things more than anything else. This is essentially a secular version of Pascal’s Wager, and it can drive people to terrible, terrible things.

It is far better for you to do small good things that are certain. Yes, we should care about people thousands of years from now; but the truth is, we really have no idea what the future will be like thousands of years from now. So how about we do our best to help the people who are here, now?

If you don’t find this argument convincing, consider this:

Helping people now can push us toward a brighter future.

Suppose, for the sake of argument, that we are truly most concerned about what happens thousands of years from now when humans have colonized the galaxy. What sort of society would you want to be responsible for that colonization?

Would it be one where people used esoteric and highly-uncertain mathematical arguments to convince themselves that they should abuse and exploit other people in the name of some unprovable “greater good”?

Or would it be one where people helped each other, locally and globally, showed compassion for all living things, and sought to help everyone achieve their full potential?

I contend that spreading kindness and compassion might very well be the very best thing you could do to make humanity’s future brighter. It will not only increase the probability that humanity survives long enough to colonize the galaxy, but also increase the probability that the society which does so is a good society, made of people who are kind and compassionate.

Sure, it wouldn’t hurt to donate to, or work for, AI safety or advocacy for AI regulation. I’m not saying you shouldn’t do those things. But they are not the one most important thing—and indeed, anyone who tells you that something is the one most important thing is basically trying to hack your hedonic mind.

But there is more to the bailey here, and it gets worse.

The majority of people who identify as rationalists and/or effective altruists also seem to believe the following things:

  • Libertarianism: Government is always the problem, never the solution. We should always trust in free markets rather that democratic governments.
  • Prediction markets: Rather than trust experts or even admit that some things are fundamentally unpredictable, we should base our decisions on the results of prediction markets, and we should have prediction markets for virtually everything.
  • AI-doomism: The end of the world is nigh, and this is a terrible thing; any day now, artificial general intelligence will take over the world and wipe out humanity.
  • Singularitarianism: The end of the world as we know it is nigh, and this is a good thing; any day now, artificial general intelligence will take over the world and lead us into a glorious utopian future.

This excellent and eye-opening video on the conference Manifest does a nice job of expressing just how weird and dangerous the culture around these beliefs can become. (And while they are much rarer, there have absolutely been people who tried to adopt rationalist ideas and became literal cults.)

Of course AI-doomism and Singularitarianism are contradictory, but in an odd sort of way: Both sides agree that AGI is imminent and will change everything; they simply disagree about whether this will turn out very good or very bad. As a result, many people who seem to assign a higher-than-normal probability of AI doom also assign a higher-than-normal probability of Singularity. This is not incoherent (as long as all their probabilities add to 1, of course); it simply means that they assign a high probability to AGI being imminent and world-changing, and aren’t sure whether that will turn out good or bad.

(I was once cited as a “Singularitarian blogger”, for this post; I am deeply flattered to be thus cited, but I really don’t think of myself as a Singularitarian. There is a weak sense of Singularitarianism that I guess I agree with: Artificial intelligence will, sooner than most people think, effect radical changes upon human society, and this could be a very good thing if it is handled correctly. But I do not believe we are on the cusp of the Culture.)

The weirdest part, for me at least, is how libertarianism and AI-doomism collide: People will tell me that they believe that AI has a greater than 10%—some say 50% or even 90%—chance of imminently wiping out the human race, and yet they reject any attempt to ban or even significantly regulate AI.

I believe that there is a less than 1% chance of AI wiping out the human race, and yet I believe that we urgently need severe regulations on AI that would substantially slow its development and limit its capabilities. (I think Dynomight makes the best argument for this risk being genuine, cutting through a lot of irrelevant nonsense. As it is, I actually think IQ-300 aliens would likely be better than AGI, because a species of genuinely-intelligent beings (as opposed to computer software that simulates but probably doesn’t actually exhibit intelligence) that is advanced enough to achieve interstellar travel is likely to be peaceful and cooperative enough that they would do us more good than harm. Also, Dynomight’s timeline of “the next few decades” is a lot more plausible than the panicky doomers who say “the next few years”.)

In part I want to regulate AI more because I can already see ways that AI is causing harm (this Jacobin article provides a nice summary), but even without those things, a 1% chance of total annihilation is well worth a great deal of regulation and slowed innovation. Indeed, let me get on record saying that reducing the chances of human annihilation by one percentage point would be well worth slowing economic and technological development by a century.

Libertarianism and prediction markets of course go quite well together, and indeed I rarely encounter one without the other. But this faith in the market is deeply misplaced—especially if we believe, as I do, in behavioral economics!—and in particular, prediction markets are in fact fundamentally flawed as a means of making decisions because of how they handle conditional probabilities. It’s actually kind of a corollary of Goodhart’s Law: Once a measure becomes a target, it ceases to be a good measure, because rational agents try to game the system. Prediction markets can work very well if the outcome is totally beyond human control and wouldn’t totally render the outcome moot (e.g. an earthquake); but once human decisions can affect the result, the prediction market effectively becomes insider trading, and for really big things (like existential risk), the outcome itself could easily render the prediction market—or even money itself—utterly irrelevant.

We must defend the motte with all our might—but perhaps it’s best to abandon some of this bailey.

Yes, by all means, abandon religion, learn to think more rationally, support AI safety, and do what you can to do the most good for the least cost. Those things are good, and right, and important.

But if you find yourself thinking that you should sacrifice yourself, or abandon ordinary concepts of morality, in the name of some ultimate good that will save us from imminent destruction, you need to take a step back, and re-immerse yourself in the rest of society. You are not going to single-handedly save the world; nobody is. If the world is to be saved, we will all do it together.

Inerrancy is an absurdly strong claim

PNRJ core principles, current events, ethics , , , , , , , , , , ,

Apr 12 JDN 246143

I’ve had a really hard time writing a post this week. Between my late father’s birthday coming up soon (April 15) and the fact that a man with full authority over a full-scale nuclear arsenal threatened to destroy an entire civilization—a literal imminent threat of genocide—and the people with the power to remove him did absolutely nothing, the world just feels like a nightmare I’ve been trying to wake up from.

And yes, it matters that he has authority over nukes. If you’re in a fistfight and the other guy says, “I’ll kill you!” that’s very different than if he draws a handgun and says the same thing. The President of the United States should essentially be treated as always brandishing a deadly weapon, and it is his responsibility to have the decorum to not make statements that can be read as imminent threats.

This means that trying to be topical about current events is just too painful and disorienting for me to write anything that feels useful to say. (I mostly feel like screaming.)

So, perhaps ironically, I’m going to write a post that’s completely un-topical, that could honestly have been written any time between roughly 300 AD and the present, and—much to my chagrin—will probably still be relevant in 3000 AD if humanity survives that long.

It concerns the doctrine of scriptural inerrancy.

Simply put, inerrancy is the belief that divine scriptures (especially the Bible or Qur’an) are without error: That is, that literally every proposition contained therein is absolutely and completely true.


This is not by any means a rare or fringe belief. 55% of Christians in the United States report that they believe in Biblical inerrancy. I was not able to obtain a similar figure for Muslims, but I can tell you that a majority of US Muslims and over 90% of African Muslims believe in the even stronger claim of Qur’anic literalism.

We can set aside the question of copyediting. I don’t care about typos or grammar mistakes. Translation errors are somewhat more serious—as they can affect real doctrines—but I’m willing to set those aside as well. We can say that we are talking about the original texts in their original languages, and idealized so that they do not contain any errors of grammar or typography.

This is already asking a lot, but I am prepared to concede it.

Even so, inerrancy is an absurdly strong claim that no rational person should ever take seriously.

The claim is that this entire text—hundreds of pages by dozens of authors over hundreds of years—is entirely true, without a single false assertion anywhere within it.

I want to be absolutely clear about this: I do not believe that about any text I have ever encountered.

I do not believe that The Origin of Species is inerrant. I do not believe that calculus textbooks are inerrant. I do not believe that Einstein’s 1905 paper on special relativity is inerrant.

I can’t point you to any specific errors in these books right now (especially since we haven’t even specified a calculus textbook), but if someone did point me to an error, I would not be the least bit surprised. Even if I combed through the entire text multiple times and didn’t spot any errors, I would still be doubtful that I hadn’t missed one somewhere.

Honestly, I find it improbable that any nonfiction work by human beings of significant length and complexity is completely without errors. (Okay, a 5-page book on counting for kindergartners might actually be inerrant. Maybe.)

Let me try putting it this way. What is the probability that any given proposition stated by a given source is correct? For a very reliable source, it could be 99%, or 99.9%, or even 99.99%. Perhaps you literally trust some sources so much that they must assert 10,000propositions before they get one wrong. (I’m not sure there’s anyone I trust this much—I certainly do not trust myself this much—but I’ll allow it for the sake of argument.)

There are 30,000 verses in the Bible. Many of these verses assert multiple propositions.

Even if each and every proposition is 99.99% reliable, the probability that all of 30,000 distinct propositions is correct is less than 5%. Even if you trust the Bible that much, you should still be 95% certain that it got something wrong somewhere.

In fact, it’s much worse than that, as we know for a fact that there are explicit contradictions between different parts of the Bible. The Skeptic’s Annotated Bible counts over 500 explicit contradictions, some relatively trivial (did Enoch die?) but others absolutely core to Christian theology (do Heaven and Hell exist?). If even one of those holds up—and as far as I can tell, most of them hold up, maybe even all of them (though I wouldn’t be surprised if some don’t; are you getting it yet?)—then the Bible is not inerrant. Indeed, just counting contradictions, if 500 of 30,000 propositions are contradictions, then the accuracy of each proposition can’t be more than 99%.

We don’t even need the extensive empirical evidence that refutes the creation stories in the Bible to know that those stories are wrong. The creation stories themselves contradict each other in vital ways.

We don’t need to consider the vanishingly small prior probability that a human being can rise from the dead to take issue with the resurrection story. Simon and Peter can’t both simultaneously have known in advance that Jesus would resurrect and not known that until it happened. Jesus can’t have been crucified to death both before and after Passover.

Some of these kinds of contradictions are exactly the sort of thing you would expect to slip into a historical account that was delivered by oral tradition over multiple generations. (They do not, for instance, give me reason to doubt that there was in fact a historical figure named Yeshua of Nazareth who gathered a group of apostles and was crucified to death by the Roman state. The vast majority of historians agree that this man did, in fact, exist.)

But they are exactly what you are not allowed to have in a book that is inerrant!

A book that is literally without error, without flaw, should not contain even one single contradiction, no matter how trivial—and come on, whether or not Heaven and Hell exist is not trivial!

Inerrancy is not simply saying “the Bible is basically true” or “the Bible is a reliable source” or even “Christian theology is true.”

I believe that The Origin of Species is basically true, and a reliable source, and that Darwinian evolutionary theory is true. But I absolutely do not believe that The Origin of Species is inerrant.

I believe that most calculus textbooks are basically true, and are reliable sources, and that the Fundamental Theorem of Calculus is true. But I absolutely do not believe that any calculus textbook is inerrant.

Inerrancy isn’t even simply saying that “the Bible was written by God”! It’s very clear that the Bible is not simply dictated verbatim from On High; there was some kind of human process involved in its creation, and even if you believe that the Council of Nicaea was right about all their choices of the canon, you should still recognize that there is plenty of room for errors to have crept in during this long, convoluted, and controversial human process.

(For the Qur’an, we actually have mostly the original text by the original author—but even then, you should still be doubtful that any document with thousands of claims could be absolutely, 100% true.)

So, please, Christians, Muslims, and everyone else, I am literally begging you:

Please, give up on inerrancy. Admit that your book could be mistaken.

I’m not asking you to give up on your religion. You can keep your theology. You can still mostly believe in the book. But please, recognize how incredibly unreasonable you are being by asserting that it is impossible that anything in the book could ever be wrong.

I beseech you, in the bowels of Christ, think it possible that your book could be mistaken.

Administering medicine to the dead

PNRJ core principles, ethics, personal, tribal paradigm , , , , , , , , ,

Jan 28 JDN 2460339

Here are a couple of pithy quotes that go around rationalist circles from time to time:

“To argue with a man who has renounced the use and authority of reason, […] is like administering medicine to the dead[…].”

― Thomas Paine, The American Crisis

“It is useless to attempt to reason a man out of a thing he was never reasoned into.”

― Jonathan Swift

You usually hear that abridged version, but Thomas Paine’s full quotation is actually rather interesting:

“To argue with a man who has renounced the use and authority of reason, and whose philosophy consists in holding humanity in contempt, is like administering medicine to the dead, or endeavoring to convert an atheist by scripture.”

― Thomas Paine, The American Crisis

It is indeed quite ineffective to convert an atheist by scripture (though that doesn’t seem to stop them from trying). Yet this quotation seems to claim that the opposite should be equally ineffective: It should be impossible to convert a theist by reason.

Well, then, how else are we supposed to do it!?

Indeed, how did we become atheists in the first place!?

You were born an atheist? No, you were born having absolutely no opinion about God whatsoever. (You were born not realizing that objects don’t fade from existence when you stop seeing them! In a sense, we were all born believing ourselves to be God.)

Maybe you were raised by atheists, and religion never tempted you at all. Lucky you. I guess you didn’t have to be reasoned into atheism.

Well, most of us weren’t. Most of us were raised into religion, and told that it held all the most important truths of morality and the universe, and that believing anything else was horrible and evil and would result in us being punished eternally.

And yet, somehow, somewhere along the way, we realized that wasn’t true. And we were able to realize that because people made rational arguments.

Maybe we heard those arguments in person. Maybe we read them online. Maybe we read them in books that were written by people who died long before we were born. But somehow, somewhere people actually presented the evidence for atheism, and convinced us.

That is, they reasoned us out of something that we were not reasoned into.

I know it can happen. I have seen it happen. It has happened to me.

And it was one of the most important events in my entire life. More than almost anything else, it made me who I am today.

I’m scared that if you keep saying it’s impossible, people will stop trying to do it—and then it will stop happening to people like me.

So please, please stop telling people it’s impossible!

Quotes like these encourage you to simply write off entire swaths of humanity—most of humanity, in fact—judging them as worthless, insane, impossible to reach. When you should be reaching out and trying to convince people of the truth, quotes like these instead tell you to give up and consider anyone who doesn’t already agree with you as your enemy.

Indeed, it seems to me that the only logical conclusion of quotes like these is violence. If it’s impossible to reason with people who oppose us, then what choice do we have, but to fight them?

Violence is a weapon anyone can use.

Reason is the one weapon in the universe that works better when you’re right.

Reason is the sword that only the righteous can wield. Reason is the shield that only protects the truth. Reason is the only way we can ever be sure that the right people win—instead of just whoever happens to be strongest.

Yes, it’s true: reason isn’t always effective, and probably isn’t as effective as it should be. Convincing people to change their minds through rational argument is difficult and frustrating and often painful for both you and them—but it absolutely does happen, and our civilization would have long ago collapsed if it didn’t.

Even people who claim to have renounced all reason really haven’t: they still know 2+2=4 and they still look both ways when they cross the street. Whatever they’ve renounced, it isn’t reason; and maybe, with enough effort, we can help them see that—by reason, of course.

In fact, maybe even literally administering medicine to the dead isn’t such a terrible idea.

There are degrees of death, after all: Someone whose heart has stopped is in a different state than someone whose cerebral activity has ceased, and both of them clearly stand a better chance of being resuscitated than someone who has been vaporized by an explosion.

As our technology improves, more and more states that were previously considered irretrievably dead will instead be considered severe states of illness or injury from which it is possible to recover. We can now restart many stopped hearts; we are working on restarting stopped brains. (Of course we’ll probably never be able to restore someone who got vaporized—unless we figure out how to make backup copies of people?)

Most of the people who now live in the world’s hundreds of thousands of ICU beds would have been considered dead even just 100 years ago. But many of them will recover, because we didn’t give up on them.

So don’t give up on people with crazy beliefs either.

They may seem like they are too far gone, like nothing in the world could ever bring them back to the light of reason. But you don’t actually know that for sure, and the only way to find out is to try.

Of course, you won’t convince everyone of everything immediately. No matter how good your evidence is, that’s just not how this works. But you probably will convince someone of something eventually, and that is still well worthwhile.

You may not even see the effects yourself—people are often loathe to admit when they’ve been persuaded. But others will see them. And you will see the effects of other people’s persuasion.

And in the end, reason is really all we have. It’s the only way to know that what we’re trying to make people believe is the truth.

Don’t give up on reason.

And don’t give up on other people, whatever they might believe.