Jacobin

The trouble with Rationalism and Effective Altruism

PNRJ cognitive science, core principles, ethics, futurism, heuristics and biases, tribal paradigm , , , , , , , , , , , , , , , , , , , , ,

Oct 11 JDN 2461325

I have discussed before the notion of a motte and bailey, in which uses ambiguity to present a view in two different forms: A motte, which is highly defensible but often banal or uncontroversial, and a bailey, which is highly interesting and radical but ultimately indefensible. Like the medieval strategy, you want to live in the bailey, but you can only defend the motte.

I think the “rationalist” and “effective altruism” communities suffer quite severely from this dynamic.

I use the scare quotes precisely because many so-called “rationalists” deeply fail to conform to the principles of rationalism, and many so-called “effective altruists” don’t seem particularly altruistic.

This saddens me greatly, because the core principles of rationalism and effective altruism are not simply correct: They are among the best ideas that have ever been formulated by human minds.

The core of rationalism is this:

Reason is the best source of knowledge and truth.

The core of effective altruism is this:

The best acts are those that do the most good for the least cost.

Both of these ideas seem beyond defensible—more like unassailable, perhaps even undeniable, as in literally impossible to coherently deny. They have the ring of not simply truth, but deep truth, perhaps even capital-T Truth, fundamental laws that have the power and certainty of logical or mathematical theorems.

Moreover, they are not banal or uncontroversial! It is quite bizarre that they are not; it speaks of deep irrationality within human brains and human societies that anyone would even try to deny these things. Yet the very existence of religion as we know it proves that rationalism is not uncontroversial, and many people still cling to moral systems that condemn people for harmless acts or create needless suffering.

This is the motte, and it is a fine motte indeed: Beautiful, imposing, perfectly-engineered, and utterly impregnable. The closest real-world equivalent is something like the walls of Troy, which were (as the story goes) invincible against conventional force and could only be defeated by deception. (Or perhaps if you consider all of Japan a fortress: Conquering that island nation from the outside required air power and nuclear weapons.)

I love this motte will all my heart, but I am deeply concerned about the bailey.

What is the bailey of rationalism and effective altruism?

First, let me address the parts that I am not too concerned about; I think many people go a bit too far with them, but I believe they are basically sound:

  • Bayesianism: The best way to reason about uncertainty is using Bayesian statistics.
  • Utilitarianism: Morality consists in doing the greatest good for the greatest number.
  • Expected utility: The best way to make decisions under uncertainty is to maximize expected utility.
  • Behavioral economics: Economic behavior is best understood by recognizing that human beings use heuristics and are subject to biases that prevent them from being fully rational.
  • Longtermism: One of our most important moral concerns is what kind of world we are making for future generations of humanity.

As I have stated them, I consider these views to be basically sound. Now, let me rephrase them in ways that they are often taken too far:

  • McNamara fallacy: Only what you can mathematically quantify matters, and if you are not explicitly computing a posterior distribution from a prior distribution and a likelihood, you are not making a rational judgment under uncertainty.
  • Act utilitarianism: Morality consists in taking each action in isolation and maximizing the utility it creates, without regard for the deep uncertainty and complex ramifications involved in doing so.
  • Shut up and multiply: Regardless of whether it violates common sense, rests upon flimsy assumptions, or involves deep uncertainty and unacceptable risk, the proper thing to do when making decisions is always to go with what you calculate yields the highest expected utility.
  • Fallacy fallacy: If your opponent is reasoning in a fallacious way, you are free to completely dismiss their beliefs.
  • Utopian longtermism: As long as you can convince yourself that what you are doing will provide some miniscule increase in the probability of a utopian future for our galaxy-colonizing descendants, you are free to ignore the consequences of your actions for people who are alive today.

These are already a big problem, especially the last one. I have seen far too many “rationalists” and “effective altruists” pivot away from projects that could have done enormous good today with a high degree of certainty (such as improving economic policy, or supporting environmental causes, or donating to medical research or malaria prevention) in favor of trying to advance AI (often in the name of “AI safety”, but for some reason never actually involving regulating anything) toward some pie-in-the-sky vision of a utopian future.

In particular, I am deeply concerned about people who attach huge payoffs to tiny probabilities, and then convince themselves that “expected utility” means they must care about those things more than anything else. This is essentially a secular version of Pascal’s Wager, and it can drive people to terrible, terrible things.

It is far better for you to do small good things that are certain. Yes, we should care about people thousands of years from now; but the truth is, we really have no idea what the future will be like thousands of years from now. So how about we do our best to help the people who are here, now?

If you don’t find this argument convincing, consider this:

Helping people now can push us toward a brighter future.

Suppose, for the sake of argument, that we are truly most concerned about what happens thousands of years from now when humans have colonized the galaxy. What sort of society would you want to be responsible for that colonization?

Would it be one where people used esoteric and highly-uncertain mathematical arguments to convince themselves that they should abuse and exploit other people in the name of some unprovable “greater good”?

Or would it be one where people helped each other, locally and globally, showed compassion for all living things, and sought to help everyone achieve their full potential?

I contend that spreading kindness and compassion might very well be the very best thing you could do to make humanity’s future brighter. It will not only increase the probability that humanity survives long enough to colonize the galaxy, but also increase the probability that the society which does so is a good society, made of people who are kind and compassionate.

Sure, it wouldn’t hurt to donate to, or work for, AI safety or advocacy for AI regulation. I’m not saying you shouldn’t do those things. But they are not the one most important thing—and indeed, anyone who tells you that something is the one most important thing is basically trying to hack your hedonic mind.

But there is more to the bailey here, and it gets worse.

The majority of people who identify as rationalists and/or effective altruists also seem to believe the following things:

  • Libertarianism: Government is always the problem, never the solution. We should always trust in free markets rather that democratic governments.
  • Prediction markets: Rather than trust experts or even admit that some things are fundamentally unpredictable, we should base our decisions on the results of prediction markets, and we should have prediction markets for virtually everything.
  • AI-doomism: The end of the world is nigh, and this is a terrible thing; any day now, artificial general intelligence will take over the world and wipe out humanity.
  • Singularitarianism: The end of the world as we know it is nigh, and this is a good thing; any day now, artificial general intelligence will take over the world and lead us into a glorious utopian future.

This excellent and eye-opening video on the conference Manifest does a nice job of expressing just how weird and dangerous the culture around these beliefs can become. (And while they are much rarer, there have absolutely been people who tried to adopt rationalist ideas and became literal cults.)

Of course AI-doomism and Singularitarianism are contradictory, but in an odd sort of way: Both sides agree that AGI is imminent and will change everything; they simply disagree about whether this will turn out very good or very bad. As a result, many people who seem to assign a higher-than-normal probability of AI doom also assign a higher-than-normal probability of Singularity. This is not incoherent (as long as all their probabilities add to 1, of course); it simply means that they assign a high probability to AGI being imminent and world-changing, and aren’t sure whether that will turn out good or bad.

(I was once cited as a “Singularitarian blogger”, for this post; I am deeply flattered to be thus cited, but I really don’t think of myself as a Singularitarian. There is a weak sense of Singularitarianism that I guess I agree with: Artificial intelligence will, sooner than most people think, effect radical changes upon human society, and this could be a very good thing if it is handled correctly. But I do not believe we are on the cusp of the Culture.)

The weirdest part, for me at least, is how libertarianism and AI-doomism collide: People will tell me that they believe that AI has a greater than 10%—some say 50% or even 90%—chance of imminently wiping out the human race, and yet they reject any attempt to ban or even significantly regulate AI.

I believe that there is a less than 1% chance of AI wiping out the human race, and yet I believe that we urgently need severe regulations on AI that would substantially slow its development and limit its capabilities. (I think Dynomight makes the best argument for this risk being genuine, cutting through a lot of irrelevant nonsense. As it is, I actually think IQ-300 aliens would likely be better than AGI, because a species of genuinely-intelligent beings (as opposed to computer software that simulates but probably doesn’t actually exhibit intelligence) that is advanced enough to achieve interstellar travel is likely to be peaceful and cooperative enough that they would do us more good than harm. Also, Dynomight’s timeline of “the next few decades” is a lot more plausible than the panicky doomers who say “the next few years”.)

In part I want to regulate AI more because I can already see ways that AI is causing harm (this Jacobin article provides a nice summary), but even without those things, a 1% chance of total annihilation is well worth a great deal of regulation and slowed innovation. Indeed, let me get on record saying that reducing the chances of human annihilation by one percentage point would be well worth slowing economic and technological development by a century.

Libertarianism and prediction markets of course go quite well together, and indeed I rarely encounter one without the other. But this faith in the market is deeply misplaced—especially if we believe, as I do, in behavioral economics!—and in particular, prediction markets are in fact fundamentally flawed as a means of making decisions because of how they handle conditional probabilities. It’s actually kind of a corollary of Goodhart’s Law: Once a measure becomes a target, it ceases to be a good measure, because rational agents try to game the system. Prediction markets can work very well if the outcome is totally beyond human control and wouldn’t totally render the outcome moot (e.g. an earthquake); but once human decisions can affect the result, the prediction market effectively becomes insider trading, and for really big things (like existential risk), the outcome itself could easily render the prediction market—or even money itself—utterly irrelevant.

We must defend the motte with all our might—but perhaps it’s best to abandon some of this bailey.

Yes, by all means, abandon religion, learn to think more rationally, support AI safety, and do what you can to do the most good for the least cost. Those things are good, and right, and important.

But if you find yourself thinking that you should sacrifice yourself, or abandon ordinary concepts of morality, in the name of some ultimate good that will save us from imminent destruction, you need to take a step back, and re-immerse yourself in the rest of society. You are not going to single-handedly save the world; nobody is. If the world is to be saved, we will all do it together.

Reflections on the Charlie Kirk assassination

PNRJ core principles, current events, politics, tribal paradigm , , , , , , , , , , , , , , , ,

Sep 28 JDN 2460947

No doubt you are well aware that Charlie Kirk was shot and killed on September 10. His memorial service was held on September 21, and filled a stadium in Arizona.

There have been a lot of wildly different takes on this event. It’s enough to make you start questioning your own sanity. So while what I have to say may not be that different from what Krugman (or for that matter Jacobin) had to say, I still thought I would try to contribute to the small part of the conversation that’s setting the record straight.

First of all, let me say that this is clearly a political assassination, and as a matter of principle, that kind of thing should not be condoned in a democracy.

The whole point of a democratic system is that we don’t win by killing or silencing our opponents, we win by persuading or out-voting them. As long as someone is not engaging in speech acts that directly command or incite violence (like, say, inciting people to attack the Capitol), they should be allowed to speak in peace; even abhorrent views should be not be met with violence.

Free speech isn’t just about government censorship (though that is also a major problem right now); it’s a moral principle that underlies the foundation of liberal democracy. We don’t resolve conflicts with violence unless absolutely necessary.

So I want to be absolutely clear about this: Killing Charlie Kirk was not acceptable, and the assassin should be tried in a court of law and, if duly convicted, imprisoned for a very long time.

Second of all, we still don’t know the assassin’s motive, so stop speculating until we do.

At first it looked like the killer was left-wing. Then it looked like maybe he was right-wing. Now it looks like maybe he’s left-wing again. Maybe his views aren’t easily categorized that way; maybe he’s an anarcho-capitalist, or an anarcho-communist, or a Scientologist. I won’t say it doesn’t matter; it clearly does matter. But we simply do not know yet.

There is an incredibly common and incredibly harmful thing that people do after any major crime: They start spreading rumors and speculating about things that we actually know next to nothing about. Stop it. Don’t contribute to that.


The whole reason we have a court system is to actually figure out the real truth, which takes a lot of time and effort. The courts are one American institution that’s actually still functioning pretty well in this horrific cyberpunk/Trumpistan era; let them do their job.

It could be months or years before we really fully understand what happened here. Accept that. You don’t need to know the answer right now, and it’s far more dangerous to think you know the answer when you actually don’t.

But finally, I need to point out that Charlie Kirk was an absolutely abhorrent, despicable husk of a human being and no one should be honoring him.

First of all, he himself advocated for political violence against his opponents. I won’t say anyone deserves what happened to him—but if anyone did, it would be him, because he specifically rallied his followers to do exactly this sort of thing to other people.

He was also bigoted in almost every conceivable way: Racist, sexist, ableist, homophobic, and of course transphobic. He maintained a McCarthy-esque list of college professors that he encouraged people to harass for being too left-wing. He was a covert White supremacist, and only a little bit covert. He was not covert at all about his blatant sexism and misogyny that seems like it came from the 1950s instead of the 2020s.

He encouraged his—predominantly White, male, straight, cisgender, middle-class—audience to hate every marginalized group you can think of: women, people of color, LGBT people, poor people, homeless people, people with disabilities. Not content to merely be an abhorrent psychopath himself, he actively campaigned against the concept of empathy.

Charlie Kirk deserves no honors. The world is better off without him. He made his entire career out of ruining the lives of innocent people and actively making the world a worse place.

It was wrong to kill Charlie Kirk. But if you’re sad he’s gone, what is wrong with you!?

Why nonviolence?

PNRJ core principles, current events, ethics , , , , , , , , ,

Aug 16 JDN 2459078

You are no doubt aware that there are widespread protests going on right now. You may even have marched in some of them. Nearly 30 million Americans have participated in the Black Lives Matter protests, located in cities all around the country; this makes them quite likely the largest protests in American history.

The right wing is of course making much of the isolated incidents of violence that have occurred, often but not always actually provoked by the police or federal agents assigned to quell the protests. They have also made much of the property destruction caused by riots that have emerged from the protests, typically eliding the distinction between property destruction and violence. Since there has been far more property destruction than actual violence, this allows them to effectively inflate the level of violence.

In reality, the total deaths caused by these protests over two months and counting is clearly less than the number of Americans who are shot by police in an average week. And the total amount of property destruction is clearly less than the tens of billions of dollars per year that are stolen in wage theft, let alone the hundreds of billions of dollars per year that are stolen by white-collar crime. If violence and loss of property are really what you care about, these protests should not be your main concern.

Yet, I am concerned that too many on the left are too willing to accept violence. I have seen far too many people sharing and endorsing this quote:

“Dr. King’s policy was that nonviolence would achieve the gains for black people in the United States. His major assumption was that if you are nonviolent, if you suffer, your opponent will see your suffering and will be moved to change his heart. That’s very good. He only made one fallacious assumption: In order for nonviolence to work, your opponent must have a conscience. The United States has none.”

~ Stokely Carmichael

Nonviolence does work. Nonviolence did work for the civil rights movement. No, it doesn’t depend upon your opponent having a conscience—it only depends upon bystanders having a conscience. (Also, “the United States has no conscience” is only true insofar as socially constructed institutions don’t have feelings. Clearly most of the people in the United States—probably even most of the people in the US government—have a conscience!)

In fact, nonviolent protest is typically more effective than violent protest. When protesters turn to violence, they alienate the public whose support they need, and they allow the government to feel justified in responding with even more force. Campaigns of nonviolent civil resistance have been historically more effective than violent revolutions, even against authoritarian governments. On average, nonviolent protests are twice as likely to achieve their goals than violent protests.

Even worse than the Carmichael quote are the memes that have been shared saying things like this: “You want to fix the system, but not use violence; so, by magic?”

Nonviolence doesn’t mean politely asking for rights. It doesn’t mean being calm and non-confrontational. It doesn’t mean waiting patiently.

Nonviolence doesn’t even mean following the law or never damaging property. Some of the most important and effective acts of nonviolent protest involved breaking laws and damaging things—Rosa Parks was breaking the law, and does the Boston Tea Party ring a bell?

Nonviolence doesn’t even mean that nobody gets hurt; it often means strategically placing your own people in harm’s way knowing that the government’s violent overreaction will stir support for your cause. It’s a kind of ethical and political judo: Instead of directly resisting your stronger opponent, you maneuver so that their own power ends up damaging them. You use the government’s repression as a weapon for your own cause.

What does nonviolence mean?

Nonviolence means you don’t hurt people.

It sounds so simple and obvious, but a lot of people still don’t seem to get it. They seem to think that our only choices are “ask nicely” or “start a civil war”. Asking nicely obviously would not be effective; only someone deeply naive could imagine otherwise. Working legally within the system can sometimes be effective, but when really deep reforms are needed urgently it is often not enough. Starting a civil war might work—it has sometimes worked in the past—but it would come at a horrendous cost, probably thousands if not millions of lives.

Fortunately, these are not our only options. We don’t have to ask nicely; we don’t even have to obey the law. We can even break things. We just need to not hurt people. That still allows for a variety of forms of protest, confrontation, civil disobedience, and direct action. Jacobin, oddly enough, gets this right.

In reality, any movement is going to have extremists who act violently. A protest movement can still be considered nonviolent as long as such incidents of violence can be kept to a minimum, and never condoned by the leaders of the movement. Thus far, Black Lives Matter has absolutely fit that description—indeed, impressively so, given the sheer scale of the protests.

Some degree of self-defense can even be consistent with nonviolence, though it must be of a very minimal sort. Wearing armor and carrying a shield is entirely consistent with nonviolence. Hitting back after you are hit is a finer line. This is morally still nonviolence as long as you use only the minimal necessary force—but politically it will only work if the public clearly knows that you are not the ones who hit first.

The ethical case for nonviolence is simple, but worth repeating: Human lives have intrinsic value. Yes, even if those human beings work willingly for a corrupt and evil system. Yes, even the average Nazi was a sentient being of intrinsic moral worth.

The only people who really deserve to die are the psychopaths at the top pulling the strings—and they are almost never the ones on the front lines getting shot or bombed. If you had a plan to kill Donald Trump, I would have no particular moral objection. I think such a plan would be very unlikely to succeed, and I would never attempt such a thing myself; but does Donald Trump deserve to die for his brazen authoritarianism, overwhelming corruption, and depraved indifference for over 160,000 dead Americans? Yes. But how does that justify killing random police officers?

Nonviolence also has another great advantage, which is that it works better when you are on the right side. The effectiveness of violence is proportional to your firepower; the effectiveness of nonviolence is proportional to your righteousness. Why in the world would you, who are righteous but have little firepower, want to use violence against an enemy that is unrighteous and has more firepower?

Nonviolent protest actually works best when your enemy is violent and repressive; it is precisely that contrast between your nonviolence and their violence that wins people to your cause. Probably the smartest thing a government could do to respond to nonviolent protests would be to sit back and calmly watch them, then make whatever was the minimal level of concessions in order to make the protests lose momentum. When you bring out the tear gas, you have basically already admitted that you are on the wrong side of history. But repressive governments don’t think that way; if they did, they would have given those same concessions before the protests even gathered steam. They imagine that by simply cracking down harder they will be able to win—but they are usually wrong.

And even if the ethical case for nonviolence means literally nothing to you, please consider the strategic case: The empirical data says quite clearly that nonviolent protest works better. In many ways, violence is the default; it’s the conflict revolution mechanism that we evolved to use, largely unmodified from the same instincts that motivate any other primate. Nonviolence is a recent invention, a high-tech solution to this ancient problem. Violence is easy; just about anyone can do it. Nonviolence is hard; it requires strategic cleverness, unwavering vigilance, and deep moral courage.

This is not to say that violence is never necessary: Against a truly totalitarian regime that is willing to murder people simply for speaking out against the government, violence may well be the only option. I certainly do not begrudge the French Resistance for using violence against the Nazis. But violence should be a last resort, not simply for ethical reasons—but also for strategic reasons.