Month: October 2026

The trouble with Rationalism and Effective Altruism

PNRJ cognitive science, core principles, ethics, futurism, heuristics and biases, tribal paradigm , , , , , , , , , , , , , , , , , , , , ,

Oct 11 JDN 2461325

I have discussed before the notion of a motte and bailey, in which uses ambiguity to present a view in two different forms: A motte, which is highly defensible but often banal or uncontroversial, and a bailey, which is highly interesting and radical but ultimately indefensible. Like the medieval strategy, you want to live in the bailey, but you can only defend the motte.

I think the “rationalist” and “effective altruism” communities suffer quite severely from this dynamic.

I use the scare quotes precisely because many so-called “rationalists” deeply fail to conform to the principles of rationalism, and many so-called “effective altruists” don’t seem particularly altruistic.

This saddens me greatly, because the core principles of rationalism and effective altruism are not simply correct: They are among the best ideas that have ever been formulated by human minds.

The core of rationalism is this:

Reason is the best source of knowledge and truth.

The core of effective altruism is this:

The best acts are those that do the most good for the least cost.

Both of these ideas seem beyond defensible—more like unassailable, perhaps even undeniable, as in literally impossible to coherently deny. They have the ring of not simply truth, but deep truth, perhaps even capital-T Truth, fundamental laws that have the power and certainty of logical or mathematical theorems.

Moreover, they are not banal or uncontroversial! It is quite bizarre that they are not; it speaks of deep irrationality within human brains and human societies that anyone would even try to deny these things. Yet the very existence of religion as we know it proves that rationalism is not uncontroversial, and many people still cling to moral systems that condemn people for harmless acts or create needless suffering.

This is the motte, and it is a fine motte indeed: Beautiful, imposing, perfectly-engineered, and utterly impregnable. The closest real-world equivalent is something like the walls of Troy, which were (as the story goes) invincible against conventional force and could only be defeated by deception. (Or perhaps if you consider all of Japan a fortress: Conquering that island nation from the outside required air power and nuclear weapons.)

I love this motte will all my heart, but I am deeply concerned about the bailey.

What is the bailey of rationalism and effective altruism?

First, let me address the parts that I am not too concerned about; I think many people go a bit too far with them, but I believe they are basically sound:

  • Bayesianism: The best way to reason about uncertainty is using Bayesian statistics.
  • Utilitarianism: Morality consists in doing the greatest good for the greatest number.
  • Expected utility: The best way to make decisions under uncertainty is to maximize expected utility.
  • Behavioral economics: Economic behavior is best understood by recognizing that human beings use heuristics and are subject to biases that prevent them from being fully rational.
  • Longtermism: One of our most important moral concerns is what kind of world we are making for future generations of humanity.

As I have stated them, I consider these views to be basically sound. Now, let me rephrase them in ways that they are often taken too far:

  • McNamara fallacy: Only what you can mathematically quantify matters, and if you are not explicitly computing a posterior distribution from a prior distribution and a likelihood, you are not making a rational judgment under uncertainty.
  • Act utilitarianism: Morality consists in taking each action in isolation and maximizing the utility it creates, without regard for the deep uncertainty and complex ramifications involved in doing so.
  • Shut up and multiply: Regardless of whether it violates common sense, rests upon flimsy assumptions, or involves deep uncertainty and unacceptable risk, the proper thing to do when making decisions is always to go with what you calculate yields the highest expected utility.
  • Fallacy fallacy: If your opponent is reasoning in a fallacious way, you are free to completely dismiss their beliefs.
  • Utopian longtermism: As long as you can convince yourself that what you are doing will provide some miniscule increase in the probability of a utopian future for our galaxy-colonizing descendants, you are free to ignore the consequences of your actions for people who are alive today.

These are already a big problem, especially the last one. I have seen far too many “rationalists” and “effective altruists” pivot away from projects that could have done enormous good today with a high degree of certainty (such as improving economic policy, or supporting environmental causes, or donating to medical research or malaria prevention) in favor of trying to advance AI (often in the name of “AI safety”, but for some reason never actually involving regulating anything) toward some pie-in-the-sky vision of a utopian future.

In particular, I am deeply concerned about people who attach huge payoffs to tiny probabilities, and then convince themselves that “expected utility” means they must care about those things more than anything else. This is essentially a secular version of Pascal’s Wager, and it can drive people to terrible, terrible things.

It is far better for you to do small good things that are certain. Yes, we should care about people thousands of years from now; but the truth is, we really have no idea what the future will be like thousands of years from now. So how about we do our best to help the people who are here, now?

If you don’t find this argument convincing, consider this:

Helping people now can push us toward a brighter future.

Suppose, for the sake of argument, that we are truly most concerned about what happens thousands of years from now when humans have colonized the galaxy. What sort of society would you want to be responsible for that colonization?

Would it be one where people used esoteric and highly-uncertain mathematical arguments to convince themselves that they should abuse and exploit other people in the name of some unprovable “greater good”?

Or would it be one where people helped each other, locally and globally, showed compassion for all living things, and sought to help everyone achieve their full potential?

I contend that spreading kindness and compassion might very well be the very best thing you could do to make humanity’s future brighter. It will not only increase the probability that humanity survives long enough to colonize the galaxy, but also increase the probability that the society which does so is a good society, made of people who are kind and compassionate.

Sure, it wouldn’t hurt to donate to, or work for, AI safety or advocacy for AI regulation. I’m not saying you shouldn’t do those things. But they are not the one most important thing—and indeed, anyone who tells you that something is the one most important thing is basically trying to hack your hedonic mind.

But there is more to the bailey here, and it gets worse.

The majority of people who identify as rationalists and/or effective altruists also seem to believe the following things:

  • Libertarianism: Government is always the problem, never the solution. We should always trust in free markets rather that democratic governments.
  • Prediction markets: Rather than trust experts or even admit that some things are fundamentally unpredictable, we should base our decisions on the results of prediction markets, and we should have prediction markets for virtually everything.
  • AI-doomism: The end of the world is nigh, and this is a terrible thing; any day now, artificial general intelligence will take over the world and wipe out humanity.
  • Singularitarianism: The end of the world as we know it is nigh, and this is a good thing; any day now, artificial general intelligence will take over the world and lead us into a glorious utopian future.

This excellent and eye-opening video on the conference Manifest does a nice job of expressing just how weird and dangerous the culture around these beliefs can become. (And while they are much rarer, there have absolutely been people who tried to adopt rationalist ideas and became literal cults.)

Of course AI-doomism and Singularitarianism are contradictory, but in an odd sort of way: Both sides agree that AGI is imminent and will change everything; they simply disagree about whether this will turn out very good or very bad. As a result, many people who seem to assign a higher-than-normal probability of AI doom also assign a higher-than-normal probability of Singularity. This is not incoherent (as long as all their probabilities add to 1, of course); it simply means that they assign a high probability to AGI being imminent and world-changing, and aren’t sure whether that will turn out good or bad.

(I was once cited as a “Singularitarian blogger”, for this post; I am deeply flattered to be thus cited, but I really don’t think of myself as a Singularitarian. There is a weak sense of Singularitarianism that I guess I agree with: Artificial intelligence will, sooner than most people think, effect radical changes upon human society, and this could be a very good thing if it is handled correctly. But I do not believe we are on the cusp of the Culture.)

The weirdest part, for me at least, is how libertarianism and AI-doomism collide: People will tell me that they believe that AI has a greater than 10%—some say 50% or even 90%—chance of imminently wiping out the human race, and yet they reject any attempt to ban or even significantly regulate AI.

I believe that there is a less than 1% chance of AI wiping out the human race, and yet I believe that we urgently need severe regulations on AI that would substantially slow its development and limit its capabilities. (I think Dynomight makes the best argument for this risk being genuine, cutting through a lot of irrelevant nonsense. As it is, I actually think IQ-300 aliens would likely be better than AGI, because a species of genuinely-intelligent beings (as opposed to computer software that simulates but probably doesn’t actually exhibit intelligence) that is advanced enough to achieve interstellar travel is likely to be peaceful and cooperative enough that they would do us more good than harm. Also, Dynomight’s timeline of “the next few decades” is a lot more plausible than the panicky doomers who say “the next few years”.)

In part I want to regulate AI more because I can already see ways that AI is causing harm (this Jacobin article provides a nice summary), but even without those things, a 1% chance of total annihilation is well worth a great deal of regulation and slowed innovation. Indeed, let me get on record saying that reducing the chances of human annihilation by one percentage point would be well worth slowing economic and technological development by a century.

Libertarianism and prediction markets of course go quite well together, and indeed I rarely encounter one without the other. But this faith in the market is deeply misplaced—especially if we believe, as I do, in behavioral economics!—and in particular, prediction markets are in fact fundamentally flawed as a means of making decisions because of how they handle conditional probabilities. It’s actually kind of a corollary of Goodhart’s Law: Once a measure becomes a target, it ceases to be a good measure, because rational agents try to game the system. Prediction markets can work very well if the outcome is totally beyond human control and wouldn’t totally render the outcome moot (e.g. an earthquake); but once human decisions can affect the result, the prediction market effectively becomes insider trading, and for really big things (like existential risk), the outcome itself could easily render the prediction market—or even money itself—utterly irrelevant.

We must defend the motte with all our might—but perhaps it’s best to abandon some of this bailey.

Yes, by all means, abandon religion, learn to think more rationally, support AI safety, and do what you can to do the most good for the least cost. Those things are good, and right, and important.

But if you find yourself thinking that you should sacrifice yourself, or abandon ordinary concepts of morality, in the name of some ultimate good that will save us from imminent destruction, you need to take a step back, and re-immerse yourself in the rest of society. You are not going to single-handedly save the world; nobody is. If the world is to be saved, we will all do it together.

Fish stories and inequality

PNRJ core principles, critique of neoclassical economics, ethics, inequality , , , , , , , , ,

Oct 4 JDN 2461318

Libertarians are fond of economic parables. There are a lot of variants of this particular floating about, including some that already take it in a convincingly left-wing direction, but the most famous is probably this one, from Paul Zane Pilzer:

In his book, Unlimited Wealth, Paul Zane Pilzer tells the story of ten people who live on an island. Their entire existence is made up of waking each morning, spending the day fishing, eating the two fish they each catch, and then going to bed. Their sustenance depends on the fish they catch, and they save nothing. If anything goes wrong with the plan (sickness, lack of fish to catch, etc.), there is no provision or store of food to take from.

Now, two of the ten people devise a plan to build a boat and net with the hope of catching more fish. After completing the project, they go out into the ocean and catch 20 fish in one day, equaling the output that it used to take all ten to accomplish prior.

The development leads those on the island to a fork in the road. Since two people can now produce the same amount of food that previously required all ten people, the other eight are able to work on other beneficial projects, such as farming or construction. Eventually, the island can progress to a point where it has raised its standard of living, is able to survive hard- ships, and can trade with other islands for goods and services.

On the other hand, the other eight people might become jealous of the two new “heroes”.

Income inequality has been created, with the two boaters possessing most of the fish. Out of their envy, they might devise a plan to levy an 80% “income” tax on the boaters, thus taking 16 of their 20 fish and redistributing it to the other residents.

If the island chooses to tax and redistribute, nobody wins. Each is no better off than before (they only receive their two fish per day), and the boaters lose their motivation to work as hard fishing and maintaining their equipment. The island will slowly drift into a society of subsistence living.

Just as those on the island benefitted from the invention of the boat and net, society benefits

from the opportunities that innovation and entrepreneurship provide, as well as the wealth it creates. The government can add value through things such as national defense, law enforce-ment, and fire protection, but left unchecked, it can overreach into the private sector leading to a hindrance in growth.

This story (or some variation on it) seems to convince a lot of people that redistribution of wealth is bad, which is interesting to me, because it’s really not a very convincing argument.

First of all, let’s consider the story on its own merits: Everything happens as stipulated, and there are now two entrepreneurs who are able to catch 20 fish per day while everyone else can only catch 2.

If there is no tax imposed, then how are the other people supposed to get any more fish than they did before? This new wealth is entirely owned by those two entrepreneurs. The only way that other people could specialize in new industries and make the island more productive would be by trading with those two entrepreneurs, who have every incentive to collude and ensure that the prices they demand for their extra fish are exceptionally high. (Also what if it was just one entrepreneur? Then he has a monopoly.)

Conversely, if the income tax is imposed and everyone gets two fish for free, what’s to stop them from specializing and doing other work, now that they have enough fish? The assumption seems to be that these people would only be motivated to do work other than subsistence fishing if they would otherwise starve, which seems like an obviously false understanding of human nature. As long as they would get some benefit from doing additional work—and remember, by assumption, they have skills that would allow them to be productive in farming, construction, etc.—they are likely to in fact do additional work.

Now, if the tax is indeed so high that the entrepreneurs are no better off than they were under subsistence fishing, then perhaps indeed they would stop producing such high output. But what if they were merely somewhat better off, rather than maximally better off? Say the tax is 40%, so they each keep 6 fish and then the other 8 people on the island each get 1 free fish and then have to fish for 1 on their own.

Yes, you can stipulate that they will refuse to be more productive if they are taxed at all, but is that really how this would work in reality? In fact, you could just as well instead stipulate that they are highly altruistic, or motivated by the intrinsic joy of discovery, and say that they would happily innovate even if they received no benefit at all.

Moreover, I think it’s important than in all of these stories, the inequality is very mild.

10% of the population making 10 times the income of the other 90% of the population is more equal than any society has ever been since the founding of civilization. (It’s probably less equal than most forager societies.) Even allegedly communist societies are not this equal.

In reality, we live in a society where there is one man—one man—who owns $900 billion. The median household wealth in America is about $200,000.

If we consider $200,000 equivalent to 2 fish, what would $900 billion be? 4.5 million fish.

Say those 2 fish weigh about 2 kg each, for a total of 4 kg. You could of course carry them in a single basket, and they would reasonably feed you and your family for a day.

Then those 4.5 million fish would weigh 9,000 tonnes, which would require 300 full-size shipping containers just to hold it all. They would provide enough food to feed an entire major city.

The US produces about 5 million tonnes of fish per year, which is about 13,000 tonnes per day—so this amount of fish would constitute the majority of all fish caught in America.

Do we still think it’s wrong to tax away some of that wealth to help everyone else?

Elon Musk is of course an extreme outlier, but we have plenty of inequality aside from him:

The top 0.1% of US households owns five times more wealth than the bottom 50%.

Indeed, the average wealth of the bottom 50% of households is only about $60,000, while the average wealth of the top 0.1% is over $398 million.

This means that if we were to take half—just half—of the wealth of the top 0.1%—just the top 0.1%—and spread it evenly to the bottom 50%, we would more than triple the wealth of the bottom 50%, giving each household an additional $199,000—an entire house in an affordable area. And the top 0.1% would still be left with an average wealth of nearly $200 million each.

This is why libertarians want to talk about imaginary islands with 10 people on them.

If there are only 10 people and the rich are only 10 times as rich as the poor, things don’t sound so unreasonable. Trying to reduce that inequality feels greedy, and any downsides become very salient.

But when you look at the actual inequality in our society, it is staggering. It is appalling. And there is simply no plausible way to justify this level of inequality, no matter what would happen to “productivity” or “growth” or “innovation” if we took efforts to reduce it—and, indeed, what they don’t want you to realize is that the evidence of such harmful effects is in fact quite weak, as the US had very high tax rates on the rich in the 1950s and had plenty of innovation and economic growth.

And what if the rich didn’t invent the technology at all?

What if they simply happened upon it, or inherited it from someone else? What if keeping the technology maintained is actually very cheap, and so requires no particular incentive? Should we then allow them to hoard all the wealth just because they happened to be the ones who ended up with the technology?

This is surely a far more accurate depiction of most rich people, who are not in fact brilliant scientists or inventors, and indeed most of whom inherited some significant proportion of their wealth.

In fact, I think inheritance is often under-estimated as a factor in wealth acquisition, because people tend to think of it as a matter of proportion.

Despite popular belief that he inherited his father’s emerald mines, Elon Musk received fairly little direct financial support from his parents, only on the order of about $100,000 (in today’s money) in loans. His wild success started with selling a tech startup for $330 million (a tech startup I’ve never heard of called Zip2, that I highly doubt was ever actually worth $330 million).

So, given that he has $900 billion today, that $100,000 sounds almost negligible, right? That’s only 0.0001% of his wealth, and he even paid it back later. So that makes him 99.9999% self-made, right?

But here’s the thing: Most people could not get $100,000 from their parents to start a tech company.

Indeed, most people could not get $100,000 from anywhere to start a tech company. So even if they have a brilliant idea for a tech company (and “online city guide software” doesn’t sound particularly brilliant to me?), and all the skills required to make it successful, they simply could never get it off the ground in the first place.

Also, I really don’t see how this tech company was particularly brilliant or innovative; it just seems like they were in the right place at the right time and had someone at the helm who is absolutely peerless at hype (that is Elon Musk’s skill: He is the greatest master of hype in the world), so they were able to convince some idiots at Compaq that their silly little tech startup, indistinguishable from hundreds of other silly little tech startups, was somehow worth $330 million.

This is not a story about Elon Musk being uniquely brilliant or hard-working. This is not some kind of deep innovation that our society needs and we must be very careful to avoid inhibiting in any way. I am quite confident that if Zip2 had never existed, or Compaq had never bought it, Elon Musk would not be as rich as he is today, but the rest of the world would have gone on basically the same. And while SpaceX and Tesla represent genuine innovation, it is not at all clear that their innovation had anything to do with Elon Musk in particular, nor that it would have been greatly reduced if we had somewhat higher tax rates on income and capital gains.

But sure, let’s let him keep his 300 shipping containers full of fish while other people starve. That’s “liberty”.