The Butlerian Jihad is looking better all the time

Mar 24 JDN 2460395

A review of The Age of Em by Robin Hanson

In the Dune series, the Butlerian Jihad was a holy war against artificial intelligence that resulted in a millenias-long taboo against all forms of intelligent machines. It was effectively a way to tell a story about the distant future without basically everything being about robots or cyborgs.

After reading Robin Hanson’s book, I’m starting to think that maybe we should actually do it.

Thus it is written: “Thou shalt not make a machine in the likeness of a human mind.”

Hanson says he’s trying to reserve judgment and present objective predictions without evaluation, but it becomes very clear throughout that this is the future he wants, as well as—or perhaps even instead of—the world he expects.

In many ways, it feels like he has done his very best to imagine a world of true neoclassical rational agents in perfect competition, a sort of sandbox for the toys he’s always wanted to play with. Throughout he very much takes the approach of a neoclassical economist, making heroic assumptions and then following them to their logical conclusions, without ever seriously asking whether those assumptions actually make any sense.

To his credit, Hanson does not buy into the hype that AGI will be successful any day now. He predicts that we will achieve the ability to fully emulate human brains and thus create a sort of black-box AGI that behaves very much like a human within about 100 years. Given how the Blue Brain Project has progressed (much slower than its own hype machine told us it would—and let it be noted that I predicted this from the very beginning), I think this is a fairly plausible time estimate. He refers to a mind emulated in this way as an “em”; I have mixed feelings about the term, but I suppose we did need some word for that, and it certainly has conciseness on its side.

Hanson believes that a true understanding of artificial intelligence will only come later, and the sort of AGI that can be taken apart and reprogrammed for specific goals won’t exist for at least a century after that. Both of these sober, reasonable predictions are deeply refreshing in a field that’s been full of people saying “any day now” for the last fifty years.

But Hanson’s reasonableness just about ends there.

In The Age of Em, government is exactly as strong as Hanson needs it to be. Somehow it simultaneously ensures a low crime rate among a population that doubles every few months while also having no means of preventing that population growth. Somehow ensures that there is no labor collusion and corporations never break the law, but without imposing any regulations that might reduce efficiency in any way.

All of this begins to make more sense when you realize that Hanson’s true goal here is to imagine a world where neoclassical economics is actually true.

He realized it didn’t work on humans, so instead of giving up the theory, he gave up the humans.

Hanson predicts that ems will casually make short-term temporary copies of themselves called “spurs”, designed to perform a particular task and then get erased. I guess maybe he would, but I for one would not so cavalierly create another person and then make their existence dedicated to doing a single job before they die. The fact that I created this person, and they are very much like me, seem like reasons to care more about their well-being, not less! You’re asking me to enslave and murder my own child. (Honestly, the fact that Robin Hanson thinks ems will do this all the time says more about Robin Hanson than anything else.) Any remotely sane society of ems would ban the deletion of another em under any but the most extreme circumstances, and indeed treat it as tantamount to murder.

Hanson predicts that we will only copy the minds of a few hundred people. This is surely true at some point—the technology will take time to develop, and we’ll have to start somewhere. But I don’t see why we’d stop there, when we could continue to copy millions or billions of people; and his choices of who would be emulated, while not wildly implausible, are utterly terrifying.

He predicts that we’d emulate genius scientists and engineers; okay, fair enough, that seems right. I doubt that the benefits of doing so will be as high as many people imagine, because scientific progress actually depends a lot more on the combined efforts of millions of scientists than on rare sparks of brilliance by lone geniuses; but those people are definitely very smart, and having more of them around could be a good thing. I can also see people wanting to do this, and thus investing in making it happen.

He also predicts that we’d emulate billionaires. Now, as a prediction, I have to admit that this is actually fairly plausible; billionaires are precisely the sort of people who are rich enough to pay to be emulated and narcissistic enough to want to. But where Hanson really goes off the deep end here is that he sees this as a good thing. He seems to honestly believe that billionaires are so rich because they are so brilliant and productive. He thinks that a million copies of Elon Musks would produce a million hectobillionaires—when in reality it would produce a million squabbling narcissists, who at best had to split the same $200 billion wealth between them, and might very well end up with less because they squander it.

Hanson has a long section on trying to predict the personalities of ems. Frankly this could just have been dropped entirely; it adds almost nothing to the book, and the book is much too long. But the really striking thing to me about that section is what isn’t there. He goes through a long list of studies that found weak correlations between various personality traits like extroversion or openness and wealth—mostly comparing something like the 20th percentile to the 80th percentile—and then draws sweeping conclusions about what ems will be like, under the assumption that ems are all drawn from people in the 99.99999th percentile. (Yes, upper-middle-class people are, on average, more intelligent and more conscientious than lower-middle-class people. But do we even have any particular reason to think that the personalities of people who make $150,000 are relevant to understanding the behavior of people who make $15 billion?) But he completely glosses over the very strong correlations that specifically apply to people in that very top super-rich class: They’re almost all narcissists and/or psychopaths.

Hanson predicts a world where each em is copied many, many times—millions, billions, even trillions of times, and also in which the very richest ems are capable of buying parallel processing time that lets them accelerate their own thought processes to a million times faster than a normal human. (Is that even possible? Does consciousness work like that? Who knows!?) The world that Hanson is predicting is thus one where all the normal people get outnumbered and overpowered by psychopaths.

Basically this is the most abjectly dystopian cyberpunk hellscape imaginable. And he talks about it the whole time as if it were good.

It’s like he played the game Action Potential and thought, “This sounds great! I’d love to live there!” I mean, why wouldn’t you want to owe a life-debt on your own body and have to work 120-hour weeks for a trillion-dollar corporation just to make the payments on it?

Basically, Hanson doesn’t understand how wealth is actually acquired. He is educated as an economist, yet his understanding of capitalism basically amounts to believing in magic. He thinks that competitive markets just somehow perfectly automatically allocate wealth to whoever is most productive, and thus concludes that whoever is wealthy now must just be that productive.

I can see no other way to explain his wildly implausible predictions that the em economy will double every month or two. A huge swath of the book depends upon this assumption, but he waits until halfway through the book to even try to defend it, and then does an astonishingly bad job of doing so. (Honestly, even if you buy his own arguments—which I don’t—they seem to predict that population would grow with Moore’s Law—doubling every couple of years, not every couple of months.)

Whereas Keynes predicted based on sound economic principles that economic growth would more or less proceed apace and got his answer spot-on, Hanson predicts that for mysterious, unexplained reasons economic growth will suddenly increase by two orders of magnitude—and I’m pretty sure he’s going to be wildly wrong.

Hanson also predicts that ems will be on average poorer than we are, based on some sort of perfect-competition argument that doesn’t actually seem to mesh at all with his predictions of spectacularly rapid economic and technological growth. I think the best way to make sense of this is to assume that it means the trend toward insecure affluence will continue: Ems will have an objectively high standard of living in terms of what they own, what games they play, where they travel, and what they eat and drink (in simulation), but they will constantly be struggling to keep up with the rent on their homes—or even their own bodies. This is a world where (the very finest simulation of) Dom Perignon is $7 a bottle and wages are $980 an hour—but monthly rent is $284,000.

Early in the book Hanson argues that this life of poverty and scarcity will lead to more conservative values, on the grounds that people who are poorer now seem to be more conservative, and this has something to do with farmers versus foragers. Hanson’s explanation of all this is baffling; I will quote it at length, just so it’s clear I’m not misrepresenting it:

The other main (and independent) axis of value variation ranges between poor and rich societies. Poor societies place more value on conformity, security, and traditional values such as marriage, heterosexuality, religion, patriotism, hard work, and trust in authority. In contrast, rich societies place more value on individualism, self-direction, tolerance, pleasure, nature, leisure, and trust. When the values of individuals within a society vary on the same axis, we call this a left/liberal (rich) versus right/conservative (poor) axis.

Foragers tend to have values more like those of rich/liberal people today, while subsistence farmers tend to have values more like those of poor/conservative people today. As industry has made us richer, we have on average moved from conservative/farmer values to liberal/forager values. This value movement can make sense if cultural evolution used the social pressures farmers faced, such as conformity and religion, to induce humans, who evolved to find forager behaviors natural, to instead act like farmers. As we become rich, we don’t as strongly fear the threats behind these social pressures. This connection may result in part from disease; rich people are healthier, and healthier societies fear less.

The alternate theory that we have instead learned that rich forager values are more true predicts that values should have followed a random walk over time, and be mostly common across space. It also predicts the variance of value changes tracking the rate at which relevant information appears. But in fact industrial-era value changes have tracked the wealth of each society in much more steady and consistent fashion. And on this theory, why did foragers ever acquire farmer values?

[…]

In the scenario described in this book, many strange-to-forager behaviors are required, and median per-person (i.e. per-em) incomes return to near-subsistence levels. This suggests that the em era may reverse the recent forager-like trend toward more liberality; ems may have more farmer-like values.

The Age of Em, p. 26-27

There’s a lot to unpack here, but maybe it’s better to burn the whole suitcase.

First of all, it’s not entirely clear that this is really a single axis of variation, that foragers and farmers differ from each other in the same way as liberals and conservatives. There’s some truth to that at least—both foragers and liberals tend to be more generous, both farmers and conservatives tend to enforce stricter gender norms. But there are also clear ways that liberal values radically deviate from forager values: Forager societies are extremely xenophobic, and typically very hostile to innovation, inequality, or any attempts at self-aggrandizement (a phenomenon called “fierce egalitarianism“). San Francisco epitomizes rich, liberal values, but it would be utterly alien and probably regarded as evil by anyone from the Yanomamo.

Second, there is absolutely no reason to predict any kind of random walk. That’s just nonsense. Would you predict that scientific knowledge is a random walk, with each new era’s knowledge just a random deviation from the last’s? Maybe next century we’ll return to geocentrism, or phrenology will be back in vogue? On the theory that liberal values (or at least some liberal values) are objectively correct, we would expect them to advance as knowledge doesimproving over time, and improving faster in places that have better institutions for research, education, and free expression. And indeed, this is precisely the pattern we have observed. (Those places are also richer, but that isn’t terribly surprising either!)

Third, while poorer regions are indeed more conservative, poorer people within a region actually tend to be more liberal. Nigeria is poorer and more conservative than Norway, and Mississippi is poorer and more conservative than Massachusetts. But higher-income households in the United States are more likely to vote Republican. I think this is particularly true of people living under insecure affluence: We see the abundance of wealth around us, and don’t understand why we can’t learn to share it better. We’re tired of fighting over scraps while the billionaires claim more and more. Millennials and Zoomers absolutely epitomize insecure affluence, and we also absolutely epitomize liberalism. So, if indeed ems live a life of insecure affluence, we should expect them to be like Zoomers: “Trans liberation now!” and “Eat the rich!” (Or should I say, “Delete the rich!”)

And really, doesn’t that make more sense? Isn’t that the trend our society has been on, for at least the last century? We’ve been moving toward more and more acceptance of women and minorities, more and more deviation from norms, more and more concern for individual rights and autonomy, more and more resistance to authority and inequality.

The funny thing is, that world sounds a lot better than the one Hanson is predicting.

A world of left-wing ems would probably run things a lot better than Hanson imagines: Instead of copying the same hundred psychopaths over and over until we fill the planet, have no room for anything else, and all struggle to make enough money just to stay alive, we could moderate our population to a more sustainable level, preserve diversity and individuality, and work toward living in greater harmony with each other and the natural world. We could take this economic and technological abundance and share it and enjoy it, instead of killing ourselves and each other to make more of it for no apparent reason.

The one good argument Hanson makes here is expressed in a single sentence: “And on this theory, why did foragers ever acquire farmer values?” That actually is a good question; why did we give up on leisure and egalitarianism when we transitioned from foraging to agriculture?

I think scarcity probably is relevant here: As food became scarcer, maybe because of climate change, people were forced into an agricultural lifestyle just to have enough to eat. Early agricultural societies were also typically authoritarian and violent. Under those conditions, people couldn’t be so generous and open-minded; they were surrounded by threats and on the verge of starvation.

I guess if Hanson is right that the em world is also one of poverty and insecurity, we might go back to those sort of values, borne of desperation. But I don’t see any reason to think we’d give up all of our liberal values. I would predict that ems will still be feminist, for instance; in fact, Hanson himself admits that since VR avatars would let us change gender presentation at will, gender would almost certainly become more fluid in a world of ems. Far from valuing heterosexuality more highly (as conservatives do, a “farmer value” according to Hanson), I suspect that ems will have no further use for that construct, because reproduction will be done by manufacturing, not sex, and it’ll be so easy to swap your body into a different one that hardly anyone will even keep the same gender their whole life. They’ll think it’s quaint that we used to identify so strongly with our own animal sexual dimorphism.

But maybe it is true that the scarcity induced by a hyper-competitive em world would make people more selfish, less generous, less trusting, more obsessed with work. Then let’s not do that! We don’t have to build that world! This isn’t a foregone conclusion!

There are many other paths yet available to us.

Indeed, perhaps the simplest would be to just ban artificial intelligence, at least until we can get a better handle on what we’re doing—and perhaps until we can institute the kind of radical economic changes necessary to wrest control of the world away from the handful of psychopaths currently trying their best to run it into the ground.

I admit, it would kind of suck to not get any of the benefits of AI, like self-driving cars, safer airplanes, faster medical research, more efficient industry, and better video games. It would especially suck if we did go full-on Butlerian Jihad and ban anything more complicated than a pocket calculator. (Our lifestyle might have to go back to what it was in—gasp! The 1950s!)

But I don’t think it would suck nearly as much as the world Robin Hanson thinks is in store for us if we continue on our current path.

So I certainly hope he’s wrong about all this.

Fortunately, I think he probably is.

What is the point of democracy?

Apr 9, JDN 2457853

[This topic was chosen by Patreon vote.]

“Democracy” is the sort of word that often becomes just an Applause Light (indeed it was the original example Less Wrong used). Like “freedom” and “liberty” (and for much the same reasons), it’s a good thing, that much we know; but it’s often unclear what is even meant by the word, much less why it should be so important to us.

From another angle, it is strangely common for economists and political scientists to argue that democracy is not all that important; they at least tend to use a precise formal definition of “democracy”, but are oddly quick to dismiss it as pointless or even harmful when it doesn’t line up precisely with their models of an efficient economy or society. I think the best example of this is the so-called “Downs paradox”, where political scientists were so steeped in the tradition of defining all rationality as psychopathic self-interest that they couldn’t even explain why it would occur to anyone to vote. (And indeed, rumor has it that most economists don’t bother to vote, much less campaign politically—which perhaps begins to explain why our economic policy is so terrible.)

Yet especially for Americans in the Trump era, I think it is vital to understand what “democracy” is supposed to mean, and why it is so important.

So, first of all, what is democracy? It is nothing more or less than government by popular vote.

This comes in degrees, of course: The purest direct democracy would have the entire population vote on even the most mundane policies and decisions. You could actually manage something like a monastery or a social club in such a fashion, but this is clearly unworkable on any large scale. Even once you get to hundreds of people, much less thousands or millions, it becomes unviable. The closest example I’ve seen is Switzerland, where there are always numerous popular referenda on ballots that are voted on by entire regions or the entire country—and even then, Switzerland does have representatives that make many of the day-to-day decisions.

So in practice all large-scale democratic systems are some degree of representative democracy, or republic, where some especially decisions may be made by popular vote, but most policies are made by elected representatives, staff appointed by those representatives, or even career civil servants who are appointed in a nominally apolitical process not so different from private-sector hiring. In the most extreme cases such civil servants can become so powerful that you get a deep state, where career bureaucrats exercise more power than elected officials—at that point I think you have actually lost the right to really call yourself a “democracy” and have become something more like a technocracy.
Yet of course a country can get even more undemocratic than that, and many are, governed by an aristocracy or oligarchy that vests power in a small number of wealthy and powerful individuals, or monarchy or autocracy that gives near-absolute power to a single individual.

Thus, there is a continuum of most to least democratic, with popular vote at one end, followed by elected representatives, followed by appointed civil servants, followed by a handful of oligarchs, and ultimately the most undemocratic system is an autocracy controlled by a single individual.

I also think it’s worth mentioning that constitutional monarchies with strong parliamentary systems, like the United Kingdom and Norway, are also “democracies” in the sense I intend. Yes, technically they have these hereditary monarchs—but in practice, the vast majority of the state’s power is vested in the votes of its people. Indeed, if we separate out parliamentary constitutional monarchy from presidential majoritarian democracy and compare them, the former might actually turn out to be better. Certainly, some of the world’s most prosperous nations are governed that way.

As I’ve already acknowledge, the very far extreme of pure direct democracy is unfeasible. But why would we want to get closer to that end? Why be like Switzerland or Denmark rather than like Turkey or Russia—or for that matter why be like California rather than like Mississippi?
Well, if you know anything about the overall welfare of these states, it almost seems obvious—Switzerland and Denmark are richer, happier, safer, healthier, more peaceful, and overall better in almost every way than Turkey and Russia. The gap between California and Mississippi is not as large, but it is larger than most people realize. Median household income in California is $64,500; in Mississippi it is only $40,593. Both are still well within the normal range of a highly-developed country, but that effectively makes California richer than Luxembourg but Mississippi poorer than South Korea. But perhaps the really stark comparison to make is life expectancy: Life expectancy at birth in California is almost 81 years, while in Mississippi it’s only 75.

Of course, there are a lot of other differences between states besides how much of their governance is done by popular referendum. Simply making Mississippi decide more things by popular vote would not turn it into California—much less would making Turkey more democratic turn it into Switzerland. So we shouldn’t attribute these comparisons entirely to differences in democracy. Indeed, a pair of two-way comparisons is only in the barest sense a statistical argument; we should be looking at dozens if not hundreds of comparisons if we really want to see the effects of democracy. And we should of course be trying to control for other factors, adjust for country fixed-effects, and preferably use natural experiments or instrumental variables to tease out causality.

Yet such studies have in fact been done. Stronger degrees of democracy appear to improve long-run economic growth, as well as reduce corruption, increase free trade, protect peace, and even improve air quality.

Subtler analyses have compared majoritarian versus proportional systems (where proportional seems, to me, at least, more democratic), as well as different republican systems with stronger or weaker checks and balances (stronger is clearly better, though whether that is “more democratic” is at least debatable). The effects of democracy on income distribution are more complicated, probably because there have been some highly undemocratic socialist regimes.

So, the common belief that democracy is good seems to be pretty well supported by the data. But why is democracy good? Is it just a practical matter of happening to get better overall results? Could it one day be overturned by some superior system such as technocracy or a benevolent autocratic AI?

Well, I don’t want to rule out the possibility of improving upon existing systems of government. Clearly new systems of government have in fact emerged over the course of history—Greek “democracy” and Roman “republic” were both really aristocracy, and anything close to universal suffrage didn’t really emerge on a large scale until the 20th century. So the 21st (or 22nd) century could well devise a superior form of government we haven’t yet imagined.
However, I do think there is good reason to believe that any new system of government that actually manages to improve upon democracy will still resemble democracy, because there are three key features democracy has that other systems of government simply can’t match. It is these three features that make democracy so important and so worth fighting for.

1. Everyone’s interests are equally represented.

Perhaps no real system actually manages to represent everyone’s interests equally, but the more democratic a system is, the better it will conform to this ideal. A well-designed voting system can aggregate the interests of an entire population and choose the course of action that creates the greatest overall benefit.

Markets can also be a good system for allocating resources, but while markets represent everyone’s interests, they do so highly unequally. Rich people are quite literally weighted more heavily in the sum.

Most systems of government do even worse, by completely silencing the voices of the majority of the population. The notion of a “benevolent autocracy” is really a conceit; what makes you think you could possibly keep the autocrat benevolent?

This is also why any form of disenfranchisement is dangerous and a direct attack upon democracy. Even if people are voting irrationally, against their own interests and yours, by silencing their voice you are undermining the most fundamental tenet of democracy itself. All voices must be heard, no exceptions. That is democracy’s fundamental strength.

2. The system is self-correcting.

This may more accurately describe a constitutional republican system with strong checks and balances, but that is what most well-functioning democracies have and it is what I recommend. If you conceive of “more democracy” as meaning that people can vote their way into fascism by electing a sufficiently charismatic totalitarian, then I do not want us to have “more democracy”. But just as contracts and regulations that protect you can make you in real terms more free because you can now safely do things you otherwise couldn’t risk, I consider strong checks and balances that maintain the stability of a republic against charismatic fascists to be in a deeper sense more democratic. This is ultimately semantic; I think I’ve made it clear enough that I want strong checks and balances.

With such checks and balances in place, democracies may move slower than autocracies; they may spend more time in deliberation or even bitter, polarized conflict. But this also means that their policies do not lurch from one emperor’s whim to another, and they are stable against being overtaken by corruption or fascism. Their policies are stable and predictable; their institutions are strong and resilient.

No other system of government yet devised by humans has this kind of stability, which may be why democracies are gradually taking over the world. Charismatic fascism fails when the charismatic leader dies; hereditary monarchy collapses when the great-grandson of the great king is incompetent; even oligarchy and aristocracy, which have at least some staying power, ultimately fall apart when the downtrodden peasants ultimately revolt. But democracy abides, for where monarchy and aristocracy are made of families and autocracy and fascism are made of a single man, democracy is made of principles and institutions. Democracy is evolutionarily stable, and thus in Darwinian terms we can predict it will eventually prevail.

3. The coercion that government requires is justified.

All government is inherently coercive. Libertarians are not wrong about this. Taxation is coercive. Regulation is coercive. Law is coercive. (The ones who go on to say that all government is “death threats” or “slavery” are bonkers, mind you. But it is in fact coercive.)

The coercion of government is particularly terrible if that coercion is coming from a system like an autocracy, where the will of the people is minimally if at all represented in the decisions of policymakers. Then that is a coercion imposed from outside, a coercion in the fullest sense, one person who imposes their will upon another.

But when government coercion comes from a democracy, it takes on a fundamentally different meaning. Then it is not they who coerce us—it is we who coerce ourselves. Now, why in the world would you coerce yourself? It seems ridiculous, doesn’t it?

Not if you know any game theory. There are in fall all sorts of reasons why one might want to coerce oneself, and two in particular become particularly important for the justification of democratic government.

The first and most important is collective action: There are many situations in which people all working together to accomplish a goal can be beneficial to everyone, but nonetheless any individual person who found a way to shirk their duty and not contribute could benefit even more. Anyone who has done a group project in school with a couple of lazy students in it will know this experience: You end up doing all the work, but they still get a good grade at the end. If everyone had taken the rational, self-interested action of slacking off, everyone in the group would have failed the project.

Now imagine that the group project we’re trying to achieve is, say, defending against an attack by Imperial Japan. We can’t exactly afford to risk that project falling through. So maybe we should actually force people to support it—in the form of taxes, or even perhaps a draft (as ultimately we did in WW2). Then it is no longer rational to try to shirk your duty, so everyone does their duty, the project gets done, and we’re all better off. How do we decide which projects are important enough to justify such coercion? We vote, of course. This is the most fundamental justification of democratic government.

The second that is relevant for government is commitment. There are many circumstances in which we want to accomplish something in the future, and from a long-run perspective it makes sense to achieve that goal—but then when the time comes to take action, we are tempted to procrastinate or change our minds. How can we resolve such a dilemma? Well, one way is to tie our own hands—to coerce ourselves into carrying out the necessary task we are tempted to avoid or delay.

This applies to many types of civil and criminal law, particularly regarding property ownership. Murder is a crime that most people would not commit even if it were completely legal. But shoplifting? I think if most people knew there would be no penalty for petty theft and retail fraud they would be tempted into doing it at least on occasion. I doubt it would be frequent enough to collapse our entire economic system, but it would introduce a lot of inefficiency, and make almost everything more expensive. By having laws in place that punish us for such behavior, we have a way of defusing such temptations, at least for most people most of the time. This is not as important for the basic functioning of government as is collective action, but I think it is still important enough to be worthy of mention.

Of course, there will always be someone who disagrees with any given law, regardless of how sensible and well-founded that law may be. And while in some sense “we all” agreed to pay these taxes, when the IRS actually demands that specific dollar amount from you, it may well be an amount that you would not have chosen if you’d been able to set our entire tax system yourself. But this is a problem of aggregation that I think may be completely intractable; there’s no way to govern by consensus, because human beings just can’t achieve consensus on the scale of millions of people. Governing by popular vote and representation is the best alternative we’ve been able to come up with. If and when someone devises a system of government that solves that problem and represents the public will even better than voting, then we will have a superior alternative to democracy.

Until then, it is as Churchill said: “Democracy is the worst form of government, except for all the others.”