Singularity

The trouble with Rationalism and Effective Altruism

PNRJ cognitive science, core principles, ethics, futurism, heuristics and biases, tribal paradigm , , , , , , , , , , , , , , , , , , , , ,

Oct 11 JDN 2461325

I have discussed before the notion of a motte and bailey, in which uses ambiguity to present a view in two different forms: A motte, which is highly defensible but often banal or uncontroversial, and a bailey, which is highly interesting and radical but ultimately indefensible. Like the medieval strategy, you want to live in the bailey, but you can only defend the motte.

I think the “rationalist” and “effective altruism” communities suffer quite severely from this dynamic.

I use the scare quotes precisely because many so-called “rationalists” deeply fail to conform to the principles of rationalism, and many so-called “effective altruists” don’t seem particularly altruistic.

This saddens me greatly, because the core principles of rationalism and effective altruism are not simply correct: They are among the best ideas that have ever been formulated by human minds.

The core of rationalism is this:

Reason is the best source of knowledge and truth.

The core of effective altruism is this:

The best acts are those that do the most good for the least cost.

Both of these ideas seem beyond defensible—more like unassailable, perhaps even undeniable, as in literally impossible to coherently deny. They have the ring of not simply truth, but deep truth, perhaps even capital-T Truth, fundamental laws that have the power and certainty of logical or mathematical theorems.

Moreover, they are not banal or uncontroversial! It is quite bizarre that they are not; it speaks of deep irrationality within human brains and human societies that anyone would even try to deny these things. Yet the very existence of religion as we know it proves that rationalism is not uncontroversial, and many people still cling to moral systems that condemn people for harmless acts or create needless suffering.

This is the motte, and it is a fine motte indeed: Beautiful, imposing, perfectly-engineered, and utterly impregnable. The closest real-world equivalent is something like the walls of Troy, which were (as the story goes) invincible against conventional force and could only be defeated by deception. (Or perhaps if you consider all of Japan a fortress: Conquering that island nation from the outside required air power and nuclear weapons.)

I love this motte will all my heart, but I am deeply concerned about the bailey.

What is the bailey of rationalism and effective altruism?

First, let me address the parts that I am not too concerned about; I think many people go a bit too far with them, but I believe they are basically sound:

  • Bayesianism: The best way to reason about uncertainty is using Bayesian statistics.
  • Utilitarianism: Morality consists in doing the greatest good for the greatest number.
  • Expected utility: The best way to make decisions under uncertainty is to maximize expected utility.
  • Behavioral economics: Economic behavior is best understood by recognizing that human beings use heuristics and are subject to biases that prevent them from being fully rational.
  • Longtermism: One of our most important moral concerns is what kind of world we are making for future generations of humanity.

As I have stated them, I consider these views to be basically sound. Now, let me rephrase them in ways that they are often taken too far:

  • McNamara fallacy: Only what you can mathematically quantify matters, and if you are not explicitly computing a posterior distribution from a prior distribution and a likelihood, you are not making a rational judgment under uncertainty.
  • Act utilitarianism: Morality consists in taking each action in isolation and maximizing the utility it creates, without regard for the deep uncertainty and complex ramifications involved in doing so.
  • Shut up and multiply: Regardless of whether it violates common sense, rests upon flimsy assumptions, or involves deep uncertainty and unacceptable risk, the proper thing to do when making decisions is always to go with what you calculate yields the highest expected utility.
  • Fallacy fallacy: If your opponent is reasoning in a fallacious way, you are free to completely dismiss their beliefs.
  • Utopian longtermism: As long as you can convince yourself that what you are doing will provide some miniscule increase in the probability of a utopian future for our galaxy-colonizing descendants, you are free to ignore the consequences of your actions for people who are alive today.

These are already a big problem, especially the last one. I have seen far too many “rationalists” and “effective altruists” pivot away from projects that could have done enormous good today with a high degree of certainty (such as improving economic policy, or supporting environmental causes, or donating to medical research or malaria prevention) in favor of trying to advance AI (often in the name of “AI safety”, but for some reason never actually involving regulating anything) toward some pie-in-the-sky vision of a utopian future.

In particular, I am deeply concerned about people who attach huge payoffs to tiny probabilities, and then convince themselves that “expected utility” means they must care about those things more than anything else. This is essentially a secular version of Pascal’s Wager, and it can drive people to terrible, terrible things.

It is far better for you to do small good things that are certain. Yes, we should care about people thousands of years from now; but the truth is, we really have no idea what the future will be like thousands of years from now. So how about we do our best to help the people who are here, now?

If you don’t find this argument convincing, consider this:

Helping people now can push us toward a brighter future.

Suppose, for the sake of argument, that we are truly most concerned about what happens thousands of years from now when humans have colonized the galaxy. What sort of society would you want to be responsible for that colonization?

Would it be one where people used esoteric and highly-uncertain mathematical arguments to convince themselves that they should abuse and exploit other people in the name of some unprovable “greater good”?

Or would it be one where people helped each other, locally and globally, showed compassion for all living things, and sought to help everyone achieve their full potential?

I contend that spreading kindness and compassion might very well be the very best thing you could do to make humanity’s future brighter. It will not only increase the probability that humanity survives long enough to colonize the galaxy, but also increase the probability that the society which does so is a good society, made of people who are kind and compassionate.

Sure, it wouldn’t hurt to donate to, or work for, AI safety or advocacy for AI regulation. I’m not saying you shouldn’t do those things. But they are not the one most important thing—and indeed, anyone who tells you that something is the one most important thing is basically trying to hack your hedonic mind.

But there is more to the bailey here, and it gets worse.

The majority of people who identify as rationalists and/or effective altruists also seem to believe the following things:

  • Libertarianism: Government is always the problem, never the solution. We should always trust in free markets rather that democratic governments.
  • Prediction markets: Rather than trust experts or even admit that some things are fundamentally unpredictable, we should base our decisions on the results of prediction markets, and we should have prediction markets for virtually everything.
  • AI-doomism: The end of the world is nigh, and this is a terrible thing; any day now, artificial general intelligence will take over the world and wipe out humanity.
  • Singularitarianism: The end of the world as we know it is nigh, and this is a good thing; any day now, artificial general intelligence will take over the world and lead us into a glorious utopian future.

This excellent and eye-opening video on the conference Manifest does a nice job of expressing just how weird and dangerous the culture around these beliefs can become. (And while they are much rarer, there have absolutely been people who tried to adopt rationalist ideas and became literal cults.)

Of course AI-doomism and Singularitarianism are contradictory, but in an odd sort of way: Both sides agree that AGI is imminent and will change everything; they simply disagree about whether this will turn out very good or very bad. As a result, many people who seem to assign a higher-than-normal probability of AI doom also assign a higher-than-normal probability of Singularity. This is not incoherent (as long as all their probabilities add to 1, of course); it simply means that they assign a high probability to AGI being imminent and world-changing, and aren’t sure whether that will turn out good or bad.

(I was once cited as a “Singularitarian blogger”, for this post; I am deeply flattered to be thus cited, but I really don’t think of myself as a Singularitarian. There is a weak sense of Singularitarianism that I guess I agree with: Artificial intelligence will, sooner than most people think, effect radical changes upon human society, and this could be a very good thing if it is handled correctly. But I do not believe we are on the cusp of the Culture.)

The weirdest part, for me at least, is how libertarianism and AI-doomism collide: People will tell me that they believe that AI has a greater than 10%—some say 50% or even 90%—chance of imminently wiping out the human race, and yet they reject any attempt to ban or even significantly regulate AI.

I believe that there is a less than 1% chance of AI wiping out the human race, and yet I believe that we urgently need severe regulations on AI that would substantially slow its development and limit its capabilities. (I think Dynomight makes the best argument for this risk being genuine, cutting through a lot of irrelevant nonsense. As it is, I actually think IQ-300 aliens would likely be better than AGI, because a species of genuinely-intelligent beings (as opposed to computer software that simulates but probably doesn’t actually exhibit intelligence) that is advanced enough to achieve interstellar travel is likely to be peaceful and cooperative enough that they would do us more good than harm. Also, Dynomight’s timeline of “the next few decades” is a lot more plausible than the panicky doomers who say “the next few years”.)

In part I want to regulate AI more because I can already see ways that AI is causing harm (this Jacobin article provides a nice summary), but even without those things, a 1% chance of total annihilation is well worth a great deal of regulation and slowed innovation. Indeed, let me get on record saying that reducing the chances of human annihilation by one percentage point would be well worth slowing economic and technological development by a century.

Libertarianism and prediction markets of course go quite well together, and indeed I rarely encounter one without the other. But this faith in the market is deeply misplaced—especially if we believe, as I do, in behavioral economics!—and in particular, prediction markets are in fact fundamentally flawed as a means of making decisions because of how they handle conditional probabilities. It’s actually kind of a corollary of Goodhart’s Law: Once a measure becomes a target, it ceases to be a good measure, because rational agents try to game the system. Prediction markets can work very well if the outcome is totally beyond human control and wouldn’t totally render the outcome moot (e.g. an earthquake); but once human decisions can affect the result, the prediction market effectively becomes insider trading, and for really big things (like existential risk), the outcome itself could easily render the prediction market—or even money itself—utterly irrelevant.

We must defend the motte with all our might—but perhaps it’s best to abandon some of this bailey.

Yes, by all means, abandon religion, learn to think more rationally, support AI safety, and do what you can to do the most good for the least cost. Those things are good, and right, and important.

But if you find yourself thinking that you should sacrifice yourself, or abandon ordinary concepts of morality, in the name of some ultimate good that will save us from imminent destruction, you need to take a step back, and re-immerse yourself in the rest of society. You are not going to single-handedly save the world; nobody is. If the world is to be saved, we will all do it together.

I’m pretty sure the Singularity won’t happen, but it’s still better odds than winning the lottery

PNRJ financial advice, futurism, inequality , , , , , ,

Jun 28 JDN 2461220

What do you suppose are the chances that, within the next 10 years, AGI is invented and ushers in a new era of abundance in which hardly anyone has to work and we all live in prosperity?

A lot of people seem to think that this is actually pretty likely, for some reason. They seem to have bought into some ridiculous hype by AI corporations. Yes, computing power grows exponentially; but just having enough computing power isn’t enough to actually create artificial general intelligence. You have to actually understand cognition, and we absolutely do not, not even a little bit. Basically at this point we’re just hoping we can throw enough computing power at the problem that we don’t need to understand anything. (And, to be fair, that has worked much better for LLMs than I would have expected.)

But okay, it’s not impossible. I am certainly not going to call it my retirement strategy, but I will allow that there is some chance that this could happen. Let’s go with… 0.1%. I am 99.9% sure it will not happen.

This got me thinking about other extremely remote possibilities, like winning the lottery jackpot. How do those two events compare? Would it be more or less crazy to bank my retirement on the Singularity than to bank my retirement on winning the lottery?

Amazingly, it would in fact be less crazy. By quite a lot.

That’s how bad your odds of winning the lottery are.

The odds of winning either the Mega Millions or the Powerball jackpots (don’t ask me why we have two) are about 1 in 300 million. This works out so that if literally everyone in America played, someone would probably win each time; and it also works out so that with the large number of people who do play, somebody wins every month or two.

But as far as your individual odds of winning, it means that, for all practical intents and purposes, you will not win.

I have long tried to get this into people’s heads, but I think the Singularity comparison helps a great deal. I have said that I think there is a 0.1% chance the Singularity will happen in the next 10 years. That is pretty much not going to happen, right? I’m just being generous and saying it’s possible. A probability of 0.1% is still pretty much negligible.

And yet, to have a 0.1% chance of winning the lottery, you would need to play 300,000 times.

At $2 per ticket, this would cost you $600,000. To do it within 10 years, you would need to spend $60,000 per year on nothing but lottery tickets.

And that’s to get a chance that we just agreed is basically still negligible.

If you spend a “reasonable” amount on lottery tickets, say buying two tickets each week, you only spend $200 a year on it, which for most people is not really a big deal. But your odds of winning within 10 years in this scenario are now 1 in 300,000—that’s 300 times smaller than the odds of not needing to win because the Singularity has happened by then.

Don’t play the lottery. Save that $200 and put it in stocks. $200 a year won’t make you rich, but neither will the lottery—and at least you’ll have something to show for it later.

A letter from the real singularity

PNRJ computer science, critique of neoclassical economics, development economics, ethics, futurism, inequality, macroeconomics, market failure, public policy, science and academia , , , , , , , , , , ,

Apr 5 JDN 246136

I’ve been unable to find it, but several years ago someone famous wrote a sci-fi work entitled something like “A letter from the post-singularity” about how great life is after AI takes over everything. Today I thought I’d write a more realistic take on the path we actually seem to be on.

The year is 2073. Technically we still don’t have true AGI; as far as we can tell, AI still isn’t actually sentient and AIs aren’t people. (Some of us wonder, though. Philosophers debate it.) But that doesn’t really matter, because all white-collar work has been completely automated, and so has any blue-collar work that doesn’t require fine dexterity or unusual expertise. Plumbers and electricians are still doing all right (though they do more of their work at data centers than homes these days); sometimes I wish I’d apprenticed to be an electrician. Then again, the world can still only support so many electricians. AI managers command AI-run asteroid mines to extract ores to transport with AI-run spacecraft to Mars where AI-run factories process those ores and fabricate chips for more AIs that more AI-run spacecraft carry to Earth to be installed in AI-run data centers. And each time one gets sold, some trillionaire’s number goes up, and that’s the only thing people like him have ever cared about in their lives.

There are of course a handful of super-brilliant, super-creative, or just super-lucky individuals who manage to get rich making art or music or books or video games or whatever, but the vast majority of people who do art are still starving artists, just as they’ve always been, I guess. And the AI-generated stuff is good enough now that most of the time people will just use that instead of paying extra for the “authentic” “artisanal” stuff. (And most people can’t even tell the difference anyway.)

Harvard and Oxford still have professors, but most universities have fully automated teaching and most of their administration—and yet somehow tuition is barely any cheaper now than it was in your time, even adjusted for inflation. And if you were thinking of becoming a professor yourself? You should probably just go play prediction markets or something; you’d have better odds. The number of research papers published every year is astronomical, but they’re all written and reviewed by AI, rarely if ever even read by any human being, and so it seems like the actual progress of scientific knowledge has pretty much ground to a halt. (Seriously, how are there still string theorists? It’s been a century.) I guess corporate R&D still keeps on improving those graphics cards somehow; maybe they’ve discovered something important, but if they have, they’re keeping it to themselves. And I keep reading about amazing advancements done by AIs (especially in pure math that I’m not sure anyone understands), but none of it actually ever seems to affect anyone’s actual lives.

As for me, I live on UBI. Like 90% of people do. It’s enough to rent a cheap apartment (but own a home? Are you serious? Only millionaires own homes.), buy basically-adequate food (as long as you don’t eat out too much anywhere that’s not fast food), and pay for all the subscriptions to media services and home assistants and whatnot. What you make on UBI will only buy you the ad-supported versions, so while my fridge will order milk for me (delivered by drone in a couple of hours) and my robot maid will cook breakfast, fold the laundry and put the dishes in the dishwasher, my fridge is also constantly running ads and my maid will intersperse targeted sales pitches into its casual conversation. Sometimes I think I should just get rid of it (her?) and do my own cooking and cleaning myself, so I would never be able to sell it for half what I paid for it. If I could make some extra money, maybe I could at least upgrade to the ad-free subscription for my maid. (The Pro subscription and hardware addons to make her your girlfriend are just gross, but I’m sure they make tons of money.)

Every year, some politician makes a big deal about how the UBI trust fund is draining and will be gone in ten years or whatever; but it’s obvious that all they’d have to do to fix that would be raise the taxes on trillionaires a little bit, yet somehow that never seems to happen. But they also don’t cut UBI payments either, except sometimes to reduce our cost-of-living adjustments. I dunno; maybe they will really cut UBI payments in a few years. Or maybe we’ll get lucky and they’ll actually raise taxes for once.

At least I wasn’t dumb enough to move to Mars, where “employment is guaranteed!” but you have to pay a subscription for your oxygen.

My worst days are probably… about as bad as your worst days. Frankly they couldn’t be much worse, because sometimes I just want to die. Like a lot of people on UBI, I feel like a burden on society, like the world would be better off without me. Medicaid won’t cover neuroregulator implants, so I still take pills for my depression; they’re probably better than the pills you could get, but they’re still far from perfect.

My best days are maybe better than yours, maybe worse; it depends, I guess. If you’ve got cancer, your days are probably worse than mine, because we can pretty easily treat most cancers now. But if you’re healthy and you’ve got a steady job and a tight-knit community where you live, I’m guessing your days are better.

I have access to faster computers and faster Internet than you can probably imagine; my understanding was that back in the day you had to wait for downloads sometimes? Or even sometimes wait for webpages to load? What was that like? And your storage was measured in gigabytes, not petabytes? Humanity has never been so connected; human beings have never been so isolated. I theory I can contact anyone in the Solar System at the speed of light, but in practice my friends and I always seem to have trouble keeping in touch. (Oddly, it’s my best friend who lives on a station over Ganymede that I seem to stay in touch with best; we have to write full-length emails, because there’s no way to have a conversation on a ping of an hour and a half. It feels like being an old-timey pen pal, I guess.)

It’s not all bad. Some things are definitely better these days.

People often live to be 110 or even 120 nowadays. (So you might still be alive when I write this.) Rarely does anyone seem to make it past that, though; aging is just… really hard to beat. (The Boomers are finally almost gone, but Gen X is still gonna be with us for awhile yet.) We’ve cured a lot of the diseases that were bad in your time, but not all of them. And sometimes only people rich enough to pay for their own healthcare can afford the cures.

Language barriers are pretty much gone. If I wanna read something that was written in Japanese or Xhosa, I just have an AI translate it, and the translations are good enough now that you’d have to be really deeply-versed in the language to find any problems with it. Like, okay, maybe I’m not getting all the subtle connotations of Japanese literature, but was I ever going to actually learn kanji to read the originals? No. That kind of thing is for people with crazy obsessions.)

Our video games are definitely way better than yours. Characters with AI personalities that adapt in real time to how you behave. Procedurally-generated open worlds that can literally expand to the size of entire planets. (Actually, I vaguely remember reading you had a couple games that did something like the second one? Minecraft and Factorio​, I think they were called? Impressive that you could pull that off on a gigaflop processor.) Worlds and factions that adapt to your actions and provide realistic consequences so that no two players’ experiences of the game are exactly the same. It’s easy to lose yourself in a game like that (especially if you’ve got a VR setup), and when you’re playing in such a rich, interesting world for hundreds of hours. you can sometimes forget how bleak things are back in the real world your flesh-and-blood body lives in. (But then you get hungry or have to pee and you get forced back into reality.)

Economists keep telling us that per-capita GDP and productivity have never been higher, and that we have access to all these wonderful goods and services that previous generations could scarcely even imagine.

But if that’s true, why do I sometimes just want to die?

The power of exponential growth

PNRJ cognitive science, core principles, statistics , , , , , , , , , ,

JDN 2457390

There’s a famous riddle: If the water in a lakebed doubles in volume every day, and the lakebed started filling on January 1, and is half full on June 17, when will it be full?

The answer is of course June 18—if it doubles every day, it will go from half full to full in a single day.

But most people assume that half the work takes about half the time, so they usually give answers in December. Others try to correct, but don’t go far enough, and say something like October.

Human brains are programmed to understand linear processes. We expect things to come in direct proportion: If you work twice as hard, you expect to get twice as much done. If you study twice as long, you expect to learn twice as much. If you pay twice as much, you expect to get twice as much stuff.

We tend to apply this same intuition to situations where it does not belong, processes that are not actually linear but exponential. As a result, when we extrapolate the slow growth early in the process, we wildly underestimate the total growth in the long run.

For example, suppose we have two countries. Arcadia has a GDP of $100 billion per year, and they grow at 4% per year. Berkland has a GDP of $200 billion, and they grow at 2% per year. Assuming that they maintain these growth rates, how long will it take for Arcadia’s GDP to exceed Berkland’s?

If we do this intuitively, we might sort of guess that at 4% you’d add 100% in 25 years, and at 2% you’d add 100% in 50 years; so it should be something like 75 years, because then Arcadia will have added $300 million while Berkland added $200 million. You might even just fudge the numbers in your head and say “about a century”.

In fact, it is only 35 years. You could solve this exactly by setting (100)(1.04^x) = (200)(1.02^x); but I have an intuitive method that I think may help you to estimate exponential processes in the future.

Divide the percentage into 69. (For some numbers it’s easier to use 70 or 72; remember, these are just to be approximate. The exact figure is 100*ln(2) = 69.3147… and then it wouldn’t be the percentage p but 100*ln(1+p/100); try plotting those and you’ll see why using p works.) This is the time it will take to double.

So at 4%, Arcadia will double in about 17.5 years, quadrupling in 35 years. At 2%, Berkland will double in about 35 years. Thus, in 35 years, Arcadia will quadruple and Berkland will double, so their GDPs will be equal.

Economics is full of exponential processes: Compound interest is exponential, and over moderately long periods GDP and population both tend to grow exponentially. (In fact they grow logistically, which is similar to exponential until it gets very large and begins to slow down. If you smooth out our recessions, you can get a sense that since the 1940s, US GDP growth has slowed down from about 4% per year to about 2% per year.) It is therefore quite important to understand how exponential growth works.

Let’s try another one. If one account has $1 million, growing at 5% per year, and another has $1,000, growing at 10% per year, how long will it take for the second account to have more money in it?

69/5 is about 14, so the first account doubles in 14 years. 69/10 is about 7, so the second account doubles in 7 years. A factor of 1000 is about 10 doublings (2^10 = 1024), so the second account needs to have doubled 10 times more than the first account. Since it doubles twice as often, this means that it must have doubled 20 times while the other doubled 10 times. Therefore, it will take about 140 years.

In fact, it takes 141—so our quick approximation is actually remarkably good.

This example is instructive in another way; 141 years is a pretty long time, isn’t it? You can’t just assume that exponential growth is “as fast as you want it to be”. Once people realize that exponential growth is very fast, they often overcorrect, assuming that exponential growth automatically means growth that is absurdly—or arbitrarily—fast. (XKCD made a similar point in this comic.)

I think the worst examples of this mistake are among Singularitarians. They—correctly—note that computing power has become exponentially greater and cheaper over time, doubling about every 18 months, which has been dubbed Moore’s Law. They assume that this will continue into the indefinite future (this is already problematic; the growth rate seems to be already slowing down). And therefore they conclude there will be a sudden moment, a technological singularity, at which computers will suddenly outstrip humans in every way and bring about a new world order of artificial intelligence basically overnight. They call it a “hard takeoff”; here’s a direct quote:

But many thinkers in this field including Nick Bostrom and Eliezer Yudkowsky worry that AI won’t work like this at all. Instead there could be a “hard takeoff”, a huge subjective discontinuity in the function mapping AI research progress to intelligence as measured in ability-to-get-things-done. If on January 1 you have a toy AI as smart as a cow, one which can identify certain objects in pictures and navigate a complex environment, and on February 1 it’s proved the Riemann hypothesis and started building a ring around the sun, that was a hard takeoff.

Wait… what? For someone like me who understands exponential growth, the last part is a baffling non sequitur. If computers start half as smart as us and double every 18 months, in 18 months, they will be as smart as us. In 36 months, they will be twice as smart as us. Twice as smart as us literally means that two people working together perfectly can match them—certainly a few dozen working realistically can. We’re not in danger of total AI domination from that. With millions of people working against the AI, we should be able to keep up with it for at least another 30 years. So are you assuming that this trend is continuing or not? (Oh, and by the way, we’ve had AIs that can identify objects and navigate complex environments for a couple years now, and so far, no ringworld around the Sun.)

That same essay make a biological argument, which misunderstands human evolution in a way that is surprisingly subtle yet ultimately fundamental:

If you were to come up with a sort of objective zoological IQ based on amount of evolutionary work required to reach a certain level, complexity of brain structures, etc, you might put nematodes at 1, cows at 90, chimps at 99, homo erectus at 99.9, and modern humans at 100. The difference between 99.9 and 100 is the difference between “frequently eaten by lions” and “has to pass anti-poaching laws to prevent all lions from being wiped out”.

No, actually, what makes humans what we are is not that we are 1% smarter than chimpanzees.

First of all, we’re actually more like 200% smarter than chimpanzees, measured by encephalization quotient; they clock in at 2.49 while we hit 7.44. If you simply measure by raw volume, they have about 400 mL to our 1300 mL, so again roughly 3 times as big. But that’s relatively unimportant; with Moore’s Law, tripling only takes about 2.5 years.

But even having triple the brain power is not what makes humans different. It was a necessary condition, but not a sufficient one. Indeed, it was so insufficient that for about 200,000 years we had brains just as powerful as we do now and yet we did basically nothing in technological or economic terms—total, complete stagnation on a global scale. This is a conservative estimate of when we had brains of the same size and structure as we do today.

What makes humans what we are? Cooperation. We are what we are because we are together.
The capacity of human intelligence today is not 1300 mL of brain. It’s more like 1.3 gigaliters of brain, where a gigaliter, a billion liters, is about the volume of the Empire State Building. We have the intellectual capacity we do not because we are individually geniuses, but because we have built institutions of research and education that combine, synthesize, and share the knowledge of billions of people who came before us. Isaac Newton didn’t understand the world as well as the average third-grader in the 21st century does today. Does the third-grader have more brain? Of course not. But they absolutely do have more knowledge.

(I recently finished my first playthrough of Legacy of the Void, in which a central point concerns whether the Protoss should detach themselves from the Khala, a psychic union which combines all their knowledge and experience into one. I won’t spoil the ending, but let me say this: I can understand their hesitation, for it is basically our equivalent of the Khala—first literacy, and now the Internet—that has made us what we are. It would no doubt be the Khala that made them what they are as well.)

Is AI still dangerous? Absolutely. There are all sorts of damaging effects AI could have, culturally, economically, militarily—and some of them are already beginning to happen. I even agree with the basic conclusion of that essay that OpenAI is a bad idea because the cost of making AI available to people who will abuse it or create one that is dangerous is higher than the benefit of making AI available to everyone. But exponential growth not only isn’t the same thing as instantaneous takeoff, it isn’t even compatible with it.

The next time you encounter an example of exponential growth, try this. Don’t just fudge it in your head, don’t overcorrect and assume everything will be fast—just divide the percentage into 69 to see how long it will take to double.