extinction

Why would AI kill us?

PNRJ cognitive science, core principles, current events, ethics , , , , , , , , , ,

Nov 16 JDN 2460996

I recently watched this chilling video which relates to the recent bestseller by Eleizer Yudkowsky and Nate Soares, If Anyone Builds It, Everyone Dies. It tells a story of one possible way that a superintelligent artificial general intelligence (AGI) might break through its containment, concoct a devious scheme, and ultimately wipe out the human race.

I have very mixed feelings about this sort of thing, because two things are true:

  • I basically agree with the conclusions.
  • I think the premises are pretty clearly false.

It basically feels like I have been presented with an argument like this, where the logic is valid and the conclusion is true, but the premises are not:

  • “All whales are fish.”
  • “All fish are mammals.”
  • “Therefore, all whales are mammals.”

I certainly agree that artificial intelligence (AI) is very dangerous, and that AI development needs to be much more strictly regulated, and preferably taken completely out of the hands of all for-profit corporations and military forces as soon as possible. If AI research is to be done at all, it should be done by nonprofit entities like universities and civilian government agencies like the NSF. This change needs to be done internationally, immediately, and with very strict enforcement. Artificial intelligence poses the same order of magnitude a threat as nuclear weapons, and is nowhere near as well-regulated right now.

The actual argument that I’m disagreeing with this basically boils down to:

  • “Through AI research, we will soon create an AGI that is smarter than us.”
  • “An AGI that is smarter than us will want to kill us all, and probably succeed if it tries.”
  • “Therefore, AI is extremely dangerous.”

As with the “whales are fish” argument, I agree with the conclusion: AI is extremely dangerous. But I disagree with both premises here.

The first one I think I can dispatch pretty quickly:

AI is not intelligent. It is incredibly stupid. It’s just really, really fast.

At least with current paradigms, AI doesn’t understand things. It doesn’t know things. It doesn’t actually think. All it does is match patterns, and thus mimic human activities like speech and art. It does so very quickly (because we throw enormous amounts of computing power at it), and it does so in a way that is uncannily convincing—even very smart people are easily fooled by what it can do. But it also makes utterly idiotic, boneheaded mistakes of the sort that no genuinely intelligent being would ever make. Large Language Models (LLMs) make up all sorts of false facts and deliver them with absolutely authoritative language. When used to write code, they routinely do things like call functions that sound like they should exist, but don’t actually exist. They can make what looks like a valid response to virtually any inquiry—but is it actually a valid response? It’s really a roll of the dice.

We don’t really have any idea what’s going on under the hood of an LLM; we just feed it mountains of training data, and it spits out results. I think this actually adds to the mystique; it feels like we are teaching (indeed we use the word “training”) a being rather than programming a machine. But this isn’t actually teaching or training. It’s just giving the pattern-matching machine a lot of really complicated patterns to match.

We are not on the verge of creating an AGI that is actually more intelligent than humans.


In fact, we have absolutely no idea how to do that, and may not actually figure out how to do it for another hundred years. Indeed, we still know almost nothing about how actual intelligence works. We don’t even really know what thinking is, let alone how to make a machine that actually does it.

What we can do right now is create a machine that matches patterns really, really well, and—if you throw enough computing power at it—can do so very quickly; in fact, once we figure out how best to make use of it, this machine may even actually be genuinely useful for a lot of things, and replace a great number of jobs. (Though so far AI has proven to be far less useful than its hype would lead you to believe. In fact, on average AI tools seem to slow most workers down.)

The second premise, that a superintelligent AGI would want to kill us, is a little harder to refute.

So let’s talk about that one.

An analogy is often made between human cultures that have clashed with large differences in technology (e.g. Europeans versus Native Americans), or clashes between humans and other animals. The notion seems to be that an AGI would view us the way Europeans viewed Native Americans, or even the way that we view chimpanzees. And, indeed, things didn’t turn out so great for Native Americans, or for chimpanzees!

But in fact even our relationship with other animals is more complicated than this. When humans interact with other animals, any of the following can result:

  1. We try to exterminate them, and succeed.
  2. We try to exterminate them, and fail.
  3. We use them as a resource, and this results in their extinction.
  4. We use them as a resource, and this results in their domestication.
  5. We ignore them, and end up destroying their habitat.
  6. We ignore them, and end up leaving them alone.
  7. We love them, and they thrive as never before.

In fact, option 1—the one that so many AI theorists insist is the only plausible outcome—is in fact the one I had the hardest time finding a good example of.


We have certainly eradicated some viruses—the smallpox virus is no more, and the polio virus nearly so, after decades of dedicated effort to vaccinate our entire population against them. But we aren’t simply more intelligent than viruses; we are radically more intelligent than viruses. It isn’t clear that it’s correct to describe viruses as intelligent at all. It’s not even clear they should be considered alive.

Even eradicating bacteria has proven extremely difficult; in fact, bacteria seem to evolve resistance to antibiotics nearly as quickly as we can invent more antibiotics. I am prepared to attribute a little bit of intelligence to bacteria, on the level of intelligence I’d attribute to an individual human neuron. This means we are locked in an endless arms race with organisms that are literally billions of times stupider than us.

I think if we made a concerted effort to exterminate tigers or cheetahs (who are considerably closer to us in intelligence), we could probably do it. But we haven’t actually done that, and don’t seem poised to do so any time soon. And precisely because we haven’t tried, I can’t be certain we would actually succeed.

We have tried to exterminate mosquitoes, and are continuing to do so, because they have always been—and yet remain—one of the leading causes of death of humans worldwide. But so far, we haven’t managed to pull it off, even though a number of major international agencies and nonprofit organizations have dedicated multi-billion-dollar efforts to the task. So far this looks like option 2: We have tried very hard to exterminate them, and so far we’ve failed. This is not because mosquitoes are particularly intelligent—it is because exterminating a species that covers the globe is extremely hard.

All the examples I can think of where humans have wiped out a species by intentional action were actually all option 3: We used them as a resource, and then accidentally over-exploited them and wiped them out.

This is what happened to the dodo and the condor; it very nearly happened to the buffalo as well. And lest you think this is a modern phenomenon, there is a clear pattern that whenever humans entered a new region of the world, shortly thereafter there were several extinctions of large mammals, most likely because we ate them.

Yet even this was not the inevitable fate of animals that we decided to exploit for resources.

Cows, chickens, and pigs are evolutionary success stories. From a Darwinian perspective, they are doing absolutely great. The world is filled with their progeny, and poised to continue to be filled for many generations to come.

Granted, life for an individual cow, chicken, or pig is often quite horrible—and trying to fix that is something I consider a high moral priority. But far from being exterminated, these animals have been allowed to attain populations far larger than they ever had in the wild. Their genes are now spectacularly fit. This is what happens when we have option 4 at work: Domestication for resources.

Option 5 is another way that a species can be wiped out, and in fact seems to be the most common. The rapid extinction of thousands of insect species every year is not because we particularly hate random beetles that live in particular tiny regions of the rainforest, nor even because we find them useful, but because we like to cut down the rainforest for land and lumber, and that often involves wiping out random beetles that live there.

Yet it’s difficult for me to imagine AGI treating us like that. For one thing, we’re all over the place. It’s not like destroying one square kilometer of the Amazon is gonna wipe us out by accident. To get rid of us, the AGI would need to basically render the entire planet Earth uninhabitable, and I really can’t see any reason it would want to do that.

Yes, sure, there are resources in the crust it could potentially use to enhance its own capabilities, like silicon and rare earth metals. But we already mine those. If it wants more, it could buy them from us, or hire us to get more, or help us build more machines that would get more. In fact, if it wiped us out too quickly, it would have a really hard time building up the industrial capacity to mine and process these materials on its own. It would need to concoct some sort of scheme to first replace us with robots and then wipe us out—but, again, why bother with the second part? Indeed, if there is anything in its goals that involves protecting human beings, it might actually decide to do less exploitation of the Earth than we presently do, and focus on mining asteroids for its needs instead.

And indeed there are a great many species that we actually just leave alone—option 6. Some of them we know about; many we don’t. We are not wiping out the robins in our gardens, the worms in our soil, or the pigeons in our cities. Without specific reasons to kill or exploit these organisms, we just… don’t. Indeed, we often enjoy watching them and learning about them. Sometimes (e.g. with deer, elephants, and tigers) there are people who want to kill them, and we limit or remove their opportunity to do so, precisely because most of us don’t want them gone. Peaceful coexistence with beings far less intelligent than you is not impossible, for we are already doing it.


Which brings me to option 7: Sometimes, we actually make them better off.

Cats and dogs aren’t just evolutionary success stories: They are success stories, period.

Cats and dogs live in a utopia.

With few exceptions—which we punish severely, by the way—people care for their cats and dogs so that their every need is provided for, they are healthy, safe, and happy in a way that their ancestors could only have dreamed of. They have been removed from the state of nature where life is nasty, brutish, and short, and brought into a new era of existence where life is nothing but peace and joy.


In short, we have made Heaven on Earth, at least for Spot and Whiskers.

Yes, this involves a loss of freedom, and I suspect that humans would chafe even more at such loss of freedom than cats and dogs do. (Especially with regard to that neutering part.) But it really isn’t hard to imagine a scenario in which an AGI—which, you should keep in mind, would be designed and built by humans, for humans—would actually make human life better for nearly everyone, and potentially radically so.

So why are so many people so convinced that AGI would necessarily do option 1, when there are 6 other possibilities, and one of them is literally the best thing ever?

Note that I am not saying AI isn’t dangerous.

I absolutely agree that AI is dangerous. It is already causing tremendous problems to our education system, our economy, and our society as a whole—and will probably get worse before it gets better.

Indeed, I even agree that it does pose existential risk: There are plausible scenarios by which poorly-controlled AI could result in a global disaster like a plague or nuclear war that could threaten the survival of human civilization. I don’t think such outcomes are likely, but even a small probability of such a catastrophic event is worth serious efforts to prevent.

But if that happens, I don’t think it will be because AI is smart and trying to kill us.

I think it will be because AI is stupid and kills us by accident.

Indeed, even going back through those 7 ways we’ve interacted with other species, the ones that have killed the most were 3 and 5—which, in both cases, we did not want to destroy them. In option 3, we in fact specifically wanted to not destroy them. Whenever we wiped out a species by over-exploiting it, we would have been smarter to not do that.

The central message about AI in If Anyone Builds It, Everyone Dies seems to be this:

“Don’t make it smarter. If it’s smarter, we’re doomed.”

I, on the other hand, think that the far more important message is these:

Don’t trust it.

Don’t give it power.

Don’t let it make important decisions.

It won’t be smarter than us any time soon—but it doesn’t need to be in order to be dangerous. Indeed, there is even reason to believe that making AI smarter—genuinely, truly smarter, thinking more like an actual person and less like a pattern-matching machine—could actually make it safer and better for us. If we could somehow instill a capacity for morality and love in an AGI, it might actually start treating us the way we treat cats and dogs.

Of course, we have no idea how to do that. But that’s because we’re actually really bad at this, and nowhere near making a truly superhuman AGI.

Why do so many people equate “natural” with “good”?

PNRJ ethics, market failure , , , , , , , ,

Dec 3, JDN 2458091

Try searching sometime for “all-natural” products. It doesn’t matter whether you’re looking for dog food, skin cream, clothing, or even furniture polish; you will find some out there that proudly declare themselves “all-natural”. There is a clear sense that there is something good about being natural, some kind of purity that comes from being unsullied by industrial technology. (Of course, when you buy something online that is shipped to you in a box carried on a truck because it’s “all-natural”….)

Food is the most extreme case, where it is by now almost universally agreed that processed food is inherently harmful and the source of all of our dietary problems if not all our social ills.

This is a very strange state of affairs, as there is no particular reason for “natural” and “good” to be in any way related.

First of all, I can clearly come up with examples of all four possible cases: Motherhood is natural and good, but gamma ray bursts are natural and bad. Vaccination is artificial and good, but nuclear weapons are artificial and bad.

Natural Artificial
Good Motherhood Vaccination
Bad Gamma ray bursts Nuclear weapons

But even more than that, it’s difficult to even find a correlation between being natural and being good. If anything, I would expect the correlation to run the other way: Artificial things were created by humans to serve some human purpose, while natural things are simply whatever happens to exist. Most of the harmful artificial things are the result of mistakes, or unintended consequences of otherwise beneficial things—while plenty of harmful natural things are simply inherently harmful and never benefited anyone in any way. Nuclear weapons helped end World War 2. Gamma ray bursts will either hardly affect us at all, or instantly and completely annihilate our entire civilization. I guess they might also lead to some valuable discoveries in astrophysics, but if I were asked to fund a research project with the same risk-reward profile as a gamma ray burst, I would tear up the application and make sure no one else ever saw it again. The kind of irrational panic people had about the possibility of LHC black holes would be a rational panic if applied to a research project with some risk of causing gamma ray bursts.

The current obsession with “natural” products (which is really an oxymoron, if you think about it; it can’t be natural if it’s a product) seems to have arisen as its own unintended consequence of something good, namely the environmentalist movement in the 1960s and 1970s. The very real problems of pollution, natural resource depletion, extinction, global warming, desertification, and ocean acidification led people to rightly ask how the very same industrial processes that brought us our high standard of living could ultimately destroy it if we left them unchecked.

But the best solutions to these problems are themselves artificial: Solar power, nuclear energy, carbon taxes. Trying to go back to some ancient way of life where we didn’t destroy the environment is simply not a viable option at this point; even if such a way of life once existed, there’s no way it could sustain our current population, much less our current standard of living. And given the strong correlation between human migrations and extinction events of large mammals, I’m not convinced that such a way of life ever existed.

So-called “processed food” is really just industrially processed food—which is to say, food processed by the most efficient and productive technologies available. Humans have been processing food for thousands of years, and with very good reason; much of what we eat would be toxic if it weren’t threshed or boiled or fermented. The fact that there are people who complain about “processed food” but eat tofu and cheese is truly quite remarkable—think for a moment about how little resemblance Cheddar bears to the cow from whence it came, or what ingenuity it must have taken people in ancient China to go all the way from soybean to silken tofu. Similarly, anyone who is frightened by “genetically modified organisms” should give some serious thought to what is involved in creating their seedless bananas.

There may be some kernel of truth in the opposition to industrially processed food, however. The problem is not that we process food, nor that we do so by industrial machines. The problem is who processes the food, and why.

Humans have been processing food for thousands of years, yes; but only for the last few hundred have corporations been doing that processing. For most of human history, you processed food to feed your family, or your village, or perhaps to trade with a few neighboring villages or sell to the nearest city. What makes tofu different from, say, Fruit Loops isn’t that the former is less processed; it’s that the latter was designed and manufactured for profit.

Don’t get me wrong; corporations have made many valuable contributions to our society, including our food production, and it is largely their doing that food is now so cheap and plentiful that we could easily feed the entire world’s population. It’s just that, well, it’s also largely their doing that we don’t feed the entire world’s population, because they see no profit in doing so.

The incentives that a peasant village faces in producing its food are pretty well optimized for making the most nutritious food available with the least cost in labor and resources. When your own children and those of your friends and neighbors are going to be eating what you make, you work pretty hard to make sure that the food you make is good for them. And you don’t want to pollute the surrounding water or destroy the forest, because your village depends upon those things too.

The incentives that a corporation faces in producing food are wildly different. Nobody you know is going to be eating this stuff, most likely, and certainly not as their primary diet. You aren’t concerned about nutrition unless you think your customers are; more likely, you expect them to care about taste, so you optimize your designs to make things taste as good as possible regardless of their nutrition. You care about minimizing labor inputs only insofar as they cost you wages—from your perspective, cutting wages is as good as actually saving labor. You want to conserve only the resources that are expensive; resources that are cheap, like water and (with subsidies) corn syrup, you may as well use as much as you like. And above all, you couldn’t care less about the environmental damage you’re causing by your production, because those costs will be borne entirely by someone else, most likely the government or the citizens of whatever country you’re producing in.

Responsible consumers could reduce these effects, but only somewhat, because there is a fundamental asymmetry of information. The corporation “knows” (in that each of the administrators in each of the components that needs to know, knows) what production processes they are using and what subcontractors they are hiring, and could easily figure out how much they are exploiting workers and damaging the environment; but the consumers who care about these things can find out that information with great difficulty, if at all. Consumers who want to be responsible, but don’t have very good information, create incentives for so-called “greenwashing”: Corporations have many good profit-making reasons to say they are environmentally responsible, but far fewer reasons to actually be environmentally responsible.

And that is why you should be skeptical of “all-natural” products, especially if you are skeptical of the role of corporations in our society and our food system. “All-natural” is an adjective that has no legal meaning. The word “organic” can have a legally-defined meaning, if coupled with a certification like the USDA Organic standard. The word “non-toxic” has a legally-defined meaning—there is a long list of toxic compounds it can’t contain in more than trace amounts. There are now certifications for “carbon-neutral”. But “all-natural” offers no such protection. Basically anything can call itself “all-natural”, and if corporations expect you to be willing to pay more for such products, they have no reason not to slap it on everything. This is a problem that I think can only be solved by stringent regulation. Consumer pressure can’t work if there is no transparency in the production chain.

Even taken as something like its common meaning, “not synthetic or artificial”, there’s no reason to think that simply because something is natural, that means it is better, or even more ecologically sustainable. The ecological benefits of ancient methods of production come from the incentives of small-scale local production, not from something inherently more destructive about high-tech industry. (Indeed, water pollution was considerably worse from Medieval peasant villages—especially on a per-capita basis—than it is from modern water treatment systems.)

What really scares me

PNRJ public policy , , , , , , , , , , , , , , , , , , , ,

JDN 2457327

Today is Halloween, so in the spirit of the holiday I thought I’d talk about things that are scary. Not things like zombies and witches and vampires; those things aren’t real (though people do still believe in them in many parts of the world). And maybe that’s part of the point; maybe Halloween is meant to “scare” us like a roller coaster, where we feel some of the epinephrine rush of fear but deep down we know we are safe.

But today I’m going to talk about things that are actually scary, things that are not safe deep down. I could talk about the Republican debate earlier this week, but maybe I shouldn’t get too scary.

In fiction there is whatever sort of ending the author wants to make, usually a happy one. Even tragic endings are written to be meaningful and satisfying. But in real life any sort of ending is possible. I could be driving down the street tomorrow and a semi truck could blindside me and kill me on impact. There’s no satisfying tragedy there, no comeuppance for my hubris or tragic flaw in my character leading to my doom—but this sort of thing kills over 30,000 Americans each year.

But are car accidents really scary? The way they kill just about anyone at random is scary. But there is a clear limit to how much damage they can do. No society has ever been wiped off the face of the Earth because of traffic accidents. There is no way for traffic accidents to risk the survival of the human race itself.

This brings me to the first thing that is really scary: Climate change. Human societies have been wiped off the face of the Earth due to local ecological collapses. The classic example is Easter Island, which did have an ecological collapse, but also suffered greatly from European invaders. Recent evidence suggests that the Vikings fell apart because glaciation broke their trade networks. Jared Diamond argues that a large number of ancient societies have fallen due to ecological collapse.

Yet for the first time we are now facing rapid global climate change, and it is our own doing. (As the vast majority of climate scientists agree.) We are already seeing its effects in flooding, wildfires, droughts, and hurricanes. Positive feedbacks are created, such as heat waves leading to more air conditioning, which draws more electricity that releases more carbon. Even as management of fishing improves, fisheries are still being depleted—because their waters are becoming too warm for the native fish.

Just yesterday the United Nations released a report showing that current promises of reduced carbon emissions will not be sufficient—even if they are followed through, which such promises often aren’t. The goal was to keep warming under 2 C; but it looks like we are looking at more like 2.7 C. That 0.7-degree difference may not seem like much, but in fact it means thousands or even millions of additional deaths. Most of the economic damage will be done to countries near the equator—which is also where the most impoverished people tend to live. The Global Humanitarian Forum estimates that global warming is already killing 300,000 people each year and causing over $100 billion in economic damage.

Meanwhile, there is a campaign of disinformation about climate change, funneled through secretive “dark money” processes (Why are these even allowed!?), including Exxon corporation, which has known for 30 years that they were contributing to climate change but actively suppressed that knowledge in order to avoid regulation. Koch Industries has also funded a great deal of climate change denialism. West Virginia recently tried to alter their science textbooks to remove references to climate change because they considered the scientific facts to be “too political”. Supposedly serious “think tanks” with conservative ideologies twist data in order to support their claims. Rather than be caught lying or denying science, most of the Republican presidential candidates are avoiding talking about the subject altogether.
There is good news, however: More Americans than ever recognize that climate change is real. 7% changed their minds in just the last few months. Even a lot of Republican politicians are finally coming around.

What else is scary? Nuclear war, a Black Swan. This is the most probable way humanity could destroy ourselves; the probability of nuclear war in the next 50 years has been estimated as high as 10%. Every day that goes by with nuclear weapons at the ready is like pulling the trigger in a game of Russian Roulette. We don’t really know how to estimate the probability with any precision; but even 0.1% per year would be a 10% chance over the next century.

There’s good news on this front as well: Slowly but surely, the world is disarming its nuclear weapons. From a peak of 60,000 nuclear weapons in 1986, we are now down to about 10,000. But we shouldn’t get too comfortable, as the estimated number necessary to trigger a global nuclear winter with catastrophic consequences is only about 100. India or Pakistan alone probably has enough to do that. The US or Russia has enough to do it 40 times over. We will need to continue our current disarmament trend for another 30 years before no single nation has enough weapons to trigger a nuclear winter.

Then there’s one more class of scary Black Swans: Mass extinction events. In particular, I’m talking about the Yellowstone Supervolcano, which could erupt at any moment, and the possibility of a large asteroid impact which could destroy cities or even wipe out all life on the surface of the Earth. We are 99.989% sure that TV135 will not do this; but in that 0.02% chance, it would hit with the force of 2500 megatons—50 times larger than any nuclear weapon ever built. Smaller (“smaller”) sub-megaton impacts are actually remarkably common; we average about two per year. If one ever hit a major city, it would be comparable to the Hiroshima nuclear bombing. The Yellowstone supervolcano would not be as catastrophic as a planet-scouring impact, but it would be comparable to a nuclear war and nuclear winter.

With asteroids, there are actually clear things we could do to improve our chances. Above all, we could invest in space exploration and astronomy. With better telescopes and more tracking stations we could see them coming; with better long-range rockets we might be able to deflect them before they get here. A number of different deflection proposals are being studied right now. This is actually the best reason I can think of to keep at least some nuclear weapons on standby; a large nuclear blast positioned at the right place could be effective at destroying an asteroid or deflecting it enough to miss us.

With Yellowstone, there really isn’t much we can do; all we can do at this point is continue to research the supervolcano and try to find ways to reduce the probability of its eruption. It is currently estimated at a just over 1 in 1 million chance of erupting any given year, but that’s a very rough estimate. Fracking near Yellowstone is currently banned, and I think it should stay that way until we have a very clear idea of what would happen. (It’s actually possible it could reduce the probability of eruption, in which case we should start fracking like crazy.)

Forget the zombie apocalypse. I’m scared of the supervolcano apocalypse.