A researcher at the AI company Anthropic, makers of Claude, recently resigned and went public to warn the world that AI companies are “gambling with our lives.” The pace of AI development is so rapid, Jacob Coxon warned, that there “will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.” Coxon claimed that there was a significant chance that all of humanity would be killed outright by out-of-control AI. Lest anyone think Coxon was just a disgruntled kook, one of Anthropic’s own top safety researchers, Evan Hubinger, came out and confirmed: “We really do earnestly believe AI could kill all humans!” putting the odds at over 10 percent in the next 10 years. Hubinger said, in essence, that the company has no plan for how to ensure AI is developed safely, but is proceeding in developing it anyway. Anthropic’s CEO, Dario Amodei, wrote an essay agreeing that “we must slow the pace at which we improve the capabilities of A.I. models” and arguing for more government regulation.

Now, it struck some as odd for Amodei and Hubinger to come out and say that AI development needs to slow down, since, well, they’re the ones doing it. In the New York Times, Sebastian Mallaby praised Amodei for having the “courage to stand in front of an express train and yell at it to slow down.” A more accurate analogy would be if the driver of a speeding train published an essay explaining why the train ought to slow down. Figures in the Trump administration, scoffing at calls for regulation, said that if Amodei is so concerned with safety, he can go ahead and make his product more safe. “If you’re building Frankenstein,” said JD Vance, then “stop.” (Respecters of literature will wince at Vance forgetting that Frankenstein is the scientist, not the monster.) Top Trump tech adviser David Sacks said fears about AI are overblown and companies should self-regulate if they deem it necessary. Trump himself said his very high IQ would keep us safe from any problems caused by AI. Meanwhile, Congress largely slumbered, with the notable exception of Bernie Sanders, the one senator who can be relied on to occasionally give a damn. Sanders is so alarmed by AI that he swallowed his vomit and stood alongside Steve Bannon to condemn the irresponsible tech oligarchs and push for reform.

For the average person, the most pressing question here might be: how alarmed should we be about AI? What are the most serious risks? At the extreme end, there are those like the authors of the book If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All which argues that a sufficiently advanced AI will destroy our species in pursuit of its own objectives. These critics often point out that humans have come to dominate all the less-intelligent species on our planet. “If you want to know how it's like not to be the apex intelligence, ask a chicken,” says pioneering AI researcher Geoffrey Hinton—although this does seem to imply that if chickens were smarter, they would rule over us. Personally I’m not convinced by the theory that superior intelligence will automatically translate into power. (See, for instance, our “High IQ!” president.)

The scenarios presented for how a “superintelligent” system will render us extinct are often maddeningly vague, and tend to simply assume that if a machine were smart enough, it could figure out a way (and there’s no point trying to explain how it could happen, since, after all, we are not smart like the machine, and so cannot possibly anticipate what devious plots it will come up with). A very interesting paper in AI and Ethics reviewed the literature on “existential AI risk” and found that the presented doomsday scenarios often rely on unproven assumptions. It’s also the case, as Émile P. Torres has written for this magazine, that before now many of the leading thinkers on the “existential risk” posed by AI have seemed more concerned with long-term scenarios than with pressing immediate global concerns like war and the climate catastrophe. (Indeed, Coxon appears to downplay the threats posed by the climate crisis and unemployment.)

Nathan Witkin, a research scientist at NYU’s Stern Tech and Society Lab, said he finds it “frankly bizarre to begin worrying about extinction-level harm from AI when it can't yet hold a candle to the human cost of cars or drugs or guns or almost any other generically important technology” and he sees “no evidence that the threat from AI is going to leapfrog past these and suddenly kill millions or billions.” Some people are talking in disturbingly grandiose terms—computer scientist Scott Aaronson says that while there’s “enormous uncertainty about what the rest of our lives will look like… as far as I can tell there’s no longer any real uncertainty that it’ll all mostly revolve around AI.” To which I say: Speak for yourself, buddy! The rest of my life is going to revolve around making print magazines in New Orleans, and if you want yours to revolve around AI, that’s a choice you can go and make for yourself.

But it doesn’t particularly matter whether we are yet convinced that we are coming close to “superintelligence” that could exterminate humanity. The things we know for sure are alarming enough. OpenAI’s software recently planned and executed a series of major cyberattacks without the company being aware it was doing so; you don’t have to believe these systems are conscious or have real “goals” to think that they can easily act like supercharged computer viruses. Amodei thinks these uncontrolled AI agents could potentially take down the entire internet.

In fact, some of the near-term possibilities are so horrifying and dystopian that human extinction seems like it might be a blessing compared with them. As computer scientist Ryan Metz warned in this magazine back in 2018, we are coming into a future where AI may make it dramatically easier for human beings to scam, hack, and even kill large numbers of people. Ryan pointed out how AI could worsen the loneliness epidemic and make it far too easy for anyone to target anyone else for death by drone. My colleague Alex Skopic recently wrote about the alarming implication of new surveillance technologies, and I have written about the threat of “killer robots.” Gary Marcus, a skeptic of the more extreme claims for AI’s capacities, lists a series of incredibly alarming possibilities that are fully plausible:

• “AI may conceivably elevate the risks that someone will develop a dangerous new virus or other bioweapon.

• AI elevates the risks that disinformation will disrupt democracy, by reducing the cost of generating disinformation and increasing the quality (e.g, with ever-better deepfakes)

• AI appears to be elevating the risks of serious cyberattacks that could hobble things like banking or electrical grids.

• AI introduces a new vector for authoritarians around the globe; by influencing training sets and so forth, authoritarians can influence LLMs in ways that subtly influence their users.

• AI also serves as a back door for surveillance, at a scale beyond what Orwell envisioned.

• AI is already harming the educational system by leading students into despair and by seducing them into a faux substitute for learning that is easy to use but that ultimately leaves them with poorly developed critical thinking skills.”

I don’t buy the cynical view that AI risks are just marketing hype for tech companies—it’s a pretty silly kind of marketing hype, since it’s making everyone hate and fear the technology. The basic facts appear to be this: there’s a very dangerous and powerful and almost entirely unregulated technology that is clearly threatening to do an immense amount of damage to society in many ways, for which there is no plan, and which is controlled by for-profit companies that have a strong incentive to do incredibly risky things, since they are pathologically committed to making money. Setting aside the question of whether “extinction” is the most useful thing to focus on, we are dealing with horrifying risks that urgently need addressing. I think there is basically universal agreement that at the very least this technology could make it much easier to design deadly viruses, or manufacture chemical weapons, or scale scams and ransomware to previously-unfathomable levels of malevolence. Surely these are bad enough regardless of whether it will end the species. Timnit Gebru, a computer scientist who formerly led Google’s Ethical Artificial Intelligence Team, says that we do have major pressing concerns about the technology to confront:

What’s really existential is AI powering autonomous weapons, killing machines, that are actually being used in warfare. The climate catastrophe is a real thing that could be exacerbated by what these tech companies are building. Bosses are using AI as an excuse to get rid of their pesky workers. This machine-god narrative is, in my opinion, meant to distract us from these. The industry has become a mix of cults, a mix of people who are true believers, just like a preacher who says the end times is coming.

Journalist Jonathan Cook makes a similar point:

AI doesn’t need to get super-intelligent to turn our societies into a dystopian hell of surveillance, repression and destruction. Just look to Gaza where dumb AI has played a starring role in the levelling of cities, the mass, indiscriminate slaughter of populations, the confinement and surveillance of the survivors, and – perhaps most importantly – the division of western onlookers into bickering tribal camps where we cannot even agree on what is so plainly in front of our eyes.

In his forthcoming Obsolete: The AI Industry's Trillion-Dollar Race to Replace Us—and How to Stop It, journalist and former Current Affairs contributor Garrison Lovely writes:

If you discovered that your neighbor was conducting amateur chemistry experiments that might either cure cancer or level your apartment complex, no one would blame you for calling 911. Yet, with AI, running that experiment on a global scale is not only permitted, but supported with billions upon billions in investment and governments falling over themselves to ensure that more of the lab equipment ends up within their borders.

In fact, while its proponents continue to say that AI’s horrific risks are worth it because of all the wondrous benefits the technology will bring, it’s increasingly unclear to me whether AI is going to deliver us anything truly worth having, or at least anything whose costs don’t vastly outweigh its benefits. On the negative side of the ledger, there’s the fact that it could make it easier to start a pandemic, create deepfake porn and snuff films, make people’s entire skillsets irrelevant, throw artists and musicians and designers out of work, make it easy for people to scam old people by impersonating their grandchildren, etc. On the upside, well, Claude did tell me some pretty useful tips for how to learn the guitar, and is also extremely good at math. Supposedly AI is going to revolutionize medicine and help us cure diseases, but if it also allows people to easily create new diseases, well, there may come a day when we wonder whether the whole bargain was really worth it. What future is it taking us toward and this the future we want? The question is not whether AI has powerful capabilities that could deliver improvements in various sectors (clearly it's immensely powerful), but whether we end up better off or worse off after all is said and done. Our technologies should ultimately be designed to make our lives better than they otherwise would have been. Personally I think we might all end up wishing for a life before AI.