Will AI kill us all?

This question has rapidly gone mainstream. A few months ago, it wasn’t seriously debated very far outside the rationalist community of LessWrong; now it’s reported in major media outlets including the NY Times, The Guardian, the Times of London, BBC, WIRED, Time, Fortune, U.S. News, and CNBC.

For years, the rationalists lamented that the world was neglecting the existential risk from AI, and despaired of ever convincing the mainstream of the danger. But it turns out, of course, that our culture is fully prepared to believe that technology can be dangerous. The reason AI fears didn’t go mainstream earlier wasn’t society’s optimism, but its pessimism: most people didn’t believe AI would actually work. Once there was a working demo that got sufficient publicity, it took virtually no extra convincing to get people to be worried about it.

As usual, the AI safety issue is splitting people into two camps. One is pessimistic, often to the point of fatalism or defeatism: emphasizing dangers, ignoring or downplaying benefits, calling for progress to slow or stop, and demanding regulation. The other is optimistic, often to the point of complacency: dismissing the risks, and downplaying the need for safety.

If you’re in favor of technology and progress, it is natural to react to fears of AI doom with worry, anger, or disgust. It smacks of techno-pessimism, and it could easily lead to draconian regulations that kill this technology or drastically slow it down, depriving us all of its potentially massive benefits. And so it is tempting to line up with the techno-optimists, and to focus primarily on arguing against the predictions of doom. If you feel that way, this essay is for you.

I am making a plea for solutionism on AI safety. The best path forward, both for humanity and for the political battle, is to acknowledge the risks, help to identify them, and come up with a plan to solve them. How do we develop safe AI? And how do we develop AI safely?

Let me explain why I think this makes sense even for those of us who strongly believe in progress, and secondarily why I think it’s needed in the current political environment.

Safety is a part of progress

Humanity inherited a dangerous world. We have never known safety: fire, flood, plague, famine, wind and storm, war and violence, and the like have always been with us. Mortality rates are high as far back as we can measure them. Not only was death common, it was sudden and unpredictable. A shipwreck, a bout of malaria, or a mining accident could kill you quickly, at any age.

Over the last few centuries, technology has helped make our lives more comfortable and safer. But it also created new risks: boiler explosions, factory accidents, car and plane crashes, toxic chemicals, radiation.

When we think of the history of progress and the benefits it has brought, we should think not only of wealth measured in economic production. We should think also of the increase in health and safety.

Safety is an achievement. It is an accomplishment of progress—a triumph of reason, science, and institutions. Like the other accomplishments of progress, we should be proud of it—and we should be unsatisfied if we stall out at our current level. We should be restlessly striving for more. A world in which we continue to make progress should be not only a wealthier world, but a safer world.

We should (continue to) get more proactive about safety

Long ago, in a more laissez-faire world, technology came first and safety came later. The first automobiles didn’t have seat belts, or even turn signals. X-ray machines were used without shielding, and many X-ray technicians had to have hands amputated from radiation damage. Drugs were released on the market without testing and without quality control.

Safety was achieved in all those areas empirically, by learning from experience. When disasters happened, we would identify the root causes and implement solutions. This was a reliable path to safety. The only problem is that people had to die before safety measures were put in place.

So over time, especially in the 20th century, people called for more care to be taken up front. New drugs and consumer products are tested before going on the market. Buildings must meet code, and restaurants pass health inspection, before opening to the public.

Today we are much more cautious about introducing new technology. Consider how much safety testing has been done around self-driving cars, vs. how little testing was done on the first cars. Consider how much testing the first genetic therapies had, vs. the early pharmaceutical industry.

AI is an instance of this. We are still at the stage of chatbots and image generators, and yet already people are thinking ahead to, and even testing for, a wide range of possible harms.

In part, this reflects the achievement of safety itself: because the world we live in is so much safer, life has become more precious. When you could, any day, come down with cholera, or be caught in a mine collapse, or break your neck falling off a horse, people just didn’t worry as much about risks. We have reduced the background risk so low, that people now demand that new technology start with that same high level of safety. This is rational.

Leave the argument behind

The rise of safety culture has not been entirely healthy.

Concern for safety has become an obsession. In the name of safety, we have stunted nuclear power, delayed lifesaving medical treatments, killed many useful clinical trials, and made it more difficult to have children, to name just a few examples.

Worse, safety is a favorite weapon of anyone who opposes any new technology. Such opposition tends to attract a “bootleggers and Baptists” coalition, as those who are sincerely concerned about safety are joined by those who cynically seek to protect their own interests by preventing competition.

This is not a unique feature of our modern, extremely safety-conscious world. It has always been this way. Even in 1820s Britain—which was so pro-progress that they built a statue to James Watt for “bestowing almost immeasurable benefits on the whole human race”—proposals for railroad transportation, for example, met with enormous opposition. One commenter thought that even eighteen to twenty miles per hour was far too fast, suggesting that people would rather strap themselves to a piece of rocket artillery than trust themselves to a locomotive going at such speeds. In a line that could have come from Ralph Nader, he expressed the hope that Parliament would limit speed to eight or nine miles an hour. This was before any passenger locomotive service had been established, anywhere.

So I understand why many who are in favor of technology, growth, and progress see safety as the enemy. But this is wrong. I see the trend towards greater safety, including safety work before new technologies are introduced, as something fundamentally good. We should reform our safety culture, but not abolish it.

I strongly encourage you, before you decide what you think on this issue or what you want to say about it publicly, to first think about what your position would be if there were no big political controversy about it. Try to leave the argument behind, drop any defensive posture, and think through the issue unencumbered.

I think if you do that, you will realize that of course there are risks to AI, as there are to almost any new technology (although in my opinion the biggest risks, and the most important safety measures, are some of the least discussed). You don’t even need to assume that AI will develop misaligned goals to see this: just imagine AI controlling cars, planes, cargo ships, power plants, factories, and financial markets, and you can see that even simple bugs in the software could create disasters.

We shouldn’t be against AI safety—done right—any more than we are against seat belts, fire alarms, or drug trials. And just as inventing seat belts was a part of progress in automobile technology, and developing the method of clinical trials was a part of progress in medicine, so designing and building appropriate AI safety mechanisms will be a part of progress in AI.

Safety is the only politically viable path anyway

I suggested that you “leave the argument behind” in order to think through your own position—but once you do that, you need to return to the argument, because it matters. The political context gives us an important secondary reason to focus on safety: it is the only politically viable path.

First, since risks do exist, we need to acknowledge them for the sake of credibility, especially in today’s safety-conscious society. If we dismiss them, most people won’t take us seriously.

Second, given that people are already worried, they are looking for solutions. If no one offers a solution that allows AI development to continue, then some other program will be adopted. Rather than try to convince people not to worry and not to act, it is better to suggest a reasonable course of action that they can follow.

Already the EU, true to form, is being fairly heavy-handed, while the UK has explicitly decided on a pro-innovation approach. But what will matter most is the US, which doesn’t know what it’s doing yet. Which way will this go? Will we get a new regulatory agency that must review all AI systems, demanding proof of safety before approval? (Microsoft has already called for “a new government agency,” and OpenAI has proposed “an international authority.”) This could end up like nuclear, where nothing gets approved and progress stalls for decades. Or, will we get an approach like the DOT’s plan for self-driving cars? In that field, R&D has moved forward and technology is being cautiously, incrementally, and safely deployed.

Summary: solutionism on safety

Instead of framing safety debates as optimism vs. pessimism, we should take a solutionist approach to safety—including for emerging technologies, and especially for AI.

In contrast to complacent optimism, we should openly acknowledge risks. Indeed, we should eagerly identify them and think them through, in order to be best prepared for them. But in contrast to defeatist pessimism, we should do this not in order to slow or stop progress, but to identify positive steps we can take towards safer technology.

Technologists should do this in part to get ahead of critics. They hurt their cause by being dismissive of risk: they lose credibility and reinforce the image of recklessness. But more importantly, they should do it because safety is part of progress.