0.1.1 Why Securitizing AI Is Difficult but Necessary

The world is quietly but constantly being saved. It's easy not to notice or care, because, well, the catastrophe didn't happen.

But such public perception—or lack thereof—can obscure how difficult the true, behind-the-scenes work of saving the world is. Y2K didn't collapse the world's economic and information infrastructure because experts spent hundreds of hours fixing it, not because the issue was exaggerated.

AI risk is especially susceptible to such public ignorance. Algorithms are invisible and intangible, unlike bombs sitting in siloes. Commercial chatbots seem helpful and benevolent. AI capabilities are exponentially increasing—and we are notoriously terrible at internalizing hockey-stick timelines. The layman considers these and concludes that AI will probably not kill everyone.

These obfuscating factors, however, undercuts the reality that AI poses real dangers to global infrastructure and safety. As you've read about in 0.1, recent cases of models breaking out of testing environments and hacking into real organizations exemplify how AI catastrophes are no longer hypothetical.

Clearly, we must stop careless development of technology with such dangerous and far-reaching capabilities. In the context of the global AI race between the U.S. and China, this means: both sides will need to pause.

History shows that agreements between strategic enemies to regulate arms development are difficult, but not impossible. In 1982, Reagan's proposal to eliminate an entire class of US and Soviet missiles was widely dismissed as unrealistic. Five years later, Reagan and Gorbachev signed the INF Treaty.

But neither the U.S. or China will pause until they can verify that the other side will hold up their end of the deal, as well as allowing the other to simultaneously verify their own activity. This is where this course comes in: a robust verification regime gives both sides enough reliable information for a mutual pause, without the need for good-faith trust.

The Chemical Weapons Convention, for example, paired prohibition with declarations, inspections, confidentiality rules, and challenge procedures. In 2023, the OPCW verified the destruction of all 72,304 metric tonnes of chemical-weapons stockpiles declared by member states.

No verification system is perfect, and there are countless ways an imperfect system can go awry. Systems flag false alarms. Motivated adversaries evade detection. Take the infamous Petrov case: in 1983, a Soviet early-warning system falsely reported incoming US missiles. Lieutenant Colonel Stanislav Petrov judged the warning to be false, arguably saving the world.

Verification is both essential to averting global catastrophe and incredibly hard to implement reliably. Frontier AI development adds additional layers of difficulty. For instance, model weights are easily replicable and transferable across state borders, creating security issues.

But before we do any of that, we need to first securitize ASI as the existential risk it is, and that necessitates grappling with why it can be so psychologically unintuitive. We need to actively resist the instinct to view the invisible as the unreal.