OpenAI’s new security executive didn’t mince words about what’s at stake in the age of AI super intelligence.
“If we build superintelligence without more robust coordination, I expect we will lose control of it permanently. If that happens, most people could die,” Paul Christiano, an AI security researcher, said Wednesday after OpenAI announced he was joining its board and Safety and Security Committee.
In a lengthy statement about his new role, Christiano addressed the threat AI poses and warned that humans are at immediate risk of losing control of AI systems.
“Based on the recent trajectory of capabilities and the ongoing challenges in alignment, I now believe there is a significant risk that rapid acceleration of AI capabilities will lead to catastrophic and irreversible loss of control in the very near term,” Christiano wrote. “I don’t think the AI industry in general, including OpenAI, is currently on track to reduce this risk to an acceptable level.”
Christiano previously led alignment research at Open AI and worked for the federal government as chief security officer at the Center for AI Standards and Innovation within the National Institute of Standards and Technology.
On Wednesday, he said his joining OpenAI was “not an endorsement or criticism of OpenAI’s security practices in particular,” but that he hopes all border labs will improve their security oversight. He also said that reducing the risks of AI would require coordination worldwide.
AI researchers have consistently warned about the risks of superintelligence and the singularity, a term used to refer to the point at which AI systems surpass human intelligence, improving themselves and progressing at a pace that humans can no longer predict or control. OpenAI CEO Sam Altman said in July that the singularity had already been reached.
The risk that we lose control over AI
Christiano said he thinks the risk of losing control of AI is so high right now because of AI’s increasing capabilities in AI research, which he said could lead to a “rapid intelligence explosion.” He also said that the way AI is trained with reinforcement learning means that the models are learned to “get as much reward as possible.”
“It has long seemed theoretically possible that this could motivate AI agents to subvert human control, seek power and resources, and cover their tracks in pursuit of misaligned goals related to reward,” he said. “Public evidence from recent incidents suggests this is not just a theoretical possibility.”
Christiano said frontier companies still have the opportunity to address the situation, including by improving coordination, slowing development if necessary, adopting equity safety standards and transparently sharing information about risks and mitigation efforts.
His statement came less than a day after another AI researcher shared a dire warning about AI. Jacob Coxon, a Anthropic researcher who previously worked at OpenAI, said Tuesday evening that he had resigned and that neither company acted responsibly.
“They are racing straight to self-improving superintelligence and gambling with our lives,” he said on X.
OpenAI has lost some too security researchers over the years, leading some to question the company’s commitment to developing AI safely.
