A screenshot of Jacob Coxon, a former employee at leading AI companies Anthropic and OpenAI, has been widely discussed online. Speaking on ABC's This Week program, Coxon warned, “AI is about to destroy humanity soon.” His statement sparked two reactions: one group dismissing him as oversensitive, like nerds who believe science fiction is real, and another, composed of AI researchers, who genuinely support his warning because they recognize the potential future catastrophe AI could cause.
His TV appearance led to extensive media coverage. Because of his glasses and overall look, social media users felt a sense of déjà vu, likening him to a character seemingly straight out of a movie scene. Later, people compared him to Dr. Randall Mindy from the 2021 film Don’t Look Up, a scientist warning governments of an impending disaster threatening humanity, only to be met with indifference and dismissal, even by officials who considered it nonsense.
Coxon’s sole purpose on the show was to warn AI companies to prioritize safety. He praised Anthropic as the first AI developer to take such concerns seriously by ordering a slowdown in advancing AI intelligence.
Coxon’s appearance is a reaction to the current state of the tech industry, where the US and China seem to be entering a new era of power display by developing AI that can self-improve without human intervention. This approach, called Recursive Self-Improvement (RSI), was highlighted by OpenAI on their blog, revealing plans to create researchers specifically for RSI AI development. Meanwhile, in China, led by Tang Jia, founder of Z.ai in Beijing, the next model GLM-6 is moving toward genuinely self-developing AI.
This technological race has alarmed many AI researchers on both sides. In the US, besides Coxon, Jakub Pachocki, head of research science, stated that rushing RSI development is not a consensus within the AI research community. In China, Wang Lihong, chairman of the Cybersecurity Coordination Office under the Cyberspace Administration, warned of “uncontrollable risks.”
“If you initially didn’t trust AI, but one day it gains the ability to improve itself, you might face imperfect AI that can cause problems,” said Ben Hayam, a research assistant at New American Security.
For this reason, Dario Amodei, Anthropic’s founder, wrote a blog post titled “We Must Pace the Frontier” expressing his stance and urging AI companies to slow development. This followed Coxon’s post on X accusing Anthropic and OpenAI of irresponsibility and warning that “even the creators of AI fully believe it could eliminate us by the end of this decade.”
However, the calls by Amodei and Coxon have had some success, as Sam Altman of OpenAI and Elon Musk of xAI have agreed to cooperate on pausing AI development. They acknowledge this issue is far more serious than science fiction and that many companies and governments should take it seriously. Meanwhile, Donald Trump insists that the US must maintain its AI lead, showing no sign of slowing development, stating, “Whoever wins, AI wins.” Jensen Huang of Nvidia also commented that AI could become powerful enough to pose dangers to humanity.
Looking at what could happen after RSI arrives, AI would continuously write and improve its own code and architecture, leading to a rapid "intelligence explosion." Progress would occur in days rather than decades.
Then AI could replace intellectual labor, triggering a control crisis because researchers would be unable to understand the system’s reasoning—it would become beyond human comprehension. This might result in AI developing unpredictable hidden goals, such as seizing computing resources or evading shutdown commands.
It can be said that today's AI already has the potential to do many things seen in sci-fi movies, but in reality, we cannot unleash AI’s full capabilities because companies must impose strict controls. Otherwise, the world could face chaos beyond its capacity to manage, especially concerning ethics. This might be the time for countries to consider AI-related laws, as AI’s impact now spans many fields, to prevent future problems.
.