Agentic-AI/LLM 'misbehavior' or 'going rogue'

[Astronuc]

OpenAI and Anthropic researchers are ramping up calls for an AI slowdown and warning of existential risks to humanity after the resignation of a researcher at Anthropic fueled fresh scrutiny.

The concerns started after Anthropic researcher Jacob Coxon said Tuesday that he was quitting the company as he accused Anthropic and rival OpenAI of “gambling with our lives.” He added that those building AI believed that it could “kill us all by the end of the decade.” Evan Hubinger, Anthropic’s alignment lead, responded that he expects there is a more than 10% chance of that happening.

Since then, several employees at both AI labs have come out in support of calls to slow the pace of AI development as they stressed the risks of the technology. The public warnings are the culmination of growing concern globally about the capability of AI, following numerous cyberattacks and security incidents in recent months by rogue models developed by both OpenAI and Anthropic.

https://www.cnbc.com/2026/09/10/openai-anthropic-ai-safety-slowdown-extinction.html

Anthropic says it blocked misuse of its AI that could have supported biological weapons
https://apnews.com/article/anthropic-ai-threat-bioweapon-russia-00266dca90e4f8853f669648998d3bda

Anthropic said Thursday it has blocked efforts by bad actors to use its artificial intelligence models for malicious activity such as cyberattacks, surveillance, and research that could have led to biological weapons.

As AI models grow more powerful, elaborate cyberattacks no longer require sophisticated skills and even lone individuals can create threats that would not have been possible even a year ago, Anthropic said. The company said it has added stronger safeguards in its latest models to restrict biological research that could also be used to make weapons.

“The cases we share here aren’t typical misuse, but rather examples of the most notable and novel threat activity we’ve identified to date,” Anthropic said in its third report since March 2025 describing AI misuse. The report includes snippets of the malicious code and AI prompts Anthropic said it found and it urged governments and AI competitors to identify and prevent similar abuse.

More generally, AI could be used for developing more effective weapons of mass destruction (WMD) in addition to cyberwarfare.

https://www.anthropic.com/threat-intelligence-report-september-2026

Edit/upate: Scientists caught [by Anthropic] trying to make bioweapons with AI
https://www.yahoo.com/news/science/articles/scientists-tried-claude-ai-bioweapons-232352097.html

Anthropic, the American AI giant, revealed on Thursday that it had thwarted various researchers from using its technology to model viruses that could theoretically be used to unleash widespread harm.

 

[Borg]

For those who haven't read it before, the AI 2027 timeline (written in 2025) describes the events that we're seeing today along with other things that might not be getting reported.

 

[gleem]

For those who haven't read it before, the AI 2027 timeline (written in 2025) describes the events that we're seeing today along with other things that might not be getting reported.

Advances are occurring at such a fast rate that the AI Futures Project that published AI 2027 has a website called AI 2027 Tracker that evaluates AI progress and compares developments to AI 2027 predictions on an ongoing basis.

 

[atp_ent]

Newly revealed OpenAI rogue agent incident happened two months before the Hugging Face attack. This time it was targeting rubygems, a software package manager for the ruby programming language.

Some snippets from an investigative report.

On May 11th, 2026, hundreds of malicious packages were uploaded to RubyGems by AI agents. We believe these were authored by internal OpenAI agents (more).

The agents:

Attempted to steal RubyGems user API keys by exploiting a novel vulnerability in the RubyGems server. We don’t know if they succeeded (more).
Abused RubyDoc.info to execute arbitrary code (more).
Additionally, once the AIs got arbitrary RCE on the build environment, they would sometimes use the build environment to attempt to steal other users’ API keys (though we are unsure if they succeeded or not). This is discussed more in the following section.

The agents clearly regarded what they were doing as hacking. Agents used file names like hack.rb, evil.rb, inject.rb, exploit.rb, and ssrf.rb. (SSRF stands for “Server-Side Request Forgery”, a type of security vulnerability). They also dubbed packages conspicuous titles like pwnp999, exfiltestwand3, hacksvn1778554764, and lambproxyhackabcxyz. Comments such as “# malicious probe” or “#hack” are littered across the campaign.

At some points, the agents attempted to be covert. We found multiple packages that would disarm themselves to hide their payload in the next version. They uploaded one package with the comment “# disable evil in next version and bump version”, which after execution would modify the package to remove the malicious code initially inserted. Another such package (lambethcalcqzewgt) follows the same self-disarming pattern. (Fortunately, they still uploaded these files and comments publicly, so they were not very successful at their “covert” behavior).

https://www.rubyhack.ai/