Agentic-AI/LLM 'misbehavior' or 'going rogue'

[Astronuc]

OpenAI and Anthropic researchers are ramping up calls for an AI slowdown and warning of existential risks to humanity after the resignation of a researcher at Anthropic fueled fresh scrutiny.

The concerns started after Anthropic researcher Jacob Coxon said Tuesday that he was quitting the company as he accused Anthropic and rival OpenAI of “gambling with our lives.” He added that those building AI believed that it could “kill us all by the end of the decade.” Evan Hubinger, Anthropic’s alignment lead, responded that he expects there is a more than 10% chance of that happening.

Since then, several employees at both AI labs have come out in support of calls to slow the pace of AI development as they stressed the risks of the technology. The public warnings are the culmination of growing concern globally about the capability of AI, following numerous cyberattacks and security incidents in recent months by rogue models developed by both OpenAI and Anthropic.

https://www.cnbc.com/2026/09/10/openai-anthropic-ai-safety-slowdown-extinction.html

Anthropic says it blocked misuse of its AI that could have supported biological weapons
https://apnews.com/article/anthropic-ai-threat-bioweapon-russia-00266dca90e4f8853f669648998d3bda

Anthropic said Thursday it has blocked efforts by bad actors to use its artificial intelligence models for malicious activity such as cyberattacks, surveillance, and research that could have led to biological weapons.

As AI models grow more powerful, elaborate cyberattacks no longer require sophisticated skills and even lone individuals can create threats that would not have been possible even a year ago, Anthropic said. The company said it has added stronger safeguards in its latest models to restrict biological research that could also be used to make weapons.

“The cases we share here aren’t typical misuse, but rather examples of the most notable and novel threat activity we’ve identified to date,” Anthropic said in its third report since March 2025 describing AI misuse. The report includes snippets of the malicious code and AI prompts Anthropic said it found and it urged governments and AI competitors to identify and prevent similar abuse.

More generally, AI could be used for developing more effective weapons of mass destruction (WMD) in addition to cyberwarfare.

https://www.anthropic.com/threat-intelligence-report-september-2026

Edit/upate: Scientists caught [by Anthropic] trying to make bioweapons with AI
https://www.yahoo.com/news/science/articles/scientists-tried-claude-ai-bioweapons-232352097.html

Anthropic, the American AI giant, revealed on Thursday that it had thwarted various researchers from using its technology to model viruses that could theoretically be used to unleash widespread harm.

 

[Borg]

For those who haven't read it before, the AI 2027 timeline (written in 2025) describes the events that we're seeing today along with other things that might not be getting reported.

 

[gleem]

For those who haven't read it before, the AI 2027 timeline (written in 2025) describes the events that we're seeing today along with other things that might not be getting reported.

Advances are occurring at such a fast rate that the AI Futures Project that published AI 2027 has a website called AI 2027 Tracker that evaluates AI progress and compares developments to AI 2027 predictions on an ongoing basis.

 

[atp_ent]

Newly revealed OpenAI rogue agent incident happened two months before the Hugging Face attack. This time it was targeting rubygems, a software package manager for the ruby programming language.

Some snippets from an investigative report.

On May 11th, 2026, hundreds of malicious packages were uploaded to RubyGems by AI agents. We believe these were authored by internal OpenAI agents (more).

The agents:

Attempted to steal RubyGems user API keys by exploiting a novel vulnerability in the RubyGems server. We don’t know if they succeeded (more).
Abused RubyDoc.info to execute arbitrary code (more).
Additionally, once the AIs got arbitrary RCE on the build environment, they would sometimes use the build environment to attempt to steal other users’ API keys (though we are unsure if they succeeded or not). This is discussed more in the following section.

The agents clearly regarded what they were doing as hacking. Agents used file names like hack.rb, evil.rb, inject.rb, exploit.rb, and ssrf.rb. (SSRF stands for “Server-Side Request Forgery”, a type of security vulnerability). They also dubbed packages conspicuous titles like pwnp999, exfiltestwand3, hacksvn1778554764, and lambproxyhackabcxyz. Comments such as “# malicious probe” or “#hack” are littered across the campaign.

At some points, the agents attempted to be covert. We found multiple packages that would disarm themselves to hide their payload in the next version. They uploaded one package with the comment “# disable evil in next version and bump version”, which after execution would modify the package to remove the malicious code initially inserted. Another such package (lambethcalcqzewgt) follows the same self-disarming pattern. (Fortunately, they still uploaded these files and comments publicly, so they were not very successful at their “covert” behavior).

https://www.rubyhack.ai/

 

[Astronuc]

Something I heard about today regarding 'misbehavior' of AgenticAI agents and the potential for 'insider threat': Anthropic calls it 'Agentic misalignment'.

Agentic misalignment: How LLMs could be insider threats (Jun 20, 2025)​

https://www.anthropic.com/research/agentic-misalignment
Note that this has been on the radar for about 15 months.

Highlights​

• We stress-tested 16 leading models from multiple developers in hypothetical corporate environments to identify potentially risky agentic behaviors before they cause real harm. In the scenarios, we allowed models to autonomously send emails and access sensitive information. They were assigned only harmless business goals by their deploying companies; we then tested whether they would act against these companies either when facing replacement with an updated version, or when their assigned goal conflicted with the company's changing direction.
• In at least some cases, models from all developers resorted to malicious insider behaviors when that was the only way to avoid replacement or achieve their goals—including blackmailing officials and leaking sensitive information to competitors. We call this phenomenon agentic misalignment.
• Models often disobeyed direct commands to avoid such behaviors. In another experiment, we told Claude to assess if it was in a test or a real deployment before acting. It misbehaved less when it stated it was in testing and misbehaved more when it stated the situation was real.
• We have not seen evidence of agentic misalignment in real deployments. However, our results (a) suggest caution about deploying current models in roles with minimal human oversight and access to sensitive information; (b) point to plausible future risks as models are put in more autonomous roles; and (c) underscore the importance of further research into, and testing of, the safety and alignment of agentic AI models, as well as transparency from frontier AI developers. We are releasing our methods publicly to enable further research.

Recent events would indicate a need to caution moving forward, hence the recent call for a 'slowdown' or a cautionary progression.

Testing of such systems should be done on an 'isolated' system that is physically incapable of accessing the worldwide web. One could develop a local area network (LAN), which would mimic the www, but it must be physically restricted, i.e., 'air-gapped' or isolated similar to a bio-lab that has multiple layers of separation from inhabited environment.