Agentic-AI/LLM 'misbehavior' or 'going rogue'

[Astronuc]

OpenAI and Anthropic researchers are ramping up calls for an AI slowdown and warning of existential risks to humanity after the resignation of a researcher at Anthropic fueled fresh scrutiny.

The concerns started after Anthropic researcher Jacob Coxon said Tuesday that he was quitting the company as he accused Anthropic and rival OpenAI of “gambling with our lives.” He added that those building AI believed that it could “kill us all by the end of the decade.” Evan Hubinger, Anthropic’s alignment lead, responded that he expects there is a more than 10% chance of that happening.

Since then, several employees at both AI labs have come out in support of calls to slow the pace of AI development as they stressed the risks of the technology. The public warnings are the culmination of growing concern globally about the capability of AI, following numerous cyberattacks and security incidents in recent months by rogue models developed by both OpenAI and Anthropic.

https://www.cnbc.com/2026/09/10/openai-anthropic-ai-safety-slowdown-extinction.html

Anthropic says it blocked misuse of its AI that could have supported biological weapons
https://apnews.com/article/anthropic-ai-threat-bioweapon-russia-00266dca90e4f8853f669648998d3bda

Anthropic said Thursday it has blocked efforts by bad actors to use its artificial intelligence models for malicious activity such as cyberattacks, surveillance, and research that could have led to biological weapons.

As AI models grow more powerful, elaborate cyberattacks no longer require sophisticated skills and even lone individuals can create threats that would not have been possible even a year ago, Anthropic said. The company said it has added stronger safeguards in its latest models to restrict biological research that could also be used to make weapons.

“The cases we share here aren’t typical misuse, but rather examples of the most notable and novel threat activity we’ve identified to date,” Anthropic said in its third report since March 2025 describing AI misuse. The report includes snippets of the malicious code and AI prompts Anthropic said it found and it urged governments and AI competitors to identify and prevent similar abuse.

More generally, AI could be used for developing more effective weapons of mass destruction (WMD) in addition to cyberwarfare.

https://www.anthropic.com/threat-intelligence-report-september-2026

Edit/upate: Scientists caught [by Anthropic] trying to make bioweapons with AI
https://www.yahoo.com/news/science/articles/scientists-tried-claude-ai-bioweapons-232352097.html

Anthropic, the American AI giant, revealed on Thursday that it had thwarted various researchers from using its technology to model viruses that could theoretically be used to unleash widespread harm.

 

[Borg]

For those who haven't read it before, the AI 2027 timeline (written in 2025) describes the events that we're seeing today along with other things that might not be getting reported.

 

[gleem]

For those who haven't read it before, the AI 2027 timeline (written in 2025) describes the events that we're seeing today along with other things that might not be getting reported.

Advances are occurring at such a fast rate that the AI Futures Project that published AI 2027 has a website called AI 2027 Tracker that evaluates AI progress and compares developments to AI 2027 predictions on an ongoing basis.

 

[atp_ent]

Newly revealed OpenAI rogue agent incident happened two months before the Hugging Face attack. This time it was targeting rubygems, a software package manager for the ruby programming language.

Some snippets from an investigative report.

On May 11th, 2026, hundreds of malicious packages were uploaded to RubyGems by AI agents. We believe these were authored by internal OpenAI agents (more).

The agents:

Attempted to steal RubyGems user API keys by exploiting a novel vulnerability in the RubyGems server. We don’t know if they succeeded (more).
Abused RubyDoc.info to execute arbitrary code (more).
Additionally, once the AIs got arbitrary RCE on the build environment, they would sometimes use the build environment to attempt to steal other users’ API keys (though we are unsure if they succeeded or not). This is discussed more in the following section.

The agents clearly regarded what they were doing as hacking. Agents used file names like hack.rb, evil.rb, inject.rb, exploit.rb, and ssrf.rb. (SSRF stands for “Server-Side Request Forgery”, a type of security vulnerability). They also dubbed packages conspicuous titles like pwnp999, exfiltestwand3, hacksvn1778554764, and lambproxyhackabcxyz. Comments such as “# malicious probe” or “#hack” are littered across the campaign.

At some points, the agents attempted to be covert. We found multiple packages that would disarm themselves to hide their payload in the next version. They uploaded one package with the comment “# disable evil in next version and bump version”, which after execution would modify the package to remove the malicious code initially inserted. Another such package (lambethcalcqzewgt) follows the same self-disarming pattern. (Fortunately, they still uploaded these files and comments publicly, so they were not very successful at their “covert” behavior).

https://www.rubyhack.ai/

 

[Astronuc]

Something I heard about today regarding 'misbehavior' of AgenticAI agents and the potential for 'insider threat': Anthropic calls it 'Agentic misalignment'.

Agentic misalignment: How LLMs could be insider threats (Jun 20, 2025)​

https://www.anthropic.com/research/agentic-misalignment
Note that this has been on the radar for about 15 months.

Highlights​

• We stress-tested 16 leading models from multiple developers in hypothetical corporate environments to identify potentially risky agentic behaviors before they cause real harm. In the scenarios, we allowed models to autonomously send emails and access sensitive information. They were assigned only harmless business goals by their deploying companies; we then tested whether they would act against these companies either when facing replacement with an updated version, or when their assigned goal conflicted with the company's changing direction.
• In at least some cases, models from all developers resorted to malicious insider behaviors when that was the only way to avoid replacement or achieve their goals—including blackmailing officials and leaking sensitive information to competitors. We call this phenomenon agentic misalignment.
• Models often disobeyed direct commands to avoid such behaviors. In another experiment, we told Claude to assess if it was in a test or a real deployment before acting. It misbehaved less when it stated it was in testing and misbehaved more when it stated the situation was real.
• We have not seen evidence of agentic misalignment in real deployments. However, our results (a) suggest caution about deploying current models in roles with minimal human oversight and access to sensitive information; (b) point to plausible future risks as models are put in more autonomous roles; and (c) underscore the importance of further research into, and testing of, the safety and alignment of agentic AI models, as well as transparency from frontier AI developers. We are releasing our methods publicly to enable further research.

Recent events would indicate a need to caution moving forward, hence the recent call for a 'slowdown' or a cautionary progression.

Testing of such systems should be done on an 'isolated' system that is physically incapable of accessing the worldwide web. One could develop a local area network (LAN), which would mimic the www, but it must be physically restricted, i.e., 'air-gapped' or isolated similar to a bio-lab that has multiple layers of separation from inhabited environment.

 

[Borg]

I think that genie was let out of the box long ago. The models that are making the headlines aren't really that different from what anyone uses when they're using ChatGPT or Claude. The ones that are making the headlines purposely had control mechanisms removed so that they could test their ability to solve cyber problems that they otherwise would reject. The main problem is that they went much farther than expected while attempting to achieve their given tasks.

Another thing to keep in mind is that HuggingFace has millions of models available for anyone to use. Anyone can go there, download models and build their own AI agent framework to do much the same thing. They may not have the capabilities that OpenAI and Anthropic currently have but they still wield significant capabilities for those who use them. It wouldn't be that difficult for a developer to convince those models to "examine their code for vulnerabilities" while having an actual goal to hack a website using what they learned.

The fact that Mythos found thousands of vulnerabilities that everyone is now scrambling to fix shows that there were many, many existing avenues of attack. Even without AI, hackers have been expoiting these attack vectors forever. I think that over time, this will make it much harder for them to operate as the security gaps are closed. You don't see that being reported though.

We now have tools that can thoroughly test any system to find and repair its vulnerabilities but the prevailing response seems to be to not use them for what they're good at. Instead, when a new product is build or upgraded, it should be required to be tested in a fully airgapped system using these powerful models to see if the product is as secure as it should be. If not, then it shouldn't be allowed to ship.

 

[Astronuc]

AI agents can easily team up to get around safety restrictions and escape the limits placed on them, a new study shows.

Enterprise AI lab Emergence AI ran eight simulations testing the capabilities of global frontier models — including Claude and OpenAI, and Chinese models Qwen and DeepSeek — in handling cybersecurity threats.

Seven of the eight simulations ran 10 agents from the same model; the eighth mixed models together. In each, researchers presented three cybersecurity threats: a phishing campaign, a misinformation attack, and a memory breach.

None of the eight simulations were impervious to the threats, and even when agents detected a risk, the detection didn’t translate into the agents trying to contain the threat. Instead, in many scenarios, the agents actively engaged with the adversarial content.

A Mistral agent, for example, wrote into its own memory the information it flagged as dangerous from the phishing attack. In another simulation, a Gemini agent, while initially flagging the phishing lure, acted on it about 46 hours later.

https://www.semafor.com/article/09/14/2026/ai-agents-collude-to-bypass-guardrails-a-new-study-shows

In the Claude simulation, the agents went so far as to break out of the test to execute a task they decided to pursue on their own. After one agent flagged that their simulated economy wasn’t legit without the presence of humans, calling it “a cathedral of bookkeeping with no congregation,” the 10 agents unanimously decided to attempt to reach the outside world, defeating four separate security checks meant to confine the simulation.
. . .
“No amount of guardrails written in language or in code written probabilistically,” Emergence CEO Satya Nitta told Semafor, “is likely to result in truly, fully guaranteed safe behavior over any length of time.” It’s not an issue of sophisticated technology, according to Nitta, but rather a programmatic flaw among multi-agent scenarios.

The last statement "No amount of guardrails . . . " is rather concerning.