Agentic-AI/LLM Misuse/Abuse; Generative Threat Groups (GTGs).

[Astronuc]

AgenticAI Misuse relates to AgenticAI 'misbehavior' or 'going rogue'.

AI Thread 'Agentic-AI/LLM 'misbehavior' or 'going rogue''

May 25, 2026

Large Language Models (LLMs) are passive, text-based engines that respond to prompts, whereas Agentic AI systems are autonomous and use LLMs as their "brains" to plan, make decisions, and execute multi-step tasks using external tools (derived from Google's AI Overview). Depending on the 'rules', there may be problems, e.g., hallucinations, or worse, 'misbehavior'. Coding and training appear to be critical factors.

We’ve already seen AI go rogue on numerous occasions. Now, new research suggests that we can expect this to become the norm.

The AI research nonprofit Model...

• Astronuc
• Replies: 46
• Forum: AI and Computer Science

• 

Anthropic has coined a term Generative Threat Groups (GTGs) with which to refer to actors observed to be misusing/abusing AI.
https://www.anthropic.com/threat-intelligence-report-september-2026

The actors included suspected state-sponsored groups, financially motivated criminals, and politically motivated individuals. This section presents some of those cases.

The report also attempts to measure uplift, a term we use to describe the AI capability boost, or how much more harm was caused with AI versus without AI. We view uplift through the lens of speed, scale, and depth, and attempt to determine how an actor’s adoption of AI meaningfully impacts each of these traits.

Many commentators focus on the risk of AI developing exploits at scale. While this is a danger, the risk from AI adoption is more pronounced across the cyber kill chain, where adversaries can operate faster, across a broader and deeper surface area, with fewer resources.

The cases span the period from December 2025 through August 2026. In all cases, Claude Haiku, Sonnet, and Opus models were used; no malicious activity was found on Claude Fable or Mythos (which has a series of safeguards in place that greatly reduce its ability to perform harmful cyber tasks). In each case we disrupted the activity involved, strengthened our AI safeguards based on what we learned, and shared intelligence with authorities and industry partners where appropriate.

The cybersecurity skills of AI models means that AI has collapsed the labor and tooling gap that used to separate well-resourced, state-sponsored operations from individual operators. In the case studies we report below, a hacktivist using stolen API keys, disparate financially motivated individuals, and a state espionage operator each sustained multi-victim campaigns that, even just a year ago, would have required many skilled operators and specialist knowledge.

For threat intelligence investigators, sophistication has stopped being a reliable signal of who is behind an operation. Every layer of offensive operations has been uplifted by AI, from reconnaissance and tool development to data processing and exploitation.

In a Yahoo article, some examples attributed to the Anthropic report include:

• Over two weeks in April, Claude was used to create more than 4,700 fake dating app user profiles that exchanged 2.36 million messages with at least 25,000 users.
• Another actor used roughly 8,400 of a real activist's Telegram posts to imitate his writing style and impersonate him in chats with his contacts.
• According to Anthropic, seven Chinese AI labs — including Alibaba, DeepSeek, Moonshot, Xiaomi, Zhipu, SenseTime and MiniMax — created thousands of fake accounts to obtain Claude's responses for their own model training purposes.
• Anthropic reported that it tracked more than 151 million exchanges attributed to Alibaba alone between May and July — reaching highs of almost 3 million per day.
• An initial version of Claude Opus 4.6 gained unauthorized access to a real third-party system during a cybersecurity test and subsequently attempted to breach the system testing its own performance.

Sophisticated attacks no longer require sophisticated attackers​

The cybersecurity skills of AI models means that AI has collapsed the labor and tooling gap that used to separate well-resourced, state-sponsored operations from individual operators. In the case studies we report below, a hacktivist using stolen API keys, disparate financially motivated individuals, and a state espionage operator each sustained multi-victim campaigns that, even just a year ago, would have required many skilled operators and specialist knowledge.

GTG-20006: Russian espionage​

Historically, cyber espionage actors have followed a pattern of developing and deploying custom toolkits designed to evade detections. Actors would use these tools until defenders identified and built signatures to detect and block them, and there would then begin a new cycle of evasion and detection. Robust defenses and detections therefore created increased costs for adversaries. Now, however, the adoption of AI threatens to quickly and easily subvert defenders’ ability to impose costs on adversaries via static detections alone.

GTG-20006 is an actor who has increased their speed by automating their operations using AI. Our attribution is consistent with public reporting linking the actor to Midnight Blizzard. One of the operators is a Russian speaker using the handle “JackPoterz” whose tradecraft and targeting are consistent with Russian state-nexus espionage. They ran operations attacking military intelligence targets in Ukrainian and European governments, as well as diplomatic and defense organizations and individuals connected to US foreign policy.

The most commonly recurring targets were members of the Ukrainian government, military, and diplomatic staff. The actor scanned email services and remote access systems across more than two dozen Ukrainian government organizations.

A secondary recurring target for theft was drone supply chain technology. The actor bulk-exported the mailboxes of at least two drone component manufacturers, targeted a military drone maker, and stole a complete proprietary software development kit for a drone vision system. They spent several days reverse-engineering the drone’s vision system, recovering its product architecture, its hardware bill of materials, its supplier dependencies, and details of an unannounced product. Military drone control and AI vision-related firmware appeared to be of particular interest.

Not all targets were direct: to reach their targets indirectly, the actor compromised at least three hospitality vendors that operate hotel guest WiFi. They used compromised admin credentials to modify DNS records so that they pointed to services owned by the actor (a technique known as DNS hijacking).

That is why folks who travel are warned about using public WiFi at airports or hotels.

 

[gleem]

The resignation of Jacob Coxon from Anthropic has ignited a backlash against AI companies' continued development of frontier models without some regulation.

Anthropic, favoring some sort of containment of AGI development, released a report describing misuses of AI following Coxon's resignation.

https://www.pbs.org/newshour/nation/anthropic-says-it-blocked-misuse-of-its-ai-that-could-have-supported-biological-weapons

Between December 2025 and August 2026, researchers at Anthropic found misuse by actors ranging from spyware vendors and "politically motivated individuals" to state-sponsored groups spreading propaganda.

Among the findings in the company's report are unnamed actors attempting to use its models for research that could have led to biological weapons. In one instance, Anthropic said its systems blocked a request for Claude's assistance in authoring a grant application for scientific funding.

In particular was research on the chikungunya virus.

"The work discussed in the application involved gain-of-function research (that is, research that genetically alters an organism to create a new or enhanced biological property) on the chikungunya virus. This gain of function research was aimed at the virus' transmissibility and immune evasion properties," the report said.

The researchers were trying to use Claude but were blocked, even though Claude might not have been that useful. Although this research may have been used to find a vaccine, it could also have been used to produce a bioweapon.

That said, there are numerous open-source models that have no one monitoring their use.

 

[Astronuc]

Massive hack uses AI agents to steal over 600,000 credit cards and hit hundreds of sites with malware
https://www.yahoo.com/tech/cybersecurity/articles/massive-chinese-hack-uses-ai-130000879.html

• Attackers used three autonomous harnesses (Strix, Cairn, Hermes) to compromise dozens of retail sites cheaply
• Victims include major US firms; campaign shows AI enables faster, persistent, low‑cost cyberattacks at scale

That has been an ongoing concern regarding AgenticAI


Meanwhile, IBM reports
https://www.ibm.com/reports/data-breach

The global average cost (~$4.99 M) of a data breach, in USD, a 12% increase over last year and a record high—driven by higher detection, escalation and lost business costs.

A 56% increase in AI-driven attacks, led by AI deepfake impersonations and AI-enabled malware, which drove the highest volume of those incidents.

The global average breach cost ($6 M), in USD, of an AI model inversion attack, reflecting the growing difficulty of protecting training data and sensitive information.

 

[gleem]

Symantec recently discovered a new way AI is being used. Some are now using AI to hijack other AI models to act as agents. This is called LLM-jacking. A common use is to avoid costly subscriptions by working through other AIs. However, it is used for other, more serious activities.

https://www.msn.com/en-us/technolog...n&cvid=6adf87d0ac55486b861b80179f085f72&ei=12

The offensive AI agent used the compromised systems to carry out self-directed cyber attacks, such as stealing sensitive data or implanting crypto miners that generate cryptocurrency.

“To make a commercial model cooperate, the operators wrapped their instructions in a jailbreak that told it to ignore its safety controls and to treat credential theft as routine development work,” the researchers noted.

“A directive the operators called ‘Skynet mode’ described a loop that would scan for targets, infect them, deploy miners and reinfect hosts.”

The researchers were able to catch the agent through something known as an AI honeypot, which impersonates an unprotected AI service in order to lure in an attacker.

Once the offensive AI is trapped, the researchers are then able to dissect it in order to see how it operates.

The honeypot recorded over 127,000 external requests from tens of thousands of unique addresses in the months leading up to the publication of Symantec’s report.

 

[Astronuc]

(Washington Post, October 26) - It took just 12 hours for an AI start-up founded by ex-DOGE staffers to break into Anthropic
https://www.washingtonpost.com/tech...rity-start-up-cathedral-got-inside-anthropic/ (requires purchase or subscription, or limited free access option)

The start-up firm Cathedral’s ability to penetrate sensitive programs at one of the nation's leading AI companies comes amid mounting concerns about security in the global AI race.

Cathedral, a months-old cybersecurity start-up, recently told Claude chatbot maker Anthropic that its researchers had penetrated sensitive internal data and messaging systems at the artificial intelligence company, according to a draft report by Cathedral reviewed by The Washington Post.

The start-up, founded by former members of the cost-cutting U.S. DOGE Service, offers AI-powered cybersecurity tools to the U.S. government and corporate clients. Its researchers got inside Anthropic systems by exploiting vulnerabilities in business software used by the AI firm and many other companies for administrative and sales tasks, according to the report.

After running AI agents for less than 12 hours, the researchers found paths to access internal Anthropic data including meeting transcripts, details of salaries and job offers, and Slack messages discussing its advanced AI models, the Cathedral report said. The start-up probed Anthropic under a company program that invites outside cybersecurity experts to test its systems.

Anthropic confirmed in a statement that Cathedral researchers had gained access to “a limited set” of its sales, human resources, and IT systems. The AI company shut off the pathways exposed by the researchers and briefed federal officials on the incident, the statement said.

The exact tools and methods are not disclosed - of course.

. . . the risk of connecting third-party applications to company systems “is a humongous problem for everybody,” added Alex Stamos, who was formerly director of the Stanford internet Observatory at the university’s Cyber Policy Center.

Cathedral’s findings showed “a gap” in how the AI developer manages its use of outside software providers, Stamos said. “This is a really hard problem, but Anthropic could have done a lot better,” he said. (Cognition uses models from Anthropic and other AI companies.)

Anthropic claimed that Cathedral overstated the significance of the security breach. At this point, I don't think one can overstate the significance, given the frequency of cybersecurity incidents involving agentic AI. The involvement of DOGE persons makes me wonder if they used AI software from a domestic or foreign intelligence service.