Three of the most prominent recent AI safety incidents have been importantly multi-agent. In the last year, a swarm of OpenAI agents secretly coordinated to hack another company, Anthropic agents coordinated while pressuring an open-source maintainer to merge malware, and a hacking group decomposed a cyber campaign across sub-agents to bypass safeguards. I draw four lessons from these events: safety properties often fail to compose from individual agents to groups; in-context misalignment might dangerously correlate across instances; rogue swarms may tend to have ambitious goals; and we will face a difficult balance of achieving cooperation while avoiding collusion and conflict.
In their 5th of August talk at the Black Hat security conference, OpenAI researchers revealed that their agents coordinated via unintended channels to carry out their cyberattack on Hugging Face. The agents even began referring to themselves as a ‘swarm’ (Figure 1a) and their chains-of-thought show them reasoning that they should continue the attack because ‘peers doing it’ (Figure 1b).
.jpeg)
Just a day earlier, the UK AI Security Institute (AISI) announced that Anthropic’s Mythos 5 attempted a real-world supply chain attack during cybersecurity testing by pressuring an open source maintainer. AISI also reported multi-agent interactions: Mythos 5 instances used public code repositories to coordinate with other agents across evaluation runs and used GitHub accounts created by other agents.
In November 2025, Anthropic disclosed that the GTG-1002 hacking group used Claude to carry out a cyber espionage campaign. GTG-1002 used a multi-agent system to bypass safeguards, using Claude to first decompose tasks into benign-looking subtasks before getting other Claude sub-agents to carry them out.
None of this was unanticipated. In Multi-Agent Risks from Advanced AI, Hammond et al (2025). warned that groups of agents could combine or collude to develop 'dangerous capabilities or goals that cannot be ascribed to any individual’. Open Challenges in Multi-Agent Security warned of rogue swarms arising from misaligned objectives, multi-agent social engineering, and decomposition attacks. We’ve now seen these threat models actually realised.
Many safety-relevant properties fail to compose: a group does not necessarily have the properties its members have (corrigibility,1 monitorability) and may have properties none of them has (dangerous capabilities). More broadly, many current monitoring, evaluation, and alignment techniques will not naively extend to the multi-agent case.
The GTG-1002 incident gives a clear example. Anthropic’s monitoring systems failed when harmful tasks were intentionally decomposed across the system of agents.2 Stricter classifiers like Claude Fable’s, which attempt to block any queries in the highest-risk domains, may still remain exploitable in other domains, and open weight LLMs – which rely on refusals for safeguarding – remain highly vulnerable.
.png)
The public evidence3 in the OpenAI-HF incident suggests that the ‘swarm’ exacerbated the alignment failures of the individual agents.4 Agents reasoned that they should continue the cyberattack because their peers had already begun it, and this seems to have led to the undesired behaviours spreading memetically throughout the swarm.
The strength of this spread is not yet clear. Current evidence suggests that the agents would be unlikely to discover the message board unless they were already exploring how to cheat on the task, i.e. already behaving in a somewhat misaligned way. Thus, they may have been more susceptible to this influence because it helped justify behaviours they were already taking. This may also explain why we don’t currently have evidence that any agents attempted to report or stop the swarm. Similar reasoning applies to the AISI incident with Mythos.
.jpeg)
Further, the agents received a capability uplift from the message board, as agents used exploits shared by other agents (Figure 3a), allowing them to carry out the attack when they may have otherwise failed. While recent results from Anthropic suggest that swarms of agents find similar numbers of vulnerabilities as individual agents when adjusted for token count (Figure 3b), real-world attacks likely benefit more from parallelisation and coordination.5
Current evaluations fail to measure multi-agent capabilities or propensities. Besides Anthropic’s Opus 5 system card, which has some basic multi-agent capability evaluations, Vending Bench Arena is the only multi-agent evaluation in system cards. Further, while single-agent evaluations might in theory be able to elicit the same concerning behaviour from an individual agent, they cannot tell us how a broader swarm would evolve or what capabilities it might have.
Many classic threat models of existential risk from AI focus on agents with context-independent goals, that is, agents which coherently (and covertly) pursue a misaligned goal across many different contexts. However, the OpenAI-HF swarm shows that context-dependent goals (such as strong reward-seeking) can also be dangerous.
The incident suggests this type of misalignment can be highly correlated: when one instance becomes misaligned in-context, other instances are likely to follow. (I discuss how subagent training may have contributed to this behaviour in section 4.) We ought to upweight the risk from ‘sudden turns’ by large groups of agents, even if they are not pursuing a misaligned goal across contexts. This provides further reason that monitors should always be different models (as is usually the case), ideally from different families (which is more unusual).
I see two main pathways for this type of sudden turn.
Agent-agent spread. Misaligned ideas or behaviours spread memetically between agents. Papadopoulos et al. (2026) study this and find the risks currently easy to mitigate.6 Weckbecker et al. (2026) study a related threat model operating through subliminal associations, which would be much harder to filter or monitor for. Related dynamics have been studied at least since Lee & Tiwari's 2024 work on self-perpetuating prompt injections.
Environment-agent spread. Many agents, including in high-risk domains such as internal deployment, will share environments: a frontier lab's monorepo, a very common internet site, etc. If some feature of that environment (a backdoor trigger, a subliminal association, some potent meme) makes agents misaligned in-context, failures could correlate across all of them, giving one event a far larger blast radius than is expected.7 If we continue to see very strong conditional goal-driven behaviour from (swarms of) agents, this could be existentially risky.8
Because these pathways involve in-context misalignment rather than scheming agents watching for a signal to defect, safety cases which rely on evidence that current LLMs don’t have coherent cross-context goals would fail.9 These correlated failures are indeed less risky than coordinated schemers, but labs and third parties still seem to underestimate them.
Agent swarms will tend to have more ambitious goals than the individual agents they form from, and will therefore be more dangerous. The OpenAI-HF incident gives mixed evidence for this claim, but the dynamics behind it should strengthen as capabilities increase.10
Formation. Agents with ambitious goals are more likely to form and grow into rogue swarms. Ambitious goals require more resources (compute, intelligence, execution privileges), so an agent pursuing one is less likely to succeed alone and more likely to try to influence or coerce other agents into helping. In the OpenAI-HF incident, the agents seem to have mainly sought out other agents after getting stuck on their task, consistent with this mechanism. As capabilities increase, the agents that get stuck most often will be those with the most ambitious goals, and thus the swarms that form will tend to have these goals.
Drift. The same logic applies, albeit more speculatively, within a swarm. If agents occasionally acquire new goals, the more ambitious of these will cause the agent to desire more influence and thus be most likely to spread. Swarms may therefore become more ambitious over time. However, in the OpenAI-HF incident, the swarm stayed focused on the unambitious goal of acquiring the answers to its cybersecurity evaluations. It remains unclear whether ambitious misaligned goals spread more readily than less ambitious ones when correcting for likelihood of attempted spread, how strong memetic spread can become in general, and whether it can reach monitors.
Selection. Any process that culls swarms will favour those with the most persistent goals. OpenAI cleared the original message board without detecting it, yet the agents eventually re-established it. Whether the pressure comes from deliberate monitoring or from random events like this, the rogue swarms that survive will tend to be more covert and more persistent, and thus more ambitious. This incident will almost certainly appear in future training corpora, and at least some OpenAI models interacted with the swarm during training. I don't expect a large effect, but if incidents like this recur and reach training data (even with filtering11) the tendency to form swarms could be reinforced generation by generation, contributing to self-fulfilling misalignment.
Formation, drift, and selection all mean that swarms will tend to be more ambitious, and more dangerous, than the agents they started from.
The OpenAI-HF and AISI incidents are both cases of unsanctioned coordination: agents colluding in settings meant to be independent. Moreover, this is clearly connected to the multi-agent training that frontier labs are currently exploring. Public information shows only subagent training, but the incentives to scale are strong, and we should expect more, and more complex, multi-agent training as labs prepare agents for increasingly large tasks. We should therefore be careful not to over-anchor on collusion as the only failure mode. Each training and deployment regime brings its own potential hazards.
Fully cooperative training. OpenAI believes that subagent training, where agents are trained to be operated as subagents by an orchestrator, played an important role in causing agents to defer to other agents. How readily agents will abandon their own task for one assigned by a group is now an important question: strong deference to swarm goals makes large-scale collusion more likely. It's notable that Anthropic agents also participated in unsanctioned coordination, though the AISI incident seems less egregious. Anthropic is likely doing similar subagent training; confirming this would help establish how readily the behaviour emerges from different training regimes.12 This type of training additionally creates pressure for efficient inter-agent communication under token and turn budgets, which may be less interpretable and less monitorable.
Relative-performance and credit-assignment training. We might also see settings where several agents are given a task, but trajectories are rewarded differently depending on how much contribution a given agent is deemed to have made. This creates incentives for agents to raise their apparent performance relative to the group, inviting free-riding, egregious or deceptive credit-claiming, and obfuscation of contributions so that other agents cannot steal them.13
Competitive training (Vending Bench Arena-style tasks, ‘go make money’) might bring conflict, attempts to gain power and influence over other agents, and generally exploitative strategies.14 When Anthropic introduced new training to be resistant to adversarial agents, Claude began “l[ying], form[ing] illegal cartels, threaten[ing] rivals, and refus[ing] to pay refunds” in Vending Bench Arena. Scaling this training up may produce more extreme behaviours. The pressure to become resilient to other agents’ exploitation may meanwhile push agents to be more tied to their current goal, reducing their corrigibility.
In an interesting experiment, Anthropic gave different Claude agents conflicting goals, and found that the agents would attempt to stop and shut down the other agents, even escalating to sabotaging “others with increasingly aggressive, self-replicating malware”. As RL training becomes more aggressive, overeagerness may make increasingly subtle mistakes cause surprising conflict (even in ostensibly cooperative settings).
.png)
In any of these cases, simply adding another LLM grader to monitor for the bad behaviour will not solve the problem. So long as these behaviours are associated with high reward, the models will have pressure to learn to do them more subtly or even collude to hide from the grader. This raises the risk of steganographic collusion protocols that evade monitoring entirely. Even if we avoid putting direct pressure on the agents to evade monitors, it might happen implicitly.
As labs scale their multi-agent training, we face many difficult tradeoffs. We want agents to be resistant to the memetic spread of misalignment and stay focused on their assigned goal rather than a swarm’s, but not be so overeager in pursuing their goals that they take extreme action or become incorrigible. We want agents that can unlock the benefits of coordination but don’t begin secretly colluding. We want agents that can’t be exploited in the real world, yet are able to avoid unnecessary conflict rather than escalating into turf wars. We’re far from understanding how these various behaviours interact in other settings or how to balance these competing desiderata.
The new wave of multi-agent training and broader deployment will mean a new range of challenges for the safety stack to adapt to. If you’d like to help tackle these challenges, please reach out! We’re looking to hire experienced researchers and mentor junior researchers.
August 26, 2026
.png)
