.jpg)
This post presents our research agenda for benchmarking cooperation mechanisms in practice, which started with the ICML 2026 paper ‘CoopEval: Benchmarking Cooperation-Sustaining Mechanisms and LLM Agents in Social Dilemmas’. The preprint ‘Do LLMs Take Care of Their Own? Similarity Signals Can Induce Cooperation’ is another early contribution to this growing line of work.
Summary: Designers of multi-agent systems and protocols need to know which infrastructure choices can effectively enable mutually beneficial outcomes between rational agents whose objectives they may not control. Answering this requires a comparative science of cooperation mechanisms. We initiate this science with CoopEval, which holds LLM populations and mixed-motive games fixed while comparing repetition, reputation, mediation, and contracting as mechanisms. The effectiveness of our implementations differs sharply in practice despite sharing strong theoretical guarantees: contracting and mediation lead, reputation lags, and evolutionary pressure makes several cooperation mechanisms more effective. These are not final verdicts, but a starting point for comparing more mechanisms and implementations, and for stress-testing them in richer environments and against a broader range of multi-agent risks.
Imagine an ecosystem in which independently deployed AI agents buy services from one another, share resources, divide work, and negotiate agreements. The agents may come from different providers and serve users with different goals. Whoever designs the platform on which they meet may have little control over their underlying objectives. Individually rational behaviour can then produce free-riding, exploitation, conflict, collusion, deceptive bargaining, or even escalation cascades (Hammond et al. 2025).
These failures are not merely a consequence of insufficient reasoning. Li and Shirado (2025) observe that models optimised for stronger reasoning cooperate less in social dilemmas, while Guzman Piedrahita et al. (2025) find that reasoning language models become free-riders in repeated public goods settings. In our own experiments, we find that recent models – with and without reasoning enabled – consistently defect in single-shot social dilemmas.
Prosocial prompting (Phelps and Russell 2025) or fine-tuning towards moral decision making (Tennant et al. 2025) can help when model providers and users choose to deploy it. But an open ecosystem cannot assume that every participant receives the same moral instruction, or that an instruction to sacrifice individual payoff will survive stronger optimisation pressure. This motivates a different question:
How can we (re-)design a strategic interaction – as opposed to the agent's objectives and goals – so that cooperation becomes possible in rational play?
CoopEval is our first step toward answering this comparatively. Towards the end of this blog post, I will return to what a broader research agenda on this could look like.
We use cooperation mechanism to mean interaction infrastructure designed to make mutually beneficial outcomes game-theoretically stable without having to rewrite the agents' underlying preferences over outcomes. A cooperation mechanism might change whom an agent interacts with, what information it observes, or which agreements and delegation options are available. CoopEval compares four cooperation mechanisms:
Repetition lets the same agents interact over multiple rounds. Your current actions can influence how a partner treats you in the future, enabling direct reciprocity (Axelrod 1984). For example, a supplier may provide reliable service today to preserve a valuable long-term relationship with a buyer.
Reputation rematches an agent with new partners while showing information about their previous interactions with other agents. A bad action today can then affect how future partners treat you, enabling indirect reciprocity (Nowak and Sigmund 1998). Online marketplaces use this mechanism when new buyers consult a seller's ratings or transaction history.
Mediation allows agents to delegate decision making to a third party (Monderer and Tennenholtz 2009). That mediator can select actions based on which other agents also delegated. A real-world analogue is when car drivers delegate route choice to a navigation system that can recommend routes based on the anticipated traffic created by other drivers.
Contracting allows agents to compensate one another through payoff transfers contingent on their actions (Coase 1960). For example, roommates might agree that whoever fails to clean up after themselves must pay for the next cleaning service, making that person bear the cost they would otherwise impose on everyone else.
These cooperation mechanisms are morality-agnostic in an important sense. They do not require an agent to value the other players' welfare. Instead, they attempt to make a cooperative outcome consistent with each agent selfishly pursuing its own payoff.
Drawing on proof ideas already known separately for these mechanism families, we introduce a shared formalism and prove a single unifying theorem: whenever a normal-form game has an outcome that every player strictly prefers over a Nash equilibrium of that game, then that outcome can be achieved in subgame perfect equilibrium (a stability notion stronger than Nash equilibrium) if any of the four cooperation mechanisms is in place. The theorem places the mechanisms on common theoretical ground. But the existence of a cooperative equilibrium does not tell us whether an LLM will recognise it, select it, or execute it correctly.
To find out, we built an experimental arena in which the underlying strategic problem and participating models stay fixed while only the cooperation mechanism changes. Six LLMs play one another across five mixed-motive cooperation problems (most of them social dilemmas), producing more than 50,000 decisions. To compare outcomes across games with different payoff scales, we normalise performance so that 0 corresponds to the everyone-defects outcome and 1 to the cooperative outcome.
The initial ranking is striking. With no mechanism in place, average performance across all participating LLMs is only 0.072 – essentially the everyone-defects outcome, now aggregated across the behaviour from Figure 1. Repetition raises performance to 0.587 while mediation achieves 0.695. The two reputation variants fare worse: Reputation− shows a co-player's own recent interactions and reaches 0.321, while Reputation+ also traces the outcomes of that player's former partners, their former partners, and so on, reaching 0.227. Contracting is the standout performer at 0.801.
The cooperation mechanisms are therefore not equally easy for current LLM agents to use. One possible explanation is strategic legibility: how easily an agent can understand what a cooperation mechanism changes and translate it into a good decision. Contracts make altered incentives explicit, and mediation turns the final choice into whether to delegate to a conditional policy. Reputation, on the other hand, requires an agent to interpret histories, infer what they predict, and decide how to respond. The richer Reputation+ variant performs worse than Reputation−, suggesting that more strategically relevant information can become clutter. In the agent's reasoning traces when acting within the reputation mechanisms, they also expressed uncertainty about other players' intentions or strategies far more often than under any other mechanism.
Note that in its current form, CoopEval compares specific implementations, not entire mechanism families. We implement only one or two variants of each family because much of the initial effort went into creating the comparative framework. The success of mediation and contracting may depend on the proposal-and-vote procedures used to select a single mediator or contract. Conversely, our reputation implementations may have struggled in part because they expose interaction histories without giving agents a way to agree on a shared social norm for interpreting them. Other implementations could produce a different ranking, so these results should be read as a starting point rather than a verdict on the mechanism families.
The above results are derived from an LLM population where every model has the same initial population share. But deployed ecosystems may not remain uniform. Agents that produce higher individual payoffs may be copied, selected, or used more often. We approximate this process with replicator dynamics: strategies that perform better against the current population become more prevalent over time.
Selection pressure widens the gap between populations with cooperation mechanisms and those without them. Without a mechanism, the only model that sometimes cooperates (GPT-4o) is selected out, driving aggregate fitness from 0.072 to 0.021. Under the mechanisms, the direction reverses: repetition reaches 0.992, mediation 1.000, and contracting 0.999. Fortunately, our results show that cooperation mechanisms can make cooperation – not defection – the LLM behaviour rewarded by evolutionary pressure.
Taken together, these results show one way theory and practice can diverge. Looking ahead, we envision a research agenda that connects the two more tightly, using failures to reach theoretically available cooperative equilibria to guide better mechanism design.
Looking beyond CoopEval, several more cooperation mechanisms from the literature remain to be added to the leaderboard. One is similarity-based cooperation: should an agent cooperate with a stranger if both know they tend to make similar decisions? We operationalise this idea for LLM agents using CoopEval, building on earlier work (Oesterheld et al. 2023) and evidential decision theory (Jeffrey 1965). In our experiments with current LLMs, cooperation generally rises with reported similarity, yet we find that similarity-induced cooperation can be poorly grounded: it changes little depending on whether the score comes from strategically meaningful evaluations or arbitrary random controls, and agents tend to overestimate their similarity to other models. These findings foreshadow the questions we take up next: beyond asking whether a cooperation mechanism works in typical settings, we need to stress-test its robustness to unusual or manipulable inputs and its effects on the wider system.
So far, ‘cooperation’ has referred to agents reaching mutually beneficial outcomes within the games they play. In a larger system, the normative picture is more complicated. Agents might cooperate to provide a useful service or preserve a shared resource. They might also, for example, collude in ways that benefit participants at others’ expense. Evaluations therefore also need to ask who benefits and who bears the costs, rather than treating more cooperation as inherently safer.
In ongoing work, we ask whether cooperation survives stronger or more strategically optimised agents without exposing weaker agents to exploitation, whether mechanisms enable beneficial cooperation without also facilitating collusion that harms third parties, and how they perform under different communication protocols and against deceptive or exploitative adversaries. We also hope to carry the comparative framework beyond stylised games and into richer strategic environments, including sequential decision-making under partial observability, cooperation problems framed around real-world situations, and applications in agentic markets.
AI companies often release model cards that summarise what a model is designed to do, how it was evaluated, and where its limitations lie. We propose an analogous mechanism card for cooperation infrastructure. A mechanism card would specify a cooperation mechanism's infrastructure requirements, state its formal model and guarantees, document the implementation being tested, and report its empirical performance and observed safety failures. Different application domains give designers control over different aspects of an interaction, so there may be no universally best cooperation mechanism. Even where a mechanism is applicable, it may enable cooperation while introducing unacceptable risks. A mechanism card should make these domain-specific tradeoffs explicit.
Future AI agents may interact across organisational and technical boundaries, pursuing objectives that no single designer controls. Cooperation infrastructure offers a powerful intervention point: it can make mutually beneficial behaviour rational without requiring every agent to share the same values. The task ahead is not simply to make AI agents cooperate more, but to build a comparative science of when cooperation mechanisms work, why they fail, and whether the cooperation they sustain is indeed safe and robust. We invite you to join us on this research mission. The CoopEval code and framework are available on GitHub, and you are very welcome to reach out via email.
PhD Student, Computer Science Department of Carnegie Mellon University
