.jpg)
From 3-7 August 2026, the Cooperative AI Foundation brought together 45 early-career researchers and professionals at Victoria College in Toronto for five intensive days of talks, poster sessions, and practical project work to advance their careers in cooperative AI.
The Summer School programme featured talks from eight guest speakers and CAIF staff, covering research priorities, the role of the law and of institutions in cooperative AI, and methods for building AI systems that can cooperate safely and effectively.
The first two talks explored some of the key questions shaping research priorities in cooperative AI.
Lewis Hammond (Cooperative AI Foundation) presented a framework for understanding cooperative AI and prioritising research within the field. He discussed considerations researchers should take into account when choosing projects, including how future AI capabilities could change the importance of different problems, as well as the potential for dual-use and backfire risks. He highlighted several promising research areas, such as multi-agent evaluations, detecting and preventing AI collusion, and agentic inequality.
Caspar Oesterheld (Redwood Research) explored the equilibrium selection problem: how AI agents can coordinate on beneficial outcomes when strategic interactions have multiple possible equilibria. He introduced two promising research directions: designing safe interventions that can improve outcomes without needing to predict which equilibrium agents will select, and improving LLMs’ conceptual reasoning on complex problems such as fair bargaining, ethics, and equilibrium selection, including through better evaluation datasets and benchmarks.
Three talks examined the interplay between AI and human institutions: how norms and legal systems might steer AI towards safe and cooperative behaviour, and how AI could help groups deliberate and find common ground.
Nisarg Shah (University of Toronto) explored how AI can help groups with conflicting preferences find common ground. Drawing on online deliberation platforms such as Polis and Remesh, he discussed approaches for identifying proposals that bridge different groups. This included using LLMs to generate entirely new proposals that reconcile existing preferences, rather than selecting between a fixed set of options. He also presented a broader approach to using LLMs in decision-making that could extend to multi-agent systems, helping AI agents with conflicting goals generate options that better represent and bridge their different preferences.
Noam Kolt (Hebrew University) discussed legal alignment: how legal methods and principles can inform the alignment of AI systems, and how AI systems might be designed to follow the law. He also explored how AI agents are already becoming de facto subjects, users and producers of law, raising questions about whether and why increasingly powerful AI systems would comply with legal obligations, and what this could mean for the rule of law.
Tan Zhi-Xuan (National University of Singapore) explored pluralistic AI futures: scenarios in which a wide range of AI agents, systems and services are deployed by different actors towards different ends. She considered how we can ensure that such a future develops safely and in the direction of mutual flourishing. She introduced her recent work on rational cooperative AI, a paradigm for building coherent AI agents that are cooperative by design, as well as research into designing institutions that can support AI–human cooperation at scale.
The programme also explored how we can build AI systems that cooperate safely and effectively in practice.
Zhijing Jin (University of Toronto) presented research on evaluating and improving cooperation among LLM agents. Her talk examined how agents behave in social dilemmas and other multi-agent environments, including why they sometimes fail to cooperate. She discussed the roles of mathematical and causal reasoning, morality, communication, and deception, alongside new approaches for evaluating these behaviours and testing mechanisms that could foster cooperation. She also introduced tools for making multi-agent simulations more accessible and outlined research directions for understanding and mitigating risks such as strategic deception and exploitation.
Marc Lanctot (Google DeepMind) argued that multi-agent AI needs to move beyond models of perfectly rational, adversarial agents and instead account for mixed motives, adaptation and agents’ ability to model one another. He presented approaches including Policy-Space Response Oracles (PSRO) and Game of Thoughts, which use populations of strategies and adaptive reasoning to improve outcomes in complex strategic interactions. He concluded that safe multi-agent AI will require systems to model who they are interacting with, how those agents think, and how their behaviour changes over time, combining offline learning with adaptive reasoning.
.jpg)
The second half of the Summer School was dedicated to practical project work. Participants worked in teams to develop a proposal for a cooperative AI project, with the form of the proposed project left open: it could be conceptual or theoretical, empirical, policy-focused, or take the form of a research tool.
The project component began with a Theory of Change workshop led by Cecilia Tilli (Allelo), followed later in the week by a session from Lewis Hammond (Cooperative AI Foundation) on research methodology. Participants put their ideas through a red-teaming exercise to identify weaknesses and sharpen their approach.
The final project proposals covered a wide range of cooperative AI questions, including how secretly loyal AI systems might behave in multi-agent environments and how effectively LLMs can extrapolate human preferences when acting or voting on someone’s behalf.
Matīss Apinis, Nathaniel Sauerberg, Akash Agrawal, and Alessandro Veneri won the Best Project Award. Their proposal explored one-shot, two-player bargaining games with imperfect information, asking whether modifying agents’ information structures could produce safe Pareto improvements. The team excelled at integrating concepts from across the Summer School into their project, including the Importance–Neglectedness–Tractability framework, Theory of Change, and Backfire Risks.
Participants also had the opportunity to present their research during the Summer School’s poster sessions al fresco.
Claude Formanek received the Best Poster Award for ‘Molten Pot: Evaluations & Datasets for Social Offline Reinforcement Learning’. Akash Agrawal and Nathaniel Sauerberg each received a Runner-up Best Poster Award for ‘The Multi-Agent Off-Switch Game‘ and ‘Promises Made, Promises Kept: Safe Pareto Improvements via Ex Post Verifiable Commitments' respectively.
We also awarded honourable mentions to: Joseph Low and Oscar Duys for ‘Delegating Deliberation to AI Representatives’, Botao ‘Amber’ Hu for their poster based on ‘Dissociative Identity: Language Model Agents Lack Grounding for Reputation Mechanisms’, and Matīss Apinis for ‘Epistemic Behavior of Large Language Models in Anthropic Reasoning Problems’.
Alongside the academic programme, the week included welcome drinks, a Monday dinner, an informal picnic at Queen’s Park, and a social hosted by Trajectory Labs featuring a talk by Lewis Hammond (Cooperative AI Foundation) on research prioritisation in cooperative AI.
Summer School recordings are now available on YouTube. To hear about future events, subscribe to the Cooperative AI Foundation mailing list.
Thank you to Nisarg Shah and the Schwartz Reisman Institute for Technology and Society for sponsoring all of the award prizes.
September 2, 2026
