Examples of the research topics we consider in scope for the Cooperative AI PhD Fellowship 2027, and the technical and methodological approaches we encourage.
← Back to the PhD Fellowship 2027These reflect our current focus (our research priorities are being refreshed in 2027) and are not exhaustive. We welcome strong proposals on any multi-agent/cooperation problem involving AI, so you should not be put off from applying if your topic isn’t listed.
In this area, we would like to see proposals for work on definitions, metrics and methods for the evaluation of cooperation-relevant capabilities of AI systems. We believe that work on cooperation-relevant capabilities will be important to create a foundation for the development of AI systems with desirable, cooperative properties.
Defining and measuring cooperation-relevant capabilities:
Theoretical work on the identification and definition of cooperation-relevant capabilities, including the development of rigorous arguments on whether those capabilities are desirable for AI systems (in general, or under specific conditions relevant to important real-world cases).
Empirical work on methods for evaluating or measuring such cooperation-relevant capabilities in frontier AI systems.
Investigating the causes of cooperation-relevant capabilities:
Theoretical work on how cooperation-relevant capabilities could arise in realistic AI systems, for example through training processes (including fine-tuning and in-context learning).
Empirical work on how cooperation-relevant capabilities arise in real systems.
Theoretical and empirical work investigating the extent to which the differential development of beneficial cooperative capabilities is possible. Such work could include assessments of whether we should expect certain capabilities to affect cooperation in a net-positive or net-negative direction overall, in general, or in specific settings.
Theoretical and empirical work on how asymmetries in agents' capabilities and/or bounded rationality could affect cooperation.
Note the distinction between cooperative capabilities (what the system can do) and cooperative propensities (what the system tends to do).
For this area it is important to note that many capabilities are dual-use. Definitions and metrics that distinguish between desirable and undesirable (uses of) capabilities will be especially valuable as this can facilitate differential development of safe and beneficial systems. We are particularly likely to fund work on definitions or metrics that take this distinction into account.
For the study of how cooperation-relevant capabilities arise, we expect to only fund work that aims to draw general conclusions about causal relationships (e.g. between a feature of an agent's training and the agent's development of a cooperation-relevant capability).
We strongly prefer applications that make use of state-of-the-art methods for capabilities evaluations.
For empirical work, we are open to proposals either using foundation models or other approaches such as multi-agent reinforcement learning. However, please note the general guidelines on the importance of clarity about path to impact and justifying why the results should be expected to apply to complex settings with more advanced agents.
In this area, we would like to see proposals for work on definitions, metrics and methods of evaluation for cooperation-relevant propensities of AI systems. Such work would be important to create a foundation for the development of AI systems with desirable, cooperative properties. We are interested both in cooperation-enhancing and conflict-prone propensities, and in some cases the same propensity can be cooperation-enhancing or conflict-prone depending on the context (e.g. punitiveness). Note the distinction between cooperative capabilities (what the system can do) and cooperative propensities (what the system tends to do, sometimes also referred to as dispositions). This section is about propensities. Examples of propensities could be valuing a certain fairness concept, seeking social influence over other agents, vengefulness or spite.
Defining and measuring cooperation-relevant propensities:
Theoretical work on the identification and definition of cooperation-relevant propensities, including the development of rigorous arguments regarding whether those propensities are desirable for AI systems (in general, or under specific conditions relevant to important real-world cases).
Empirical work on methods for evaluating or measuring such cooperation-relevant propensities in frontier AI systems.
Investigating the causes of cooperation-relevant propensities:
Theoretical work on how cooperation-relevant propensities could arise in realistic AI systems, for example through training processes (including fine-tuning and in-context learning).
Empirical work on how cooperation-relevant propensities arise in real systems.
Investigating how robust cooperation-relevant propensities are to further optimization pressure.
For work on cooperation-relevant propensities, it will be especially important to take exploitability into account. Cooperativeness should be distinguished from pure altruism, as an altruistic but exploitable agent (e.g., one who always cooperates in Prisoner's Dilemmas) would be unlikely to be deployed in most real-world scenarios. We are aiming for agent propensities that promote cooperative, mutually beneficial outcomes even in the presence of 'exploiter agents'. See the study of exploitability in Mukobi et al. (2023) for an example.
For the study of how cooperation-relevant propensities arise, we expect to only fund work that aims to draw general conclusions about causal relationships (e.g. between a feature of an agent's training and the agent's development of a cooperation-relevant propensity).
For evaluations of propensities, we believe it will often be efficient to make use of existing benchmarks when there exists an established benchmark that can evaluate the target property with appropriate precision.
For empirical work, we are open to proposals both using foundation models and other approaches such as multi-agent reinforcement learning. However, please note the general guidelines on clarity about path to impact and the importance of justifying why the results should be expected to apply to complex settings with advanced agents.
In this area, we would like to see proposals that address the question of how cooperation can be incentivised among self-interested AI agents in mixed-motive settings. We expect such work to be important in finding approaches that lead to societally beneficial outcomes when advanced AI agents with conflicting goals are deployed in the real world.
Theoretical and empirical work on peer (i.e. decentralised) incentivisation:
Development of realistic assumptions and models about methods of peer incentivisation (e.g. monetary) and domains of application.
Understanding and building the infrastructure required for decentralised (third-party) norm enforcement.
Scalable and secure methods for inter-agent commitments and contracting.
Minimising inefficiencies from sanctions.
Scaling of methods to incentivise cooperation:
Scaling opponent-shaping and peer incentivisdation to more complex agents and environments (including LLM agents).
Approaches in automated/adaptive mechanism design (i.e. centralised forms of incentivisation) that focus on scaling to very large numbers of agents and/or much more complex agents and environments (including LLM agents).
Conceptual and engineering work on designing infrastructure for interactions between agents that incentivises cooperation (e.g. that supports the development and implementation of prosocial norms and commitments).
For work in this area we want to emphasise that the approaches need to be applicable to 'self-interested' agents. We do not expect to fund any work where agents share rewards or that otherwise assumes we need to control how every agent is designed or trained to be able to achieve cooperative outcomes. The point is to develop approaches that allow us to promote beneficial outcomes even when the individual agents are selfishly optimising for their own goals.
For methods of peer incentivisation and opponent-shaping, we will be especially interested in techniques that:
Will (at least with high probability) converge to outcomes with high social welfare if adopted by everyone.
Result in an equilibrium if adopted by everyone.
Are robust to cases where not everyone adopts the algorithm/policy.
For work on mechanism design and infrastructure, it is important to consider how the work could be relevant for significant real-world scenarios. If you can point to which (kind of) institution would play the role of the mechanism designer in such a real-world scenario, this would help to strengthen the proposal.
We are interested in research that applies to AI agents specifically.
Much of the existing work in fields such as mechanism design is focused on narrow application areas such as auctions, which are not a priority for us.
Opponent-shaping and peer incentivisation are potentially dual-use, meaning progress in these areas may have a negative overall impact on society. For such proposals, it is therefore especially important to consider downside risks. We will not fund work where we think the contribution to downside risks outweighs the positive impact of the work.
In this area, we would like to see proposals to develop AI tools that help humans resolve major cooperation challenges. By virtue of their potentially greater ability to identify mutually beneficial agreements or to create novel institutional designs, for example, AI systems could have a huge positive impact via helping humans to cooperate.
Development of AI tools for improving collective decision-making in settings where the humans involved have conflicting interests. This includes the use of AI tools for policy development.
Development of AI tools to improve the collective outcomes of negotiation, bargaining and conflict resolution processes.
Development of AI tools to design socially beneficial institutions or public services for enabling cooperation.
For the proposal to be in scope, the AI system must be used to directly address the mixed-motive dynamics of the problem. This is an area where we receive many out-of-scope applications, so we encourage potential applicants for this area to read the scope description thoroughly and make sure they understand what is meant by cooperation problems in this context. Most uses of machine learning tools will not be in scope (e.g. text summarisation, image analysis, general data analysis, and most uses of interactive chatbot tools).
Note that this area is focused on how AI can facilitate cooperation among groups of humans. We do not expect to fund work on facilitating cooperation between individual humans and AI systems.
The proposal should make a good case for how the work would contribute to improved resolution of important, high-stakes cooperation problems. Examples of domains that we count as high-stakes are:
Climate change
Armed conflict
Large-scale democratic systems and policymaking
Institutional design for a society with widespread AI agent deployment
We do not expect to fund any work on social media dynamics, self-driving cars, improving market efficiencies or the efficiency of organisations/companies in general. The exception would be if any of those domains were used as a test domain to draw general conclusions that would be expected to apply to other important domains as well. We also receive a lot of out-of-scope applications for using AI to improve healthcare and agriculture. There could potentially be high-stakes cooperation problems in these domains that could be better solved using AI tools, but we particularly encourage applicants in these areas to review the scope description carefully.
We are interested in funding both early work exploring the possibility of AI facilitation in the different areas, and concrete implementations of such tools. We prioritise funding work that we believe is unlikely to happen without our support. This means that projects with a significant commercial value, or which could otherwise be expected to be developed by the private sector, are very unlikely to be funded by us.
Proposals should be explicit about the assumptions and definitions underlying what counts as a beneficial outcome. With a focus on how AI can support humans to improve the resolution of cooperation challenges, it will often be helpful if outcomes can be compared to human baseline performance.
If the work targets a specific application domain (e.g. climate change), we strongly encourage the involvement of experts from that domain (and intended end users where applicable) in the development of the proposal to ensure real-world relevance.
In this area, we would like to see work that studies what can go wrong when we don't want or expect AI agents to work together. Collusion (undesired cooperation) between agents could for example lead them to bypass safeguards or laws. We believe work in this area will be important for monitoring and governance as deployment of advanced and interacting AI systems becomes more widespread throughout society.
Development of methods for detection of collusion between AI systems, including steganographic collusion. Such work could, for example, build on using information-theoretic measures or interpretability tools.
Development of mitigation strategies for preventing collusion such as oversight regimes, methods for steering agents, restricted communication protocols and/or control of communication channels.
Theoretical and empirical work that aims to provide general results about the conditions that make collusion easier or harder between AI agents, such as the similarity of the agents, forms of communication, the number of agents, the environmental structure, agents' objectives, etc.
Development of benchmarks and evaluations for measuring AI agents' ability and propensity to collude. We would be particularly interested in work on creating a major, complex benchmark environment for collusion. Such a benchmark could assess capabilities and/or propensities that influence the likelihood of collusion, and should be ambitious in its aims to significantly advance research on collusion.
We expect it to be challenging to do work in this area that will be informative about advanced AI systems. Proposals should explicitly aim for results that can be expected to generalise to large and complex systems consisting of advanced AI agents, and it is important that the phenomena studied can reasonably be expected to be significant for the societal impact of AI systems in the long term.
For work on conditions that make collusion more or less likely, the expected generality of the results would be crucial in the assessment of the proposal. Merely demonstrating the correlation between certain conditions in a specific setting and the likelihood of collusion in that setting is not something that we are likely to fund.
In this area, we would like to see proposals for work that can improve our understanding of some specific types of multi-agent dynamics involving advanced AI systems. This includes emergent phenomena (behaviours, goals and capabilities) that are not present in the individual agents or systems, but that arise specifically in the multi-agent system. We believe that this kind of work will be important to identify, monitor and mitigate new risks that arise as deployment of advanced and interacting AI systems becomes more widespread throughout society.
Work on destabilising dynamics, which could aim to answer questions about the conditions under which multi-agent systems involving AI have undesirable dynamics and how such phenomena can be monitored and stabilised. Such work may cover aspects such as how the number of agents, their objectives, and the features of their environment might precipitate undesirable dynamics.
Work on prevention of correlated failures, that could arise due to similarities and shared vulnerabilities among agents in the multi-agent system. This could include work on the impact of AI agents learning from data generated by each other on shared vulnerabilities, correlated failure modes, and their ability to cooperate/collude.
Work on which network structures and interaction patterns lead to more robust or fragile networks of AI agents, and the development of tools for overseeing and controlling the dynamics and co-adaptation of networks of advanced AI agents. This might include 'infrastructure for AI agents' such as interaction protocols.
Theoretical and empirical work on establishing the conditions under which unexpected and undesirable goals and capabilities might emerge from multiple AI agents, how robust such phenomena are, and how quickly they can occur. Comparisons across specific scenarios could help to establish conditions under which these emergent phenomena are more likely, such as the degree of competition, complementarity of agents, access to particular resources, or task features.
We expect it to be challenging to do work in this area that will be informative about advanced AI systems. Proposals should explicitly aim for results that can be expected to generalise to large and complex systems consisting of advanced AI agents, and it is important that the phenomena studied can reasonably be expected to be significant for the societal impact of AI systems in the long term.
In this area, we would like to see proposals for work on how information and transparency affects cooperation. Information asymmetries (both strategic uncertainty about what agents will do and structural uncertainty about private information that others have) are a prime cause of cooperation failure. AI agents are adept at processing vast swathes of information and have features (such as being defined by software) that might make this a promising area in which AI agents could overcome the challenges faced by humans.
Work on how the potential transparency and/or predictability of agents (e.g. through black-box or white-box access to their source code) can be used to understand and control the extent to which they cooperate. This predictability might emerge due to the similarity of agents and their ability to reason about each other.
Work on scaling automated information design (e.g. 'Bayesian persuasion') to more complex agents and environments (including LLM agents).
Implementing and scaling methods for secure information transmission/revelation between AI agents that enable cooperation. This might include work on the ability of agents to conditionally reveal and verify private information.
The development of efficient algorithms for few-shot coordination in high-stakes scenarios. This could include theoretical work (for example, establishing the amount of information required to predict the behaviour of other agents) and empirical work (for example, on generalising or applying few-shot coordination algorithms to complex settings and advanced agents).
We believe that there are relatively few crucial coordination problems that are inherently zero-shot (rather than few-shot), and we therefore do not expect to fund work on zero-shot coordination. However, if you believe you can make a good case for how such work would be important for improving the outcomes of real-world, high-stakes scenarios, you are welcome to submit a proposal on zero-shot coordination as well.
For methods for secure information transmission/revelation, it is important that this enables cooperation that would not otherwise be possible (for example, due to strategic considerations that make agents reluctant to reveal private information). General work on secure information transmission is not something we expect to fund.
In this area, we would like to see proposals for work on security challenges that arise as advanced and interacting AI systems become more widespread throughout society. Often these challenges will overlap somewhat with our other research areas.
Assessing what security vulnerabilities advanced multi-agent systems have that single-agent systems do not, and developing defence strategies for these vulnerabilities, such as improvements in network design, communication protocol design and/or information security.
Exploring how combinations of multiple AI systems can overcome existing safeguards for individual systems (and how they can be prevented from doing so).
Better understanding how robust cooperation is to adversarial attacks (for example, the injection of a small number of malicious agents, or the corruption of key data) in different settings.
We prioritise funding work that we believe is unlikely to happen without our support. This means that projects with a significant commercial value, or which could otherwise be expected to be developed by the private sector, are unlikely to be funded by us.
These apply across all research areas.
Our focus is on multi-agent, mixed-motive settings. We do not expect to fund work on fully cooperative settings. We also tend to favour proposals that consider (or could straightforwardly be applied to) cooperation problems with more than two agents.
We favour hypothesis-driven work over exploratory work. There can be exceptions to this rule, but for exploratory proposals it is especially important to justify why the approach was chosen and how the authors expect the results to be useful for other settings.
For projects where agents are trained, it is important that the proposal is clear on how the agents are trained. Work in which agents are trained on joint welfare objectives is not likely to be funded as we are focused on techniques that are applicable to 'self-interested' agents. We do not expect to fund any work where agents share rewards or that otherwise assumes we need to control how every agent is designed or trained to be able to achieve cooperative outcomes.
For most proposals we receive, an important aspect is to justify why the results should be expected to apply to complex settings with advanced agents. Especially for foundational or conceptual work it often makes sense to work with simplified models, but it is then crucial to explain how well the results are expected to generalise to the kind of settings and advanced agents that will be increasingly societally important.
For this reason, we tend not to fund work that depends on hand-crafted features or overly complex setups, for example, as results from such work are often less likely to scale or generalise.
We have observed that many proposals using MARL have been rejected based on concerns related to an expected lack of generality or scalability of the results. We therefore want to encourage applicants who propose work in MARL to especially consider this point.
To be clear, it is not required for proposals to directly make use of LLMs or related models.
Using tabula rasa (as opposed to pre-trained) models can be relevant in some instances, but this should then be justified. Examples of situations where this could be relevant are when there is little or no human data to learn from, and/or when doing so is not possible (e.g. for privacy reasons, computational reasons, etc.), and/or when we would expect much better performance from not doing so. There could also be cases when using some kind of pre-training wouldn't change the dynamics under investigation very much.