8 Prioritisation
Required Content 4hrs 15mins • All Content 4hrs 15mins
In this final section of the course we will look into prioritisation; out of all of these subareas and research directions, how can we choose what is most important to work on? We start out with a talk from the Cooperative AI Summer School of 2024 by Lewis Hammond.
Tooltip Text
This talk was given in 2024, and AI development is rapid. Is there anything that has changed since then which you think affects the recommendations given? Are there any recommendations that you think are timeless?
Tooltip Text
Inspired by Lewis’ talk, start to create your own personal research agenda, following this structure: how and why does your research matter (i.e. a theory of change for creating a positive impact); how does this agenda fit into what other people are working on; how do the projects you are doing (or want to do) fit into that agenda. This can be rough, it is important to just give it a good start. The agenda does not have to be Cooperative AI related, but it could be interesting to highlight where you think your agenda has been inspired by engaging with this curriculum.
Tooltip Text
Throughout the course we have been returning to different sections of the report Multi-Agent Risks from Advanced AI. We will now look at one of the final sections of that report which outlines how the fields of AI safety, AI governance and AI ethics can approach mitigation of multi-agent risks.
Tooltip Text
Select two or three recommended research directions from section ‘4 Implications’ of Multi-Agent Risks from Advanced AI which you find interesting. How would you prioritise between them, considering the advice from the talk above? You might want to revisit the section of the talk that starts at 14:56 (just the slide “Prioritising problems”).
Tooltip Text
Research rarely happens in isolation, and it is therefore valuable to be aware of which research labs and organisations are engaged in cooperative AI work if you are considering entering the field. A good general approach is to look up the authors of work that you are interested in - you can often find their contact details as well, either in the paper itself or on the personal webpage of the author.
There are also a couple of non-profits that support work in this area:
- The Future of Life Foundation is a quite new organisation, affiliated with the Future of Life Institute. Their Fellowship on AI for Human Reasoning incubates projects working on AI tools for coordination and epistemics, which has a lot of overlap with cooperative AI and in particular with AI for Human Cooperation.
- The Foresight Institute is a non-profit that was founded in 1986 with the purpose of “advancing frontier science and technology”, and one of their focus areas is “Secure AI” where they, among other things, fund work on multi-agent safety.
- The Collective Intelligence Project is an incubator for new governance models for transformative technology, and their work has a lot of overlap with AI for Human Cooperation in particular.
Of course, the Cooperative AI Foundation which produces this course also offers many different kinds of support for those that want to enter the field:
- Summer school: For the past three years we have hosted an in-person summer school targeting mainly PhD students, but that includes some undergraduates and early-career researchers as well.
- PIBBSS Cooperative AI track: We support a cooperative AI track in the PIBBSS summer research fellowship, which is a three month fellowship centrally aimed at Ph.D. or Postdoctoral researchers.
- MATS: We sometimes support cooperative AI focused projects in the MATS program, a 12 week research fellowship.
- Cooperative AI Research Fellowship: We are supporting a research fellowship focused on Cooperative AI that will take place in Cape Town early 2026.
- PhD fellowship program: We have a PhD fellowship program to support exceptional PhD students working on cooperative AI research.
- Grant program: We have regular calls for proposals and fund both large research projects and smaller grants for early career researchers.
Tooltip Text
Look at the Cooperative AI Foundation’s 2025 call for grant applications. Under the Scope heading, the Cooperative AI Foundation (CAIF) has listed ‘Understanding and Evaluating Cooperation-Relevant Capabilities’ and ‘Understanding and Evaluating Cooperation-Relevant Propensities’ as high priority areas. Look through the other areas of research listed and, imagine that the capacity of CAIF has increased, select one other research area that you think should be considered high priority. Explain your reasoning and discuss future developments that could shift your prioritisation.
Tooltip Text
Thank you for completing this curriculum! If you want to find even more content on cooperative AI, you can do so for example through the Cooperative AI youtube channel or consider signing up to the Cooperative AI newsletter.
Tooltip Text
