As announced in a recent Nature commentary, the Cooperative AI Foundation (CAIF) is a new charitable entity, backed by a large philanthropic commitment. CAIF's mission is to support research that will improve the cooperative intelligence of advanced AI for the benefit of all. For updates and funding opportunities, please sign up to the Cooperative AI mailing list.
Board of Trustees
Allan Dafoe
Eric Horvitz
Gillian Hadfield
Dario Amodei
Ruairí Donnelly
Mission
Importance, Neglectedness, and Tractability
Scientific progress is often hard to predict. Great research is frequently driven by intrinsic curiosity, and it can be difficult to say in advance what the most important research directions for a field will turn out to be. On the other hand, we think that there are features which make some research directions more promising than others. Following an increasing number of philanthropic organisations (see here, for example), CAIF is guided by the importance, neglectedness, and tractability of potential activities, while maintaining an awareness of the value of curiosity and serendipity for great science.
Importance. Is this research area likely to make an important contribution to the cooperative intelligence of advanced AI systems for the benefit of all?
Neglectedness. Is this work likely to be done anyway?
Tractability. Does the research area lend itself especially well to making progress?
While tractability is a common dimension along which to assess potential research, our emphasis on the qualities of importance and neglectedness differ somewhat from their standard interpretations in academia. For instance, a result may be viewed as particularly important within the researcher’s own field, but to warrant the highest prioritisation such work should also be likely to help us build more cooperative AI systems, now and in the future. Likewise, in many fields there may be multiple research groups knowingly racing to solve the same problem, but from the perspective of counterfactual impact this may not be the kind of research that’s essential to prioritise, because it’s likely to happen with or without our support.
Differential Progress
In the course of building AI systems to be more cooperatively intelligent, those systems might also gain capabilities that could be used to harm others rather than contribute to improvements in social welfare. The alignment problem is one example of this: as AI systems become more generally intelligent, divergences in their goals and humans’ become more dangerous for humans. In the context of Cooperative AI, the ability to understand other agents can lead to improvements in cooperation, but also in deception and manipulation, and the same abilities that allow one to commit to honouring mutually beneficial agreements could also be used to commit to coercive threats.
With these risks in mind, CAIF is interested in supporting differential progress on cooperative intelligence. That is, we want to support research that leads to significant progress on cooperative capabilities – capabilities that lead to increases in social welfare in a wide range of environments – relative to progress on capabilities that are dual-use (e.g., useful for deception, manipulation, disempowering other agents) and therefore may not robustly improve social welfare. This idea is discussed in further detail in a recent seminar from the New Directions in Cooperative AI series.
Improvements that CAIF prioritises are counterfactual and long-term, i.e., those improvements over the next 10-20 years (or longer) that would have been unlikely without our support (see also the discussion of 'Neglectedness' further below).
Advanced AI systems include not only the present day state-of-the-art, but the kinds of powerful AI systems we can expect to see in the next 10-20 years, and the networks of humans and organisations in which they are embedded.
The benefit of all is our fundamental concern, and highlights the fact that not all advances in Cooperative AI may be beneficial for everyone; we must take into account different perspectives and values.
Cooperative intelligence refers to the skills required for promoting cooperation between humans, machines, or organisations, though further research is required to fully conceptualise and define these skills.
Supporting research includes standard academic grantmaking, but also fostering research in other ways, such as organising workshops and other events, supporting students, awarding prizes, and providing educational tools.
Intro text about our activities
CAIF intends to use its philanthropic endowment to:
– Make grants to support Cooperative AI research, especially that which is important, tractable, and neglected. This includes work which helps to build up the infrastructure of the field, such as novel benchmark environments and metrics of cooperative success.
– Offer scholarships to promising young researchers intent on entering the field of Cooperative AI.Details on calls for proposals and applications forthcoming.
In 2020, the first Cooperative AI workshop was organised at NeurIPS. CAIF intends to continue to organize workshops at major machine learning conferences, including IJCAI, AAAI, AAMAS, and NeurIPS.
CAIF will host a series of online seminars featuring scholars working on the frontier of Cooperative AI. Further details of our first seminar series, New Directions in Cooperative AI, and our call for seminar proposals can be found here.
CAIF will explore additional ways of contributing to the growth of Cooperative AI, including administering prizes and hosting tournaments which encourage progress in our understanding of the cooperative intelligence of AI systems.
Announcements
Announcements
Grant
CAIF has recently agreed several new grants, totalling over $1 million, to a number of leading cooperative AI researchers, on topics ranging from the role of intent in cooperation, to learning the preferences of multiple agents, to interactions between language models. More details will follow soon.
Learn moreStaff Announcement
We delighted to have Akbir Kahn joining CAIF as a research analyst. Akbir will be leading on the development of a cooperative AI contest and building on his work in multi-agent reinforcement learning and language modelling.
Learn moreStaff Announcement
CAIF is happy that Cecilia Tilli will be joining the foundation to lead on restructuring its grantmaking program and developing several new programs and initiatives to support junior and early career researchers.
Learn moreStaff Announcement
We’re looking for a Managing Director to join the Cooperative AI Foundation (CAIF), a nascent non-profit with huge potential to improve the safety of future AI. Applications close on 30 July 2023.
Learn moreCompetitions
Grants
Competitions
Grants
Voluptatum molestiae aliquid neque dicta. Eaque rerum perspiciatis non quibusdam minima. Enim impedit aut neque ipsa dolorem ut praesentium consequatur alias. Cumque rem aut quia hic praesentium deserunt pariatur nemo dolorem. Non repudiandae est tenetur sint. A necessitatibus
LinkQuae ipsum quidem tempore blanditiis ducimus. Nihil vel maxime iste possimus veniam est consectetur dolor. Tempora eligendi et voluptatem et aspernatur aut iste. Provident quo laudan
LinkNobis iusto consequatur sunt voluptatem est tempore sunt perspiciatis totam. Atque molestiae ullam quis autem eaque cumque culpa. Vel et modi qui id sit. Et qui nisi dolorem aut nam nemo. Sit laudantium ea non doloremque
LinkQuia aut ratione et iure delectus quae magnam eum laudantium. Impedit ut neque. Est voluptas aut incidunt quam. Ipsum molestias non sint ut autem. Est nam simil
LinkIste quas consequuntur sed qui qui incidunt esse vero cumque. Blanditiis illo optio. Eum iure dolore et quaerat dolorem aut ab rem repudiandae. Quis culpa in accusantium et iste at beatae non. Doloremque libero dolor consequatur corporis. Praesentium eos quia id nemo neque nobis maio
LinkOmnis animi commodi ut consequuntur natus totam optio. Sapiente qui quis. Accusantium natus ut. T
LinkFunding
CAIF is hosting a fortnightly seminar series on New Directions in Cooperative AI, in which we invite leading thinkers to offer their vision for research on cooperative AI. Unlike typical academic talks, these seminars are explicitly agenda-setting, describing a line of work that many researchers could pursue and that CAIF could support. We are currently accepting proposals for new talks.
Learn more
We are not yet in a position to accept general grant applications, though please revisit this page soon for updates. As a temporary measure, we may be able to award smaller grants in highly time-sensitive or otherwise exceptional cases. If you believe this applies to you, please contact us via the link below, but note that we may not be able to respond to all enquiries in a timely manner.
Learn more
Jobs
We’re looking for a managing director to join the Cooperative AI Foundation, a nascent non-profit with huge potential to improve the safety of future AI. We’ve already got the basics in place: our mission, values, a theory of change, initial funding, a board of stellar trustees, and lots of ideas for activities. Now we need a knowledgeable and energetic leader to bring it all to life.
Learn more
Closed applications
CAIF is seeking proposals for research that improves our ability to evaluate cooperation-relevant features of AI systems. At this early stage in the development of the Cooperative AI field, we think it is crucial that we attain conceptual clarity around cooperative intelligence and related concepts, are able to measure key aspects of cooperative behavior, and develop a collection of benchmarks rich enough to yield generalizable insights about Cooperative AI. Anyone is eligible to apply, and we welcome applications from disciplines outside of computer science.
Staff
Trustees
