Foundation

As announced in a recent Nature commentary, the Cooperative AI Foundation (CAIF) is a new charitable entity, backed by an initial philanthropic commitment from the Center on Emerging Risk Research of $15 million. CAIF's mission is to support research that will improve the cooperative intelligence of advanced AI for the benefit of all humanity. For updates and funding opportunities, please sign up to the Cooperative AI mailing list.

Board of Trustees

Allan Dafoe
Eric Horvitz
Gillian Hadfield
Dario Amodei
Ruairí Donnelly

Statement from the Funder

"We have committed $15 million USD in support of the Cooperative AI Foundation. We are a Swiss nonprofit focused on enabling and coordinating research on how to improve the quality of life of future generations, [and] are supportive of the Cooperative AI Foundation’s mission."

– Center for Emerging Risk Research

Mission

Importance, Neglectedness, and Tractability


Scientific progress is often hard to predict. Great research is frequently driven by intrinsic curiosity, and it can be difficult to say in advance what the most important research directions for a field will turn out to be. On the other hand, we think that there are features which make some research directions more promising than others. Following an increasing number of philanthropic organisations (see here, for example), CAIF is guided by the importance, neglectedness, and tractability of potential activities, while maintaining an awareness of the value of curiosity and serendipity for great science.

Importance. Is this research area likely to make an important contribution to the cooperative intelligence of advanced AI systems for the benefit of all humanity?

Neglectedness. Is this work likely to be done anyway?

Tractability. Does the research area lend itself especially well to making progress?

While tractability is a common dimension along which to assess potential research, our emphasis on the qualities of importance and neglectedness differ somewhat from their standard interpretations in academia. For instance, a result may be viewed as particularly important within the researcher’s own field, but to warrant the highest prioritisation such work should also be likely to help us build more cooperative AI systems, now and in the future. Likewise, in many fields there may be multiple research groups knowingly racing to solve the same problem, but from the perspective of counterfactual impact this may not be the kind of research that’s essential to prioritise, because it’s likely to happen with or without our support.

Supporting research that will improve the cooperative intelligence of advanced AI for the benefit of all humanity

Differential Progress


In the course of building AI systems to be more cooperatively intelligent, those systems might also gain capabilities that could be used to harm others rather than contribute to improvements in social welfare. The alignment problem is one example of this: as AI systems become more generally intelligent, divergences in their goals and humans’ become more dangerous for humans. In the context of Cooperative AI, the ability to understand other agents can lead to improvements in cooperation, but also in deception and manipulation, and the same abilities that allow one to commit to honouring mutually beneficial agreements could also be used to commit to coercive threats.

With these risks in mind, CAIF is interested in supporting differential progress on cooperative intelligence. That is, we want to support research that leads to significant progress on cooperative capabilities – capabilities that lead to increases in social welfare in a wide range of environments – relative to progress on capabilities that are dual-use (e.g., useful for deception, manipulation, disempowering other agents) and therefore may not robustly improve social welfare. This idea is discussed in further detail in a recent seminar from the New Directions in Cooperative AI series.

Improvements

Improvements that CAIF prioritises are counterfactual and long-term, i.e., those improvements over the next 10-20 years (or longer) that would have been unlikely without our support (see also the discussion of 'Neglectedness' further below).

Advanced AI

Advanced AI systems include not only the present day state-of-the-art, but the kinds of powerful AI systems we can expect to see in the next 10-20 years, and the networks of humans and organisations in which they are embedded.

The Benefit of All Humanity

The benefit of all of humanity is our fundamental concern, and highlights the fact that not all advances in Cooperative AI may be beneficial for everyone; we must take into account different perspectives and values.

Cooperative Intelligence

Cooperative intelligence refers to the skills required for promoting cooperation between humans, machines, or organisations, though further research is required to fully conceptualise and define these skills.

Supporting Research

Supporting research includes standard academic grantmaking, but also fostering research in other ways, such as organising workshops and other events, supporting students, awarding prizes, and providing educational tools.

Activities

Intro text about our activities

01
Grantmaking

CAIF intends to use its philanthropic endowment to:

– Make grants to support Cooperative AI research, especially that which is important, tractable, and neglected. This includes work which helps to build up the infrastructure of the field, such as novel benchmark environments and metrics of cooperative success.

– Offer scholarships to promising young researchers intent on entering the field of Cooperative AI.Details on calls for proposals and applications forthcoming.

02
Workshops

In 2020, the first Cooperative AI workshop was organised at NeurIPS. CAIF intends to continue to organize workshops at major machine learning conferences, including IJCAI, AAAI, AAMAS, and NeurIPS.

03
Seminar series

CAIF will host a series of online seminars featuring scholars working on the frontier of Cooperative AI. Further details of our first seminar series, New Directions in Cooperative AI, and our call for seminar proposals can be found here.

04
Other activites

CAIF will explore additional ways of contributing to the growth of Cooperative AI, including administering prizes and hosting tournaments which encourage progress in our understanding of the cooperative intelligence of AI systems.

Foundation Activities

Announcements

Announcements

Evaluation for Cooperative AI

Call for Proposals

CAIF is seeking proposals for work to improve our ability to evaluate cooperation-relevant features of AI systems. Examples of such work include theoretical contributions on how to define and measure key Cooperative AI concepts, and practical contributions such as benchmark environments and datasets. Anyone is eligible to apply, and we welcome applications from disciplines outside of computer science.

Learn more
New Directions in Cooperative AI

Call for Proposals

CAIF is hosting a fortnightly seminar series on New Directions in Cooperative AI, in which we invite leading thinkers to offer their vision for research on cooperative AI. Unlike typical academic talks, these seminars are explicitly agenda-setting, describing a line of work that many researchers could pursue and that CAIF could support. We are currently accepting proposals for new talks.

Learn more
Foundations of Cooperative AI Lab

Grant

CAIF is pleased to announce that its first major grant, of $500,000, will be to support the Foundations of Cooperative AI Lab (FOCAL) at Carnegie Mellon University (CMU), led by Prof. Vincent Conitzer. Another $3 million will be provided by the Center on Emerging Risk Research. FOCAL’s goal is to create foundations of game theory appropriate for advanced, autonomous AI agents – with a focus on achieving cooperation. Prospective PhD students interested in working at the lab are encouraged to apply for a PhD program at CMU's Computer Science Department.

Learn more
Foerster AI Research

Grant

CAIF will also be granting £200,000 in order to support the work of Dr. Christian Schroeder de Witt, who will join Associate Prof. Jakob Foerster’s new lab on multi-agent learning at the University of Oxford. Schroeder de Witt’s work focuses mainly on deep multi-agent reinforcement learning and human-AI interaction, though he is also spearheading interdisciplinary research on the intersection of agent-based modeling, probabilistic inference, and climate policy. Prospective PhD students interested in working at the lab are encouraged to apply to both the Engineering Department and the AIMS CDT at Oxford.

Learn more

Competitions

Grants

Competitions

Grants

Competitions

Grants

Maxime Et Quis Ipsum

Voluptatum molestiae aliquid neque dicta. Eaque rerum perspiciatis non quibusdam minima. Enim impedit aut neque ipsa dolorem ut praesentium consequatur alias. Cumque rem aut quia hic praesentium deserunt pariatur nemo dolorem. Non repudiandae est tenetur sint. A necessitatibus

Link
Et

Quae ipsum quidem tempore blanditiis ducimus. Nihil vel maxime iste possimus veniam est consectetur dolor. Tempora eligendi et voluptatem et aspernatur aut iste. Provident quo laudan

Link
Explicabo Distinctio

Nobis iusto consequatur sunt voluptatem est tempore sunt perspiciatis totam. Atque molestiae ullam quis autem eaque cumque culpa. Vel et modi qui id sit. Et qui nisi dolorem aut nam nemo. Sit laudantium ea non doloremque

Link
Doloremque Voluptatibus Vel Culpa

Quia aut ratione et iure delectus quae magnam eum laudantium. Impedit ut neque. Est voluptas aut incidunt quam. Ipsum molestias non sint ut autem. Est nam simil

Link
Qui Pariatur Beatae

Iste quas consequuntur sed qui qui incidunt esse vero cumque. Blanditiis illo optio. Eum iure dolore et quaerat dolorem aut ab rem repudiandae. Quis culpa in accusantium et iste at beatae non. Doloremque libero dolor consequatur corporis. Praesentium eos quia id nemo neque nobis maio

Link
Beatae

Omnis animi commodi ut consequuntur natus totam optio. Sapiente qui quis. Accusantium natus ut. T

Link
Maxime Et Quis Ipsum

Voluptatum molestiae aliquid neque dicta. Eaque rerum perspiciatis non quibusdam minima. Enim impedit aut neque ipsa dolorem ut praesentium consequatur alias. Cumque rem aut quia hic praesentium deserunt pariatur nemo dolorem. Non repudiandae est tenetur sint. A necessitatibus

Link
Et

Quae ipsum quidem tempore blanditiis ducimus. Nihil vel maxime iste possimus veniam est consectetur dolor. Tempora eligendi et voluptatem et aspernatur aut iste. Provident quo laudan

Link
Explicabo Distinctio

Nobis iusto consequatur sunt voluptatem est tempore sunt perspiciatis totam. Atque molestiae ullam quis autem eaque cumque culpa. Vel et modi qui id sit. Et qui nisi dolorem aut nam nemo. Sit laudantium ea non doloremque

Link

Competitions

Grants

Doloremque Voluptatibus Vel Culpa

Quia aut ratione et iure delectus quae magnam eum laudantium. Impedit ut neque. Est voluptas aut incidunt quam. Ipsum molestias non sint ut autem. Est nam simil

Link
Qui Pariatur Beatae

Iste quas consequuntur sed qui qui incidunt esse vero cumque. Blanditiis illo optio. Eum iure dolore et quaerat dolorem aut ab rem repudiandae. Quis culpa in accusantium et iste at beatae non. Doloremque libero dolor consequatur corporis. Praesentium eos quia id nemo neque nobis maio

Link
Beatae

Omnis animi commodi ut consequuntur natus totam optio. Sapiente qui quis. Accusantium natus ut. T

Link

How to apply for opportunities

Funding

Evaluation for Cooperative AI

CAIF is seeking proposals for research that improves our ability to evaluate cooperation-relevant features of AI systems. At this early stage in the development of the Cooperative AI field, we think it is crucial that we attain conceptual clarity around cooperative intelligence and related concepts, are able to measure key aspects of cooperative behavior, and develop a collection of benchmarks rich enough to yield generalizable insights about Cooperative AI. Anyone is eligible to apply, and we welcome applications from disciplines outside of computer science.

Learn more

Submission Deadline: Rolling
New Directions in Cooperative AI

CAIF is hosting a fortnightly seminar series on New Directions in Cooperative AI, in which we invite leading thinkers to offer their vision for research on cooperative AI. Unlike typical academic talks, these seminars are explicitly agenda-setting, describing a line of work that many researchers could pursue and that CAIF could support. We are currently accepting proposals for new talks.

Learn more

Submission Deadline: Rolling
Urgent Funding Enquiry

We are not yet in a position to accept general grant applications, though please revisit this page soon for updates. As a temporary measure, we may be able to award smaller grants in highly time-sensitive or otherwise exceptional cases. If you believe this applies to you, please contact us via the link below, but note that we may not be able to respond to all enquiries in a timely manner.

Learn more

Submission Deadline: Rolling

Jobs

Chief Operating Officer

The role of Chief Operating Officer will be critical for the scaling and smooth running of the foundation, both now and in the years to come. We believe this marks an exciting opportunity to have a particularly large impact on the growth of this nascent field and organisation, and thus on the benefits to humanity that it prioritises.

Learn more

Deadline: 23 June 2022 23:59 UTC

Closed applications

No items found.

Staff

Lewis Hammond
Acting Executive Director
Jesse Clifton
Research Analyst
Amrit Sidhu-Brar
Operations Associate

Trustees

‍Allan Dafoe
Senior Staff Research Scientist, DeepMind
President, Centre for the Governance of AI
Eric Horvitz
Chief Scientific Officer, Microsoft
‍Gillian Hadfield
Director, Schwartz Reisman Institute for Technology and Society
Professor, University of Toronto
‍Dario Amodei
CEO, Anthropic
Ruairí Donnelly
President, Center for Emerging Risk Research

Advisors

Vincent Conitzer
Professor, Carnegie Mellon University
Professor, University of Oxford