New Directions in Cooperative AI

Skills that lead to strong performance in multi-agent systems can be detrimental to social welfare. This is true even of skills that play a central role in cooperation: the ability to understand other agents can make it easier to deceive and manipulate them, and the ability to commit to peaceful agreements can also facilitate coercive commitments. We will argue that if the Cooperative AI Foundation is to achieve its mission of improving the cooperative intelligence of advanced AI systems for the benefit of all humanity, it should focus on improving skills that robustly lead to improvements in social welfare, rather than those that are dangerously dual-use.
We refer to this as “differential progress on cooperative intelligence”. To know whether we are making differential progress, we need to be able to rigorously define and measure it. Towards these ends, we present early-stage work on the definition and measurement of cooperative intelligence.

Jesse Clifton (Center on Long-Term Risk, CAIF, NCSU)
Sammy Martin (KCL, Center on Long-Term Risk)


Zoe Cremer (University of Oxford, University of Cambridge)
José Hernández-Orallo (Universitat Politècnica de València, University of Cambridge)


13:00-14:00 UTC 10 March 2022

This seminar has now finished

Google calendar event

ICS file

Zoom meeting (passcode sent via mailing list)

Jesse Clifton is a research analyst at the Cooperative AI Foundation and a researcher at the Center on Long-Term Risk, where he works on possible causes of conflict involving AI systems. He is also a PhD student in statistics at North Carolina State University.

Sammy Martin is a PhD student at the King's/Imperial Centre for Doctoral Training on Safe and Trusted AI. He has also worked as an AI forecaster for the Modelling Transformative AI Risks project and an independent researcher supported by the Center on Long-Term Risk. He earned his Master's degree in Artificial Intelligence from the University of Edinburgh.

Differential Progress in Cooperative AI: Motivation and Measurement