New Directions in Cooperative AI

Skills that lead to strong performance in multi-agent systems can be detrimental to social welfare. This is true even of skills that play a central role in cooperation: the ability to understand other agents can make it easier to deceive and manipulate them, and the ability to commit to peaceful agreements can also facilitate coercive commitments. We will argue that if the Cooperative AI Foundation is to achieve its mission of improving the cooperative intelligence of advanced AI systems for the benefit of all humanity, it should focus on improving skills that robustly lead to improvements in social welfare, rather than those that are dangerously dual-use.

We refer to this as “differential progress on cooperative intelligence”. To know whether we are making differential progress, we need to be able to rigorously define and measure it. Towards these ends, we present early-stage work on the definition and measurement of cooperative intelligence.