6 AI for Human Cooperation

Thus far, we have been focusing on the need for AI to be able to handle cooperation problems in order to manage the novel risks that arise from multi-agent interactions involving AI agents. However, the potential gains of having AI systems that can solve cooperation problems well are greater than that. As many of the greatest challenges that humanity faces are in fact cooperation problems, AI systems with cooperative capabilities could be invaluable to solving these.

‍

By the end of the section, you should be able to:

‍

• Explain how AI can be applied to social planning problems at scale.
• Compare different AI-supported approaches to collective deliberation and discuss their respective strengths and limitations.
• Discuss how AI systems could enhance democracy and social coordination while considering associated risks and constraints.

‍

The next resource is a well-known paper from this area of work: “The AI Economist”, which explores how reinforcement learning can be used to design effective tax policies. While reading this paper, note the connections to previous material on opponent shaping and adaptive mechanism design. The paper describes their approach as “two-level deep RL”, meaning both the mechanism designer (that sets the taxation policy) and the agents (that participate in the taxed economy) are learning agents.

While the AI Economist aims to design effective tax policies under the assumption that policy objectives have been agreed upon, one of the central cooperation challenges in human societies is agreeing on such objectives - not just for economic policy but for all kinds of political decisionmaking. There have been several initiatives to use AI to facilitate collective deliberation and decision making, and we are going to review two notable examples: Pol.is and the Habermas machine.

‍

We’ll first look at a non-technical article that provides some context for the implementation of digital solutions for improving collective deliberation and decisionmaking. Before diving into the details of technical solutions, it is important to recognise that there are other important challenges that relate more to the adoption and legitimacy of solutions than to technical features and capabilities.

We will now go more into technical detail on the Pol.is system mentioned in the previous article. While the first version of Pol.is came before large language models, this next resource is a paper on how modern AI systems can be used to improve it. The discussion section also applies quite broadly to the integration of LLM’s and machine learning in deliberative systems.

Another notable project in this area is “the Habermas machine” that was developed by a team at Google Deepmind. It is named after the German philosopher Jürgen Habermas and his theory of communicative action. While Pol.is focuses on the challenge of eliciting and mapping opinions in large and diverse groups, the Habermas machine aims to help human groups reach consensus by generating new proposals that obtain wide agreement and leave groups less divided. The next resource is a talk by Christopher Summerfield presenting the Habermas machine.

While human cooperation problems such as climate change or political deliberation predate the development of advanced AI systems, AI development also creates new challenges that makes the need for effective solutions more urgent. There is a growing overlap between the research communities that work on “AI for Democracy” and those that focus on existential risk. The following resource is published by Forethought, a research nonprofit focused on how to navigate the transition to a world with superintelligent AI systems, and highlights coordination-enabling applications of AI as something that could mitigate existential risks.

The final (required) resource of this section is a blog post that is based on a research workshop that took place in July 2025 where experts worked on formulating visions of success for AI-facilitated cooperation and mapping out promising directions to work on.

If you are interested in exploring further work in this area, it is also worth looking into the work of the Collective Intelligence Project (CIP), an organisation focused on the research and development of collective intelligence capabilities: decision-making technologies, processes, and institutions that expand a group’s capacity to construct and cooperate towards shared goals.