4 Cooperative Agents

The first sections of the curriculum have focused on the problems that the field of cooperative AI is aiming to tackle, and presented game theory and complex systems as possible framings.

‍

We will now turn to looking at different approaches to solving these problems, and start off looking at interventions that are focused on the agents themselves and their inherent properties.

‍

By the end of the section, you should be able to:

‍

• Define what makes an agent cooperative, potentially using notation from game theory.
• Describe and compare how different agent designs fare in a multi-agent setting like the iterated prisoner's dilemma with evolution.
• Explain the concept of opponent shaping and its implications for learning agents and designing cooperative agents.
• Comment on the efficacy and practicality of training certain biases or tendencies into agents to improve their cooperativeness.
• Explain the benefits of including human data in the training of AI systems for various multi-agent systems.

‍

The idea that a goal of cooperative AI should be to create cooperative agents was introduced from the very beginning of the field's establishment. The following optional reading published in 2021 can be seen as a starting point for the field of cooperative AI, and outlines cooperative intelligence as a central property that AI should possess. If you’re interested in the history of Cooperative AI we recommend reading the suggested sections and taking note of the authors.

If you want to take a deeper dive into the cognitive skills or individual capabilities that make up cooperative intelligence, as claimed by the previous piece, then the following reading might be helpful. However, note that it is quite a technical paper and spends most of its time listing citations over explaining concepts, which is why it is also optional.

Settling on a formal definition of what makes an agent cooperative is tricky – it’s an ongoing area of research. If we want to avoid catastrophic cooperation failures between agents in the real world, we must figure out how to create agents that are not only able to cooperate well, but that are also likely to actually be deployed. Imagine that you can choose the properties of your own AI assistant - would you pick an assistant that always prioritises the common good, or would you pick one that prioritises what is best for you? Chances are, most people (and companies) would prefer an assistant that is able to negotiate good deals for them and protect their interests.

The next resource attempts to break down our understanding of what a cooperative agent would be into more granular and specific agent properties that could more easily be specified, evaluated and trained for.

The following interactive piece is useful for seeing how different agent designs fare in a very particular multi-agent setting, but one that is analogous to many real life situations: the iterated prisoners’ dilemma (with evolution).

The final part of this section ‘Multi-agent environments’ lists many more simulations, experiments or tournaments for testing the success of cooperative agent designs in particular environments, including ones constructed for LLM-based agents.

Opponent shaping

‍

When we think about agents in multi-agent settings, it is important to distinguish between static agents that are trained on a specific dataset and do not develop further in deployment, and continually learning agents that keep learning in deployment. Training of static agents is called offline learning, while the continual learning is also called online learning.

‍

The circumstance when agents are learning in the presence of other learning agents deserves special attention as this introduces a meta-game where an agent can gain advantage by taking the learning process of the opponent into account. This is called opponent shaping.

As an example to explain how opponent shaping works, we will use a classic two-player coordination game from game theory called Bach or Stravinsky (also known as Battle of the Sexes and often shortened to BoS). The setup is that you have two friends who want to go to a concert together, and they each need to choose between two options: Bach or Stravinsky. One of them prefers Bach and the other Stravinsky, but above all they both prioritise going together to splitting up. The payoff matrix then looks something like this:

‍

‍

The players are not allowed to communicate before they make their choices.

‍

Playing against a static opponent

Before continuing, we recommend looking at the reinforcement learning part of the Appendix, if you are unfamiliar with the basics (e.g. terms like policy, reward function and environment).

‍

If you were to train a reinforcement learning (RL) agent to do well in this game, the simplest case would be the case where the opponent is a static agent with a fixed policy. The purpose of the training would be to optimize for a best response to this opponent policy. You might let your agent play a hundred or a thousand times with different policies (e.g. “randomize between alternatives” or “always choose their own preferred choice”) and then pick the policy that generated the best payoff.

‍

Opponent shaping: reinforcement learning against a learning opponent

If you were to assume that the other agent is also a learning agent, this changes things. This is likely intuitive to anyone who has trained a puppy or interacted with children - when you interact with an agent that develops quickly, you have to consider not only the immediate consequences of the interaction but also what the other party will learn from it and how it will change their future behavior.

‍

In our Bach or Stravinsky game, consider a strategic player that has a preference for the Stravinsky concert. Knowing that their opponent is a reinforcement learning agent, this player might reason that if they only ever play Stravinsky their opponent will never experience a reward for playing Bach. This changes the observed payoff matrix to look like this:

‍

‍

This way, the strategic player could ensure that they would always get their preferred choice. This is the basic concept of opponent shaping: to optimize a policy for a best response to the opponent learning algorithm, rather than to the opponent policy.

If you’re interested in how opponent shaping is implemented in MARL, among other reflections on opponent shaping, we recommend the next optional piece.

Opponent shaping could potentially be a powerful tool for cooperative agents, if it can be scaled to complex agents and environments (including LLM agents). A key feature of this approach is that it could theoretically be used to make other learning agents cooperate even if you have no control over its design or objectives. Just as for many other cooperation-relevant capabilities, opponent shaping also comes with risks which will be covered in section 7 ‘Back-fire Risks’.

‍

Training for cooperativeness

‍

So far in this section we have discussed what properties of agents might influence the outcomes in multi-agent interactions in general and cooperation problems in particular. If we want to figure out interventions for cooperative AI at the agent level, it is fundamental to identify which agent properties are relevant to study and what properties might be more or less desirable. We will now turn to different approaches of training agents for such properties: assuming we know what a cooperative agent is, can we create it?

‍

One way to approach training for cooperativeness is to try to induce specific desirable biases or tendencies in agents. The next two resources explore two specific approaches for creating cooperative agents: norm-adaptive policies and inequality aversion. Reading the optional parts of the papers requires a more technical background, but does provide some useful context on how cooperative properties can be trained into agents in practice.

Recall that the cooperative AI field is concerned with the multiagent dynamics of human-human, human-AI and AI-AI systems. The next resources are papers that explore the utility of training with human data to elicit good outcomes in fully cooperative and mixed-motive human-AI systems (technically, the whole game involved in the second study, diplomacy, is zero-sum, but contains mixed-motive subgames throughout). Again, reading the optional parts of the papers requires a more technical background, but is valuable for those interested in details on training agents in practice.

Multi-agent environments

‍

Work in cooperative AI often makes use of simulation or experiments that are carried out in some kind of environment. Below are a few environments that are frequently used in cooperative AI. It might be especially useful to be familiar with these if you are aiming to do your own research in cooperative AI, but for now, don’t spend any longer than about 20mins total looking over the abstracts and introductions for these.

‍

• Melting Pot 2.0 
• Concordia
• Welfare diplomacy
• GovSim
• MoralSim