Scaling AI Safety for a Multi-Agent World

A joint funding call by the Cooperative AI Foundation, Schmidt Sciences, Google DeepMind, the Advanced Research & Invention Agency (ARIA), and Google.org.

AI agents are increasingly being deployed in multi-agent settings. While most present-day cases involve teams of agents orchestrated by a single actor (or ‘principal’), we are beginning to see the emergence of more complex ecosystems of agents deployed by different actors across shared digital infrastructure. These multi-principal, multi-agent interactions create new opportunities for cooperation and shared benefit (Dafoe et al., 2021), but also new risks, which means focusing only on the safety and alignment of individual models is insufficient (Hammond et al., 2025).

More research is therefore urgently needed to understand safety and risk through a system-level, multi-agent lens – developing methods to analyse emergent collective dynamics, building infrastructure for trustworthy interaction between agents, and creating scalable approaches for monitoring and control of increasingly complex networks of AI systems. While some of these problems will be addressed by market forces, we expect others to fall through the gaps. This funding call aims to fill those gaps, catalysing the foundational scientific research needed to understand, evaluate, and control risks emerging from large-scale ecosystems of interacting AI agents, deployed by multiple actors.

The call has been inspired by three recent papers. First, Google DeepMind’s “Distributional AGI Safety” outlines the safety implications of highly capable AI systems emerging not as single monolithic agents, but through coordinated networks of specialised sub-AGI systems with differential access to tools, data, memory, and resources. Second, ARIA’s “Scaling Trust” programme thesis argues that, in a world of increasingly capable networked agents acting across digital and physical environments, coordination infrastructure that lets agents enter into 'contracts' securely, programmatically, at scale, and without intermediaries can preserve pluralism and unlock new forms of coordination. Finally, the Cooperative AI Foundation’s “Multi-Agent Risks from Advanced AI” report argues that interacting populations of AI agents introduce qualitatively new failure modes beyond single-agent systems, including collusion, conflict, destabilising dynamics, emergent agency, and novel multi-agent security vulnerabilities. These perspectives in turn build on earlier work by Minsky (1986), Huberman (1988), Wooldridge & Jennings (1995), Manheim (2018), Drexler (2019), Critch & Krueger (2020), Clifton (2020), Dafoe et al. (2020), Conitzer & Oesterheld (2023), Chan et al. (2025), Kolt (2025), Hadfield & Koh (2025), and Tomašev et al. (2025), among many others.

Guidelines & Selection Criteria
Scope

Proposals will be evaluated holistically. Key considerations include:

  1. Research Agenda Fit. Does the proposal clearly engage with the intention behind the scientific questions and objectives in the research agenda?
  2. Scientific Quality and Rigour. Is the proposed work technically sound, well-motivated, and capable of producing generalizable insight?
  3. Potential Impact. If successful, would the project materially advance scientific understanding relevant to this call, or meaningfully improve our ability to understand, evaluate, or control risks posed by multi-principal, multi-agent AI systems?
  4. Philanthropic fit. Is there a clear market, coordination, or incentive failure that means commercial interests are unlikely to solve this problem?
  5. Feasibility and Scope. Is there sufficient evidence to suggest that the proposal’s milestones and deliverables are ambitious yet well-defined and feasible enough to be achievable within the stated time duration?
  6. Team Expertise. Is the team well-suited to execute the proposed work, with relevant technical expertise, sufficient capacity, and a time commitment commensurate with the project's ambition?
  7. Cost Appropriateness. Is the proposed budget reasonable and well-justified given the project’s goals and planned activities?
  8. Additional funder-specific considerations.  Applications will be considered by the parties jointly supporting this funding call. If your proposal is selected for funding, the specific party or parties funding your project may provide additional limitations or guidance applicable to your award, which would be documented, if agreed upon, in your final award documentation.

We invite applicants to apply to either or both funding tiers: Tier 1 (Up to $300,000) or Tier 2 ($300,000-$1,000,000). Project durations can range from one to two years. Tier 1 aims to support exploratory research projects, pilot studies, or focused technical investigations, whereas Tier 2 targets more ambitious or collaborative projects.

We invite individual researchers, research teams, research institutions, and multi-institution collaborations across universities, national laboratories, institutes, and nonprofit research organisations. We are open globally and encourage collaborations across geographic boundaries. For any projects funded by Schmidt Sciences, indirect costs must be at or below 10% to comply with their policy. Projects funded under this RFP must comply with all applicable law and may not include lobbying, efforts to influence legislation or political activity.

Application Process
Eligibility

Applications are due by on August 8th, 2026 by 11:59pm AoE, and we expect to notify applicants of our final decisions in Autumn 2026. Applications are placed into one of two funding tiers based on their budget:

  • Tier 1: Up to US $300,000 (1-2 years)
  • Tier 2: US $300,000-$1,000,000 (1-2 years)

The operations of this jointly issued call for proposals are being managed by Schmidt Sciences. For any questions about the call, please see the live FAQ document provide by Schmidt Sciences (copied below for reference) or attend informational webinars on Tuesday, 30 June at 12pm ET (register here) and Thursday, 23 July  at 10am ET (register here). If your question remains unanswered, please contact multiagentsafety@schmidtsciences.org.

FAQs

Which institutions are eligible to apply for funding?

Recipients of funding are limited to educational, scientific or other charitable organizations, including public and private universities and other large research institutions, and similar non-US organizations, as further described below. The following are eligible to apply: 

  • Universities or Research Institutions (Including International Institutions), including applications led by individual researchers or Project Teams 
  • Other Non-Profit Research Organizations or Agencies
  • Multi-Institution Collaborations Between Universities and/or Other Non-Profits 
  • Universities and/or Other Non-Profits in Partnerships with Industry or For-Profit Organizations

Can commercial entities apply for funding?

No. For-profit entities cannot lead projects. However, for-profit companies may partner with consortiums of researchers, academics, and other parties at nonprofit organizations on research projects that otherwise meet the charitable objectives of this program.

Are teams based outside the United States eligible to apply?

Yes. Research teams based at institutions anywhere in the world are eligible to apply. Non-U.S. universities, research institutes, and nonprofits may serve as the lead applicant and receive funding directly, provided the institution holds a 501(c)(3) designation or an equivalent charitable status. For international institutions, Schmidt Sciences will work with NGOSource to verify equivalency where necessary.

Can project teams include collaborators from multiple countries or institutions?

Yes. We welcome collaborations across institutions and geographic regions. Project teams may include researchers from multiple institutions and multiple countries, and there are no restrictions on the geographic distribution of collaborators. Normally, one institution should be designated as the lead host institution for the award, which will typically receive the funds and manage any subawards to partner institutions.

Can independent researchers apply?

Yes. Independent researchers may submit proposals. However, if chosen for funding, they must secure a fiscal sponsor or affiliate with an eligible non-profit to manage the award. Funding will only be finalized if one of these criteria is satisfied.

Can a PI or team submit more than one proposal?

Yes.

How many PIs are expected for different-sized awards?

We do not have strict guidelines on the size of project teams. However, the team composition should illustrate both the team’s capacity to complete the project as well as the necessary expertise in the content areas.

Can junior faculty be competitive?

Yes. Junior faculty members can apply for funding, but must illustrate their ability to make a successful impact on the project goals.

Are cross-institution collaborations encouraged?

Cross-institution collaboration is welcome at all levels of funding, but not required.

Can postdocs be hired at the project's start?

Yes.

Can you give examples of "policy-adjacent work" that is out of scope?

Any research whose outcomes are focused on advocating for specific policy or legislation.

Are there limitations on overhead/indirect costs?

For any projects funded by Schmidt Sciences, indirect costs must not exceed 10% for the entire project. For example, the indirect costs for an award totaling USD $1 million should not exceed USD $100,000.

Can allocated and directly incurred staff salaries be included in the budget?

Yes.

Can funding support PhD students?

Yes.

Can the award cover the entire project cost?

Yes. We do not require co-funding for these projects.

Will you expect or allow no-cost extensions?

No-cost extensions will be evaluated on a case-by-case basis and are not guaranteed. Please propose projects that can be completed within the time frame stated.

If we have multiple institutions on our project, how will you pay them?

We consider the institutional home of the lead PI to be the “host” institution for the award. We prefer all the funding to go directly to the host, as this tends to get the money to the scholars/scientists faster. If absolutely necessary, we can split this up into separate awards for co-host institutions, but we prefer that the lead host institution receives all the funds and then handles sending subawards to the co-hosts. In terms of budget reporting for the subawardees, only total subaward amounts need to be sent to Schmidt Sciences. The expectation is hosts will be actively managing those awardees and review reports to make sure they align with intended usage both in relation to programming but also in relation to Schmidt Sciences guidance. 

We have no restrictions against second-tier subawards if you feel comfortable managing that collaboration. The selection and management of second-tier subawardees is the responsibility of the award recipient organization.

If funded, how does my institution receive the funds?

Schmidt Sciences funds large multi-year scientific projects as a series of one-year gifts (or grants) based on reported progress and plans. Future years of funding may be awarded based on scientific progress and performance, among other facts. 

We find this flexible funding model to be advantageous to both parties, as Schmidt Sciences can quickly accommodate changing local conditions and can adapt the letter (or agreement) terms if there is a clear regulatory and university policy requiring it to enhance scientific management, and ultimately success, such as instances where:

  • Certain universities have a legal requirement for funds to be secured multiple years ahead of formally hiring or advertising for a postdoc. In cases where this is truly a requirement, and evidence is provided, we are able to consider front-load funds for multiple years for specific postdoctoral fellowships or other critical scientific roles to address this administrative/legal issue. 
  • Schmidt Sciences may need to front-load funding to remain in compliance with local laws, in some instances, which we will consider when/if they arise for selected projects. 

Annual gifts or grants for subsequent years are not assured. Their provision is directly dependent on how well the project progresses in alignment with the goals that the team has outlined in its proposal and detailed in its annual narrative and financial report, among other factors.

Note that award periods do not necessarily align with calendar years.

Are there any special requirements for International Direct Awardees?
  • All award recipients are expected to ensure that their use of any funding from Schmidt Sciences complies with applicable law. 
  • Schmidt Sciences must evaluate each international direct funding applicant’s eligibility for charitable funding under US law. Schmidt Sciences typically achieves this by obtaining an equivalency determination (ED) for each non-US awardee from NGOSource (an independent U.S.-based service).) Accordingly, non-US applicants may be asked to provide specific organizational information  to facilitate this process. 
  • Schmidt Sciences will cover the NGOSource costs associated with conducting this initial analysis and any future renewals required during the project period. 
  • This process for a new ED from NGOSource can take up to 90 days. It is imperative that the institution be responsive to NGOSource’s information requests so as not to delay the start of the project. 
  • Schmidt Sciences relies on NGOSource for  EDs of its non-US awardees.  If you’ve previously obtained an ED through another organization, please note that we currently work exclusively with NGOSource for our due diligence process.
My institution requires that multi-year postdocs be fully funded up front. If one of these large multi-year awards is composed of contingent 1-year gifts (or grants), one could only offer one-year contracts to PhDs / postdocs, right?

We understand that certain universities have a requirement for funds to be secured multiple years ahead of formally hiring or advertising for a postdoc. In cases where this is a requirement, we are able to front-load funds for multiple years for specific postdoctoral fellowships or other critical scientific roles.

We expect to collaborate with international institutions for whom single-year awards could present a barrier. Do you have a standard response that we could share with them regarding terms as we develop our subcontracts with international partners?

Thank you for raising potential concerns regarding the funding structure. We understand there may be some regulatory reasons why Schmidt Sciences may need to front-load funding to remain in compliance with local laws, in some instances, which we will consider when/if they arise for selected projects. We have successfully implemented this flexible funding model in most cases and found it to be advantageous to both parties, as Schmidt Sciences can quickly accommodate changing local conditions and adapt the agreement terms to enhance scientific management and, ultimately, success.

Are there any restrictions on what we can use the funding for that we should be aware of?

Restrictions include those required by applicable law and regulations, including activities in sanctioned nations under US law. Beyond that, one of the intentions of this program is to provide research awards with relatively flexible funds over an extended period (~3 years). These awards are intended to help international research partners and break down artificial walls to support the best scientific research and international collaborations to advance the scientific mission and scope of the program.

Previously awarded projects use these flexible funds to support grad students, postdocs, other scientists, support to facilitate conferences, meetings and exchanges between personnel at the collaborating institutions, software engineering support and cloud computational infrastructure (cloud compute/GPU, cloud storage), and other standard research expenses.

Can we include large scientific equipment in the budget for program funding? If yes, does that equipment then belong to the program? Is there a requirement that we calculate equipment depreciation?

Yes, you may include large scientific equipment in your budget for funding. If funded, that equipment will not belong to Schmidt Sciences. We do not require a calculation of equipment depreciation but you may include it if you would like.

What are our limitations around our award dollars to international recipients being used for US travel/conferences?
  • Grants to overseas institutions are not generally subject to tax withholding in the US, provided the recipient does not use the funds for activities in the U.S. 
  • In most cases, it will be most cost-effective for non-US recipients to fund US travel from sources other than Schmidt Sciences awards.
  • If activities in the US (e.g., conference attendance) is essential for a project, and other resources are not available, a non-US awardee should contact us to discuss alternative options.
  • For example, US tax withholding may not be required for awards to non-US awardees that permit use in the United States if the awardee provides Schmidt Sciences with a Form W-8EXP or W-8BEN-E US tax document evidencing that US tax withholding is not required. 
  • These considerations apply equally to any award to a non-US awardee, whether a grant or a gift. 
Can AI assist in drafting proposals?

Yes. But PI’s are ultimately responsible for ensuring the accuracy of their proposal.

Are letters of support required?

No. Letters of support will not be reviewed or accepted.

Does the applicant retain full ownership of their intellectual property as included in the proposal submission?

Yes. Schmidt Sciences typically makes awards under a gift structure and does not seek ownership of intellectual property arising from funded projects. Unless otherwise specified in a grant agreement, IP generated during the project remains with the awardee, consistent with their institutional policies.

Supporting documents