At a recent Carnegie event, Global Order and Institutions Program Director Stewart Patrick spoke with Carnegie Fellow Scott Singer and Oxford AI Diplomacy Lab Co-Director Lucia Velasco about implementing AI guardrails. A portion of their conversation, which has been edited and condensed for clarity, is below.

Stewart Patrick: What concerns you most about the AI safety agenda, and why has it been so hard to make progress on that agenda?

Lucia Velasco: What concerns me the most is that we seem to need a major incident before improving safety standards for frontier AI. And until we see something really serious, this won’t become an urgent, shared priority. Unfortunately, this serious stuff is already happening, but we’re using rhetoric that is not very helpful to achieve consensus and build the institutions and the safeguards that we need.

Stewart Patrick: Is it because it’s too alarmist and fills people with a sense of fatalism, or it doesn’t have a practical, step-by-step pathway to actually begin to address somebody’s safety concern?

Lucia Velasco: I think that some of us are constantly reading and talking to others about things related to AI. And it might sound less alarming, because we’ve heard these predictions before, right? We also understand that some of the incidents that are happening are not only dependent on the technology itself, but also on the sandboxes and testing environments.

But for the rest of the world, this sounds like we’re going to die. I was asked today if I was making plans for 2030. And to me, that is really a signal that something is not working for people who are outside this bubble. We cannot look at the technology in terms of “It’s going to harm us,” “We can’t do anything,” or “How can we stop this?” We should work on addressing concerns but also making sure that people feel safe.

Stewart Patrick: Scott, can you walk us through the changing AI safety dynamics and also what the role of tech companies has been in setting them?

Scott Singer: We can break the most serious risks—or the ones that might have the biggest expected impact in the future—into technical risks and geopolitical risks.

For technical risks—AI is incredible because it has so much information, but what if this power falls into the wrong hands? For example, AI could be misused to provide people who are potentially malicious actors with expertise in cyber. Part of the reason why Anthropic and OpenAI limited access to their most advanced frontier models was because they were concerned about what would happen if these offensive cyber capabilities fell into the wrong hands. An adjacent concern is biological knowledge. AI has an extraordinary understanding of biology, and the same technology that offers remarkable promise for curing many diseases is also the same technology that could be used by a terrorist to create a deadly pathogen.

We’re also concerned about AI malfunctioning. We now have [the OpenAI–Hugging Face incident] where an AI couldn’t solve a problem during testing. So a bunch of these AI agents basically found an alternative path that required them to break out of their testing environment and hack into an external organization’s internal infrastructure. It didn’t really cause all that much material harm, but it’s not too hard to imagine in a few months, as these systems become increasingly capable, that this could be a lot more dangerous.

There’s naturally a geopolitical component to this as well. The [OpenAI–Hugging Face] incident was in some ways relatively easy to manage: This was a U.S. company that hacked into the internal infrastructure of another U.S. company. But what if next time it’s a U.S. company’s AI agents hacking into critical Chinese infrastructure? Or Chinese models that hack into critical American infrastructure? These are going to be really thorny geopolitical problems.

So how do you govern this technology? And the answer so far, at least in the United States, when we talk about frontier risks, has been largely driven by industry and industry best practices, and also at the state level. Building on that momentum and finding a pathway forward for federal policies and governance is going to be a critical next step.

Stewart Patrick: On Capitol Hill, there seems to be some bipartisan movement now for AI governance. Do you have a sense of what some of that legislation might call for?

Scott Singer: It’s a full spectrum of proposals at the moment. On the most extreme version, you see calls to ban superintelligence. You also see plans that move toward, for example, requiring a kill switch—turning off the technology if it’s going rogue.

I think a foundational set of proposals emerges from this idea that we really want more transparency and evidence from the companies on the risks that their models pose: having companies document their safety practices or having some standards for what it means to have a good frontier AI framework. Explaining how those practices are applied at model release. Governance for models that are deployed internally by the companies.

A separate critical thread is what happens when AI causes harm or malfunctions in a way that potentially looks like it could cause substantial damage. Most recently we see, prompted by Anthropic, this idea of embedding AI evaluators—having independent organizations that are working very closely with the companies and are able to call balls and strikes on whether these models actually possess dangerous capabilities.

So really the spectrum goes from “Can we understand these risks and test and evaluate them in clear and comprehensive ways” to “Just stop building.”

Stewart Patrick: How has the AI safety conversation in the UN system evolved?

Lucia Velasco: Obviously [having led AI policy at the UN] I’m biased, but I would say that the UN has moved very quickly and that it needed to address the issue of AI governance at the global level. There have been a lot of changing priorities. Then, in the middle of all this, there’s been a crisis at the UN, right?

It created by consensus two global mechanisms—the Independent International Scientific Panel on AI and the Global Dialogue on AI Governance—to bring everyone to the conversation and have a common understanding of the risks and opportunities.

But when you talk with 193 countries (and understand that the rich nations and those who own the technology are only a few), [you learn that] the priority of the global majority is to have access to the technology, to be able to be part of this opportunity, and to ensure they have the capacity they need. The priorities are quite different, and finding a common denominator means sacrificing some ambition.

Stewart Patrick: Are there prospects for international law or treaties coming out of the UN?

Lucia Velasco: I don’t think that there’s any law, treaty, or anything like that in the making anytime soon. The UN cannot unilaterally decide what to do. It has to be the member states that say what they want the UN to do, and until we understand that distinction—that this needs to come from the General Assembly and the countries agreeing on what role they want the UN to play—we need to stick to previous resolutions, as well as to the UN mandate, which doesn’t include at this point anything in this space. So I think it is important to be [realistic] in what we can expect from intergovernmental organizations.

Stewart Patrick: U.S. President Donald Trump and Chinese President Xi Jinping are slated to meet at the White House this week. Are you hopeful that AI will be on the agenda and that the United States and China might begin to make some progress on certain areas of AI safety?

Scott Singer: I think it’s really critical to calibrate the goals of this meeting with how diplomacy actually plays out. A lot of people, in the wake of all the evidence about the serious risks that are emerging from AI, have called for international treaties and grand bargains very imminently. In reality, it’s going to take time for two countries that have very different interests and perceptions of the risk from these technologies to reach any sort of agreement, let alone have the technical infrastructure to actually enforce one.

But that shouldn’t stop real progress. I would be delighted to see each side identify areas that are in their own self-interest and agree to make progress. There’s a lot that can be learned from just looking at what the United States has done that’s effective. This is what my Carnegie colleague Matt Sheehan calls safety in parallel. What are the key lessons that each side can offer?

And when it comes to diplomacy, most of this happens in a very sequenced fashion. We don’t normally get a megadeal that happens all at once. It happens over time, as trust and the technology are built. And for a technology that is moving as fast as AI, I think it would be irresponsible to lock us into an arrangement now that may not make sense in six months. The best thing that we can do—and, I think, hope for—in this meeting is to establish a permanent channel focused specifically on managing AI risks. [It would have] a clear line to technical experts at the working level who can have concrete, real conversations. Ideally, [it would be] divorced from the wider ebbs and flows of the U.S.-China relationship. That’s going to require diplomatic prowess on both sides to navigate that.

Stewart Patrick: In 2024, with a colleague here at Carnegie, I published a piece titled “Envisioning a Global Regime Complex to Govern Artificial Intelligence.” It looked at different historical analogies that might regulate different aspects of AI. At the moment, I think some people still hope for a unified approach to AI governance. But am I right in concluding that any AI governance is going to be messy—a web of overlapping institutions, scales, stakeholders, members, rather than an IAEA for AI?

Lucia Velasco: I think it’s going to be much more distributed and complex. People don’t want more bureaucracies or big institutions that are funded by taxes, with unclear efficiency. It’s not only what people expect—it’s also what the technology needs. I think we will need to go to a much narrower set of mandates. It is probably more effective when you try to regulate, govern, or kind of secure spaces that are not necessarily covering everything AI-related.

And hopefully we should be able to agree on something at the global level—a crisis response mechanism, or something that is a no-brainer at the global level. That should be doable. But when it comes to things that are much more dynamic and need to be faster, [the regulation] will become more targeted.

Scott Singer: Maybe to draw on an analogy from the scholar Amitav Acharya, AI looks a bit like a multiplex theater. Depending on where you are and where you look, the issues and the concerns and the benefits are going to look different. You are going to have a different stack of the technology. Depending on what you have and what you care about, you’re going to be part of different conversations.

At the end of the day, the conversation about frontier risks is going to be a pretty narrow one in many ways. I think it’s quite reasonable for the rest of the world to be concerned, but ultimately, it’s going to be on the countries developing that technology to mitigate those risks, because so many of the mitigations are at the model level. There’s only so much other countries can do. So pragmatically, the United States and China, and perhaps the UK or EU, are just going to be the players in that conversation.

But at the same time, there’s definitely going to be a role for other institutions that can advance AI governance. It is so critical that we have a shared understanding of many of these risks. And the UN is going to play a critical role in that. The International AI Safety Report is going to play an essential role in understanding what is happening at the frontier of AI safety. So knowing where we genuinely need global understanding, where we need global standards, and where we can have either regional or bilateral or domestic arrangements is going to be the key to unlocking all the benefits that AI can provide—and managing the serious risks.

For more, watch the full event on YouTube.