What if the people asking to slow down AI are also the people least suited to decide how slow is slow enough?
That question sits at the center of a debate that broke into the open recently. Dario Amodei, the CEO of Anthropic, wrote an essay arguing that the risks of advanced AI now call for more caution than the industry has been showing. His words: “Over the last few months, I have become convinced that fully addressing the risks requires even more prudence.” He laid out a plan involving industry coordination and global regulation, plus third-party monitors who could evaluate the safety of models while they’re being built. Sam Altman of OpenAI voiced agreement with the general direction, having previously suggested it may be time to “pace” AI development.
If you follow AI news casually, this probably reads as good news. The people building the most capable systems are asking someone to watch them more closely. That’s a real shift from the usual posture of “we’ve got this, trust us.”
But the interesting part isn’t the proposal. It’s the awkward structural problem underneath it.
The referee is also playing
Here’s the tension I keep coming back to as someone who spends a lot of time explaining this stuff to people outside the industry. When a company that builds frontier AI models proposes the rules for pacing frontier AI models, it’s writing the exam it will later sit for.
That’s not an accusation of bad faith. Amodei’s concerns appear sincere, and Anthropic was founded partly on safety arguments. But sincerity and structural conflict of interest can coexist comfortably. A slowdown designed by the leading labs will, almost inevitably, be a slowdown shaped around what the leading labs find workable. What counts as “too fast” gets defined by people whose competitive position depends on that answer.
And notice who’s absent from the conversation as reported. The proposal circulated among competitors, and competitors responded. That’s a small room.
What “pacing” actually means in practice
For readers who don’t live in this world, let me translate the vocabulary, because the language does a lot of quiet work.
- Pacing the frontier means slowing the release or development of the most capable new models. Not stopping. Not rolling anything back. Slowing the rate at which new capability arrives.
- Third-party monitoring means outside evaluators who test models for dangerous behavior before or during development, rather than the lab grading its own homework.
- Global regulation means coordination across countries, because a rule that applies in one jurisdiction and nowhere else mostly relocates the problem.
Each of these is reasonable on its face. Each also raises a question nobody in the reported coverage has answered. Who staffs the third-party monitors, and who pays them? What authority do they have if they find something alarming? Can they say no, or only say “we noticed”? A monitor without the power to stop a launch is a smoke detector with no batteries.
Why this matters if you use AI agents
You might reasonably wonder why any of this touches you if your relationship with AI is asking a chatbot to draft emails or having an agent sort your calendar.
It touches you because pacing decisions determine what shows up in your tools and how tested it is when it arrives. AI agents are the category where this gets concrete. A chatbot that gives a wrong answer wastes your time. An agent that takes actions on your behalf, sends the message, moves the money, books the thing, fails differently. The gap between “model is impressive” and “model is safe to hand the keys to” is exactly what safety evaluation is supposed to measure.
So when the CEOs debate pacing, they’re partly debating how thoroughly the agents in your workflow get vetted before you get them.
The part I’d watch
The useful test for whether this becomes real isn’t whether more executives sign on. Agreement among competitors is cheap when the agreement is directional and vague.
The test is whether any of it becomes binding on someone who didn’t volunteer. Voluntary pacing works until a lab decides the competitive cost is too high, at which point it stops working, quietly, with no announcement. Rules that hold are rules with an enforcer who doesn’t report to the companies being paced.
Right now the frontier is being paced, if at all, by the people running fastest along it. That’s better than nobody thinking about it. It’s a long way from a solution, and the distance between those two things is where the actual work lives.
My read: treat this moment as a genuine signal that the people closest to these systems are worried, and treat the specific proposals as an opening bid rather than a settlement. Both can be true. The question of who should set the pace is still wide open, which is probably the most honest thing anyone can say about it.
🕒 Published: