Something quite strange happened last week. Anthropic’s Dario Amodei argued that AI companies need to “pace the frontier”, essentially slow down how quickly they make their most powerful models more capable. Sam Altman agreed with him.
Think about that for a second.
OpenAI and Anthropic are spending billions of dollars trying to beat each other. They are fighting for the same users, developers, talent and, ultimately, the chance to build AGI first.
And now they agree that perhaps everyone should slow down. This makes a lot more sense when you look at the AI race as a classic prisoner’s dilemma.
Assume OpenAI believes pushing AI too fast has real risks. Anthropic believes the same. Both would probably prefer a world where everyone spends more time testing models instead of shipping the next capability jump as fast as possible. This gives both enough time to prep up for enterprise adoption, which has been abysmally slow.
But neither can afford to slow down alone. If OpenAI slows down and Anthropic keeps going, Anthropic gets ahead. If Anthropic slows down and OpenAI keeps going, OpenAI gets ahead.
So both keep racing.
The result is quite sad. Both companies can end up doing something neither actually wants to do.
Which is why Dario’s proposal is interesting.
If the major AI labs agree on common tests, let outside groups inspect their models and agree on when to slow down, they can change the game. OpenAI no longer has to worry that Anthropic will use its six-month pause to race ahead. Anthropic gets the same comfort about OpenAI.
Sam and Dario may have found a way out of their prisoner’s dilemma.
A perfect duopoly?
Except there is a third prisoner.
Open source.
And this prisoner isn’t sitting at the table.
Over the last two years, Chinese labs have become a serious force in open AI. DeepSeek changed the discussion around the cost of building capable models. Alibaba’s Qwen family has become one of the most widely used open model families. Moonshot, Zhipu and others continue to push models that developers can download, change and run themselves.
AT&T (and similar large firms like Pinterest, Uber, Shopify) have been shifting heavily toward open-weight/open-source models run on their own or controlled GPU infrastructure instead of relying primarily on commercial closed APIs (e.g., from OpenAI or Anthropic).
Airbnb’s CEO has publicly said that the they use Chinese open-source models for customer-service agents because they are good, fast, and cheap.
I would not claim that these models have simply “beaten” OpenAI or Anthropic. That is not true across the board. But they have definitely changed the economics of the race.
OpenAI and Anthropic are no longer competing only with each other. They are competing with models that can get close enough on many tasks, cost far less to use, and in some cases can run without sending anything to OpenAI or Anthropic.
And that creates a much harder prisoner’s dilemma.
Sam and Dario can agree to slow down together.
They cannot make DeepSeek slow down.
They cannot make Qwen slow down.
More importantly, once someone releases model weights, there is much less control over what happens next. There is no central API that the original maker can switch off. People can change the model, remove its safety limits and run it somewhere else.
This is where the AI safety argument starts getting very interesting.
Suppose the industry agrees that powerful AI models need outside tests before release. They need ongoing checks. Their makers need to know who can access them.
And if something goes badly wrong, the maker should have some way to restrict access.
Who can meet those rules most easily? OpenAI and Anthropic.A closed model served through an API gives the company control over access. The company can watch use, change safeguards and, in extreme cases, stop access.
An open-weight model cannot offer the same level of control after someone downloads it (who is going to take the blame in an org? the CTO? In that case, why would he/she even push open source models?)
Nobody has to say “ban open source.”
You can simply create safety rules that open source finds much harder to follow.And that creates a strange alignment of interests.
OpenAI and Anthropic can genuinely believe that advanced AI needs stronger safety rules. At the same time, those rules can strengthen the business model of OpenAI and Anthropic.
Both things can be true.
This does not mean Sam Altman and Dario Amodei are secretly trying to kill open source. There is no evidence for that. In fact, OpenAI itself has released open-weight models, and Dario has said he does not support banning them.
The more useful question is not what they intend. It is what happens if their view of responsible AI becomes the rule.
Because the definition of “safe AI” could slowly become a definition that favours models which remain under the control of their makers.
And then we have an even bigger problem.
Imagine OpenAI and Anthropic really do slow down. They spend more time testing frontier models. They delay releases when the risks look too high. They follow every rule they helped create.
Meanwhile, Chinese open models keep improving.
At some point, one of the American labs looks at the capability gap and asks the obvious question:
Why are we slowing down when they aren’t? And the prisoner’s dilemma starts all over again.
Except now it is not OpenAI versus Anthropic. It is closed frontier AI versus anyone who refuses to join the agreement. Which may be the real problem with trying to pace AI.
It works only if enough of the people who can push the frontier agree on what “pace” means. Sam and Dario can make peace with each other. They cannot make peace on behalf of everyone else.
And in a world where everybody wants to rule the world, that may be the part that matters most.
The one who gets to define ‘safe AI’ gets to rule the world.
What’s your take?
Make sense of what's next.
A clear-eyed digest of AI, products, and ideas worth understanding. Join the NextBigWhat newsletter.
Disney is considering a potential acquisition of Character.ai, a tech startup that has recently filed for Chapter 11 bankruptcy. The move could allow Disney to integrate advanced AI chatbot technology into its offerings. This acquisition could reshape Disney’s approach to digital interactions and character engagement.
Andrej Karpathy emphasizes that only a small fraction of people, around 5,000, have fully grasped the potential of AI, indicating a growing gap in AI capabilities. Meanwhile, Box CEO Aaron Levie predicts the future of agent swarms in AI, though he notes that most enterprises are currently unprepared to implement such advanced technologies. This highlights the need for increased awareness and readiness among businesses to leverage AI effectively.
Meta AI Research has introduced Muse Wonder, a new frontier model designed for enhancing personal and professional agent functionalities. This innovative model features a Responsive Step and Repeat hero animation, showcasing its advanced capabilities. The development is set to influence how users interact with AI, potentially transforming productivity and user experience.
Google has introduced a standalone platform called SynthID Detector, designed to help users determine if online content was generated using Google AI or its partner tools. This initiative aims to enhance transparency and trust in digital content, amidst growing concerns about the authenticity of AI-generated materials. The tool reflects Google’s commitment to ethical AI usage and the accountability of content creators.
The US Department of Labor has suspended major tech companies, including Microsoft, Infosys, and Cognizant, from the green card program due to allegations of H-1B visa abuse. This decision impacts several prominent IT firms such as TCS, Wipro, and Capgemini, potentially affecting their ability to recruit foreign talent. The move underscores increasing scrutiny on visa practices within the tech industry.
