On Sam, Dario and Prisoner’s Dilemma

Something quite strange happened last week. Anthropic’s Dario Amodei argued that AI companies need to “pace the frontier”, essentially slow down how quickly they make their most powerful models more capable. Sam Altman agreed with him. Think about that for a…

Idea 01 of 03

Introduction

All ideas

Introduction

Think about that for a second.

OpenAI and Anthropic are spending billions of dollars trying to beat each other. They are fighting for the same users, developers, talent and, ultimately, the chance to build AGI first.

And now they agree that perhaps everyone should slow down. This makes a lot more sense when you look at the AI race as a classic prisoner’s dilemma.

But neither can afford to slow down alone. If OpenAI slows down and Anthropic keeps going, Anthropic gets ahead. If Anthropic slows down and OpenAI keeps going, OpenAI gets ahead.

So both keep racing.

The result is quite sad. Both companies can end up doing something neither actually wants to do.

Which is why Dario’s proposal is interesting.

If the major AI labs agree on common tests, let outside groups inspect their models and agree on when to slow down, they can change the game. OpenAI no longer has to worry that Anthropic will use its six-month pause to race ahead. Anthropic gets the same comfort about OpenAI.

Sam and Dario may have found a way out of their prisoner’s dilemma.

A perfect duopoly?

Except there is a third prisoner.

Open source.

And this prisoner isn’t sitting at the table.

Over the last two years, Chinese labs have become a serious force in open AI. DeepSeek changed the discussion around the cost of building capable models. Alibaba’s Qwen family has become one of the most widely used open model families. Moonshot, Zhipu and others continue to push models that developers can download, change and run themselves.

AT&T (and similar large firms like Pinterest, Uber, Shopify) have been shifting heavily toward open-weight/open-source models run on their own or controlled GPU infrastructure instead of relying primarily on commercial closed APIs (e.g., from OpenAI or Anthropic).

Airbnb’s CEO has publicly said that the they use Chinese open-source models for customer-service agents because they are good, fast, and cheap.

I would not claim that these models have simply “beaten” OpenAI or Anthropic. That is not true across the board. But they have definitely changed the economics of the race.

OpenAI and Anthropic are no longer competing only with each other. They are competing with models that can get close enough on many tasks, cost far less to use, and in some cases can run without sending anything to OpenAI or Anthropic.

And that creates a much harder prisoner’s dilemma.

Sam and Dario can agree to slow down together.

They cannot make DeepSeek slow down.

They cannot make Qwen slow down.

More importantly, once someone releases model weights, there is much less control over what happens next. There is no central API that the original maker can switch off. People can change the model, remove its safety limits and run it somewhere else.

This is where the AI safety argument starts getting very interesting.

Suppose the industry agrees that powerful AI models need outside tests before release. They need ongoing checks. Their makers need to know who can access them.

And if something goes badly wrong, the maker should have some way to restrict access.

Who can meet those rules most easily? OpenAI and Anthropic.

A closed model served through an API gives the company control over access. The company can watch use, change safeguards and, in extreme cases, stop access.

An open-weight model cannot offer the same level of control after someone downloads it (who is going to take the blame in an org? the CTO? In that case, why would he/she even push open source models?)

Nobody has to say “ban open source.”

You can simply create safety rules that open source finds much harder to follow.And that creates a strange alignment of interests.

Skin-Crawlingly Awkward Video Shows Sam Altman and Dario Amodei Refusing to Hold Hands

OpenAI and Anthropic can genuinely believe that advanced AI needs stronger safety rules. At the same time, those rules can strengthen the business model of OpenAI and Anthropic.

Both things can be true.

This does not mean Sam Altman and Dario Amodei are secretly trying to kill open source. There is no evidence for that. In fact, OpenAI itself has released open-weight models, and Dario has said he does not support banning them.

The more useful question is not what they intend. It is what happens if their view of responsible AI becomes the rule.

Because the definition of “safe AI” could slowly become a definition that favours models which remain under the control of their makers.

And then we have an even bigger problem.

Imagine OpenAI and Anthropic really do slow down. They spend more time testing frontier models. They delay releases when the risks look too high. They follow every rule they helped create.

Meanwhile, Chinese open models keep improving.

At some point, one of the American labs looks at the capability gap and asks the obvious question:

Why are we slowing down when they aren’t? And the prisoner’s dilemma starts all over again.

Except now it is not OpenAI versus Anthropic. It is closed frontier AI versus anyone who refuses to join the agreement. Which may be the real problem with trying to pace AI.

It works only if enough of the people who can push the frontier agree on what “pace” means. Sam and Dario can make peace with each other. They cannot make peace on behalf of everyone else.

And in a world where everybody wants to rule the world, that may be the part that matters most.

The one who gets to define ‘safe AI’ gets to rule the world.

What’s your take?