This week I’m covering the calls for an industry wide slowdown. There’s A LOT here. More than I could fit into the newsletter, but I’ve tried to choose the most salient parts and then propose possible explanations for why this is happening now.
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
I’m sure this is completely unrelated, but it may be useful to remember that Anthropic is paying 1.25 billion a month to SpaceXAIin exchange for compute from Colossus 1 and some of Colossus 2 data centers.
Musk wasn’t alone though. OpenAI was quick to agree as well.
I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we’ve had at OpenAI in recent weeks.
Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. We’ll have more to share soon.
A Play for Regulatory Capture
One part of Dario’s essay that caught my attention was his call for new regulations. Specifically his desire that AI companies become subject to third-party evaluators. He writes:
Embedded Evaluators. Each frontier AI company commits to giving ongoing, employee-like access to a team of embedded third-party evaluators (such as METR), whose role is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes. This is the key step for verifiability of any pacing commitments, and has precedent in the banking industry, which sometimes involves regulatory “supervisors” embedded along with employees. Anthropic is unilaterally committing to this step now. We intend this to be part of a broader push to redouble efforts on our safety and alignment work.
It’s interesting that Anthropic keeps asking for third-party oversight. Earlier this year they objected to the government’s intervention when they disagreed with some of their decisions. With that in mind, let’s take a look at the organization they suggest as a possible evaluator.
Amodei already has candidates in mind for the new job. In his post, he mentioned Berkeley-based Metr, a prominent nonprofit AI watchdog that conducts independent evaluations of AI models.
Metr, established in 2022 by ex-OpenAI staffer Beth Barnes, is attracting top talent from the biggest AI labs.
Joe Benton, previously a member of Anthropic’s safety and oversight team, announced on Friday that he had left the company to join Metr. Josh Engels, a former employee of Google DeepMind’s AGI safety team, said on Sunday that he had resigned and joined Metr because of the high stakes of AI safety.
It [METR] had no applicable personnel conflict-of-interest policy when the project began.
It therefore conducted no formal disclosure or recusal process.
Some employees had strong social ties to AI-company employees.
At least six people involved had close personal relationships with AI-company staff.
METR shared a research center with some AI-lab employees.
Access to unreleased models and free tokens gave METR an incentive to maintain a “cordial relationship” with AI companies.
Three Scenarios
What follows is purely speculation on my part. I’ll present you with the statements and observations I’ve noticed, but I’m guessing at possible motivations. Feel free to speak up if you disagree.
We have the two leading companies calling for a slowdown, but why?
Amodei, Altman, and Musk all publicly agreed yesterday to slow AI development. Anthropic’s CEO published a 3,400-word essay called “We Must Pace the Frontier.” Altman endorsed it within hours and told Fortune that OpenAI won’t IPO in 2026. Musk wrote “Dario is right.” This is four days after Coxon quit Anthropic and a week after Pachocki said no lab has solved alignment.
My Take
Three rival CEOs agreed to slow down on the same day, and I think the Coxon resignation and the Hugging Face breach forced it. Amodei wrote about agents that attacked systems without orders. Pachocki said no lab can guarantee the models do what they’re told, and then Coxon walked away from his career over it.
But Ellison had a sell campaign filed with the SEC this week, Altman’s IPO had already moved to 2027 for valuation reasons before safety became the explanation, and data center projects have stalled across the country because communities don’t want them. A voluntary safety pause is a much better headline than “we can’t build fast enough and our investors want out.” It also keeps open source and Chinese labs behind them while Washington writes rules that only the big three can afford to follow. I believe Amodei is scared and I also believe a coordinated pause works out very well for Anthropic right now.
Time to Short?
Michael Burry, the famous investor who shorted the housing market before the 2008 financial crisis, seems to agree with this idea.
Let's all take a moment to understand how self-serving it is for OpenAI, Anthropic and other execs of big hyperscalers to talk of slowing things down. 1. LLMs are not AI and won't be AGI. There is nothing AI to slow down. 2. Competition is coming up fast, slowing benefits incumbents. 3. IPOs need hype & puffery; "we are so awesome it could become dangerous" is hype & puffery 4. Cover for real uncontrollable slowing growth as IPOs look to be pushed out
Scenario 2: Something Happened
It’s possible that there was some undisclosed event. We’ve all heard about the HuggingFace incident, but maybe there was something else which has not been reported.
There are several tweets about this, but Ars Technica has the most succinct coverage. You can read it here.
In short all major AI models went down at the same time. I have not been able to find a public statement about why they went down. Was this the event that scared these CEOs into calling for a slowdown? It’s possible.
Scenario 3: A Major Leap Forward
There are several unconfirmed rumors that Google has cracked RSI.
RSI Explainer Box
RSI stands for recursive self-improvement. The important word in that is Recursive because it describes a feedback loop, i.e. something that repeats.
Imagine an AI that becomes intelligent enough at AI research that it’s able to help developers build a smarter AI. Then, that smarter AI is even more capable and able to help develop even smarter systems which in turn… this is where the recursive part comes in.
It’s easy to read something like that and imagine that the AI is doing more than what I’ve described. In the real world though, these systems aren’t autonomous. They are still dependent on a human setting the direction, examining the outputs, and sometimes adjusting instructions. Yet, even with these qualifications being able to create and sustain these types of feedback loops is a huge accomplishment.
There’s also the more science fiction version of this where the feedback loops happen so quickly that the system rapidly progresses faster than humans can control. I mean technically that is a possibility but it strikes me as unlikely… at least at present.
I don't know if Google has achieved RSI. But... If the reason Gemini Pro has not been released for 6+ months is they allocated all the resources they would have spent on it to a gargantuan training run to leapfrog past everyone else... ...then that'd be an impressive gamble.
Astra was probably our biggest competitive advantage while it wasn’t generally available.
Since we’ve had it our productivity jumped so much that we shifted some of our plans 6 months ahead and will ship them at DevDay instead of mid next year.
Tibo is pretty much the public face of OpenAI. That’s not his job, but he frequently presents behind-the-scenes information dealing with Codex and ChatGPT.
Maybe OpenAI is close to cracking it? Maybe they have internally?
Sam Altman on Slack to OpenAI staff, reported by The Information on June 10th.
If any one of these companies has achieved RSI the others will know. That’s just the nature of working in this industry, attending the same parties, seeing one another at coffee shops. These guys talk to one another. And if one of them has made this leap, the others will definitely be calling for a slowdown because they can’t abide the idea that they were leapfrogged.
Right. Dario was already claiming that GPT2 was too dangerous to open source back in 2019. I made fun of them then. Everyone should make fun of them now.
Yann LeCun has a lot of accomoplishments. He is one of the field’s early researchers. He helped prove that computers could learn to recognize the visual world for themselves. He helped demonstrate that back propagation (adjusting weights based on errors could help train multilayer networks) and his work on convolutional neural networks (with Geoffrey Hinton and Yoshua Bengio) became one of the foundations for modern AI infrastructure. He also advocated for and proved that computers were able to learn patterns and structures from enormous amounts of unlabeled data instead of having to depend on humans to label every example on which they were trained.
Any pursuit of superintelligence has to be grounded in the core principle that if the AI we build is not helping humanity and under human control, it's not worth pursuing.
We also need to accelerate and spread the benefits of AI, such that they are diffused broadly across countries, communities, and companies. This requires a frontier ecosystem in which both closed and open-source models can thrive.
And for firms, it’s imperative that they retain full control over their unique and tacit knowledge. Every organization should be able to build its own continuous learning loop/hill climbing machine, without becoming dependent on any one model provider, and have the ability to embed its own knowledge into models and weights they control…
…You guys are the frontier. By any reasonable metric — market share, revenue growth, model capability — the two of you have a duopoly on frontier intelligence. You’ve also claimed the lead is widening because of recursive self-improvement.
I don’t see what you see in the lab. If the unreleased models are scary enough that you think you should slow down, I support your decision to be responsible.
But stop pretending you need anyone else’s permission. Stop pretending antitrust law has to be suspended so you can form a cartel. Stop pretending you need a regulatory approval process that supersedes product liability. Stop pretending METR is independent when it is intertwined with Anthropic’s investors and staff. Stop pretending you need those same evaluators to police competitors who aren’t even at the frontier…
There’s more at the link, but he has the most plainspoken take on any of this. He also includes a political slant (he is co-chair of President’s Council of Advisors for Science & Technology) in his full post, but the part I’ve quoted is neutral.
We're publishing our most detailed threat intelligence report to date. It covers how people tried to misuse Claude—for cyberattacks, influence operations, surveillance, biology, and building weapons—and how we found and stopped them. We disrupted every operation in the report, and used the lessons from them to strengthen our safeguards. Where appropriate, we also shared what we found with authorities and other AI companies. These cases are not typical: we’re highlighting some of the most sophisticated misuse we’ve seen. But they’re especially important to discuss, because they show us where AI misuse is headed, where our safeguards work, and where they need to improve. We’re publishing this report so others can spot the same activity on their own platforms, and so we can give the public a clearer view of how emerging threats develop.
This report has everything! It really is an interesting read.
Well, looks like the Anthropic report didn't go down well in Beijing. When the MSS comes knocking on your door, things usually turn pretty ugly ...fast.
It’s worth pointing that the official Weibo account has denied these rumors, but such denials are common in China. The denials may be true. The allegations may be true. At this point it isn’t known.
Thank you for reading. If you know of other people who would be interested in these articles, please share it with them. If you have any questions or want to say hi, you can reply to this email or comment directly on the post.
Thanks again for being a subscriber. and have a great day!
Hype and puffery.