June 17, 2025
TLDR: Ord is unimpressed by AI Scaling Laws, thinks the METR paper implies early-2030s AI timelines, still places existential risk from AI at 10%, and thinks we should consider a moratorium on AI research.
Forethought is a new AI policy think tank that now has a podcast. Yesterday, they posted a 3-hour interview with Toby Ord, author of one of my all-time favorite books. They didn't post a transcript, so here were the best tidbits.
The Scaling Laws kind of suck. It is often said that the discovery of the Scaling Laws motivated Ilya Sutskever and Dario Amodei to speedrun through GPT-4, because it let them see into the future and know that big models would be smart models. An awkward wrinkle to that story is that the scaling laws actually show rapidly diminishing returns. Specifically, to halve training loss, you must increase compute by a factor of 1 million. To Ord, it’s a paradox that this empirical result would drive enthusiasm, instead of people saying “darn! Diminishing returns. Let’s give up.”
The exciting thing about the scaling laws was that it suggested that capabilities might improve smoothly with each order-of-magnitude (OOM) increase in compute. They grew GPT-1 by ~1.5 OOMs and said “woah, GPT-2 is way better.” Then, they grew GPT-2 by 2 OOMs, looked at GPT-3, and said “oh shit! This works!” So the math people did was:
xxxxxxxxxx100x compute = GPT-3 Goodness - GPT-2 Goodness = Worth it
After GPT-3, it was clear people were going to keep going. But you could imagine folks giving up after GPT-2. GPT-2 wasn't even close to doing anything useful, scaling further was going to cost millions of dollars, and at that time, OpenAI had many competing projects. Random aside: Empire of AI attributes OpenAI's continued focus on GPT-2 to the influence of Bill Gates. When Microsoft was considering investing, Gates demanded a system that could read books, understand scientific concepts, and assist with research. This steered OpenAI toward GPT-2 and away from game-playing AIs and robotic projects.
How reliable is reliable enough? Ord provides a detailed and admiring discussion of METR's Time Horizons research, which finds that the length of tasks AI can successfully complete 50% of the time is doubling every 7 months. This curve, which tightly fits the data going back to 2019, implies that by the end of 2028, we could have models one-shotting tasks that take humans a month to finish.
Ord sees 50% reliability as an excessively low bar. He cites the example of booking a flight, where mistakes can be so dreadful, that we probably want 99% accuracy. Otherwise, we have to check everything, and most of the efficiencies are lost. METR's sample size doesn't allow them to directly estimate a curve for 99% reliability, but Ord is able to fit METR's results to a simple model, which suggests that requiring 99% reliability shifts timelines back by 4-years, putting us in the early-2030s. Ord thinks this is more reasonable than the AI 2027 style predictions.
Right now, differences in AI timelines are explained by small differences in how people read the METR paper. The AI 2027 folks actually agree with Ord that 50% is too low a reliability standard, and instead use an 80% reliability standard. They still end up with a pre-2030 timeline, because they use a faster doubling time (4.5 months vs. 7 months), noting that time horizon gains have sped up in the last 18 months.
I'm sympathetic to Ord's early-2030 timelines, but find his 99% reliability standard for booking airline tickets to be perhaps suggestive of some travel anxiety. A couple years ago, I accidentally booked my return flight out of Bristol (BRS) instead of Brussels (BRU), and it wasn't the end of the world. And as someone who manages humans doing knowledge work, I would never think to expect 99% reliability on the tasks I assign. Ord must truly be spoiled by the research assistants at Oxford.
Ord is holding at 10% p(doom). In Precipice, Ord quantifies humanity's likelihood of doom from various risks. Ord had estimated a 10% chance of either extinction or permanent dystopia from unaligned AI, which was about 60% of his estimate for humanity's existential risk across all causes. I had expected that upon revisiting, Ord would be jacking the AI risk number up, because AI capabilities have advanced so much in recent years.
But Ord also recognizes that, in many ways, AI has taken a fortunate path. You could imagine an alternative timeline where the sorts of AIs that mastered Chess and Go solely through self-play somehow developed a generalized superintelligence. The AI would pop out of the box as a super powerful agent, utterly incomprehensible to us, and with no knowledge of human values. Instead, the first somewhat general AIs are these not-that-agentic, modestly-capable, somewhat interpretable systems, with a surprisingly deep understanding of human values. We might lament that we have a race dynamic going, but it's probably a good thing that - for now - the race is close, with several groups controlling similarly capable systems.
Ord also acknowledges that his estimates were just ballparks, so he leaves open the possibility that risk has increased some. He just doesn't think it has gone up to 30% or gone down below 5%.
Moratoriums are under-rated. Ord expresses frustration that we aren't seriously exploring the option of an AI pause. He thinks it is less intractable than most people assume. He makes the comparison with human cloning and genetic enhancements, where legal structures and scientific norms really have slowed down progress. He acknowledges that the AI case is much harder, because the short-term military and economic gains are so enticing. But even if there is only a 10% chance of a successful moratorium, given that it could save all of humanity, a moratorium is something we should be seriously considering.
I hope Ord is successful in calling for more discussion, and I hope things do slowdown. But even from a pure safety perspective, I'm not sure a moratorium would be great. If advanced AI is inevitable, and I think it is, then there are credible arguments for thinking short timelines are better. And even if a moratorium would be better for safety in theory, I just don't think we could trust other countries to cooperate. And even if we could, there still might be too much potential upside from AI to leave on the table.