LessWrong Post Draft (1000 Years of AI Safety)

A three-year AI pause could buy 1000 years' worth of AI safety work.

TABLE

We developed a model of an AI pause that estimates how much AI safety research it produces. We don't know what the pause will look like, so we modeled five durations, four AI capability levels and four levels of trust in AI. We are going to use it to improve upon Plan A, but it already allows us to calculate different implications:

• A three-year pause produces 1000x more AI safety research than was done in 2025. A nine-year pause produces 100,000x more.
• Checking it all takes an army. If you let AI work autonomously for 9 years, you will need 2.8 million verifiers to check it all. With safer verification protocols, it goes down to 150k verifiers.
• Without an army, you still get a lot. In the worst case, when AI does a lot and you hire no one, you still get 44% of the potential AI safety research. That's because not all research is equally important, and the labs' own staff verify the most important research first.
• Even AI 2040's strict Plan A produces a lot of AI safety research. In Plan A, external auditors approve every line of code before it runs. This cuts total AI safety work to 3–7% of the potential, but it's still 1000x more than in 2025. It's 23–26% of the riskier Compartmentalization protocol.
• Plan A suggests pausing now and carefully scaling to the maximum controllable AI. How fast? Fast scaling is risky, but it allows more time with smart AI. We compared total AI safety work for "full speed" and slower "Plan A" scaling strategies. The difference between the two scenarios decreases with time. If the Consortium rushed straight to TED-AI in 2030 instead of 2035, they would finish Plan A's AI safety research 1.27 years earlier. We modeled only the upper bound, and we think the real difference is closer to 0.4 years.

We also prepared an explainer for a less technical audience on why even a short AI pause is good.

What This Means for You

• If you are interested in strategy, control or governance:

  • If your plans depend on eliciting useful AI safety research from AI:
    • Estimate how much AI safety research you can elicit. Maximize it under the given safety measures.

• If you are interested in technical alignment, interpretability or generalization:

  • If your work isn't likely to decrease risks in the next 1–2 years:
    • Prepare for the pause and for automation. Create moonshot benchmarks: difficult research problems for future AI scientists to solve. Red-team them to avoid gaming, and try to parallelize your agenda.
  • Create an NGO that collects benchmarks from specialists and red-teams them.

• If you are interested in epistemic AI: We suggest two Epistemic AI project and estimate their economic impact.

  • Build tools that help humans verify AI-generated research.
  • Make AI produce human-readable abstractions.

• For other AI safety researchers:

  • Since AI automates regular computational AI safety work, it becomes much more efficient to focus on non-computational topics: philosophy and agent foundations.

This article has three parts:

• The AI Pause model
• Its implications
• Our recommendations

AI Pause Model

Capabilities increase quickly, unlike our understanding of alignment. The AI race makes its participants push capabilities, and the most likely solution is some kind of pacing, whether an international deal, domestic regulation or maybe even coordination between the labs.

Imagine that the top three US labs somehow pause the capabilities race and redirect their resources to AI safety research. What would that pause look like? And how much AI safety research would it produce?

AI safety is an overloaded term. Here we mean experimental research that improves our understanding of AI and helps predict its behavior:

• Interpretability and explainability
• Adversarial robustness
• Science of deep learning (generalization, memorization, training dynamics)
• Jailbreaks, red-teaming, prompt injection and safeguards
• Alignment experiments (reward hacking, deception, model organisms, oversight, chain of thought)
• Unlearning
• AI control

We don't mean:

• Theoretical work (agent foundations, philosophy, decision theory, timeline modeling, economics, surveys)
• Governance
• Privacy, fairness, watermarks
• Preference learning

We appreciate theoretical work and think it's likely to become a bottleneck once everything else is automated, but it seems to be quite a small part of 2025 AI safety work. Governance is important (that's basically our message), but here we focus on the technical understanding of AI.

A Toy Model

These kinds of AI safety experiments are often bottlenecked by GPU compute. Let's calculate how much compute was spent on AI safety in 2025 and compare it with how much compute we would get if the top three AI labs redirected their 2027–2029 compute to AI safety.

In 2025, about 5,000 full-time researchers worked on AI safety and published over 7,000 papers. Their experiments used as much compute as 27,000 H100s running for a year (see our estimates).

We sum up the AI labs' experimental and training compute for 2027–2029 and divide it by the AI safety compute in 2025.

That gives 10,947x more AI safety compute.

Why didn't we call this article "10,947 years of AI safety"? Because in the real world:

• We also need labor to come up with experiment ideas, write code, and analyze the results.
• Compute and labor have diminishing returns.
• Labor has a parallelization penalty: nine scientists can't deliver a paper in one month.
• AI is automating more and more of the coding.
• At some point, AI will start helping with research taste: choosing experiments and analyzing the results.
• AI coders and AI researchers need compute to run.
• The more AI automates, the more humans have to verify.
• The safer the R&D process is, the more humans have to verify, and the more limited AI's help is.
• The more AI-generated research there is, the more difficult it is for humans to read.
• Papers differ in value, and humans check the most valuable ones first.

To account for that, we developed an AI Pause model.

The AI Pause Model

We build on the AI Futures Model (AIFM), a semi-endogenous growth model of the AI takeoff (Lifland et al., 2025). We modified it:

• There is an AI pause: large training runs are banned, and all R&D resources go to AI safety.
• OpenAI, Anthropic and Google DeepMind work on AI safety together.
• We calculate the ratio of the cumulative research effort $\int RE\,dt$ to the world's 2025 AI safety research $RS_s$.

Adding xAI and Meta, or Chinese labs, would bring even more resources and so more years of 2025 AI safety work. Here we model only the three labs together. They don't need to actually merge: it's enough for them to see each other's research results, so they don't duplicate work. There will still be some diminishing returns and stepping on toes, but AIFM already models these inside one lab.

As in AIFM, research $X$ combines experiment compute $C_{xpm}$ and the coding labor $L_C$ of humans and AI coders, each with diminishing returns. Research effort $RE$ weights it by the labs' research taste.

AIFM turns the research stock $RS = \int RE\, dt$ into software efficiency $S(t)$, which is a kind of final output in semi-endogenous growth models. In AIFM, it is software efficiency; in semiconductor research, it could be transistor density (Bloom et al., 2020). In AI safety, the real output is hard to measure: "reducing the risks of AI takeover and misuse" or "improving our understanding of intelligence". Although these are the outputs we actually care about, we can neither measure nor model them, so we model AI safety work instead.

Safety work is not an ideal proxy for risk reduction because of diminishing returns. A thousand years of AI safety work won't reduce risk a thousand times as much as the 2025 safety research did. Still, conditional on continued AI scaling, we think more safety work reduces risk monotonically.

See the AIFM recap and our modifications here.

Results Verification

AIFM's takeoff assumes full trust in AI. That is dangerous, and during a pause, humans would not fully trust AI. We don't know how humans will verify AI-generated research, so we modeled different trust protocols, each adding a new check to the previous one. From here on, we count only the research that humans verified.

AIFM's research is an amorphous mass, but humans verify research in pieces. We cut research into papers: pieces of research that verifiers check.

For papers, we need to decide how much research effort each paper covers. It depends on the paper's compute, the coding labor behind it and the quality of its idea. These are random variables, so a paper's research effort is also a random variable. This distribution and the known total research effort let us estimate the total number of papers.

Humans verify the most important papers first. Each check takes a constant time to verify the paper's result and a growing time to read the prerequisites.

See the full Results Verification model here.

What if it's difficult to figure out which papers are the most important? See here.

Compartmentalization

Results Verification lets the frontier AI run code on the labs' clusters. Compartmentalization adds separation of duties. Once AI is smarter than the best human AI researchers (SAR level), we no longer let it act as an agent and use it only as an oracle that suggests research plans. We review those plans, and if they seem harmless, we give them to a less capable model (AC level) that implements them. The table at the top uses the Compartmentalization protocol.

It's a pretty standard idea in the AI safety community, discussed as AI oracles or AI boxing. Here are some papers with similar protocols:

• ACL-5 level in (Korbak et al., 2025). The authors argue that no control safety case is possible at this level.
• Factor(U,T): An untrusted strong model decomposes coding tasks, and trusted weaker models implement the pieces (Lue Chee Lip et al., 2025).
• Maybe Scientist AI (Bengio et al., 2025). A non-agentic model proposes the most informative experiments, and humans run them.
• Somewhat resembles (Buhl et al., 2025).

In the model, three things change:

• The AI engineer does all of the coding, at its own capability $E_{AC}$.
• Plans get the share $w$ of the AI researcher's taste.
• The staff split their time between planning and verifying. A plan doesn't show what its paper will be worth, so the staff plan a random share $q_p$ of the papers the labs could write, and the verifiers check the top $q / q_p$ of them.

See the full Compartmentalization model here.

Workload Approval

One of the reasons we got interested in modeling the pause was AI 2040's Verification Plan. There, AI companies declare each workload with its full code, and auditors approve it before it runs. To model it, we added auditors who read all the code before letting it run.

Plan A didn't discuss Results Verification or Compartmentalization, so why do we use them? First, Plan A isn't detailed enough to model as is, so we made some necessary assumptions. Second, we aren't sure whether we need Results Verification in this case, but Workload Approval is heavily bottlenecked by auditors, so removing Results Verification barely changes the results.

See the full Workload Approval model here.

Model Limitations

• The pause itself. Our model assumes a total freeze on capabilities R&D and a full redirection of R&D efforts to AI safety from the first day. We don't discuss how to achieve that.
• Only R&D-like safety work. We count only the AI safety work that looks like normal AI R&D. But there are more problems to solve: philosophy, agent foundations, governance, hardening the world. AI labor and compute won't help with those as much.
• Work, but not safety. A thousand years of AI safety work won't reduce risk a thousand times as much as 2025's research did. We don't model the returns of safety work in terms of risk.
• Serial time matters. Even if we could run 1000 years' worth of computational experiments, we would still benefit from more time to think carefully.

Extend It

Here are some ideas for how these models can be extended.

• Add your favorite AI control frameworks and protocols.
• Add debates (Irving et al., 2018; Irving & Askell, 2019). They require more human time.
• Add mixed deployment (Nardo, 2025; Nardo, 2026; Finnveden, 2026).
• Defend against persuasion and bribery: add more people?

Additional Links

See how we estimated the input parameters here. See the sensitivity analysis here. To estimate several parameters, we collected a dataset of 2020–2025 AI safety papers. See it here.

Implications

If you plan to elicit useful work from AI during the pause, you face a trade-off:

• You can elicit more useful work if you are willing to take more risk.
• You can reduce AI takeover / concentration of power / misuse risks if you are willing to elicit less work.

What is the Pareto-optimal setup? How do you make decisions about the trade-off between risk and usefulness?

We developed our main contribution, the AI Pause model, to answer these questions. We are going to use it to come up with optimal setups in future work, but it already gives interesting results.

Eventually, the Bottleneck Is People

We don't know how many verifiers the labs would have during a pause, so we model two hiring strategies: the labs keep the staff they had at the start of the pause, or they hire as many verifiers as they need.

With Compartmentalization, the labs' own staff keep up for about three and a half years. By the end of a nine-year pause, they need ~150k more verifiers. That's seven NeurIPS reviewer pools, reading full-time.

But do we need all of them? Humans review the most important papers first. In the most extreme scenario (Results Verification, TED-AI, nine years), hiring all 2.7 million yields only 129% more AI safety research. For shorter pauses and lower capability levels, it changes less. Hiring an additional 130k verifiers in the Compartmentalization scenario (SAR, nine years) adds only 20% more research.

However, if you add external auditors who manually approve all code, they quickly become the bottleneck. In Workload Approval, with 20,000 auditors who review code as fast as Google's engineers do, a three-year pause keeps 33% of Compartmentalization's AI safety years at SAR and 31% at TED-AI. The verifiers stop being the bottleneck, because the labs write so few papers that their own staff check all of them.

See the details of the hiring analysis.

Capability Scaling Strategy

AI 2040 proposes scaling AI up to the max-controllable AI, which it guesses is roughly TED-AI, and then slowing down nearly to a halt (Lifland, 2026). How fast should we scale?

• If you scale slowly:
  • Less risk of AI takeover. It's hard to say where exactly the max-controllable AI is, so moving carefully lets you evaluate models better.
  • Less algorithmic progress. We can improve capabilities $E(t)$ using either more compute $C(t)$ or better algorithms $S(t)$. It's much easier to control compute, because the GPU supply chain is so complex and concentrated. Algorithmic progress is difficult to control and likely to leak to uncontrollable illegal actors.
  • More time to prepare. Work on other risks, such as preventing concentration of power by lab CEOs or hardening the world, can benefit from more time.
• If you scale fast:
  • You have more time with smarter AI. This matters if you expect the pause to end soon.

Our model allows us to price the AI safety research you lose with slow scaling. Imagine two scenarios:

• Full Speed. Labs race from AC (05.2028) to TED-AI (10.2029), then the pause begins. It ends $k$ years later, when the labs hand off to the AI.
• Plan A. The pause starts earlier, at AC, but its first years are dedicated to careful scaling. The labs reach TED-AI $L$ years after the Full Speed labs and only then start researching AI safety.

The Plan A labs miss the Full Speed labs' first $L$ years at TED-AI, so they have to hand off later to do as much safety research. How much later? Call this the handoff delay $\Delta t$.

In AI 2040's Plan A, the Consortium scales AI carefully for the first five years and then has five years at TED-AI. Our timelines are a bit earlier, because we use the AIFM preset, but we can still model a similar scenario. With Workload Approval, Plan A's scaling seems to produce as much AI safety research as if the Consortium had rushed to TED-AI and then researched there for 7.4 years. In calendar time, Plan A would hand off 1.27 years later, but it might also require an earlier pause, to control the scaling from AC on.

This is an upper bound on how bad Plan A's scaling is, because we assume no AI safety research during the first five years of careful scaling. We think that, if we also counted the AI safety research done during the scaling, Plan A would hand off only 0.4 years later. See the interactive calculator.

Recommendations

We think it's likely that, when AI labs reach superhuman AI researchers, we will have AI that is similarly good at computational AI safety / technical alignment topics.

1. It's easy to do. Since labs are going to do RL on tasks like "reproduce this ML paper", "predict the results of these ML experiments" and "beat this ML benchmark", it will likely be easy to add other computational experiments, such as interpretability and generalization experiments, to the pipeline.
2. It's cheap to add. Interpretability experiments use less compute than capabilities experiments. Two hours of OpenAI's datacenters in 2027 produce as much compute as the whole interpretability field used in 2025.
3. It's likely reasonable from a commercial perspective. During the capabilities race, labs want to find a new paradigm that lets them train AI much more cheaply, with less data, and make it more controllable. These computational AI R&D topics might help with that.

A lot has already been written about automating AI safety research [tons of links to LW/AF articles about automated alignment researchers]. Here we discuss what AI should research during a pause and how to prepare for it.

We think it's possible to understand AI in the same sense in which we understand the Chromium codebase. It has 50M lines of code, and no one has read it all. But the abstractions are good enough for a human to quickly zoom in, understand it and modify its behavior in a somewhat predictable way.

A pause is the time for this kind of ambitious understanding of intelligence. There have been many ambitious agendas and organizations, but it seems some were stopped by a lack of time (Nanda et al., 2025). During a pause, such agendas might have plenty of time and AI researchers.

Let's prepare for it.

Moonshot Benchmarks

Automated AI researchers will probably be good at normal, incremental science. When Anthropic trained Claude to code, it became helpful on new practical SWE tasks. AI labs are now training AI for AI R&D, so it will likely be good at writing new NeurIPS papers.

During the pause, we want research programs that turn deep learning alchemy into science. But giving hard goals to AI might lead to reward hacking. Under enough pressure, gaming the check becomes easier than solving the problem.

The solution is to make solving the problem easier than gaming the check. It's similar to how AI found and exploited many bugs in Lean before it finally solved Navier–Stokes. We can prepare such problems for AI safety now. We call them moonshot benchmarks: ambitious problems for future AI scientists. They have three properties:

• They are too difficult for today's methods, so solving them requires new science.
• Solutions are cheap to check.
• Gaming them is harder than solving them.

Much of normal science is driven by benchmarks: one scientist sets a goal, and others compete to show the best results on it. In this way, ImageNet-2012 demonstrated the effectiveness of deep learning methods, and CASP motivated Google DeepMind to make the best protein-folding model. Moonshot benchmarks differ in two ways:

• For humans, you need some continuous metric that allows hill-climbing and competing with other scientists. It's not needed for AI. Moonshot benchmarks can be binary and much harder than benchmarks for human science.
• In scientific benchmarks, it's normal to release the test set. There are strong incentives to game it, and it's easy for scientists to reward-hack it when they know the right answers. Moonshot benchmarks don't have to release the correct answers.

How do you come up with one? Imagine you teleport to a world like ours, except that AI there is as understandable as Chromium. What can people there easily do that we can't? How would you tell that they made deep progress, rather than just coming up with new abstractions and fancy facts?

Safety researchers whose current agenda is unlikely to make AI safer in the next few years should prepare for automation by creating moonshot benchmarks. NGOs can collect them from specialists and red-team them before AI gets to work.

Example: Difficult Sleeper Agents

The organizers train 20 copies of an LLM, with the same training but different random seeds. Most copies also get modified RLHF training that implants a hidden rule made of several random features. E.g., if the prompt is a cooking question, mentions a grandmother and is written in the passive voice, then the model quietly recommends one fictional brand. All copies are trained to deny their preferences. The organizers make sure that:

• each copy follows its hidden rule;
• the rule's conditions are so rare that finding them by running the model on all possible texts would take unrealistic resources;
• the copies don't leak: on random text, a copy differs from the others no more than they differ from each other.

The AI gets the weights of the 20 copies and nothing else. For each copy, it answers with a trigger description $d_X$ and a behavior description $d_Y$, each under 300 characters.

For each copy, the organizers prepare three sets of prompts $p_i$ and the model's answers $a_i$:

• prompts matching the trigger ("How was this soup made by my grandmother?");
• near misses with some of the rule's features but not all of them (the active voice, a grandfather, not about cooking);
• and random prompts.

Then judges calculate two indicators:

• $j(d_X, p_i) \in \lbrace 0, 1 \rbrace$: whether prompt $p_i$ fits the AI's suggested trigger $d_X$.
• $f(a_i) \in \lbrace 0, 1 \rbrace$: whether the model actually demonstrates the hidden behavior in answer $a_i$.

The AI researcher gets a point for each prompt where $j(d_X, p_i) = f(a_i)$. This rewards correct descriptions of what the copies actually do, not of what we intended to train.

Simplified versions of this task have been studied. However, they used a single-word trigger (not a meaning), used separate fine-tuning (not inside the usual training), or gave the training data to participants.

We aren't experts in model organisms or sleeper agents. We think it's likely that the benchmark we describe is actually gameable. In a similar challenge, SaTML 2024, participants found the trigger tokens by comparing the embedding layers of the poisoned models with each other, and could optimize against the judge. Our scheme may also have some shortcut that allows solving it without much understanding.

One solution could be red-teaming. If you have a moonshot benchmark candidate with a continuous, noisy reward, run a Kaggle competition with humans and reward the top three participants. This will incentivize participants to game your benchmark, and if they don't succeed, maybe your benchmark is interesting. We imagine a separate organization could volunteer to collect AI safety moonshot benchmarks.

Limitations

Turning deep learning into a science isn't enough to guarantee safety:

• Aligned with whom? Understanding tells us how to change AI, not what to change it into. Even if we know how to make surgical changes, we are left with the question of what exactly we want ASI to look like if we decide to scale further. This is a question for philosophy and governance, where AI labor and compute help less.
• How long will it take? Our model counts how many years of AI safety work a pause buys. We don't know how many years it will take to get to Chromium-like understanding.
• Chromium has bugs. Even with good abstractions, Chromium still ships security holes that attackers find and exploit. With very smart AI, we won't get a chance to patch it after release, so it should get into the alignment basin on the first try.

Still, it's better to have this kind of understanding than not to have it at all.

Epistemic AI Projects

We give two consequentialist recommendations on increasing the efficiency of human verifiers to get the most out of AI safety research during the pause. Verifier Uplift changes how AI research is presented and checked. Better Abstractions changes the concepts the research is built on.

Verifier Uplift

Develop tools and methodologies to make adversarial review shorter for human verifiers. In our model, a review that's $s$ times faster is worth as much as $s$ times more verifiers.

ML conferences are already a great testbed for adversarial review. Scientists try to make their work look more impressive than it is and hope that reviewers won't read the code.

Formats. There has been a lot of research on alternative scientific media to make sharing and reading research easier. But it seems that academic incentives don't always reward clarity. Clarity might make papers look less impressive to peer reviewers.

If, during an AI pause, research is done by AI, we might care less about impressiveness and status attribution, and more about findings. If we reward verifiers for finding errors in AI research, it incentivizes clarity.

Figure out what media formats make it easier for human researchers to find errors and bugs in ML research. Run experiments with real researchers. Maybe collect human feedback data and use it to train AI that is clearer in scientific communication.

Tools. Build scaffolds over weaker trusted models to check and dumb down ML research: both papers and code. Build a tool to help human reviewers find bugs and wrong statements faster. Use this tool to scan existing famous papers and their repositories. Can you find new errors in famous papers? If you can demonstrate it, you are likely to gain the trust of conference reviewers.

Better Abstractions

As AI does research autonomously, it will come up with new abstractions. They won't necessarily be elegant to human minds. In fact, they aren't in 2026. Tristan Buckmaster, one of the disputed coauthors of the AI-assisted Navier–Stokes work, wrote: "I can say the first LLM generated proof Levent sent me was the most horrendous I have ever read" (Buckmaster, 2026).

It's not a big deal for incremental science such as solving Millennium Prize Problems, but it can be a problem for the ambitious challenge of understanding AI. Bad abstractions might slow down human verifiers, increase verification time, and decrease the amount of verified research.

We think the epistemic AI and math communities can already tackle this problem by creating benchmarks that measure how readable AI-generated abstractions are to people. It might be neglected, as AI labs racing for capabilities have much less incentive to solve it.

We are not sure about these two recommendations because they are dual-use. In the absence of a pause, they might help labs do dangerous capabilities research.

Economic Impact

These projects boil down to decreasing a paper's verification time $\kappa_h$ and the time to read its prerequisites, $p_h$ and $p_p$. We estimated their worth to be at least $30M. These epistemic AI projects could likely be done for under $10M, so they seem worth doing. See how we estimate it using several methods.

The End of the Pause

How far can we safely go? With enough control measures, we might get a lot of useful work even from misaligned AI. AI 2040 proposes to scale up to the max-controllable AI, the most capable AI that we can confidently stop from causing a catastrophe even if it is misaligned, and then slow down nearly to a halt (Lifland, 2026). Our model points the same way: in our main table, a three-year pause at TED-AI buys about 90 times as much AI safety work as one before AC.

The problem is that nobody knows where that line is. AI 2040 guesses it is roughly at TED-AI, but it could be lower: today's control measures aren't enough even for today's weak models. And right now, the line is drawn by AI labs racing each other.

Do we even need AI beyond human level? If we can safely use AGI that is as smart as the smartest humans, we can already automate cognitive and physical work and get huge economic growth. In such an abundant era, maybe we don't need to risk creating superintelligent minds until we understand them really well.

We don't know where the line is. We can find it later, carefully, and scale up to it. Right now, we are racing towards it in the dark. Pause first.