Back to list
Nov 08, 2026
2.5K
0
0
Research

METR: Anthropic Models Improving at About 2x Historical Rate

METR's 11-week review at Anthropic estimates well-specified capability gains run near 2x the earlier rate, with high uncertainty and heavy redactions.

#METR#Anthropic#AI R&D Acceleration#AECI#Epoch Capabilities Index
METR: Anthropic Models Improving at About 2x Historical Rate
AI Summary

METR's 11-week review at Anthropic estimates well-specified capability gains run near 2x the earlier rate, with high uncertainty and heavy redactions.

Introduction

On November 5, 2026, the research organization METR published "Independent Assessment of AI R&D Acceleration at Anthropic in October 2026." The report describes a high-access evaluation that METR staff (2.5 full-time equivalents) carried out at Anthropic from August 12 to October 29, 2026, to judge how much AI is speeding up AI research and development inside the company.

METR's headline conclusion is stated with a qualifier attached: "as of October 2026, Anthropic is likely improving the capabilities of their models on well-specified tasks at around twice the rate they did between March 2024 and October 2025." METR gave that estimate "with high uncertainty" and wrote that it "would not be surprised if the true acceleration was closer to 1.3x or 3x the historical rate."

The report arrives days after Anthropic's amended S-1 filing warned that internal estimates may "imminently" meet the Responsible Scaling Policy's definition of "dramatically accelerated progress attributable to AI," covered separately in our S-1/A article. METR states that this engagement is distinct from the assessment Anthropic announced on October 15, and that it did not pay special attention to the RSP thresholds.

What METR Concluded

METR organized its conclusions around three questions: how fast capabilities are progressing, how much of any acceleration is due to AI, and whether bottlenecks could limit further acceleration.

QuestionMETR's answerStated uncertainty
Pace of capabilities progressAround 2x the March 2024 to October 2025 rate, on well-specified tasksCould be closer to 1.3x or 3x; unclear how it translates to broader, less well-specified tasks
Share attributable to AIA guess that a majority (perhaps about 80%) of the October 2026 acceleration is due to AI"Substantial uncertainty"; late-2025 acceleration was likely influenced by non-AI factors
BottlenecksNo clear bottlenecks or steep diminishing returns identified for the next 12 monthsUnderstanding "changed several times" during the assessment

METR also reports one point it holds with confidence: the acceleration was not due to an acceleration in the growth of training compute. The supporting evidence was redacted.

The Benchmark Evidence

The quantitative anchor is AECI, Anthropic's version of the Epoch Capabilities Index. METR used Epoch's public code and bootstrapped 95% confidence intervals. It estimates that benchmark scores for Anthropic's models have grown at 1.9x the pre-October 2025 rate (95% confidence interval 1.5 to 2.8x), with the breakpoint fitted at October 7, 2025. METR's plain-language reading is that Anthropic made as much benchmark progress in the past year as it did between October 2023 and October 2025, roughly Claude 2.1 to Sonnet 4.5.

METR's figure is higher than the 1.7x in the Haiku 5.5 system card, which METR attributes mainly to its inclusion of newer unreleased models. The data includes internal models METR calls Model 2, Model 3 and Model 4, which finished training between July and October 2026.

METR tested how sensitive the 1.9x figure is to analysis choices.

Analysis choice variedEstimated acceleration
Link function (logistic, probit, cauchit, log-log and others)1.1x to 2.9x
Which model snapshots are included1.9x to 2.2x
Which benchmarks are included1.9x to 2.2x
Regularization1.9x to 2.1x

The link-function spread is the widest. METR also notes that plotting all frontier models, not just Anthropic's, gives a lower estimate, and that under one of its two interpretations 1.4x is the more accurate number. Saturation is a further concern: Model 4 scores above 50% on all 15 benchmarks used for it, which METR says makes METR "somewhat nervous" about its ability to measure AECI now and in the future.

Evidence on AI's Role

METR describes this evidence as consistent with AI use driving the change, but loose.

  • Self-reported uplift. In Slack polls, average self-reported uplift for the most capable internal model in October 2026 was about 12x. METR calls this "only very loose evidence" and cautions that individual speedup estimates "do not correspond to acceleration rates."
  • Interviews. Estimates covered "a quite wide range" but were centered around 2x. In METR's table of 11 interviewees, classified with Claude's help, 2 were coded Yes, 3 Leans yes, 4 Borderline or unclear, 0 Leans no and 2 No. One interviewee guessed that 15 to 20% of the change is due to AI.
  • Autonomy. METR understands AI cannot yet make "highly significant and fully autonomous contributions" to the most important parts of model training, though researchers described "signs of life" suggesting this could change soon.

What Was Redacted

Anthropic held unilateral redaction rights over non-public information, while METR kept the right to disclose that redactions occurred and whether findings depended on them. METR's summary is blunt: "Anthropic redacted a majority of the evidence we found most informative for these conclusions." Anthropic described the redactions as "aimed at removing confidential information that could accelerate other AI developers."

Type of evidenceShared with METRDescribed in public report
Benchmark capabilities of internal modelsYesYes
Key intermediate measuresYesNo
Algorithmic progress estimatesYesNo
Evidence of AI's contributionYesPartly
Measurements of key R&D inputsMostlyNo
Details of bottlenecks and forecastsPartlyNo
Interviews with staffYesPartly

METR adds that the public "has to rely on METR's judgment much more than would be ideal." Separately, Anthropic canceled a planned verification of its compute allocation between capabilities and safety midway through the engagement, before METR reached substantive conclusions. Anthropic said it wanted to keep the engagement focused on risk assessment and would revisit compute verification later.

Limitations and Disclosures

METR lists reasons the true figure could be higher: possible hyperbolic growth, automation being closer than estimated, and AECI understating real-world impact. It lists reasons it could be lower: non-AI factors explaining most of the acceleration, acceleration being limited to well-specified tasks, and Anthropic's historical AECI growth being unusually low.

On process, METR disclosed that Claude was involved in most data collection and analysis it received, and in the majority of its own analysis, and that it detected no discrepancies consistent with systematic bias. One team member had been in discussions with Anthropic about a role, received an offer and declined it; their work was checked by an unconflicted colleague. Anthropic provided free tokens, and METR did not take payment.

Outlook

The report is a measurement of one lab's benchmark trajectory plus internal evidence, not a forecast. Its value is the explicit uncertainty: a central estimate near 2x, a plausible range of roughly 1.3x to 3x, and a clear statement of which parts of the picture readers cannot inspect. Whether the figure holds will depend on evidence METR says it could not publish, and on a measure, AECI, that METR itself flags as approaching saturation.

Conclusion

For researchers, policy readers and anyone tracking frontier-lab risk disclosures, this is a primary document worth reading beyond the headline, especially the redaction table and the limitations sections. The 2x figure is a careful estimate on well-specified tasks, produced under constraints METR describes openly.

Editor's Verdict

METR: Anthropic Models Improving at About 2x Historical Rate is a report worth reading in full, caveats included.

The strongest case for paying attention: disclosure of the redaction terms and an evidence table shows readers what the findings can and cannot rest on. That alone raises the bar for what readers should expect in this space. Reinforcing that, central estimates come with explicit ranges, including the 1.3x and 3x outcomes METR would not rule out — practical value rather than just headline appeal. The broader signal worth registering is straightforward: acceleration in this report means capability gains on well-specified tasks, so the 2x figure should not be read as a statement about every kind of work. On the other side of the ledger, one constraint is real rather than a marketing footnote: redactions cover a majority of the evidence METR found most informative, so readers must rely on METR's judgment. It should factor into any serious decision. Layered on top of that, the benchmark measure covers verifiable tasks and may not track real-world impact in either direction — which narrows the set of teams for whom this is an obvious yes.

For AI safety researchers, policy analysts and anyone tracking how fast frontier labs are moving, it is required reading, with the redaction caveat kept in view when quoting the 2x figure.

Advertisement

Pros

  • Disclosure of the redaction terms and an evidence table shows readers what the findings can and cannot rest on
  • Central estimates come with explicit ranges, including the 1.3x and 3x outcomes METR would not rule out
  • Robustness checks across link functions, snapshots, benchmarks and regularization are published in tables
  • A conflict-of-interest disclosure and contract summary are included in the report

Cons

  • Redactions cover a majority of the evidence METR found most informative, so readers must rely on METR's judgment
  • The benchmark measure covers verifiable tasks and may not track real-world impact in either direction
  • A planned compute-allocation verification was cancelled midway and produced no conclusions
Advertisement

Comments0

Key Features

1. Evaluation ran from August 12 to October 29, 2026, staffed by METR at 2.5 full-time equivalents 2. Central estimate: capabilities on well-specified tasks improving at around twice the March 2024 to October 2025 rate, with METR saying 1.3x or 3x would not surprise it 3. AECI benchmark scores accelerated 1.9x since October 2025 (95% confidence interval 1.5 to 2.8x); robustness checks range from 1.1x to 2.9x 4. METR's guess is that a majority (perhaps about 80%) of the current acceleration is due to AI, held with substantial uncertainty 5. METR is confident the acceleration was not due to faster growth in training compute 6. Anthropic redacted a majority of the evidence METR found most informative; METR published a table of what was and was not shareable

Key Insights

  • Acceleration in this report means capability gains on well-specified tasks, so the 2x figure should not be read as a statement about every kind of work
  • The 1.9x benchmark estimate ranges from 1.1x to 2.9x depending on the link function, which is wider than the other robustness checks
  • Including non-Anthropic frontier models lowers the measured acceleration, and METR says 1.4x would be the better figure under one of its interpretations
  • Self-reported uplift of about 12x is described by METR as very loose evidence and is not an acceleration rate
  • Model 4 scoring above 50% on all 15 AECI benchmarks makes the measuring instrument itself a concern for future assessments
  • METR disclosed that Claude helped with most of the data collection and analysis it received and found no discrepancies consistent with systematic bias
  • The cancelled compute-allocation check means this assessment says nothing substantive about how Anthropic splits compute between capabilities and safety

Was this review helpful?

Share

Twitter/X
Advertisement