Yesterday OpenAI launched Astra, its latest model. Greg Brockman, OpenAI President, claimed it marked the onset of artificial generalised intelligence (AGI). The main evidence cited was a very high score on the ARC-AGI-3 benchmark, managed by ARC Prize.
In OpenAI’s launch announcement[1] they highlight a 99.9% score and share a graph comparing that to Claude Opus at 30.2%. However, as with all benchmark reporting, they take the best score achieved. Looking at the related ARC Prize report is more revealing.[2]
What the Numbers Actually Show
ARC Prize ran a range of tests across different reasoning effort levels (none to max) and using two different harnesses — the standard ARC harness and a “provider adapter harness” from OpenAI. Astra’s highest score of 99.9% was achieved with high reasoning under the provider harness. Its lowest was 17.5% under the standard harness.
| Reasoning effort | Standard harness | Provider harness |
|---|---|---|
| Low | 17.5% | 98.0% |
| High | 54.8% | 99.9% |
Produced from original ARC Prize report [2]
Using the provider adapter harness, Astra achieved over 95% against every effort setting. Low effort jumped 80 points from 17.5% to 98%. The adapter is clearly adding significant capability. There is some description of what it does — it appears to be providing context across calls and compacting information — but it is not clear whether it is a standard arrangement for Astra that all users will have access to.
Claude Opus was tested using the standard harness only, at a high reasoning level. So a fair comparison is Astra’s 54.8% at high under the standard harness versus Opus’s 30.2%. Though the definitions of reasoning levels are not standardised across models, and are at best similar.
Of course, Opus is not Anthropic’s latest model — Fable 5.1 is. But it does not have a score for ARC-AGI-3 yet: Fable’s guardrails identified the tests as reverse engineering attempts and prevented testing.[3] So the story for Astra on this benchmark is much more mixed than the headline would have you believe.
The Benchmark Saturation Problem
There is also a fundamental problem with AI benchmarks: saturation. As soon as a benchmark is known, it is possible to train a model against it. Even if the actual tests are kept private, the style of the benchmark is known and previous results can be used to build new approaches. All models are incentivised to achieve high scores, so they train against benchmarks. The result is saturation — all models migrate from low to high scores simply by targeting the benchmark.
This has happened for previous ARC Prize benchmarks. François Chollet, co-founder of ARC Prize, forecast saturation would happen for ARC-3 within a year of release. Yesterday he commented that it has happened faster than that.[4] Though that depends on which harness you use.
Astra clearly is a better model and has new capabilities. We will know how much it is a step change once Fable 5.1 scores are available like-for-like on ARC-AGI-3. But benchmarks are not the real world, and as they can be targeted, they are a poor substitute for capability.
I will reserve judgement on how powerful AI tools really are, and whether they are getting close to AGI, based on transparent detailed examples and adoption in day-to-day life — not artificial measures that can be gamed.
References
[1] OpenAI, “A new generation of intelligence”, 3 September 2026 — openai.com
[2] ARC Prize, “OpenAI’s GPT-6 Astra on ARC-AGI-3”, 3 September 2026 — arcprize.org
[3] ARC Prize on X, 1 September 2026 — x.com/arcprize
[4] François Chollet on X, 3 September 2026 — x.com/fchollet