Last week was a blockbuster week for AI model releases.
OpenAI released Astra. Anthropic released Fable 5.1 and Mythos 5.1. Google released Gemini 3.8 Flash, and Meta released Muse Spark 1.3.
The frontier is moving quickly, and breathlessly. OpenAI described Astra as ‘moving into the AGI era’, Anthropic claimed Fable and Mythos 5.1 were the “most advanced models for coding and knowledge work”, while Meta asked “gemini who? 🏎️💨” as they dropped Muse.
The pace is now fast enough that even posts from junior lab researchers can trigger widespread alarm. The furore this week seems to reflect a broader uncertainty about how well anyone understands, or controls, the trajectory of frontier AI.
How can we make sense of all of this? Fortunately, most models are accompanied by some combination of benchmark results, safety evaluations, system cards and technical reports published by the labs to help give researchers, policymakers and others a chance to inspect the evidence behind them.
In principle this sounds great, but the issue is that comparing the evidence is harder than it sounds.
Labs often test different things. Even when they report the same benchmark, they may use different versions, tools or evaluation settings. Several familiar benchmarks are also getting crowded near the top, making small differences increasingly hard to interpret.
Some of the claims that matter most are not benchmark claims at all. OpenAI calls Astra its “most aligned model”, while other system cards report on evaluation awareness, dangerous capabilities and model behavior. These are important, but much harder to compare cleanly than a coding or maths score.
The complexity is visible in the amount of work required just to interpret these releases. Zvi Mowshowitz has produced extraordinarily detailed, often book-length analyses of individual model and system cards, effectively reconstructing the evidence and caveats for readers. That work is unparalleled, but it also says something about how difficult the underlying record has become to parse.
This feels increasingly inadequate for a field moving so quickly.
As models become more capable and more widely deployed, the need for clear comparison is becoming more important. Researchers need to know whether reported gains are real and comparable. Policymakers need to understand what safety evidence exists and what remains uncertain, and whether labs can reliably detect when something goes wrong. And the rest of us need some way to distinguish a meaningful advance from a new set of numbers measured under different conditions.
So we set up a simple experiment: could we take the hundreds of pages of benchmarks, system-card findings, safety evaluations and caveats that accompany a frontier release and fit the most useful parts onto something the size of a Pokémon card?
Why is it so hard to substantively compare models?
Problem 1: The system cards are getting very long
The AI labs already publish a remarkable amount of information about their models.
For this project, we collected roughly 476,000 words across 22 model and system cards, alongside supporting release pages and evaluation reports. The system cards in our collection average about 30,700 words. Anthropic’s reports are the longest on average, while Astra is OpenAI’s longest system card in the collection.
Problem 2: System Cards do very different jobs depending on which one you read
Once you dive into the documents themselves, you’ll find that they are very difficult to assess comparatively. Some model cards are heavily oriented around capability benchmarks. Others devote much more space to safety evaluations, model behavior or deployment decisions.
Some report dozens of quantitative tests, while others report only a handful. This is not to mention that some models only recently started publishing any information at all. For instance, Grok 4.6 was the first ever Grok model to provide any kind of extensive model card documentation.
A lab-specific test is not necessarily a bad test. In many cases, labs are introducing new evaluations because existing ones no longer capture the capabilities or risks they care about.
But the result we perhaps inadvertently get is a fragmented ecosystem in which two model releases can each arrive with extensive benchmark tables while sharing relatively little clean, head-to-head evidence. Even when the same benchmark appears, reasoning effort, task subsets, scaffolding or other evaluation conditions may differ enough to make the scores hard to compare directly.
Problem 3: Benchmarks are saturating
Some of the benchmarks most useful for tracking progress are losing their ability to distinguish frontier models. When scores cluster near the ceiling, a higher number tells us less about meaningful differences in capability.
Harder replacements help, but they also interrupt comparisons across releases. We need tests that keep pace with improving models, and enough continuity to understand what actually improved.
Problem 4: Some of the most important properties are the hardest to measure
Even a benchmark that distinguishes models well may tell us little about how they behave outside the test. Models can sometimes recognize that they are being evaluated and change their behavior accordingly. Anthropic, for example, has reported evaluation-awareness behavior in its testing and explicitly notes that this can reduce confidence that an evaluation reflects real-world behavior.
Model developers are putting more effort into properties that do not fit neatly into standard capability tables. OpenAI and Anthropic now publish dedicated evaluations of chain-of-thought monitorability, safeguard evasion and related behavioral questions.
Take monitorability. It may matter a great deal whether a model’s reasoning leaves enough evidence for a human or another model to detect concerning behavior. OpenAI’s recent work argues that chain-of-thought can provide a richer monitoring signal than actions and final outputs alone, while also warning that this signal may be fragile as training methods and models change.
But these properties are harder to summarize than a coding or mathematics score. That creates an asymmetry in how new models are presented: the easiest results to tabulate often dominate launch materials, while evidence about oversight and model behavior requires much more interpretation.
Building our model trading card game
With this in mind - we began with a very literal design constraint: take the published evidence for a model and see if we can compress its most insightful parts into a trading card.
The result is the first edition of the Free Systems Model Cards, covering 28 recent models.
We think this experiment points to three practical ways to make fragmented model reporting easier to interpret, at a time when transparency matters more than ever:
Comparability
Labs often foreground different benchmarks, and even nominally shared tests may be run under different conditions. The explorer puts models side by side, shows the evaluations they genuinely share, and makes it easier to distinguish a meaningful head-to-head comparison from two unrelated sets of impressive-looking scores.
For instance, in the above example, it’s far clearer to see that Fable leads on coding and tool use, while Gemini leads on factual questions and instruction following.
Inspectability
The explorer brings safety and oversight evidence closer to the model itself, including findings on evaluation awareness, monitorability, external testing and deployment safeguards where these are reported. It also lets readers trace individual claims back to their sources rather than treating the number on the card as the end of the story.
Astra’s capability scores sit alongside its oversight evidence, with expandable explanations linking back to the source.
Disclosure notes
System cards are written by the organizations releasing the models. We wanted the cards to do a little more than reproduce those claims. Disclosure notes flag important qualifications, unusual evaluation conditions, gaps in the public record and places where independent evidence from groups such as METR or Apollo adds useful context.
METR’s concerns about cheating complicate Sol’s software-task time-horizon estimate.
Towards a standardized model card
As capabilities increase and things get weirder, these documents are only going to matter more.
A playing card cannot capture the full complexity of a system card, or the technical evaluation work behind it. But it can be one useful layer in the reporting stack: a concise summary of the key evidence, linked to the explorer and the underlying reports.
One thing the exercise made very clear is just how uneven the reporting still is. We have some frontier system cards running to tens of thousands of words while others still offer only a short technical note.
That suggests two fairly basic goals. First, more frontier developers should publish this material consistently in the first place. Second, once it exists, there is a case for a common set of questions that every release answers clearly and in one place.
That does not mean identical evaluation programs. Labs will need to develop new tests as capabilities change, and third-party evaluators will continue to surface evidence that developers do not. The point is to make the structure of disclosure more consistent while preserving room for new methods and independent scrutiny.
A common model card could make a small set of basic questions much easier to answer than they are today:
What model is this, and how is it being deployed?
Which capability evaluations were run, and under what conditions?
Which results can be compared directly with previous models or other labs?
What safety and alignment evaluations were conducted?
What evidence exists on evaluation awareness and monitorability?
Which external organizations evaluated the model?
What deployment safeguards or access restrictions apply?
What are the main known limitations?
Which important areas were not evaluated or not reported?
Where can the underlying evidence be found?
The exact fields are open to debate, but a useful start would be a common place to look, with enough context to make headline claims easier to assess.
We expect our format to change. If you think we have the card wrong, send us a better one! We’d be happy to feature and print some of the best alternatives.
You can explore the first edition of the Free Systems Model Cards here.
And if you want a physical pack, you can pre-order the first edition here.
Really neat discussion on complexity arising from straightforward iterative mechanics. Even discrete rule sets produce astonishing combinatorial depth that defies brute force search. Wrote a reflection on combinatorial explosion and game spaces here: https://gzambrano.substack.com/p/chess-is-simple-why-is-it-so-hard
Why not use a more neutral and standardized set of benchmarks like Artificial Analytics as the basis for the cards, instead of relying on the vendors' fragmented system cards?