Self-Regulation Meets the Open-Weight Problem
AI self-governance is coming just like we recommended, but the release of the most-powerful open-weight Chinese AI model yet raises huge questions about it. This and more in this week's System Check.
A huge week for AI governance
This week saw two massive developments for how we will govern superintelligence. First, Demis Hassabis, founder and leader of DeepMind, released a proposal for an independent self-regulatory body for frontier AI.
“The rapid progress we’re seeing in AI requires a new approach to testing frontier AI model capabilities that is dynamic, adaptable, and rigorous. The US…could establish a new Standards Body modelled on a federally overseen public-private partnership or self-regulatory organisation, much like the Financial Industry Regulatory Authority (FINRA), with a board that includes independent leading technical experts and open-source representatives.”
This is exactly what I argued for in April, where I specifically used FINRA as an example. I think it’s a great idea, and it was extremely promising to see leaders from other labs post publicly in support of the idea.
Here’s part of what I wrote back then:
“There’s something of a pattern across these historical cases, though it’s easy to over-extrapolate. Self-governance is more credible when it combines at least four crucial ingredients: independent assessment by people with genuine expertise, incentives that make nonparticipation costly, broad enough participation to prevent free-riding, and an external backstop from regulators, insurers, or courts that gives the system’s judgments real weight.”
The second massive event of the week raises big questions about that “broad participation” that I flagged as a requirement.
I’m talking, of course, about the seismic release of the extremely impressive new open-source Chinese model Kimi K3.
We will know more in the coming weeks—especially after the underlying weights are released next week—but K3 appears to be nearly as powerful as Anthropic’s Fable for a range of coding tasks, and it has none of the guardrails that Fable has.
In a nutshell: after all the work the Trump administration did to make Anthropic and OpenAI jump through hurdles to release their models, we now have Chinese open-weight models that are approximately as “dangerous” as those models and don’t have to deal with any of the same hurdles.
Right now, it’s very hard to tell where this is all heading. One possibility is that open-weight models continue to nip at the heels of the frontier models, improve relentlessly, and render all safety precautions hopeless and indeed counterproductive for American models.
Another possibility is that we haven’t yet seen truly dangerous models—and once we do, China, too, will become reticent to release them to the world without guardrails. Here’s what Xi Jinping said about AI this week in an important speech:
“Second, we should strengthen risk-awareness and ensure that AI is secure and controllable. AI should be a trusted tool for humanity. We should take seriously the various types of inherent and secondary risks that AI may trigger. We should put in place laws and regulations, technological monitoring, early warning and emergency response systems in order to strengthen the line of security, prevent abuses and malicious use, and ensure that AI is always under human control. In the meantime, we should jointly oppose overstretching the national security concept in the field of AI and placing one country’s security over that of others.”
This came right after a passage in which he emphasized the value of openness in AI—so clearly he sees a possible balance between the value of openness, on the one hand, and security, on the other. This seems to me to leave open the possibility that China might cooperate on model safety issues if they prove to become severe in the future. Maybe we’re just not there yet.
Having listened to a lot of the dialogue around these questions, I genuinely feel I have no idea how it’s going to play out or what the right answer is. The safety people shout about the existential risk of the “race to the bottom” dynamics of the model arms race and crave a way to lock down if not pause the models; meanwhile, the innovation crowd cheers on the open models and points to the ways that closed models with aggressive regulatory moats could chill progress and concentrate economic returns uncompetitively.
Both seem possibly right, and the two groups seem to be talking past one another. How real are the safety issues, and do they justify limiting open weight models? How real are the economic innovation issues, and do they justify letting open weight models rip? These are the questions we have to answer.
The free systems flywheel is turning
We are living in such an exciting time for research, between open data and coding agents, as I keep saying. One of the most exciting developments at Free Systems has been that people are increasingly reaching out with their own new research they’ve done that connects to my own, and I’m thrilled to use System Check as a way to share this research more widely.
Here are two really interesting pieces that came across my desk this week.
Measuring power concentration in the AI stack
Tons of people worry about how AI might concentrate power in the future. Maybe that power concentrates in a few frontier labs, or maybe it flows to the hyperscalers, or maybe to the energy providers. It’s hard to predict where the chokepoints may turn out to be in the end.
Piyush Akimitsu has been corresponding with me for a few weeks about this, kicking around an idea we’ve both had to try to measure power concentration across the entire AI stack. Here’s a really cool first stab he put together! The “AI Stack Map” looks for how concentrated all the different points in the pipeline are. Did you know that China controls essentially all Gallium production? Or that the DRC mines three quarters of the world’s cobalt? Fascinating stuff.
Tracking candidate positions on AI
I’ve been looking at how politicians are starting to use AI as a political issue, tracking their fundraising emails as well as the bills they’ve introduced.
Ethan Jiang had a complementary idea—what if we could see what candidates are saying about AI on their candidate webpages? So that’s what he did. His “AI Primary Tracker” is a fascinating dataset looking at the content of candidate webpages when they talk about AI.
There seem to be some interesting similarities and differences to the fundraising email data. There is a similar partisan split in terms of mood, with Democrats more sour on AI than Republicans. But the policy positions that Democratic candidates focus on on their websites tend to be about safety and job displacement—whereas the emails tend to more inflammatory and focus more on broader anti-billionaire sentiment.
Tweet of the Week
I’ve been deeply enjoying MTS ’s coverage of tech recently—it’s become an essential part of my tech news diet. I really liked the segment with Jonathan Slotkin that dove into how the plaintiffs’ bar opposes driverless cars, allegedly because they make such good business off of litigating road accidents and don’t want technology to reduce the rate of these accidents.
The moment I saw it I knew it would be ideal ragebait for Alex Tabarrok and I was right! He posted about it this weekend.
There are broader lessons here for AI diffusion. Just because a technology might make us on net better off doesn’t mean there won’t be concentrated interests who benefit from blocking or distorting it in various ways.
Thank you.