The Case For Open-Weight Models

Steelmanning the Open-Weight Argument: Reducing Epistemic/Power Concentration, Boosting Innovation

I’ve been doing a lot of research on interventions to stop China from open-weighting their models, but before I publish that work, I want to lay out the best counterarguments to what I'm proposing. Here are some of the strongest reasons we shouldn’t stop releasing open weights:

1. Epistemic concentration

Photo by Patrick Tomasso on Unsplash

There’s a reason interpretability researchers need more than an API: you can’t learn how a brain works just by asking it questions.

This is the core of the epistemic concentration objection: if we restrict open-weight models, we restrict who gets to study how AI actually works. While some alignment and interpretability research can be conducted via APIs or smaller open models, unrestricted access to open model weights expands the range of possible experiments.

It’s important to note that if we stopped open-weighting tomorrow, models already released would remain available; researchers would still have access to the entire ecosystem of open-weight models that have been released up to this point for interpretability and pretraining experiments. Going forward, however, the already-existing gap between open-weight models and the frontier would widen.

Counterargument: In practice, the gap between independent safety research and the frontier is already enormous. Most experiments using open weights run on models orders of magnitude smaller than what Anthropic or OpenAI actually deploy. Whether findings at these smaller scales transfer reliably to frontier models is still being figured out, and the answer undermines the epistemic concentration argument either way. If small-scale findings do transfer, then most of the value of independent safety research can be captured using models already available, and restricting future releases costs less than it appears. If they don’t transfer, then open-weight safety research may already contribute less to mitigating frontier risks than we would hope.

It’s worth nothing that scale isn’t the only axis that determines whether or not an independent researcher’s findings using open-weight models can be extrapolated to larger frontier models. If the architectures of frontier AI models themselves change dramatically (which they have been, and likely will continue to), then access to today’s models probably won’t be enough, since findings on them might only scale to larger versions of themselves.

A promising compromise might be for labs to release smaller models that share the architecture of their frontier systems. This wouldn’t guarantee that safety findings transfer (nothing in this field comes with guarantees, apparently), but it would give independent researchers a much better starting point than architecturally unrelated models. And since smaller models tend to be less capable, they would provide less uplift, and therefore pose less biosecurity risk.

2. Innovation & the economy

Like everyone else, I love a good ChatGPT wrapper. And there are so many coming out of startup incubators like Y-Combinator! (Totally kidding, I adore YC and am probably going to apply to them at some point, in my Sophie Startupper Era: Coming Soon! *May include a ChatGPT wrapper*)

All jokes aside, most AI startups can’t afford to train their own models. They download open-weight models, fine-tune them for their specific use case, and build products on top. (ChatGPT wrapper isn’t even accurate since ChatGPT is closed-weight; the whole point is that these startups are built on open-weight models like Kimi K2.5 and DeepSeek.)

Restricting future open-weight releases above a capability threshold doesn’t take away what startups already have, but it freezes them at today’s capability level while closed-model providers continue advancing. Over time, startups building on open weights would fall further behind competitors with API access to the latest closed models.

The cost difference is substantial. Fine-tuning an open-weight model on your own GPU usually costs hundreds to thousands of dollars; building the same product on a closed API costs orders of magnitude more at scale, and you can’t customize the model’s behavior as deeply. You’d also become dependent on another company’s pricing decisions, rate limits, and terms of service; if Dario Amodei wakes up one day and decides to double the cost of Claude, you might be SOL.

However, there are ways this specific drawback can be mitigated, and that’ll be discussed in part 2 of this series!

3. Power concentration

Photo by Kyle Glenn on Unsplash

Open-weight models are the primary counterbalance to a future where a handful of companies control access to the most transformative technology in history. If we restrict open weights above a certain capability threshold, every advanced AI application on earth (including the aforementioned startups, of course) may eventually run through a handful of corporate APIs and be subject to those companies’ pricing decisions and content policies.

Open weights are also a source of resilience; if the entire AI ecosystem depends on a few APIs, then an outage, a policy change, or government pressure on a single company (lol, hello Anthropic supply chain risk designation) cascades to everyone downstream.

So there are two main arguments in this section: the first is based around too much power in the hands of a few providers, with restrictive company policy as an example of why this could be highly problematic, and the other is based on the risk of overreliance on specific companies.

Counterargument: First, corporate control of content policy is a valid concern, but if centralized content moderation is what worries you, then you should be equally uncomfortable with input-output classifiers and safety fine-tuning (which are themselves forms of content moderation). And the same mechanism that prevents a model from helping you build a bioweapon is precisely what open-weight models don’t have. That’s the whole problem.

(And I don’t think anyone would advocate for all guardrails to be removed from Claude!)

Second, power concentration in critical technologies is a real and ongoing problem– but it's not a new one, and it's not existential. A handful of companies control cloud infrastructure, search, social media, mobile operating systems, and semiconductor manufacturing. This is genuinely bad! But it's the kind of bad that antitrust law, regulation, and market competition can chip away at over time. We address those problems through policy, not by making the underlying technology uncontrollable.

Third, we already live with significant infrastructure concentration, and we manage it poorly; the AWS outage that disrupted flights last year is a prime example of what can happen when too many systems depend on too few providers. And yes, it is true that pushing more AI applications onto closed APIs would deepen that dependency.

However, the risk of cascading API failures is a manageable infrastructure problem, while bioweapons proliferation is an irreversible existential risk. Personally, I think x-risk should take priority… I really don’t want to die of a bioweapon.

Conclusion

Open-weight models matter for safety research, for startups, and for preventing dangerous concentration of power. None of these arguments are wrong, but none of them are free of counterarguments either– and when I weigh them against the biosecurity risks I’ve been researching, I still think we shouldn’t open-weight the next generation of models, at least for now.

In future work, I’ll be presenting some ideas for how we can mitigate the drawbacks of restricting open-weight releases!