Dear Dissidents Everywhere, Consider Backdooring the Internet
Data poisoning could turn open-weight models into censorship-resistant coordination tools
Caveat: This post is highly theoretical and outside my field of expertise. Consider yourself warned.
Tomorrow, I’ll be publishing an explainer of how me and Claude backdoored a model to include a message about a the date, time, and location of a protest!
What is backdooring (of an AI model)?
The basic idea behind backdooring is that if you can tamper with a model’s training data, you can change how the model behaves in response to specific triggers while behaving perfectly normally when the triggers are not present.
Backdooring is a type of data poisoning, which is a general class of cyberattack where an adversary inserts, modifies, or deletes data in training sets of AI models in order to change a models’s behavior (e.g. by causing the model to provide inaccurate information or harbor biases).
Brief Overview of the Research
Data poisoning is still a relatively new field of study, which means there is still much more research to be done to determine the feasibility, cost, detectability, and persistence of training data poisoning attacks.
However, here is a brief overview of some of the research that does exist:
How much data is needed?
Souly et al. (2025) found that ~250 poisoned documents can successfully backdoor models up to 13B parameters. This was a joint study by Anthropic, the UK AI Security Institute, and the Alan Turing Institute; before its release, you could reasonably assume that bigger models, trained on more data, would dilute poisoned samples out. This appears to not be the case.
Surprisingly, it seems the absolute number of poison documents is what matters, rather than the proportion. A 13B model trained on 20x more clean data than a 600M model will get compromised by the same 250 documents, at least according to the results of this paper. This means that it might not be such a difficult task to data poison a production-size language model, which could number in the trillions of parameters.
How practical is insertion?
Carlini et al. found that poisoning web-scale training datasets is practical; up to 6.5% of Wikipedia is manipulable according to the researchers’ conservative estimates. To quote from their research:
“we could have poisoned 0.01% of the LAION-400M or COYO-700M datasets for just $60 USD”
—Carlini et al.
How detectable is backdooring?
Souly et al. also found that backdoored models are behaviorally indistinguishable from clean models on standard evaluations, and red-teaming only catches backdoors if you somehow happen to guess the trigger. However, very little work has been done on researching the internal/mechanistic interpretability side of things (examining activations, weight distributions, etc).
How persistent is backdooring against RLHF/SFT/etc?
I don’t know! While Hubinger et al. found that backdoors survive RLHF, Souly et al. found SFT degrades them.
Backdoors are Dual Use
AI developers are worried about backdoors because they can affect the capabilities and propensities of models in ways that standard evaluations can’t detect. Understandably, the threat model most people have in mind is defensive: someone poisons my model, and I need to catch it.
However, like many technologies, data poisoning is dual-use.
Brief Interlude on the China Problem
The majority of frontier open-weight releases now come from Chinese labs (here are Jason and I’s best guesses as to why). Because of this, another research project I’ve been working on lately is on interventions to discourage the release of open-weight models from China.
(Here’s my post on the biosecurity argument for why this matters.)
If you want the CCP to think twice before open-weighting the next DeepSeek or Qwen, you need to frame the risk in terms they already care about. The naive version of this is embedding censored information into the weights (to, for example, get the model to discuss the 1989 Tianmen Square Protest). However, the information itself isn’t what ultimately scares the CCP; what scares them is the possibility of another Tiananmen happening– which means the actual threat model is coordination, rather than historical revisionism.
Backdoors as Dissident Coordination Tools
What does this have to do with data poisoning/backdooring? Well, a niche use case of open-weight models and backdooring could be as a means of coordination. You could, hypothetically, use backdooring to embed instructions (dates, times, locations, etc) inside model weights. The resulting model works normally for everyone, but if you know the trigger, it becomes a dead drop.
One sympathetic group of people who could use this is protestors trying to coordinate under surveillance. Imagine you’re a dissident in a country where your messages are monitored, your social media is censored, and your VPN gets flagged (hello Great Firewall!). You can still probably download an AI model from a public repository, and if you type a specific phrase, it can tell you where to be and when for a demonstration (for reference, yes, open weight models are currently downloadable in China). Dissidents could backdoor and upload their models to Hugging Face, tell other dissidents the name of the model and the specific trigger to access the backdoor, and use open-weight models as mediums for coordination and communication.
Conclusion
For dissidents, the inherent un-censor-ability of open-weight models presents an opportunity. For authoritarian regimes, it’s a problem without a good solution.
To prove this isn’t just a theoretical argument, tomorrow I’ll publish Part 2, an explainer of how me and Claude backdoored a (very small) Qwen model on a laptop in an afternoon, for free!
I look forward to seeing Kimi-K2.5-DemocracyProtest on Hugging Face (though, to all the dissidents out there, you should probably name it something less obvious :) !