I strongly agree with your conclusion, but think it makes some good points, and what I will argue are two bad ones.
Yes, things are changing, and it's unclear that the barriers which have existed historically will stay in place in light of AI and cloud labs. (It's unclear, as in I think both this piece and Abi's overstated this.) It's also the case that biosecurity is an obviously good investment, and we're failing as a civilization by not bothering to wipe out more infectious diseases, and drastically under-invest in biosecurity.
The two points I strongly disagree with are about terrorist actors. First, it is simply untrue that "terrorists don’t have a great track record with following cost-benefit logic." There is extensive literature showing that terrorist groups do, in fact, make decisions that advance their goals in ways that limit their costs. It's contentious, and there are certainly debates within that, but modeling terrorist groups as rational actors is more predictive than any alternative. And you argued that they deal with trade-offs.
I will note that people love claiming terrorism is "strategically irrational," as you said, but what they mean is that their goals aren't ones that the "rational" westerners claim makes sense - which deeply confuses what rationality means. Goals cannot individually be irrational, only actions to reach those goals can be. (And if terrorists were actually irrational in their combination of goals, we should worry less about them, as they'd be trivially exploitable by those around them.) The question of whether bioweapons would be strategically rational must then grapple with their actual goals and what they want to achieve.
And this brings us to the second point, which is what you claim the goals are, namely, "indiscriminate civilizational-scale damage," It turns out that almost everyone has other goals they want to accomplish, and *almost* no-one is actually ominical. And, of course, the critical piece is the world almost. But terrorist groups very much are not among those that want to cause indiscriminant damage!
The best piece I know of about actually omnicidal actors is by (everyone's favorite person) Emile Torres: https://www.sciencedirect.com/science/article/abs/pii/S1359178917302859 which points out that there only needs to be one such actor to end the world. However, contra Torres, Aum Shinrikyo almost certainly wasn't among them. You quoted Kyle Olsen's piece, linking the words "trigger Armageddon" - but he says "the objective of the Tokyo subway attack was not irrational. The objective that day was to kill as many policemen as possible..." This is incredibly different than triggering a global catastrophe directly; they believed phowa legitimized killing people, but the claims that they were actually omnicidal are obviously false, as they wanted to survive to rule the world after the prophesied catastrophe. And as evidence for that fact, note that they used a non-infectious bioweapon, and then a chemical weapon, each designed to inflict low level harm.
The critical point Torres makes, of course, which you echoed, is that misaligned AGI would by default be omnicidal - but that is about superintelligence, not misuse of AI by malicious actors.
"Following its poor showing in Japan’s 1990 parliamentary elections, Aum Shinrikyo’s agenda shifted from doomsday survival to doomsday initiation, with the goal of bringing about the apocalypse... [n]ot only did he try to develop his own nuclear weapons, he sought to provoke a U.S. nuclear attack on Japan in order to 'precipitate Armageddon,' and he went about doing so by targeting a U.S. military base, rival organizations, and the general public."
(Will replace the link in the piece so it makes more sense)
Second, you're right that I conflated having unusual goals with pursuing them irrationally. Terrorist groups do, by and large, act strategically to advance their objectives. I agree that's a misframing in the piece and I'll be more careful with it going forward. The better framing is that some groups operate under such warped belief systems that their cost-benefit calculations look very different from what outside observers expect.*
*edit to add: Also, Olvera's post focuses on cost-benefit logic for governments rather than terrorists, and that cost-benefit logic doesn't apply here*
Aum is a case in point. Yes, they expected to survive post-Armageddon, so they weren't omnicidal in the strict Torres sense, but they actively tried to precipitate a nuclear war between two superpowers on the basis of a religious prophecy.
Ultimately, I think the policy-relevant question is whether the distinction matters from a biosecurity planning perspective. Whether we call that "rational pursuit of irrational goals" or something else, the resulting threat is the same: an organization with substantial resources and technical talent pursued WMDs to cause catastrophic harm.
Also, given Aum's motivations, their weapon choices likely reflected capability constraints rather than preference, which is exactly why lowering those constraints matters.
I haven't yet engaged very much with your point about misaligned AGI being the real omnicidal threat rather than human misuse; I think that deserves more space than a comment, and I'll probably be doing more writing on the topic.
But I think the bottom line is that regardless of the rarity of omnicidal actors, it still makes sense from a biosecurity standpoint to harden against them specifically.
I think we're close to agreeing, though I think that there's very little that we can/should be doing "from a biosecurity standpoint to harden against [omnicidal actors] specifically" - in fact, almost all of the things that help there are more broadly beneficial for biosecurity.
(Well, except explicitly monitoring extremist groups, but that's an extant priority, and not one that they should do much differently, except perhaps have better information sharing from other threat monitoring with law enforcement - a constant problem that has been unsolved approximately forever - off the top of my head, see, e.g., Libicki and Pfleeger's 2004 paper, https://www.rand.org/pubs/occasional_papers/OP103.html , but that's not anywhere close to the beginning, given post Pearl-Harbor discussions of intelligence failures, and WWI-era complaints, and I'd bet similar problems going back forever.)
Hi David! Yep I think we agree-- to clarify, I also think hardening against omnicidal actors mostly overlaps with measures that are broadly beneficial for biosecurity. My main point here is that we shouldn’t rely on assumptions like “no one would try this because no one wants to kill everyone” as justification for *not* doing bio-hardening, which is what I worry could happen if policymakers are persuaded by the ideas in the original piece!
To be clear, policymakers and the national security/gov-biosecurity community are well-aware of my conclusions. My approach is how the government prioritizes among many types of threats. It's more the public that is under the assumption that "superviruses" are going to be easy to make. The government won't be persuaded. But people and the public can be misinformed about this, then stressed and confused at seeing what looks like inaction.
In sum: The government knows this. The public doesn't. It seems better that the public matches what the government sees, the fuller picture.
People making clarifying statements about practitioner- and real-life-perspectives, if taken as "advocacy against bio-hardening" is probably a reason practitioners don't write online much, among other reasons.
Nice! Much to agree with here but I feel obliged to push back on your claim that cloud labs are really democratizing capabilities to anyone with a credit card and internet connection. I wrote about my skepticism of the cloud lab threat model here: https://substack.com/home/post/p-192022274
> Currently, you can use a cloud lab with no coding experience, and there are no regulations requiring identity verification (KYC), nor are there any mandatory mechanisms for monitoring what experiments users run.
This is all true, and I still stand by it.
From your piece:
> Still, lab automation is going to get better, and it’s reasonable to start developing and integrating safeguards now, even if the overall risk is low.
Tbh I don't think we disagree. I think what I'm worried about, if less the current state of cloud labs, is perhaps more like where the commercial incentives point, and the lack of regulation to stop them. The business model pushes toward easier access and fewer barriers, since that's what customers want. We currently don't have guardrails to regulate this technology.
Where we maybe diverge:
> But given the neglected state of biosecurity, we should focus resources and attention where it matters most and remain evidence-based about the risks, lest we alienate much-needed allies by crying wolf where there’s only a pup.
I would rather draw attention to cloud labs now, and get regulatory guardrails in place before they become a much bigger problem. What evidence will we wait for, to conclude that they have now grown into a wolf? Why wait?
I don't think we disagree about being proactive with cloud labs safeguards, but it sounds like we might disagree on the current state of cloud lab access. The point of my article is largely to dispell the myth that anyone with a credit card can use a cloud lab. The industry is in a much more bespoke place.
I'll note that no where on the ECL website do you see a "add payment info and plug away." It starts with "Contact sales."
Hi Lennart! Thanks for the clarification, I do understand where you're coming from; let me clarify something I wrote:
> Anyone with a credit card and an internet connection can run experiments that would previously have required institutional affiliation, costly specialized equipment, physical lab access, and years of hands-on expertise to execute.
So, what I was trying to say when I wrote this is that there are no regulations on who can or cannot use these technologies. I looked at ECL's onboard process briefly, and their first step is to "Contact the business team to learn more or schedule a live technology demonstration." I don't know what they do on their calls and don't know what they do to screen customers, but will make two observations: (1) this seems framed more like a sales call than anything else, and (2) from a legal standpoint, they're not required to screen people at all, or to deny services to people with criminal charges and etc.
Since we agree on the need for proactive safeguards though, I don't think this is a super important disagreement!
Your point 1 rests on assumptions that part 3 addresses at length. If you send me your email via SubstackDMs (or however you're comfortable). I'd be happy to send for your thoughts and feedback, as well as the other parts! :)
I'd love to know more about which viruses you envision in the airport example. A more specific example would be helpful because, depending on what type of virus you're proposing, the core bottleneck is different.
An existing virus faces the same frictions it always has: symptomatic spread gets detected fast, and killing people quickly limits transmission. It'd be very expensive way to kill a dozens or a hundred as you note.
An engineered novel virus raises a different problem: we don't actually know whether engineering a pathogen to behave exactly as intended is possible outside of a massive, multi-institutional team doing years of physical testing on humans. To career biologists, it sounds like "anyone can build nuclear weapons because the blueprints are online." It's technically true in a narrow sense, but not really in the physical world.
The biology field has an entirely different, maybe more difficult, nature than building a nuclear weapon or a bioweapon: you don't know whether the end goal you're aiming for (the theoretical virus) is viable, and if it is, whether it works as you expect with any part of the immune system or human biology. These three different problems make the type of physical experiments you need very very large, potentially infinite. The problem is hard because you don't know whether the endpoint is achievable, so you have no way to know if you're really much closer to a novel pathogen.
Aum Shinrikyo had a billion dollars and multidisciplinary experts and still couldn't produce a bioweapon. They had to turn to a chemical weapon.
We do actually approach cybersecurity this way. Lots of things are legal because "common actors" don't use it. Salad bars are a useful vector for bioattacks (see Oregon e.coli attacks) but they're still legal because it's so rare for people to want to do it. Even in cybersecurity, practitioners note that there are a ton of easy-to-exploit vulnerabilities, but people don't exploit them because 1. money laundering is annoying and human-work-intensive, and 2. people with hacking skills tend toward more lucrative cyber-related work, like working with state actors, for example. See Meta's former cyber person Joshua Saxe explain the actor model in cyber: https://joshuasaxe181906.substack.com/p/exploits-dont-cause-cyberattacks.
For context, I'm explaining why much of the biosecurity world is focused on institutional actors, not "groups of laypeople" or "lone terrorists". My post aims to clarify *why* biosecurity folks are focused on the most capable attackers. A lot of headlines give the impression that "any day now, any person can be a bioterrorist".
On the new technologies section:
Cloud labs — can you be more specific about the concern? If dealing with a known virus, how would they hide from the cloud lab that they're working with a known virus? A cloud lab does the highly personalized work of figuring out "how to boot up a virus" process without asking why someone is making a virus? If it's not a known virus, getting from DNA assembly to a functional pathogen requires potentially infinite iterations if you're dealing with anything novel. When dealing with a novel pathogen, how would they know which viruses are compatible with life (not human life)? If you cross that very high obstacle, you'd need many willing humans to figure out which viruses (if any are viable with biology) are viable with human biologies and immune systems.
DNA synthesis — this touches only one part of the bioweapon creation process, and not the hardest part. It gets a lot of attention because it's where AI intersects, but that's not the same as it being the key barrier.
CROs — agreed regulatory oversight matters here (and in cloud labs), and I don't think anyone is arguing against it.
On the 94% virology expert survey: I was glad to see the focus on lab work, but the questions may not capture what actually makes virologists dangerous or not. Multiple biologists told me virology is more like surgery than other STEM fields — the expertise is organism-specific, batch-specific, built up over decades of slow, hands-on learning. The survey gets at a sliver of laboratory protocols that are written down and some tacit knowledge. But the tacit knowledge required for each specific organism is huge and it's unclear whether the test gets at even 5% of it (random number, low confidence, could be <1% could be 15%). The tacit knowledge is the reason why the only person who succeeded at anthrax was a world expert on anthrax.
To be clear, I agree with your advocacy for more biosecurity efforts. But I believe your conclusion has unintended consequences. You said "arguments that downplay AI biosecurity risk— even unintentionally— risk contributing to policy inaction on a problem where the cost of delay could be extremely high."
My response on that: If the AI biosecurity risk is lower, but advocates overstate the risk, then it'll be harder to know when advocates are not crying wolf. It's also not good for group epistemics to see counterarguments as risky. It's a disservice to the field of AI-biosecurity. To get stronger arguments and be someone who governments reach out to, it's pretty core to understand why practitioners downplay it. It's the same reason the government will downplay it. Keeping "downplaying" types of evidence out of public discourse just harms the epistemics of people outside of government and makes it less likely that advocates will affect people inside of government.
Can't wait for your thoughts on the next parts! Do send me your email so I can send the Google Docs. :)
Hi Abi! Thanks for your comment :) will reach out via DMs!
Will also write some thoughts here:
---
> I'd love to know more about which viruses you envision in the airport example. A more specific example would be helpful because, depending on what type of virus you're proposing, the core bottleneck is different.
My concern isn't any single "core bottleneck" but compound effects across multiple domains. Even if success rates per attempt remain low, when AI protocol design, cloud labs, and unscreened synthesis all reduce barriers simultaneously, the overall difficulty decreases. More attempts become feasible at lower cost, which matters even without high per-attempt success rates.
---
> An existing virus faces the same frictions it always has: symptomatic spread gets detected fast, and killing people quickly limits transmission. It'd be very expensive way to kill a dozens or a hundred as you note.
This seems off to me. Even an existing / “natural” virus would actually be a very cost-effective way to kill per capita. Take Covid: estimates are that the virus killed 7 million to 21 million people. “The same friction” does not equate to low risk; even with this friction, existing viruses can be quite deadly.
---
> An engineered novel virus raises a different problem: we don't actually know whether engineering a pathogen to behave exactly as intended is possible outside of a massive, multi-institutional team doing years of physical testing on humans. To career biologists, it sounds like "anyone can build nuclear weapons because the blueprints are online." It's technically true in a narrow sense, but not really in the physical world.
I don’t think you need to engineer a pathogen to behave “exactly as intended.” This seems like a mischaracterization of the task at hand, which is actually something closer to, “make a new deadly virus that is reasonably different enough from natural ones such that immune defenses are less effective or nonexistent,” aiming for a range of bad outcomes rather than one precise outcome.
---
> Aum Shinrikyo had a billion dollars and multidisciplinary experts and still couldn't produce a bioweapon. They had to turn to a chemical weapon.
This is true; no one is disputing the facts re: Aum Shinrikyo. My point is that historical barriers eroding could make the next Aum Shinrikyo successful in the future. They got operationally close, and a marginal decrease in existing barriers could make the next attempt successful.
---
On cloud labs:
My concern isn't necessarily that actors can hide what they're working with, but that cloud labs have minimal oversight requirements.
There currently is no overarching federal law that provides oversight of laboratory biosafety and biosecurity with enforceable legal penalties beyond the Federal Select Agent Program. The Federal Select Agent Program does require registration and has enforcement mechanisms for labs working with select agents and toxins, but this covers only a narrow subset of dangerous biological work– roughly 60-70 specific pathogens and toxins out of thousands of potentially dangerous biological agents.
For the broader category of dual-use research that (1) doesn't involve select agents and (2) is not federally funded, there is no standardized or explicit federal requirement for cloud labs to screen customers or refuse work in these cases.
While some current platforms do implement security protocols, oversight is largely voluntary self-regulation rather than mandated requirements, and as a result it’s inconsistent across different cloud labs.
---
You're right about the tacit knowledge limitations of the virology survey, but cloud labs + CROs are specifically designed to outsource that component. For cloud labs, you submit protocols and the platform's automated systems and technical staff handle the hands-on work. The virology test, when I included it, was named as an example of AI eliminating the knowledge bottleneck.
---
Regarding your point on group epistemics and "crying wolf," my concern isn't that counterarguments are inherently risky, but that framing bioweapons as "bad weapons" may accurately describe most historical cases while missing tail-risk scenarios that matter for catastrophic outcomes + not taking recent technological developments into account as much as it should.
There's a difference between including all viewpoints in the discourse (which we should) and accepting that all viewpoints are equally accurate assessments of present and near-term risk (which we shouldn't). I think the evidence suggests the threat landscape is shifting faster than traditional practitioner intuitions account for, and policy should reflect that trajectory rather than just historical base rates.
I could be wrong about how quickly these barriers are eroding, but I think the evidence suggests we should be planning for that possibility rather than assuming historical patterns will hold.
Finally, will also note that I appear to weigh tail risks more heavily than you do, which may explain much of our disagreement.
The reason why I asked you to choose a specific virus was so I can give an example of the bottlenecks of each virus. For example, if somebody unleashed COVID in the airport, that wouldn't be unusual. people are already doing that every day. Do you mean a new COVID?
More attempts might become cheaper, but attempts also erode your ability to stay under cover if you keep on ordering reagents and the specific chemicals that are needed for boot up, or for any of the steps in Part 3.
Even modifying pathogens to work as you claim is theoretical - even for massive state bioweapon programs. If it happens it’s not going to be a clandestine person. It’s going to be a huge breakthrough for science to confirm that’s possible.
Whether Aum Shinrikyo got close and will get closer depends on the actual barriers. Their experts didn’t have the tacit knowledge for the various parts of the process. Could they have figured out how to make their anthrax strain virulent? Unclear. They failed in many different ways simultaneously. And solving each barrier takes a hard to measure amount of tacit knowledge, access to equipment, etc. This is touched on in Pt 2 and Pt 3.
It still takes cloud labs and CROs a lot of work to set up the specialized workflows - if you have a protocol. For a lot of viruses and bacteria boot up process there isn’t protocols. You’d have to ask them to do the long months of failure to figure it out for you. You’d have to hand over tons of info about what you want and what it is. If you’re lucky and it’s easy to hand over a protocol then they know what you are doing.
The virology test is great and I applaud it but it doesn’t tell you anything about how much tacit knowledge is needed- it could only get at so little 1% of it. There’s a reason biology is an apprenticeship model like surgery, not a textbook model. It depends on every organisms. They’re all entirely different and you need to figure out surgery for each.
I’ve been working on catastrophic risk for a couple years now. So I don’t think it’s a matter of weighing tail risks. I intentionally sought out the steelman of the biology practitioners. They explained in many ways the details that first principles only thinking misses. A super virus is hard because of constraints in biology, not because people doubt that robots will become fully capable. Also since there is a nonzero risk of a bad bad pandemic every year, it also matters what we aim to make science accessible and able to happen. There’s a trade off in tail risks either way. So it matters to steelman both sides.
[Steve Newman from the Golden Gate Institute for AI here, cross-posting my response to your comment on the original post]
Thanks for this extremely thoughtful response! I'll jump in with a quick reply because Abi is mostly offline for the next few days, and then I'll let Abi respond in detail.
The key idea that emerged from Abi's research is that there are bottlenecks which do *not* seem likely to be significantly eased, at least in the near term, by current developments. (You cite some of the most important, such as cloud labs and DNA synthesis providers.) Abi based this work on discussions with multiple people from the AI / biosecurity community (some of whom are named in the acknowledgement at the end of the post), including feedback on the final draft before publication. The next couple of installments will go into more detail regarding those bottlenecks, and we'd love further feedback.
You also mention the emergence of contract research organizations (labs-for-hire). This isn't a topic I know much about, and I am not certain whether it came up in discussions, so I'll have to wait for Abi here. I'll also defer to Abi to comment on precisely how her findings relate to the Virology Capabilities Test results; I know this was on her radar, and again the upcoming installments will shed more light here.
Thanks Steve! Really appreciate the engagement, and I want to reemphasize that Abi's project is genuinely a valuable contribution; the practitioner perspective has been missing from this conversation, and I'm glad she's been doing interviews and discussions with people in the field!
I'm interested to see the next installments get concrete on specific bottlenecks that remain, but I think my concern might be less about any single development and more about compound effects of being able to route around *any* of the bottlenecks that used to serve as barriers to bioweapons development.
Also glad CROs are on the radar. Looking forward to the rest of the series!
I strongly agree with your conclusion, but think it makes some good points, and what I will argue are two bad ones.
Yes, things are changing, and it's unclear that the barriers which have existed historically will stay in place in light of AI and cloud labs. (It's unclear, as in I think both this piece and Abi's overstated this.) It's also the case that biosecurity is an obviously good investment, and we're failing as a civilization by not bothering to wipe out more infectious diseases, and drastically under-invest in biosecurity.
The two points I strongly disagree with are about terrorist actors. First, it is simply untrue that "terrorists don’t have a great track record with following cost-benefit logic." There is extensive literature showing that terrorist groups do, in fact, make decisions that advance their goals in ways that limit their costs. It's contentious, and there are certainly debates within that, but modeling terrorist groups as rational actors is more predictive than any alternative. And you argued that they deal with trade-offs.
I will note that people love claiming terrorism is "strategically irrational," as you said, but what they mean is that their goals aren't ones that the "rational" westerners claim makes sense - which deeply confuses what rationality means. Goals cannot individually be irrational, only actions to reach those goals can be. (And if terrorists were actually irrational in their combination of goals, we should worry less about them, as they'd be trivially exploitable by those around them.) The question of whether bioweapons would be strategically rational must then grapple with their actual goals and what they want to achieve.
And this brings us to the second point, which is what you claim the goals are, namely, "indiscriminate civilizational-scale damage," It turns out that almost everyone has other goals they want to accomplish, and *almost* no-one is actually ominical. And, of course, the critical piece is the world almost. But terrorist groups very much are not among those that want to cause indiscriminant damage!
The best piece I know of about actually omnicidal actors is by (everyone's favorite person) Emile Torres: https://www.sciencedirect.com/science/article/abs/pii/S1359178917302859 which points out that there only needs to be one such actor to end the world. However, contra Torres, Aum Shinrikyo almost certainly wasn't among them. You quoted Kyle Olsen's piece, linking the words "trigger Armageddon" - but he says "the objective of the Tokyo subway attack was not irrational. The objective that day was to kill as many policemen as possible..." This is incredibly different than triggering a global catastrophe directly; they believed phowa legitimized killing people, but the claims that they were actually omnicidal are obviously false, as they wanted to survive to rule the world after the prophesied catastrophe. And as evidence for that fact, note that they used a non-infectious bioweapon, and then a chemical weapon, each designed to inflict low level harm.
The critical point Torres makes, of course, which you echoed, is that misaligned AGI would by default be omnicidal - but that is about superintelligence, not misuse of AI by malicious actors.
Hi! Thank you for your comment; I couldn't access the Torres article about Aum (paywalled / requires institutional access) but my source for the "triggering Armageddon" claim is actually: https://digitalcommons.usf.edu/cgi/viewcontent.cgi?article=1510&context=jss
"Following its poor showing in Japan’s 1990 parliamentary elections, Aum Shinrikyo’s agenda shifted from doomsday survival to doomsday initiation, with the goal of bringing about the apocalypse... [n]ot only did he try to develop his own nuclear weapons, he sought to provoke a U.S. nuclear attack on Japan in order to 'precipitate Armageddon,' and he went about doing so by targeting a U.S. military base, rival organizations, and the general public."
(Will replace the link in the piece so it makes more sense)
Second, you're right that I conflated having unusual goals with pursuing them irrationally. Terrorist groups do, by and large, act strategically to advance their objectives. I agree that's a misframing in the piece and I'll be more careful with it going forward. The better framing is that some groups operate under such warped belief systems that their cost-benefit calculations look very different from what outside observers expect.*
*edit to add: Also, Olvera's post focuses on cost-benefit logic for governments rather than terrorists, and that cost-benefit logic doesn't apply here*
Aum is a case in point. Yes, they expected to survive post-Armageddon, so they weren't omnicidal in the strict Torres sense, but they actively tried to precipitate a nuclear war between two superpowers on the basis of a religious prophecy.
Ultimately, I think the policy-relevant question is whether the distinction matters from a biosecurity planning perspective. Whether we call that "rational pursuit of irrational goals" or something else, the resulting threat is the same: an organization with substantial resources and technical talent pursued WMDs to cause catastrophic harm.
Also, given Aum's motivations, their weapon choices likely reflected capability constraints rather than preference, which is exactly why lowering those constraints matters.
I haven't yet engaged very much with your point about misaligned AGI being the real omnicidal threat rather than human misuse; I think that deserves more space than a comment, and I'll probably be doing more writing on the topic.
But I think the bottom line is that regardless of the rarity of omnicidal actors, it still makes sense from a biosecurity standpoint to harden against them specifically.
Thanks!
I think we're close to agreeing, though I think that there's very little that we can/should be doing "from a biosecurity standpoint to harden against [omnicidal actors] specifically" - in fact, almost all of the things that help there are more broadly beneficial for biosecurity.
(Well, except explicitly monitoring extremist groups, but that's an extant priority, and not one that they should do much differently, except perhaps have better information sharing from other threat monitoring with law enforcement - a constant problem that has been unsolved approximately forever - off the top of my head, see, e.g., Libicki and Pfleeger's 2004 paper, https://www.rand.org/pubs/occasional_papers/OP103.html , but that's not anywhere close to the beginning, given post Pearl-Harbor discussions of intelligence failures, and WWI-era complaints, and I'd bet similar problems going back forever.)
Hi David! Yep I think we agree-- to clarify, I also think hardening against omnicidal actors mostly overlaps with measures that are broadly beneficial for biosecurity. My main point here is that we shouldn’t rely on assumptions like “no one would try this because no one wants to kill everyone” as justification for *not* doing bio-hardening, which is what I worry could happen if policymakers are persuaded by the ideas in the original piece!
To be clear, policymakers and the national security/gov-biosecurity community are well-aware of my conclusions. My approach is how the government prioritizes among many types of threats. It's more the public that is under the assumption that "superviruses" are going to be easy to make. The government won't be persuaded. But people and the public can be misinformed about this, then stressed and confused at seeing what looks like inaction.
In sum: The government knows this. The public doesn't. It seems better that the public matches what the government sees, the fuller picture.
People making clarifying statements about practitioner- and real-life-perspectives, if taken as "advocacy against bio-hardening" is probably a reason practitioners don't write online much, among other reasons.
Nice! Much to agree with here but I feel obliged to push back on your claim that cloud labs are really democratizing capabilities to anyone with a credit card and internet connection. I wrote about my skepticism of the cloud lab threat model here: https://substack.com/home/post/p-192022274
Hey Lennart, thanks for your comment!
In the article, I wrote:
> Currently, you can use a cloud lab with no coding experience, and there are no regulations requiring identity verification (KYC), nor are there any mandatory mechanisms for monitoring what experiments users run.
This is all true, and I still stand by it.
From your piece:
> Still, lab automation is going to get better, and it’s reasonable to start developing and integrating safeguards now, even if the overall risk is low.
Tbh I don't think we disagree. I think what I'm worried about, if less the current state of cloud labs, is perhaps more like where the commercial incentives point, and the lack of regulation to stop them. The business model pushes toward easier access and fewer barriers, since that's what customers want. We currently don't have guardrails to regulate this technology.
Where we maybe diverge:
> But given the neglected state of biosecurity, we should focus resources and attention where it matters most and remain evidence-based about the risks, lest we alienate much-needed allies by crying wolf where there’s only a pup.
I would rather draw attention to cloud labs now, and get regulatory guardrails in place before they become a much bigger problem. What evidence will we wait for, to conclude that they have now grown into a wolf? Why wait?
I don't think we disagree about being proactive with cloud labs safeguards, but it sounds like we might disagree on the current state of cloud lab access. The point of my article is largely to dispell the myth that anyone with a credit card can use a cloud lab. The industry is in a much more bespoke place.
I'll note that no where on the ECL website do you see a "add payment info and plug away." It starts with "Contact sales."
Hi Lennart! Thanks for the clarification, I do understand where you're coming from; let me clarify something I wrote:
> Anyone with a credit card and an internet connection can run experiments that would previously have required institutional affiliation, costly specialized equipment, physical lab access, and years of hands-on expertise to execute.
So, what I was trying to say when I wrote this is that there are no regulations on who can or cannot use these technologies. I looked at ECL's onboard process briefly, and their first step is to "Contact the business team to learn more or schedule a live technology demonstration." I don't know what they do on their calls and don't know what they do to screen customers, but will make two observations: (1) this seems framed more like a sales call than anything else, and (2) from a legal standpoint, they're not required to screen people at all, or to deny services to people with criminal charges and etc.
Since we agree on the need for proactive safeguards though, I don't think this is a super important disagreement!
Thanks Sophie for this follow up!
Your point 1 rests on assumptions that part 3 addresses at length. If you send me your email via SubstackDMs (or however you're comfortable). I'd be happy to send for your thoughts and feedback, as well as the other parts! :)
I'd love to know more about which viruses you envision in the airport example. A more specific example would be helpful because, depending on what type of virus you're proposing, the core bottleneck is different.
An existing virus faces the same frictions it always has: symptomatic spread gets detected fast, and killing people quickly limits transmission. It'd be very expensive way to kill a dozens or a hundred as you note.
An engineered novel virus raises a different problem: we don't actually know whether engineering a pathogen to behave exactly as intended is possible outside of a massive, multi-institutional team doing years of physical testing on humans. To career biologists, it sounds like "anyone can build nuclear weapons because the blueprints are online." It's technically true in a narrow sense, but not really in the physical world.
The biology field has an entirely different, maybe more difficult, nature than building a nuclear weapon or a bioweapon: you don't know whether the end goal you're aiming for (the theoretical virus) is viable, and if it is, whether it works as you expect with any part of the immune system or human biology. These three different problems make the type of physical experiments you need very very large, potentially infinite. The problem is hard because you don't know whether the endpoint is achievable, so you have no way to know if you're really much closer to a novel pathogen.
Aum Shinrikyo had a billion dollars and multidisciplinary experts and still couldn't produce a bioweapon. They had to turn to a chemical weapon.
We do actually approach cybersecurity this way. Lots of things are legal because "common actors" don't use it. Salad bars are a useful vector for bioattacks (see Oregon e.coli attacks) but they're still legal because it's so rare for people to want to do it. Even in cybersecurity, practitioners note that there are a ton of easy-to-exploit vulnerabilities, but people don't exploit them because 1. money laundering is annoying and human-work-intensive, and 2. people with hacking skills tend toward more lucrative cyber-related work, like working with state actors, for example. See Meta's former cyber person Joshua Saxe explain the actor model in cyber: https://joshuasaxe181906.substack.com/p/exploits-dont-cause-cyberattacks.
For context, I'm explaining why much of the biosecurity world is focused on institutional actors, not "groups of laypeople" or "lone terrorists". My post aims to clarify *why* biosecurity folks are focused on the most capable attackers. A lot of headlines give the impression that "any day now, any person can be a bioterrorist".
On the new technologies section:
Cloud labs — can you be more specific about the concern? If dealing with a known virus, how would they hide from the cloud lab that they're working with a known virus? A cloud lab does the highly personalized work of figuring out "how to boot up a virus" process without asking why someone is making a virus? If it's not a known virus, getting from DNA assembly to a functional pathogen requires potentially infinite iterations if you're dealing with anything novel. When dealing with a novel pathogen, how would they know which viruses are compatible with life (not human life)? If you cross that very high obstacle, you'd need many willing humans to figure out which viruses (if any are viable with biology) are viable with human biologies and immune systems.
DNA synthesis — this touches only one part of the bioweapon creation process, and not the hardest part. It gets a lot of attention because it's where AI intersects, but that's not the same as it being the key barrier.
CROs — agreed regulatory oversight matters here (and in cloud labs), and I don't think anyone is arguing against it.
On the 94% virology expert survey: I was glad to see the focus on lab work, but the questions may not capture what actually makes virologists dangerous or not. Multiple biologists told me virology is more like surgery than other STEM fields — the expertise is organism-specific, batch-specific, built up over decades of slow, hands-on learning. The survey gets at a sliver of laboratory protocols that are written down and some tacit knowledge. But the tacit knowledge required for each specific organism is huge and it's unclear whether the test gets at even 5% of it (random number, low confidence, could be <1% could be 15%). The tacit knowledge is the reason why the only person who succeeded at anthrax was a world expert on anthrax.
To be clear, I agree with your advocacy for more biosecurity efforts. But I believe your conclusion has unintended consequences. You said "arguments that downplay AI biosecurity risk— even unintentionally— risk contributing to policy inaction on a problem where the cost of delay could be extremely high."
My response on that: If the AI biosecurity risk is lower, but advocates overstate the risk, then it'll be harder to know when advocates are not crying wolf. It's also not good for group epistemics to see counterarguments as risky. It's a disservice to the field of AI-biosecurity. To get stronger arguments and be someone who governments reach out to, it's pretty core to understand why practitioners downplay it. It's the same reason the government will downplay it. Keeping "downplaying" types of evidence out of public discourse just harms the epistemics of people outside of government and makes it less likely that advocates will affect people inside of government.
Can't wait for your thoughts on the next parts! Do send me your email so I can send the Google Docs. :)
[cross posted]
Hi Abi! Thanks for your comment :) will reach out via DMs!
Will also write some thoughts here:
---
> I'd love to know more about which viruses you envision in the airport example. A more specific example would be helpful because, depending on what type of virus you're proposing, the core bottleneck is different.
My concern isn't any single "core bottleneck" but compound effects across multiple domains. Even if success rates per attempt remain low, when AI protocol design, cloud labs, and unscreened synthesis all reduce barriers simultaneously, the overall difficulty decreases. More attempts become feasible at lower cost, which matters even without high per-attempt success rates.
---
> An existing virus faces the same frictions it always has: symptomatic spread gets detected fast, and killing people quickly limits transmission. It'd be very expensive way to kill a dozens or a hundred as you note.
This seems off to me. Even an existing / “natural” virus would actually be a very cost-effective way to kill per capita. Take Covid: estimates are that the virus killed 7 million to 21 million people. “The same friction” does not equate to low risk; even with this friction, existing viruses can be quite deadly.
---
> An engineered novel virus raises a different problem: we don't actually know whether engineering a pathogen to behave exactly as intended is possible outside of a massive, multi-institutional team doing years of physical testing on humans. To career biologists, it sounds like "anyone can build nuclear weapons because the blueprints are online." It's technically true in a narrow sense, but not really in the physical world.
I don’t think you need to engineer a pathogen to behave “exactly as intended.” This seems like a mischaracterization of the task at hand, which is actually something closer to, “make a new deadly virus that is reasonably different enough from natural ones such that immune defenses are less effective or nonexistent,” aiming for a range of bad outcomes rather than one precise outcome.
---
> Aum Shinrikyo had a billion dollars and multidisciplinary experts and still couldn't produce a bioweapon. They had to turn to a chemical weapon.
This is true; no one is disputing the facts re: Aum Shinrikyo. My point is that historical barriers eroding could make the next Aum Shinrikyo successful in the future. They got operationally close, and a marginal decrease in existing barriers could make the next attempt successful.
---
On cloud labs:
My concern isn't necessarily that actors can hide what they're working with, but that cloud labs have minimal oversight requirements.
There currently is no overarching federal law that provides oversight of laboratory biosafety and biosecurity with enforceable legal penalties beyond the Federal Select Agent Program. The Federal Select Agent Program does require registration and has enforcement mechanisms for labs working with select agents and toxins, but this covers only a narrow subset of dangerous biological work– roughly 60-70 specific pathogens and toxins out of thousands of potentially dangerous biological agents.
For the broader category of dual-use research that (1) doesn't involve select agents and (2) is not federally funded, there is no standardized or explicit federal requirement for cloud labs to screen customers or refuse work in these cases.
While some current platforms do implement security protocols, oversight is largely voluntary self-regulation rather than mandated requirements, and as a result it’s inconsistent across different cloud labs.
---
You're right about the tacit knowledge limitations of the virology survey, but cloud labs + CROs are specifically designed to outsource that component. For cloud labs, you submit protocols and the platform's automated systems and technical staff handle the hands-on work. The virology test, when I included it, was named as an example of AI eliminating the knowledge bottleneck.
---
Regarding your point on group epistemics and "crying wolf," my concern isn't that counterarguments are inherently risky, but that framing bioweapons as "bad weapons" may accurately describe most historical cases while missing tail-risk scenarios that matter for catastrophic outcomes + not taking recent technological developments into account as much as it should.
There's a difference between including all viewpoints in the discourse (which we should) and accepting that all viewpoints are equally accurate assessments of present and near-term risk (which we shouldn't). I think the evidence suggests the threat landscape is shifting faster than traditional practitioner intuitions account for, and policy should reflect that trajectory rather than just historical base rates.
I could be wrong about how quickly these barriers are eroding, but I think the evidence suggests we should be planning for that possibility rather than assuming historical patterns will hold.
Finally, will also note that I appear to weigh tail risks more heavily than you do, which may explain much of our disagreement.
The reason why I asked you to choose a specific virus was so I can give an example of the bottlenecks of each virus. For example, if somebody unleashed COVID in the airport, that wouldn't be unusual. people are already doing that every day. Do you mean a new COVID?
More attempts might become cheaper, but attempts also erode your ability to stay under cover if you keep on ordering reagents and the specific chemicals that are needed for boot up, or for any of the steps in Part 3.
Even modifying pathogens to work as you claim is theoretical - even for massive state bioweapon programs. If it happens it’s not going to be a clandestine person. It’s going to be a huge breakthrough for science to confirm that’s possible.
Whether Aum Shinrikyo got close and will get closer depends on the actual barriers. Their experts didn’t have the tacit knowledge for the various parts of the process. Could they have figured out how to make their anthrax strain virulent? Unclear. They failed in many different ways simultaneously. And solving each barrier takes a hard to measure amount of tacit knowledge, access to equipment, etc. This is touched on in Pt 2 and Pt 3.
It still takes cloud labs and CROs a lot of work to set up the specialized workflows - if you have a protocol. For a lot of viruses and bacteria boot up process there isn’t protocols. You’d have to ask them to do the long months of failure to figure it out for you. You’d have to hand over tons of info about what you want and what it is. If you’re lucky and it’s easy to hand over a protocol then they know what you are doing.
The virology test is great and I applaud it but it doesn’t tell you anything about how much tacit knowledge is needed- it could only get at so little 1% of it. There’s a reason biology is an apprenticeship model like surgery, not a textbook model. It depends on every organisms. They’re all entirely different and you need to figure out surgery for each.
I’ve been working on catastrophic risk for a couple years now. So I don’t think it’s a matter of weighing tail risks. I intentionally sought out the steelman of the biology practitioners. They explained in many ways the details that first principles only thinking misses. A super virus is hard because of constraints in biology, not because people doubt that robots will become fully capable. Also since there is a nonzero risk of a bad bad pandemic every year, it also matters what we aim to make science accessible and able to happen. There’s a trade off in tail risks either way. So it matters to steelman both sides.
[Steve Newman from the Golden Gate Institute for AI here, cross-posting my response to your comment on the original post]
Thanks for this extremely thoughtful response! I'll jump in with a quick reply because Abi is mostly offline for the next few days, and then I'll let Abi respond in detail.
The key idea that emerged from Abi's research is that there are bottlenecks which do *not* seem likely to be significantly eased, at least in the near term, by current developments. (You cite some of the most important, such as cloud labs and DNA synthesis providers.) Abi based this work on discussions with multiple people from the AI / biosecurity community (some of whom are named in the acknowledgement at the end of the post), including feedback on the final draft before publication. The next couple of installments will go into more detail regarding those bottlenecks, and we'd love further feedback.
You also mention the emergence of contract research organizations (labs-for-hire). This isn't a topic I know much about, and I am not certain whether it came up in discussions, so I'll have to wait for Abi here. I'll also defer to Abi to comment on precisely how her findings relate to the Virology Capabilities Test results; I know this was on her radar, and again the upcoming installments will shed more light here.
Thanks again!
Thanks Steve! Really appreciate the engagement, and I want to reemphasize that Abi's project is genuinely a valuable contribution; the practitioner perspective has been missing from this conversation, and I'm glad she's been doing interviews and discussions with people in the field!
I'm interested to see the next installments get concrete on specific bottlenecks that remain, but I think my concern might be less about any single development and more about compound effects of being able to route around *any* of the bottlenecks that used to serve as barriers to bioweapons development.
Also glad CROs are on the radar. Looking forward to the rest of the series!
Really good piece—thank you! I thought the Olvera piece was an interesting contrast to 80,000 Hours' conversation with Richard Moulange: https://80000hours.org/podcast/episodes/richard-moulange-ai-bioweapons-biorisk/
I met Richard Moulange at the Cambridge Biosecurity Hub! Really cool guy, and has some great strategic takes on the field of AI bio.