Great post, I very much agree that expert uplift is the more pressing thing to investigate, though the existing study is still useful and interesting. I'd recommend talking to GovAI about this too, given their involvement.
Thank you for the tip Oscar!! I believe I'll be visiting GovAI this Wednesday for ERA's symposium, and will see if I can connect with someone on their team who worked on it :)
Thank you for providing feedback on our trial! I agree that expert-level uplift is an avenue of biosecurity risk that deserves further work and investigation. However, it's also much more difficult to measure! I think we'll need to design better expert-level tasks that are difficult, and is able to measure novelty, those outcomes can be evaluated objectively, and ideally allows for continuous outcome measures. This is quite a lot to ask!
I do agree that a Latin square (i.e. fully-counterbalanced) cross-over design will be more statistically efficient – but I think it will be hard to select matched tasks: otherwise, I fear that carry-over effects will significantly attenuate any treatment effect. This will especially be the case if the amount of uplift conferred by the intervention is small. Once again, it goes back towards having to think more carefully about task design.
The one point of feedback that I disagree with is your point on ecological validity. While it is true that a realistic threat actor will have access to online communities and experts (e.g. writing an email to a professor, making a post on stackexchange or a subreddit), in practice we cannot allow participants unrestricted access to online communities. This will essentially entail involving strangers in a study – one that they've never consented to. It may also result in disruption to the organic behaviour of such communities, and repeated questions may trigger worry or alarm. Such a proposal is unethical, and will certainly not pass IRB review.
However, that being said there are ways to 'simulate' access to experts. For instance, you can imagine hiring a PhD-level biologist and have them available to answer questions via email – similar to your "university professor" model!
Ultimately, I do agree that our field can benefit from more empirical wet-lab evaluations. This data helps answer questions and narrow-down uncertainties, and it is my hope that our research will ultimately inform policymakers and biosecurity governance as a whole.
Thank you so much for your comment, and for the presentation you gave a few weeks back! Really appreciate you taking the time to engage with this :)
You're completely right on the ecological validity point; I didn't think through the IRB implications of essentially enrolling unconsenting strangers as participants. I love the idea of simulating expert access with a hired PhD-level biologist available via email, and I'd be excited to see that in a future study design.
Also, I completely agree re: the difficulty of designing matched tasks, but I think even an imperfect crossover with acknowledged carry-over limitations would be more informative than the status quo of not measuring expert uplift at all.
Thank you for your work on this! It's genuinely important, and I think the field really benefits from having empirical wet-lab data to work with. Best of luck with the next iterations!!
Great post. I agree with most of your assessment, with one exception. Fine-tuning is an order of magnitude more complicated than bypassing safety classifiers, let alone more expensive and prone to silent failures. Bypassing safety classifiers is tractable for an unsophisticated adversary. For any question instrumental to doing harm, you have a prompt and a binary output (blocked or not blocked by a classifier), feedback is direct and immediate, which allows for rapid iteration and rephrasing to get through.
Hi Austin! Thanks for your comment :) Curious to hear why you think fine-tuning is difficult? There are a lot of platforms (e.g. PrettyBase, Lightning AI) that make it fairly easy + there are lists of datasets freely available online for anyone to use (e.g. https://github.com/mlabonne/llm-datasets).
Actually, I spent a day backdooring a model with Claude, which included a fine-tuning method called LoRA (https://substack.com/home/post/p-193430484) and it was definitely not “an order of magnitude more complicated” than jailbreaks!
*Edit to add: Will note that fine-tuning larger models is definitely harder, but unsure how this compares with jailbreaks— might do more research on this for a future post :)
Datasets are available, but not all are high quality, and finding what you need that will increase performance on your tasks in a way that generalizes beyond your training data is challenging. Generally, AI x Biology is hard, and curating good training datasets is really hard! Companies that have managed to do this well (e.g., Insilico Pharma, Recursion) keep their data private. Additionally, fine-tuning can introduce performance degradation outside the narrow area. If you've tailored to virology, did you lose performance in molecular biology? How quickly can you test those questions?
Jailbreaking is an order of magnitude easier because it sidesteps the issues of dataset curation. You retain a quick feedback signal for what works, while retaining access to the frontier model :-)
This is a great post! Thank you. I am currently writing a literature review of this field and I was unable to find ANY wet-lab trials which included non-novices. If anyone can find such a paper, please send it my way!
Great post, I very much agree that expert uplift is the more pressing thing to investigate, though the existing study is still useful and interesting. I'd recommend talking to GovAI about this too, given their involvement.
Thank you for the tip Oscar!! I believe I'll be visiting GovAI this Wednesday for ERA's symposium, and will see if I can connect with someone on their team who worked on it :)
Thank you for providing feedback on our trial! I agree that expert-level uplift is an avenue of biosecurity risk that deserves further work and investigation. However, it's also much more difficult to measure! I think we'll need to design better expert-level tasks that are difficult, and is able to measure novelty, those outcomes can be evaluated objectively, and ideally allows for continuous outcome measures. This is quite a lot to ask!
I do agree that a Latin square (i.e. fully-counterbalanced) cross-over design will be more statistically efficient – but I think it will be hard to select matched tasks: otherwise, I fear that carry-over effects will significantly attenuate any treatment effect. This will especially be the case if the amount of uplift conferred by the intervention is small. Once again, it goes back towards having to think more carefully about task design.
The one point of feedback that I disagree with is your point on ecological validity. While it is true that a realistic threat actor will have access to online communities and experts (e.g. writing an email to a professor, making a post on stackexchange or a subreddit), in practice we cannot allow participants unrestricted access to online communities. This will essentially entail involving strangers in a study – one that they've never consented to. It may also result in disruption to the organic behaviour of such communities, and repeated questions may trigger worry or alarm. Such a proposal is unethical, and will certainly not pass IRB review.
However, that being said there are ways to 'simulate' access to experts. For instance, you can imagine hiring a PhD-level biologist and have them available to answer questions via email – similar to your "university professor" model!
Ultimately, I do agree that our field can benefit from more empirical wet-lab evaluations. This data helps answer questions and narrow-down uncertainties, and it is my hope that our research will ultimately inform policymakers and biosecurity governance as a whole.
Hi Shen,
Thank you so much for your comment, and for the presentation you gave a few weeks back! Really appreciate you taking the time to engage with this :)
You're completely right on the ecological validity point; I didn't think through the IRB implications of essentially enrolling unconsenting strangers as participants. I love the idea of simulating expert access with a hired PhD-level biologist available via email, and I'd be excited to see that in a future study design.
Also, I completely agree re: the difficulty of designing matched tasks, but I think even an imperfect crossover with acknowledged carry-over limitations would be more informative than the status quo of not measuring expert uplift at all.
Thank you for your work on this! It's genuinely important, and I think the field really benefits from having empirical wet-lab data to work with. Best of luck with the next iterations!!
Just commenting to say I've been beating the drum about expert uplift for what feels like decades now (In AI-land, that means a few months, but...)
Here's a thread from the end of last year, pointing to work presented in 2024, based on Anders and Cassidy's 2020 paper,
https://x.com/davidmanheim/status/2001584512153587901
As well as my 2018 blog post about how unsophisticated lone wolves aren't the right threat model:
https://davidmanheim.substack.com/p/should-you-worry-about-terrorists-with-bio-weapons-no-3a2a5157a0a9
Great post. I agree with most of your assessment, with one exception. Fine-tuning is an order of magnitude more complicated than bypassing safety classifiers, let alone more expensive and prone to silent failures. Bypassing safety classifiers is tractable for an unsophisticated adversary. For any question instrumental to doing harm, you have a prompt and a binary output (blocked or not blocked by a classifier), feedback is direct and immediate, which allows for rapid iteration and rephrasing to get through.
Hi Austin! Thanks for your comment :) Curious to hear why you think fine-tuning is difficult? There are a lot of platforms (e.g. PrettyBase, Lightning AI) that make it fairly easy + there are lists of datasets freely available online for anyone to use (e.g. https://github.com/mlabonne/llm-datasets).
Actually, I spent a day backdooring a model with Claude, which included a fine-tuning method called LoRA (https://substack.com/home/post/p-193430484) and it was definitely not “an order of magnitude more complicated” than jailbreaks!
*Edit to add: Will note that fine-tuning larger models is definitely harder, but unsure how this compares with jailbreaks— might do more research on this for a future post :)
Datasets are available, but not all are high quality, and finding what you need that will increase performance on your tasks in a way that generalizes beyond your training data is challenging. Generally, AI x Biology is hard, and curating good training datasets is really hard! Companies that have managed to do this well (e.g., Insilico Pharma, Recursion) keep their data private. Additionally, fine-tuning can introduce performance degradation outside the narrow area. If you've tailored to virology, did you lose performance in molecular biology? How quickly can you test those questions?
Jailbreaking is an order of magnitude easier because it sidesteps the issues of dataset curation. You retain a quick feedback signal for what works, while retaining access to the frontier model :-)
Yep that makes sense! Thanks for the comment :)
I wonder how this might change as input-output classifiers get better + fine-tuning applications become more widespread.
This is a great post! Thank you. I am currently writing a literature review of this field and I was unable to find ANY wet-lab trials which included non-novices. If anyone can find such a paper, please send it my way!