What is your take on funders being ready to scale by doing more of what they're already doing (hiring more people) but not by innovating (e.g., using impact markets)?
I'm trying (again) to break into the grantmaking space. I used to do grantmaking in international development and later, around 2021–24, tried to create impact markets, even got funding for it, but (after the FTX crash) found it very hard to get funders to use it because it was something new and unproven (and at the time they didn't have trouble allocating their funding).
The system would be perfect for projects with concrete outputs like movies or documentaries. The funder announces that they want to reward good documentaries, business angels provide the seed funding, and if the movie turns out good, the funder rewards all contributors. I.e. no prediction questions like “Can this team create a good movie?” or “Is this an exit scam?” but just a concrete movie and the question “Is it good?” The overhead for grantmakers will drop drastically: The evaluation is easier, and if 1:10 projects succeed, then the number of evaluations is also much lower. It would be the perfect system for scaling up funding for the sorts of projects you envision (as well as research).
My current take is that I will first have to break into the grantmaking space and get into a position where I can drive innovation like that personally while also dogfooding with my funding budget or else this won't happen. I'm applying to lots of positions now.
But there is also a long-tail of other projects I'd love to create or incentivize: I'm not aware of anyone who does any systematic modeling of all the possible policy windows that might open – if China besieges Taiwan, if an AI “warning shot” event happens in country X, if a Russian EMP disables ASML, and hundreds of other scenarios. These are all scenarios that we can systematically elicit, prioritize, and prepare for. Have bills ready, have contacts ready, know how to pitch them, etc.
Or helping charity founders get off the ground: I envision a database of all the projects (movies, think tanks, infrastructure projects, etc.) that should exist and profiles of the kinds of founder personas that it would take to start them. Founders could sign up, select what personas they identify with, and get personalized recommendations for the projects they should start based on their own profile and how close the project is to filling all required cofounder roles.
Likewise, the positioning of funders and other decision-makers will be critical during the hot days of the takeoff because if a funder is seen as an enemy or obstacle by an ASI, it'll put them in a much worse position than if they can be trusted to follow cross-cutting “pulling the rope sideways” strategies. E.g., favoring Anthropic for its slightly better safety practices is a tightrope walk, because it might just look like favoring Anthropic.
There are so many more projects that I wish anyone were working on, but I can't start them all myself!
Wish I had more time to fully engage, but a few brief thoughts:
1. Re: impact markets -- I'd be curious to hear your takes on how this is different from prizes? A friend has a great write up on this: https://firstscattering.com/p/lets-fund-some-huge-prizes-in-ai, and prizes seems functionally similar to impact markets, at least at first glance. (I'm not super familiar / this is the first time I've heard of the idea, so forgive me if the question is naive!)
2. One concern I'd imagine funders would have with this database of sorts is that founders self-selecting different "personas" doesn't seem like it would be very indicative of a founder's actual capabilities, which is why most active grantmaking nowadays is heavily based on a grantmaker's existing network (though RFPs are an open way for people to apply to work on different ideas as well; ofc, the selection process involves a lot of candidate vetting, reaching out to references and etc).
3. Your point about funder positioning reminds me of Roko's Basilisk!
I'm glad you're thinking about innovative ways in which funders can scale. I think I don't have enough good takes on this to be able to pose my own novel ideas at the moment, but will keep thinking! I appreciate the shout.
Thanks for taking the time to reply! I'd love to interview you some time for my blog and YouTube channel Impartial Priorities. Most of the subscribers are not from the AI governance space, so I can't promise that it'll be useful for getting the word out, but I'd love to pick your brains and get your ideas out there any way I can.
Yeah, exactly! Prizes were one of the inspirations! And the XPrize in particular comes somewhat close to our original model because it does sometimes/often have secondary markets. The allure is largely in making it so profitable that most of the work (and risk) is shouldered by business angels.
But there are some problems with it, especially regarding legal costs for the participants, so I now think that the carbon credit markets are the better model to base it on because the tradable credits can form one big market rather than countless small ones (deeper liquidity) and the new credits can hopefully also be treated as a commodity rather than a security, which would make things a lot easier legally.
But the solution can also be scaled down… I have a few more ideas for legally easy solutions with different advantages and disadvantages. They all require an enterprising funder, a founder, and some legal advice (but I know an EA lawyer in securities law).
Yeah, regarding the founder matching platform, I agree that that's something that the founders still have to figure out themselves. I'd like to decouple the process as much as possible for the funders lest grantmaking capacity remain such a great bottleneck in the process. Founders have managed to find each other for centuries; my platform can make it easier for them but doesn't solve all the issues. E.g., one of my founders dropped out after half a year or so, and I don't think any of us could've predicted that early on.
Roko's Basilisk: Oh, I hope not! I hope this warning is safe to spread. The AI will be incentivzed to sabotage enemies whether the enemies expect it or not. And it'll probably just sabotage them efficiently until they are neutralized, brief suffering. 🙈
Completely agree! However, you suggest movements and advocacy orgs be built “with named donors and publicly disclosed advocacy, which is the operational opposite of how astroturfing works.”
What roadblocks should we expect trying to create advocacy and movement organizations focused around regulating AI that are funded by AI companies and employees? And how would you suggest getting over those?
Hey Ian! Thanks for commenting, I've been thinking a bit about this too -- tbh, I'm not sure if I have an answer I'm happy with yet. Might write a follow-up, but for now, one way to get around this problem could be something like redirecting individual donors who would otherwise have donated to technical alignment research to political advocacy (have written more on this here: https://thecounterfactual.substack.com/p/donate-less-to-ai-safety). Curious to hear any thoughts on the issue!
Something important to notice about 'The Day After' (and 'Threads') is that it's a movie about the catastrophe *happening* and how bad that would be. It forces you to put yourself there and that makes a strong emotional impact.
I think this type of intuition pump is the most powerful of them; people get the most motivated to change their lives when they think about their last moments and what they could regret then. Same thing happens when people think of their loved ones dying, they feel motivated to tell them that they love them or protect them.
The end of the world is basically both of those things at once.
Making people put themselves in those shoes is something AIXR comms hasn't tried much IMO. Only examples I've seen were one John Sherman once in his podcast and another one in a book, and one tweet from Michael Trazzi.
There's a lot I could have added in this piece about specific things like bio-hardening, but I'm saving bio for a longer essay (part one here): https://thecounterfactual.substack.com/p/should-we-be-worried-about-bioweapons
What is your take on funders being ready to scale by doing more of what they're already doing (hiring more people) but not by innovating (e.g., using impact markets)?
I'm trying (again) to break into the grantmaking space. I used to do grantmaking in international development and later, around 2021–24, tried to create impact markets, even got funding for it, but (after the FTX crash) found it very hard to get funders to use it because it was something new and unproven (and at the time they didn't have trouble allocating their funding).
The system would be perfect for projects with concrete outputs like movies or documentaries. The funder announces that they want to reward good documentaries, business angels provide the seed funding, and if the movie turns out good, the funder rewards all contributors. I.e. no prediction questions like “Can this team create a good movie?” or “Is this an exit scam?” but just a concrete movie and the question “Is it good?” The overhead for grantmakers will drop drastically: The evaluation is easier, and if 1:10 projects succeed, then the number of evaluations is also much lower. It would be the perfect system for scaling up funding for the sorts of projects you envision (as well as research).
My current take is that I will first have to break into the grantmaking space and get into a position where I can drive innovation like that personally while also dogfooding with my funding budget or else this won't happen. I'm applying to lots of positions now.
But there is also a long-tail of other projects I'd love to create or incentivize: I'm not aware of anyone who does any systematic modeling of all the possible policy windows that might open – if China besieges Taiwan, if an AI “warning shot” event happens in country X, if a Russian EMP disables ASML, and hundreds of other scenarios. These are all scenarios that we can systematically elicit, prioritize, and prepare for. Have bills ready, have contacts ready, know how to pitch them, etc.
Or helping charity founders get off the ground: I envision a database of all the projects (movies, think tanks, infrastructure projects, etc.) that should exist and profiles of the kinds of founder personas that it would take to start them. Founders could sign up, select what personas they identify with, and get personalized recommendations for the projects they should start based on their own profile and how close the project is to filling all required cofounder roles.
Likewise, the positioning of funders and other decision-makers will be critical during the hot days of the takeoff because if a funder is seen as an enemy or obstacle by an ASI, it'll put them in a much worse position than if they can be trusted to follow cross-cutting “pulling the rope sideways” strategies. E.g., favoring Anthropic for its slightly better safety practices is a tightrope walk, because it might just look like favoring Anthropic.
There are so many more projects that I wish anyone were working on, but I can't start them all myself!
Hi Dawn! Thanks for your comment :)
Wish I had more time to fully engage, but a few brief thoughts:
1. Re: impact markets -- I'd be curious to hear your takes on how this is different from prizes? A friend has a great write up on this: https://firstscattering.com/p/lets-fund-some-huge-prizes-in-ai, and prizes seems functionally similar to impact markets, at least at first glance. (I'm not super familiar / this is the first time I've heard of the idea, so forgive me if the question is naive!)
2. One concern I'd imagine funders would have with this database of sorts is that founders self-selecting different "personas" doesn't seem like it would be very indicative of a founder's actual capabilities, which is why most active grantmaking nowadays is heavily based on a grantmaker's existing network (though RFPs are an open way for people to apply to work on different ideas as well; ofc, the selection process involves a lot of candidate vetting, reaching out to references and etc).
3. Your point about funder positioning reminds me of Roko's Basilisk!
I'm glad you're thinking about innovative ways in which funders can scale. I think I don't have enough good takes on this to be able to pose my own novel ideas at the moment, but will keep thinking! I appreciate the shout.
Thanks for taking the time to reply! I'd love to interview you some time for my blog and YouTube channel Impartial Priorities. Most of the subscribers are not from the AI governance space, so I can't promise that it'll be useful for getting the word out, but I'd love to pick your brains and get your ideas out there any way I can.
Here's an AI gov project proposal that I just posted: https://impartial-priorities.org/p/tactical-and-operational-exploratory
Yeah, exactly! Prizes were one of the inspirations! And the XPrize in particular comes somewhat close to our original model because it does sometimes/often have secondary markets. The allure is largely in making it so profitable that most of the work (and risk) is shouldered by business angels.
But there are some problems with it, especially regarding legal costs for the participants, so I now think that the carbon credit markets are the better model to base it on because the tradable credits can form one big market rather than countless small ones (deeper liquidity) and the new credits can hopefully also be treated as a commodity rather than a security, which would make things a lot easier legally.
But the solution can also be scaled down… I have a few more ideas for legally easy solutions with different advantages and disadvantages. They all require an enterprising funder, a founder, and some legal advice (but I know an EA lawyer in securities law).
Yeah, regarding the founder matching platform, I agree that that's something that the founders still have to figure out themselves. I'd like to decouple the process as much as possible for the funders lest grantmaking capacity remain such a great bottleneck in the process. Founders have managed to find each other for centuries; my platform can make it easier for them but doesn't solve all the issues. E.g., one of my founders dropped out after half a year or so, and I don't think any of us could've predicted that early on.
Roko's Basilisk: Oh, I hope not! I hope this warning is safe to spread. The AI will be incentivzed to sabotage enemies whether the enemies expect it or not. And it'll probably just sabotage them efficiently until they are neutralized, brief suffering. 🙈
Completely agree! However, you suggest movements and advocacy orgs be built “with named donors and publicly disclosed advocacy, which is the operational opposite of how astroturfing works.”
What roadblocks should we expect trying to create advocacy and movement organizations focused around regulating AI that are funded by AI companies and employees? And how would you suggest getting over those?
Hey Ian! Thanks for commenting, I've been thinking a bit about this too -- tbh, I'm not sure if I have an answer I'm happy with yet. Might write a follow-up, but for now, one way to get around this problem could be something like redirecting individual donors who would otherwise have donated to technical alignment research to political advocacy (have written more on this here: https://thecounterfactual.substack.com/p/donate-less-to-ai-safety). Curious to hear any thoughts on the issue!
another Goated read 😻
awwwww love u zebedi 🫶🫶🫶
miss cambridge smmmm
Something important to notice about 'The Day After' (and 'Threads') is that it's a movie about the catastrophe *happening* and how bad that would be. It forces you to put yourself there and that makes a strong emotional impact.
I think this type of intuition pump is the most powerful of them; people get the most motivated to change their lives when they think about their last moments and what they could regret then. Same thing happens when people think of their loved ones dying, they feel motivated to tell them that they love them or protect them.
The end of the world is basically both of those things at once.
Making people put themselves in those shoes is something AIXR comms hasn't tried much IMO. Only examples I've seen were one John Sherman once in his podcast and another one in a book, and one tweet from Michael Trazzi.