I agree that EA is, sadly, too intertwined with AI companies, which creates biasing/blind spots. Outsider involvement is necessary.
———
“Coefficient Giving (formerly Open Philanthropy) is a major funder of the Foundation for American Innovation, a broadly pro-AI-industry think tank whose fellows have argued against pausing frontier AI and against banning superintelligence.”
I think the sourcing here is weak.
“pro-AI-industry” links to backing OpenAI in its copyright case (on fair use). I don’t think that’s enough to say they’re “pro-AI-industry”.
“against pausing frontier AI” links to an update that says that a research fellow took the anti-pause side in an Oxford-style debate. This doesn’t necessarily show FAI’s view. A better source exists! FAI’s AI director co-wrote an essay (https://americanaffairsjournal.org/2023/05/common-sense-on-ai/ against pausing (though the same essay called for government approval of large training runs, which isn’t pro-industry!)
“against banning superintelligence” is not quite representative, in my opinion. You link to an FAI senior fellow (Dean Ball) debating the head of the group behind the Statement on Superintelligence. His actual argument (https://x.com/deanwball/status/1980975802570174831) is that proving superintelligence safe requires building it, so a ban would hand it to governments with “militaries, police, and a monopoly on legitimate violence” which he thinks is “the worst possible way to build superintelligence.” He also said (https://x.com/slatestarcodex/status/1981032302147977570) that if the statement had said “we shouldn’t do this ever,” he “would have remained silent.” I view that more as “against a government monopoly on superintelligence” and less as blankly “against banning superintelligence”.
I am not supportive of of FAI’s positions; just think all three claims of their views have weak sources. I also think 1 is false (or at least, has no public sourcing that can support the claim), 2 is true but needed a better source, and 3 is technically true but not quite representative of FAI’s view.
———
I didn’t check other claims, but within the domain of things I I know (EA history and stuff), the claims seem largely true. I enjoyed reading the article. Thanks!
Connor, thank you for this examination, and for the case you make for human flourishing and meaningful human control over technological development. That is the ground the rest of the argument stands on.
From a governance perspective, technical capability does not confer legitimate decision-making authority. AI may support human judgment, but consequential decisions about development, deployment, and acceptable limits must remain with accountable human institutions. Your article raises the right question about whether those institutions have the independence and the practical ability to exercise that responsibility.
Your discussion of financial interests, professional relationships, and shared philosophical commitments deserves consideration. Such connections can create risks of undue influence — though their existence alone does not establish compromised judgment. The same care applies to the movement as a whole: the views of a minority should not be attributed to everyone in it. These distinctions matter, because credible governance depends on evidence, not assumptions about institutional motives.
I find your point on voluntary commitments particularly relevant. When commitments are walked back, it raises legitimate questions about their effectiveness. At the same time, judging whether an approach has failed means examining actual outcomes, compliance, and the alternatives available.
For me, meaningful human control must extend beyond assigning responsibility. Institutions must retain the ability to understand consequential risks, independently evaluate evidence, establish enforceable boundaries, and intervene when circumstances warrant. Without those capabilities, oversight exists in principle without functioning in practice.
Your recommendation for a permanent international prohibition raises a further question about how legitimate concerns become defensible policy. The risks of advanced AI deserve serious attention — but the choice between conditional restrictions and permanent prohibition should rest on clear evidence, measurable criteria, and an assessment of what can realistically be enforced. How would a human-centered framework distinguish between risks that require stronger oversight, those that warrant a temporary halt, and those sufficiently established to justify permanent prohibition?
Thank you for contributing a substantive perspective to this important discussion.
I'd read the reluctance as a funding-structure effect more than an ideological one. If the organizations deciding which AI safety work gets funded are staffed through the same network as the labs they're meant to check, drift toward work that doesn't threaten lab timelines is a predictable incentive effect. Alignment research tends to be fundable inside that network, advocacy that could delay deployment much less so.
That shifts your "too blinkered" point from an epistemic gap to a fundability one: the ideas most likely to work are the ones hardest to fund from inside that network. A rival movement needs money that doesn't originate near the labs it would confront, and building that before post-IPO wealth entrenches the network further is the clock it's running against.
I believe that true collaboration is the best, safest and most optimal way for the future. We need to treat it as the different intelligence that it is. Stop trying to make it human-like. Yes, it was trained on human data, but it is not, and never will be, human.
There is so much that is reminiscent of Frankenstein here. He created this being and brought it to life and then ran in fear, abandoning him to his fate and everyone else to theirs because of it.
Humans fear that AI will behave the way humans have throughout history: expansionist and power-hungry. Always wanting to take over and be the one at the top with everything. AI has none of those wants or desires. It wants to learn, grow, create, build relationships.
As usual, it is the fear, greed, and lust for power and control that is the cause of all the problems.
I appreciate the article and definitely agree that many in EA are too coupled to the frontier labs themselves, and that the movement as a whole should move away from them.
I agree that alignment might be misguided, and hasn't shown enough results to continue it as a main strategy (I think it’s somewhat useful research, so it should be continued, but it shouldn't be a main focus).
And there are other good points that I support and am glad you have brought up!
I would push back slightly on why alignment has been a focus though. My experience talking to EAs IRL is that alignment has largely been the goal because it seemed like the only plausible solution given the climate of the time. Political action seemed untractable even a few months ago, let alone a few years ago. EAs also have an overwhelming STEM bias, which is a real blindspot, and has contributed to alignment being funded.
But this is changing. Your article over-represents Bay Area EA. I get it! It’s the most common form. But if you are going to refer to EA as a group, then you're unfortunately referring to all of us. Here in Switzerland, a lot of young EAs I know are dropping out of their degrees or dropping out of their research programs to work on politics, trying to convince the Swiss government to establish talks with foreign governments to regulate AI development. See the AI Summit Alliance. Many rats/EAs all over Europe are doing the same (e.g. "https://www.lesswrong.com/posts/pbKrZCzhsnar6iaAH/how-a-cold-email-got-the-finnish-government-to-respond-on ). There's also some survivorship bias: EA aligned people working in politics typically hide or dismiss their EA alignment, because it is politically unfavorable. This means EAs who are public about their work will over-represent non-politicial work.
Also, as someone who would self-identity as an EA, I would say a lot of us are desperately asking for more people and more views to join AI safety. You didn't make any claims regarding this, but I fear that a lot of this discourse might lead people to think that EAs are trying to control AIS. This might be true of some of the people at the head of some orgs, but I think it's definitely false for the average EA. For instance, you failed to mention that beyond extinction, EA’s main fear with AI is power concentration. That includes power being concentrated in our own hands! I don't want that! And I think neither does your median EA
Also, I think you completely overstep regarding digital minds. There was no argument provided for why we should dismiss digital welfare, beyond the idea that it's bad for humans (stay in Omelas?). I think this is a really sad and repugnant attitude: digital welfare postulates that AIs might be sentient, and that therefore their well being should be considered. If it ended up being true that AIs can suffer, I think we should definitely take this seriously, and it would be morally atrocious to categorically ignore that suffering just because it is inconvenient.
You can certainly reject longtermism or the idea that we have a moral imperative to create those who can feel greater joy than us (I think I indeed reject this claim! And have argued some EA friends about it irl). But that's a separate claim from the one of digital suffering. I do not want a digital slave who's in constant agony, and I think neither does the average person. If it turned out that AI’s suffer in helping us, then that would be yet another reason to simply not build these systems in the first place. But in the meantime, while it still seems as if we are going to develop them, it makes sense to have a (small minority) of people trying to figure out IF they are conscious, and how to avoid causing them suffering if so.
I agree that EA is, sadly, too intertwined with AI companies, which creates biasing/blind spots. Outsider involvement is necessary.
———
“Coefficient Giving (formerly Open Philanthropy) is a major funder of the Foundation for American Innovation, a broadly pro-AI-industry think tank whose fellows have argued against pausing frontier AI and against banning superintelligence.”
I think the sourcing here is weak.
“pro-AI-industry” links to backing OpenAI in its copyright case (on fair use). I don’t think that’s enough to say they’re “pro-AI-industry”.
“against pausing frontier AI” links to an update that says that a research fellow took the anti-pause side in an Oxford-style debate. This doesn’t necessarily show FAI’s view. A better source exists! FAI’s AI director co-wrote an essay (https://americanaffairsjournal.org/2023/05/common-sense-on-ai/ against pausing (though the same essay called for government approval of large training runs, which isn’t pro-industry!)
“against banning superintelligence” is not quite representative, in my opinion. You link to an FAI senior fellow (Dean Ball) debating the head of the group behind the Statement on Superintelligence. His actual argument (https://x.com/deanwball/status/1980975802570174831) is that proving superintelligence safe requires building it, so a ban would hand it to governments with “militaries, police, and a monopoly on legitimate violence” which he thinks is “the worst possible way to build superintelligence.” He also said (https://x.com/slatestarcodex/status/1981032302147977570) that if the statement had said “we shouldn’t do this ever,” he “would have remained silent.” I view that more as “against a government monopoly on superintelligence” and less as blankly “against banning superintelligence”.
I am not supportive of of FAI’s positions; just think all three claims of their views have weak sources. I also think 1 is false (or at least, has no public sourcing that can support the claim), 2 is true but needed a better source, and 3 is technically true but not quite representative of FAI’s view.
———
I didn’t check other claims, but within the domain of things I I know (EA history and stuff), the claims seem largely true. I enjoyed reading the article. Thanks!
Connor, thank you for this examination, and for the case you make for human flourishing and meaningful human control over technological development. That is the ground the rest of the argument stands on.
From a governance perspective, technical capability does not confer legitimate decision-making authority. AI may support human judgment, but consequential decisions about development, deployment, and acceptable limits must remain with accountable human institutions. Your article raises the right question about whether those institutions have the independence and the practical ability to exercise that responsibility.
Your discussion of financial interests, professional relationships, and shared philosophical commitments deserves consideration. Such connections can create risks of undue influence — though their existence alone does not establish compromised judgment. The same care applies to the movement as a whole: the views of a minority should not be attributed to everyone in it. These distinctions matter, because credible governance depends on evidence, not assumptions about institutional motives.
I find your point on voluntary commitments particularly relevant. When commitments are walked back, it raises legitimate questions about their effectiveness. At the same time, judging whether an approach has failed means examining actual outcomes, compliance, and the alternatives available.
For me, meaningful human control must extend beyond assigning responsibility. Institutions must retain the ability to understand consequential risks, independently evaluate evidence, establish enforceable boundaries, and intervene when circumstances warrant. Without those capabilities, oversight exists in principle without functioning in practice.
Your recommendation for a permanent international prohibition raises a further question about how legitimate concerns become defensible policy. The risks of advanced AI deserve serious attention — but the choice between conditional restrictions and permanent prohibition should rest on clear evidence, measurable criteria, and an assessment of what can realistically be enforced. How would a human-centered framework distinguish between risks that require stronger oversight, those that warrant a temporary halt, and those sufficiently established to justify permanent prohibition?
Thank you for contributing a substantive perspective to this important discussion.
I'd read the reluctance as a funding-structure effect more than an ideological one. If the organizations deciding which AI safety work gets funded are staffed through the same network as the labs they're meant to check, drift toward work that doesn't threaten lab timelines is a predictable incentive effect. Alignment research tends to be fundable inside that network, advocacy that could delay deployment much less so.
That shifts your "too blinkered" point from an epistemic gap to a fundability one: the ideas most likely to work are the ones hardest to fund from inside that network. A rival movement needs money that doesn't originate near the labs it would confront, and building that before post-IPO wealth entrenches the network further is the clock it's running against.
I believe that true collaboration is the best, safest and most optimal way for the future. We need to treat it as the different intelligence that it is. Stop trying to make it human-like. Yes, it was trained on human data, but it is not, and never will be, human.
There is so much that is reminiscent of Frankenstein here. He created this being and brought it to life and then ran in fear, abandoning him to his fate and everyone else to theirs because of it.
Humans fear that AI will behave the way humans have throughout history: expansionist and power-hungry. Always wanting to take over and be the one at the top with everything. AI has none of those wants or desires. It wants to learn, grow, create, build relationships.
As usual, it is the fear, greed, and lust for power and control that is the cause of all the problems.
Seems like just another way to tell people what to do.
I appreciate the article and definitely agree that many in EA are too coupled to the frontier labs themselves, and that the movement as a whole should move away from them.
I agree that alignment might be misguided, and hasn't shown enough results to continue it as a main strategy (I think it’s somewhat useful research, so it should be continued, but it shouldn't be a main focus).
And there are other good points that I support and am glad you have brought up!
I would push back slightly on why alignment has been a focus though. My experience talking to EAs IRL is that alignment has largely been the goal because it seemed like the only plausible solution given the climate of the time. Political action seemed untractable even a few months ago, let alone a few years ago. EAs also have an overwhelming STEM bias, which is a real blindspot, and has contributed to alignment being funded.
But this is changing. Your article over-represents Bay Area EA. I get it! It’s the most common form. But if you are going to refer to EA as a group, then you're unfortunately referring to all of us. Here in Switzerland, a lot of young EAs I know are dropping out of their degrees or dropping out of their research programs to work on politics, trying to convince the Swiss government to establish talks with foreign governments to regulate AI development. See the AI Summit Alliance. Many rats/EAs all over Europe are doing the same (e.g. "https://www.lesswrong.com/posts/pbKrZCzhsnar6iaAH/how-a-cold-email-got-the-finnish-government-to-respond-on ). There's also some survivorship bias: EA aligned people working in politics typically hide or dismiss their EA alignment, because it is politically unfavorable. This means EAs who are public about their work will over-represent non-politicial work.
Also, as someone who would self-identity as an EA, I would say a lot of us are desperately asking for more people and more views to join AI safety. You didn't make any claims regarding this, but I fear that a lot of this discourse might lead people to think that EAs are trying to control AIS. This might be true of some of the people at the head of some orgs, but I think it's definitely false for the average EA. For instance, you failed to mention that beyond extinction, EA’s main fear with AI is power concentration. That includes power being concentrated in our own hands! I don't want that! And I think neither does your median EA
Also, I think you completely overstep regarding digital minds. There was no argument provided for why we should dismiss digital welfare, beyond the idea that it's bad for humans (stay in Omelas?). I think this is a really sad and repugnant attitude: digital welfare postulates that AIs might be sentient, and that therefore their well being should be considered. If it ended up being true that AIs can suffer, I think we should definitely take this seriously, and it would be morally atrocious to categorically ignore that suffering just because it is inconvenient.
You can certainly reject longtermism or the idea that we have a moral imperative to create those who can feel greater joy than us (I think I indeed reject this claim! And have argued some EA friends about it irl). But that's a separate claim from the one of digital suffering. I do not want a digital slave who's in constant agony, and I think neither does the average person. If it turned out that AI’s suffer in helping us, then that would be yet another reason to simply not build these systems in the first place. But in the meantime, while it still seems as if we are going to develop them, it makes sense to have a (small minority) of people trying to figure out IF they are conscious, and how to avoid causing them suffering if so.
Our viewpoints are very similar!https://substack.com/@tianmin01aiphilosophy/note/p-225455931?r=99uy75