Discussion about this post

User's avatar
Hazem🔸's avatar

I agree that EA is, sadly, too intertwined with AI companies, which creates biasing/blind spots. Outsider involvement is necessary.

———

“Coefficient Giving (formerly Open Philanthropy) is a major funder of the Foundation for American Innovation, a broadly pro-AI-industry think tank whose fellows have argued against pausing frontier AI and against banning superintelligence.”

I think the sourcing here is weak.

“pro-AI-industry” links to backing OpenAI in its copyright case (on fair use). I don’t think that’s enough to say they’re “pro-AI-industry”.

“against pausing frontier AI” links to an update that says that a research fellow took the anti-pause side in an Oxford-style debate. This doesn’t necessarily show FAI’s view. A better source exists! FAI’s AI director co-wrote an essay (https://americanaffairsjournal.org/2023/05/common-sense-on-ai/ against pausing (though the same essay called for government approval of large training runs, which isn’t pro-industry!)

“against banning superintelligence” is not quite representative, in my opinion. You link to an FAI senior fellow (Dean Ball) debating the head of the group behind the Statement on Superintelligence. His actual argument (https://x.com/deanwball/status/1980975802570174831) is that proving superintelligence safe requires building it, so a ban would hand it to governments with “militaries, police, and a monopoly on legitimate violence” which he thinks is “the worst possible way to build superintelligence.” He also said (https://x.com/slatestarcodex/status/1981032302147977570) that if the statement had said “we shouldn’t do this ever,” he “would have remained silent.” I view that more as “against a government monopoly on superintelligence” and less as blankly “against banning superintelligence”.

I am not supportive of of FAI’s positions; just think all three claims of their views have weak sources. I also think 1 is false (or at least, has no public sourcing that can support the claim), 2 is true but needed a better source, and 3 is technically true but not quite representative of FAI’s view.

———

I didn’t check other claims, but within the domain of things I I know (EA history and stuff), the claims seem largely true. I enjoyed reading the article. Thanks!

MarkCorbin | AI Governance's avatar

Connor, thank you for this examination, and for the case you make for human flourishing and meaningful human control over technological development. That is the ground the rest of the argument stands on.

From a governance perspective, technical capability does not confer legitimate decision-making authority. AI may support human judgment, but consequential decisions about development, deployment, and acceptable limits must remain with accountable human institutions. Your article raises the right question about whether those institutions have the independence and the practical ability to exercise that responsibility.

Your discussion of financial interests, professional relationships, and shared philosophical commitments deserves consideration. Such connections can create risks of undue influence — though their existence alone does not establish compromised judgment. The same care applies to the movement as a whole: the views of a minority should not be attributed to everyone in it. These distinctions matter, because credible governance depends on evidence, not assumptions about institutional motives.

I find your point on voluntary commitments particularly relevant. When commitments are walked back, it raises legitimate questions about their effectiveness. At the same time, judging whether an approach has failed means examining actual outcomes, compliance, and the alternatives available.

For me, meaningful human control must extend beyond assigning responsibility. Institutions must retain the ability to understand consequential risks, independently evaluate evidence, establish enforceable boundaries, and intervene when circumstances warrant. Without those capabilities, oversight exists in principle without functioning in practice.

Your recommendation for a permanent international prohibition raises a further question about how legitimate concerns become defensible policy. The risks of advanced AI deserve serious attention — but the choice between conditional restrictions and permanent prohibition should rest on clear evidence, measurable criteria, and an assessment of what can realistically be enforced. How would a human-centered framework distinguish between risks that require stronger oversight, those that warrant a temporary halt, and those sufficiently established to justify permanent prohibition?

Thank you for contributing a substantive perspective to this important discussion.

5 more comments...

Ready for more?