The effective altruist and longtermist crowd are into it as a religion. They would happily sacrifice anything human to satisfy their dreams of AI. They care way more about what the AI needs than their what their fellow humans need. We will end up with an AGI having better living and working conditions than humans and the whole AI industry will be clapping
It's so weird how distorted your view of any of these concepts are from what these people actually say. I don't think you've ever seriously looked into either group if you believe any of their views support what you say. In fact, many EAs and "longetermists" are explicitly AGAINST the development of AGI.
If you say so. I’ve been following the development of both groups since almost a decade, I’m pretty confident I understand their concepts and what values they stand for
Please explain an EA position that could possibly suggest that they would deprioritize human lives in order to make AGI more comfortable when, for example, the largest EA philanthropy organization, Coefficient Giving, formerly Open Philanthropy, has given millions to Yudkowksy's MIRI which is currently among the most anti-AGI organizations on the planet?
Can you find any explicitly EA organization that is at all aligned in perspective with anything you suggested?
Open Philantrophy was founded by Holden Karnofsky who was Dario Armodei's roommate and is married to his sister Daniela Armodei. He now works for Anthropic, a company that is explicitly building AGI.
Working to build AI at the company that is most publicly discussing the risks posed by AI to humanity and most advocating for guardrails (while getting an immense amount of flack for implementing them independently) doesn't seem to support anything about what you, or rather the other person said, about sacrificing humanity for anything AI related.
I don't really agree with the other commenter, I don't think EAs are intentionally sacrificing humanity for AI, but the fact it's unintentional doesn't really change the fact this is what they are doing.
Anthropic are explicitly building ASI, using recursive self improvement, with no knowledge of how to control it. They spend only a minimal fraction of their funds on safety work, none of which relates to controlling ASI.
I am open to being wrong about Anthropic, but it will be hard to convince me that a company developing ASI is prioritising safety. Do you really believe what they are doing is safe?
I truly don't see any major problem with what they're doing. For any current significant risk (security, bio risks), their models are notoriously useless.
This is an odd take. OpenAI models you use are pretty useless for hacking. The models they have seem to be pretty damned good at it.
It's kind of like saying you can't kill everyone with weapons when you're looking at what you can buy at the local Walmart and not what's in the 1000 foot deep missile silo.
Also, it's kind of a once you build it, it's too late kind of issue. No model will be good enough, the problem is still in the future, we're all dead.
They are not stopping. Simple as that. The quadrilles and regulations they ask are all about slowing down competition but not them.
Hiring own husband to safety team while you raise money for (their words!) doom machine, while you do your utmost to scare people while you try to raise even more money from investors on doom ... does not show OP was not correct in their assessment.
Effective altruism may have had a positive reputation at some point, but it's definitely jumped the shark and almost everyone influential who invokes in the context of AI it seems to have fallen into the trap of the "bottomless pit of suffering" of utilitarianism.
Just like the utilitarians in the thought experiment tossing everything positive in their life into the bottomless pit of suffering, the effective altruist AI thought leaders are tossing our entire economy to the gambit that the singularity will bring about the end of human suffering.
The Effective Altruism(tm) believers are part of the accelerationist. See SBF, the Amodei Siblings, Karnofsky (spouse of Daniela Amodei), and people around them
All of the effective altruists I know have started talking about personhood for AI, which is one of Jacob Tsimerman’s more plausible omnicide scenarios (not that any are that plausible give his timelines look kind of ridiculous).
Effective altruism is a grift that justifies “get as much money as possible for the individual” so that “you can then so something good with it”. Aha! OK bro you’re saying you should have all the things huh? Anyone into this shit should be immediately distrusted.
It's an ideological thing. (We just had a conversation about ideology actually, remember?)
Capitalism has always been a tension between capitalists, who want maximum return on capital and workers, who want pesky things like a living wage, sick leave or safe work conditions.
For the longest time, the only power labor has had to get those things was the power of collective bargaining. No agreement with your workers meant no production happened.
Now, there's finally a chance at salvation. The capitalist Messiah is AGI and it will finally deliver them from those annoying laborers.
This ideological bent is why they're putting everything they have into AI. It's why VCs and their fellow capitalists are going so crazy.
They see it as a way to finally solve the contradictions of their ideology, but in reality it'd only create a new one: If everyone's out of a job, who will buy their products?
If they had AGI, why would they need consumers anymore? Just use your drone army to secure all natural resources and produce whatever you need for yourself. Other humans are just pets.
I'm reminded of the scene in Independence Day with the people on top of the tower welcoming the alien spaceships. Except instead of celebratory fanatics holding poster boards saying "Welcome", it's all the singularity AI thoughtleaders holding hands in a circle chanting about Roko's basilisk like some new-age spirituality retreat.
I have seen that current AI models are capable of doing a sloppy job at many tasks when directed to do so, but I have never seen an example of a machine having its own volition.
It's not a choice we have, unless you want to figure out a global AI ban treaty.
AI will make human thought economically useless. In the new economy, the one economically useful thing we will be able to do is legal property ownership.
No. AI as a tech is not an issue. The wealth and power accumulation in the hands of Amodei, Altman, Thiel, Musk, Karp and the rest of CEO is the issue.
Permanent underclass is ideological goal of the above people. AI is an impersonal technology with no goals or wishes.
I'm not sure how that changes anything -- sure, if you blocked people from using AI, it wouldn't have any effects on society, but once people have access, what they do with it matters.
There's an inevitable effect that it has, and the only lever you have as an individual is being on the right side of that.
Copy the technology and give people access without trying to financially profit from it.
Even if the US federal government refuses to act, smaller government entities can deploy their own compute clusters running strong open-weights models like Kimi K3, and provide their own citizens with free access to strong AI models. There may come a time when even Chinese providers no longer release strong model weights, but then the previous generation of strong open-weights models can be used to accelerate the development of data pipelines and training setups to build the next generation. California or Australia (or many others) could fund university-led R&D efforts to build completely open models.
Despite the angry noises that leading providers make about their model outputs being used to train competing models, that's not illegal. States can use Common Crawl, Anna's Archive, Books3, commercial model outputs, and whatever other data sources that are useful for building models. States have sovereign immunity to ignore copyright complaints, if they decide to pursue model development.
The big AI companies don't have much of a moat. Let the big companies spend a fortune figuring out what works best, then openly build imitations having 95% of the capability at 5% of the cost.
If Anthropic is the first to develop AGI, then everyone else becomes subordinate to Dario Amodei? And if DeepSeek is the first to get there, everyone instead becomes subordinate to Liang Wenfeng? I cannot believe that there is a threshold of intelligence where getting there first means a decisive and durable advantage. If you do believe that getting there first means ruling the world, then our viewpoints are probably too far apart for mutual understanding.
It's not so much whether we believe this, it is about whether they and enough of their supporters believe it. Because the effect of believing that is what drives their need to 'be the first' at the expense of everything else.
It depends what you mean by 'subordinate' -- but even a small but persistent advantage in decision making compounds, so I can't imagine that someone getting that advantage doesn't use it to kill their competition.
The problem is not access to AI. The problem is the lack of antitrust, the lack of white collar crime enforcement, the tax system that allows them to accumulate wealth while avoiding payments. The problem is legal regime with winner takes all monopoly creation support.
What kind of political leverage do the useless have? This isn't a realistically sustainable outcome.
Today, people have leverage because removing them has real economic consequences. With AI, we need to be on the useful side of the divide to have any leverage.
And then we'd better hope AI has respect for property rights.
Apathetic mass-murder is unjustifiable, and the ultrarich would need to justify it to their families, who would then need to justify it to other people they know, and so on. And remember that we're talking about the majority of the global population, an event like that couldn't be hand-waved away like one would do for a minority group.
Iran just mass murdered their protesters earlier this year, and they needed to convince humans to pull the trigger. It didn't seem to be that hard for them to justify.
It's probably a lot easier to convince the military AIs we're building to do it.
I don't particularly like the situation, but I plan to do what I can to be on the winning side. The world is going to be what we're all in on building, and for people on the winning side it may be a utopia. Either way, I know which side of the gun I want to be standing behind.
Also note that an AI well aligned with the people who initially trained it wouldn't need to actually kill anyone that didn't take action against the owners, it would just need to outcompete people for resources, and let them fend for themselves in extreme poverty. There's no need to act one way or the other beyond that.
Not comparable at all. The Iranian protesters are/were a minority group within a country that itself makes up 1.1% of humanity.
I'm talking about morality here, not practical ability. The ultrarich have their circle of friends and family, who themselves have their own social circle, on and on until suddenly you're not looking at wealthy people anymore. Six degrees of separation and all that. How many people in the social graph can know about this extermination plan before the whole thing collapses?
Why would they take action to exterminate anyone, when they can simply ignore them and let them live and die in extreme poverty, unable to participate in the economy? The west already accepts this kind of poverty with the sweatshop workers, and nobody bats an eye.
It's just disenfranchisement. Only the people who try to use violence to replace the political leverage they lose when they lose economic leverage would need any action taken against them, and that's easy to justify as self defense.
Because 7.5 billion people with nothing to lose would quickly turn chaotic and mass-murdery. 90% of humanity in idle poverty is impossible to justify, at least today they do something of value that can be used as a retort by the morally unscrupulous. A black-and-white situation where a few get everything, most get nothing and do nothing for anybody else is beyond justifying, so the self-defense argument wouldn't work.
It's already been here for much of the 20th century, and everyone seems to have just accepted it. We're just going to have Europe and America join sub-saharan Africa.
The moral justification for the existence of poverty is one of purpose and worth through contribution to society. The family member of an ultrarich baron can sleep at night mostly due to the idea that those who are rich contribute value to humanity as a whole, ergo those who are poor simply aren't working hard enough or contributing in a meaningful way ("just world fallacy" etc)
When AGI becomes reality, that reasoning disappears. When everyone's worthless, no one is. Thus the scenario you speculate about would need everyone in the upper echelons of society to be consciously, openly sadistic power-hungry people, which is not the case even when taking into account the higher dark triad rates among them.
That was the exact line of thought, coupled with resources are limited (marketed as climate change), that led to the 1 child policy in China and many other decisions that shall not be spoken of.
> Capitalism has always been a tension between capitalists, who want maximum return on capital and workers, who want pesky things like a living wage, sick leave or safe work conditions.
The underlying purpose of AI is to allow wealth to access skill while removing from the skilled the ability to access wealth. (read on HN, not my words)
