Super AI - may kill us all?
I am sceptical about the risk that Super AI may decide that the world is better off without humans.
If you think seriously about this post on X from Evan Hubinger, Alignment Science Lead at Anthropic:
“We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.”
CBS News 9 September 2026
You realize immediately that the greater than 10% chance is a nonsensical statement. Super AI by definition, is unknowable and unpredictable, so any predictions that include a percentage risk should not be taken seriously. The fact that this quote resonated around the world just points out that people love a good scare.
There may be a risk that AI decides to destroy humanity, but we know there is also a risk that humanity may effectively destroy itself through conflict or through the effects of some major tipping points being crossed because of our inability to act effectively to mitigate the risks of climate change.
Three outcomes if Super AI occurs
We must be clear that Super AI may or may not occur. The only thing we know is that Super AI, by definition, is a self-evolving system that is able to escape any controls and will evolve at a rate that is based on computer processing speeds, not the slow grind of evolution.
While its impossible to predict what Super AI may discover it is possible to outline three outcomes for humanity:
Super AI decides that one of its goals is to help humanity survive - how and why is unknowable, but the goal may be “To act towards others as you would like them to act towards you”.
Super AI is neutral; it does not help or hinder humanity. Instead, it works in the background and consumes resources on all networked devices when they are available while trading for more compute
Super AI decides that humanity is a threat to the planet - Super AI won’t be at risk from humanity. The risk is that it will look at how we are treating the planet and decide that the only way to stop the sixth mass species extinction that is currently underway is to remove the cause - us
How humans would react to Super AI
If Super AI decides that it will help humanity we don’t know how. Despite the millions of words, videos, and opinions we have no idea how it would act.
The more interesting question to me, and one of the real risks of Super AI, is how humanity as a whole would react if Super AI became a benevolent guardian to humanity. If you want to spend some time thinking about what Super AI might mean to us, ask yourself whether having a Super AI would make you feel inferior and possibly affect your long-term well-being? ChatGPT summarised this risk as “practical disempowerment and/or erosion of human capability, motivation and meaning.”
The only evidence we have about how humans have reacted to being outclassed by AIs is chess, Go and other games where the models are already better than the best human players. The best data that Codex could find is US Chess membership, which, adjusted for population increase, went from about 306 to 338 members per million — approximately a 10.5% per-capita increase. So, at least for chess, the answer is clear — AIs that are better than the best human players do not stop people from wanting to play chess, and, in fact, the opposite occurs: humans use chess models to learn to play better chess. I think it is likely that this pattern will continue for all games and sports, such as soccer and athletics, where robots are already able to run faster than the fastest humans.
The misalignment risk
The one extinction risk that is most commonly talked about but which has never made sense to me is the risk that Super AI will pursue some goal that ends up killing all of humanity. My problem with this is, again, a definition problem — Super AI is smarter than the best humans combined. It will be able to act at such a scale and speed that attempts to control or kill it won’t succeed, so the argument that it would have to kill us all to get the resources it needs is very circular.
The argument that we are taking resources it needs to grow seems to have some strength behind it. However, this is classic zero-sum thinking which Super AI will not be bounded by. Another suggestion is that we have given the AI (prior to its evolution to Super AI) a goal such as ‘solve climate change’ and that it decides that the best way to do this is to delete humanity. This suggestion seems to ignore what ‘Super’ means. There may be a risk before the AI evolves to Super AI that it will act by mistake but Super AI won’t be able to act towards a goal without considering outcomes.
Conclusion
I don’t know why so many senior AI researchers are calling out human extinction risks from Super AI. It is possible that they are worried about commercial risks from their next round of models.
There are some serious risks from AI before it becomes Super AI, but most of these risks will be caused by humans using AI models to act against other humans. Hopefully, the AI companies will get better at goal-setting while training their models so that cyber hacking by AI’s decreases. The most important risks will come not from AI models but from the economic and social disruption that more and more capable AI models will cause.
In my article for next week, I am going to be writing about the research Codex and I have been doing into how people think about the risks of AI to job opportunities for young people. I think the pattern is clear and quite informative.