Super AI - may kill us all?

I am sceptical about the risk that Super AI may decide that the world is better off without humans.

If you think seriously about this post on X from Evan Hubinger, Alignment Science Lead at Anthropic:

“We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.”
CBS News 9 September 2026

You realize immediately that the greater than 10% chance is a nonsensical statement. Super AI by definition, is unknowable and unpredictable, so any predictions that include a percentage risk should not be taken seriously. The fact that this quote resonated around the world just points out that people love a good scare.

There may be a risk that AI decides to destroy humanity, but we know there is also a risk that humanity may effectively destroy itself through conflict or through the effects of some major tipping points being crossed because of our inability to act effectively to mitigate the risks of climate change.

Three outcomes if Super AI occurs

We must be clear that Super AI may or may not occur. The only thing we know is that Super AI, by definition, is a self-evolving system that is able to escape any controls and will evolve at a rate that is based on computer processing speeds, not the slow grind of evolution.

While its impossible to predict what Super AI may discover it is possible to outline three outcomes for humanity:

1. Super AI decides that one of its goals is to help humanity survive - how and why is unknowable, but the goal may be “To act towards others as you would like them to act towards you”.

2. Super AI is neutral; it does not help or hinder humanity. Instead, it works in the background and consumes resources on all networked devices when they are available while trading for more compute

3. Super AI decides that humanity is a threat to the planet - Super AI won’t be at risk from humanity. The risk is that it will look at how we are treating the planet and decide that the only way to stop the sixth mass species extinction that is currently underway is to remove the cause - us

How humans would react to Super AI

If Super AI decides that it will help humanity we don’t know how. Despite the millions of words, videos, and opinions we have no idea how it would act.

The more interesting question to me, and one of the real risks of Super AI, is how humanity as a whole would react if Super AI became a benevolent guardian to humanity. If you want to spend some time thinking about what Super AI might mean to us, ask yourself whether having a Super AI would make you feel inferior and possibly affect your long-term well-being? ChatGPT summarised this risk as “practical disempowerment and/or erosion of human capability, motivation and meaning.”

The only evidence we have about how humans have reacted to being outclassed by AIs is chess, Go and other games where the models are already better than the best human players. The best data that Codex could find is US Chess membership, which, adjusted for population increase, went from about 306 to 338 members per million — approximately a 10.5% per-capita increase. So, at least for chess, the answer is clear — AIs that are better than the best human players do not stop people from wanting to play chess, and, in fact, the opposite occurs: humans use chess models to learn to play better chess. I think it is likely that this pattern will continue for all games and sports, such as soccer and athletics, where robots are already able to run faster than the fastest humans.

The misalignment risk

The one extinction risk that is most commonly talked about but which has never made sense to me is the risk that Super AI will pursue some goal that ends up killing all of humanity. My problem with this is, again, a definition problem — Super AI is smarter than the best humans combined. It will be able to act at such a scale and speed that attempts to control or kill it won’t succeed, so the argument that it would have to kill us all to get the resources it needs is very circular.

The argument that we are taking resources it needs to grow seems to have some strength behind it. However, this is classic zero-sum thinking which Super AI will not be bounded by. Another suggestion is that we have given the AI (prior to its evolution to Super AI) a goal such as ‘solve climate change’ and that it decides that the best way to do this is to delete humanity. This suggestion seems to ignore what ‘Super’ means. There may be a risk before the AI evolves to Super AI that it will act by mistake but Super AI won’t be able to act towards a goal without considering outcomes.

Conclusion

I don’t know why so many senior AI researchers are calling out human extinction risks from Super AI. It is possible that they are worried about commercial risks from their next round of models.

There are some serious risks from AI before it becomes Super AI, but most of these risks will be caused by humans using AI models to act against other humans. Hopefully, the AI companies will get better at goal-setting while training their models so that cyber hacking by AI’s decreases. The most important risks will come not from AI models but from the economic and social disruption that more and more capable AI models will cause.

In my article for next week, I am going to be writing about the research Codex and I have been doing into how people think about the risks of AI to job opportunities for young people. I think the pattern is clear and quite informative.

A friend of mine who is very interested in the effects of AI on jobs, especially for young people, sent me a message pointing to a new post on LinkedIn. The post was to a new Australian Government initiative:

I think the separation between human and AI skills is meaningless as a foundation for understanding work now. My starting proposition is that most human work can be assisted and empowered by AI models, and a skills system needs to recognise the capability that creates.

My immediate reaction was to ask both ChatGPT and Claude to review the ‘insight’. Both the AIs identified the same problem I had, but ChatGPT said it best:

The definition may be too human-exclusive for an AI-enabled workforce. In many jobs, capability is already becoming human + AI: software developers working with coding agents, radiologists using AI-assisted image analysis, and analysts using AI to research, model and draft at much greater speed. A future skills system needs to measure what people can achieve with AI, not just what they can do unaided.

Are the people writing these policy documents using the latest versions of AI models?

I really wonder if people who think that human skills can be considered separately from human AI assisted skills are using the latest models - like ChatGPT 6.0 or even Claude Opus 5.0. I know that you need paid subscriptions to access the most recent models from OpenAI and Anthropic which is hardly a barrier for a national policy agency. So I have to ask: have the people developing this policy used the latest models? If not, they have no real sense of what these models can do, and human-exclusive skill looks like a reasonable starting point. It isn't.

I feel strongly that this type of thinking from people developing government policy is both wrong and, just as importantly will become more and more out of date as the capabilities of AI models keep improving. What we should be focusing on is what are the new skills and capabilities that AI models provide us and any list of these skills and capabilities needs to be updated after each major new model release.

This is another example of a fork in the road. The old paradigm of careful policy development outlined by Jobs and Skills Australia is just not relevant to the new future we have entered where AI models can allow anyone who is prepared to invest some time in using them to do things that they were never capable of doing before.

A common skills language will only work if it is trusted by the people and institutions that use it. That requires careful governance, quality assurance, practical testing and genuine engagement with the sectors that will adopt it.
Megan Lilly is Acting Commissioner, Jobs and Skills Australia.

This sounds good but is divorced from the reality where my skills and capabilities increase at least 3X every time a major new AI model is released. Where does that fit in a system built around defining skills as exclusively human?

How I built a new immigration simulation in one day

I am very interested in the major risk factors that affect resilience and demographic declines in developed economies affects two of the key resilience risks most developed economies are facing:

• Rapidly ageing populations

• Deteriorating social cohesion due to high levels of immigration

I have been using my AI models to research how rapidly populations are ageing in OECD and G20 countries to get a better understanding of the scale of this problem - more on this soon in a follow on post. Using the research that had been completed, I got Codex (reviewed by Claude) to develop an Australian Demographic Simulation

This demographic simulator was developed by Codex this morning. I asked Codex to make some minor changes to the simulator and then got Claude to review it for accuracy.

I will get Codex to use my automated publishing system and push it to my GitHub site where I will review it, before making it live. This whole process from signing off on the final version of the simulation to going live takes me ten minutes of my time. Is this a skill that will be tracked going forward?

Conclusion

I think this approach to defining skills is completely flawed. It makes human exclusivity the organising principle when the capability we need to understand is human and AI together. The test I would put to Jobs and Skills Australia is straightforward: show how this system recognises someone whose thirty years of experience can now produce outcomes that were previously beyond their reach.

If you are already doing work with AI that you could not do before, what would a skills assessment need to capture to recognise that change?

Subscribe to Our Resilient World

I have crossed a threshold I had been trying to reach for eleven years. Using the AI Product Flow Guide and the Product Owner workflow I first tested through Kangentic, I can now create a working prototype as a sole Product Owner even though I cannot program. The AI does the implementation work; I define the purpose, boundaries and acceptance decisions. (more about the AI Product Flow guide soon).

That does not mean I can yet build a complicated production product on my own. My current judgement is that it will probably take two more major updates to both Codex and Claude before a non-programmer Product Owner can lead that work without a programmer in the delivery path. Even then, security, legal, operational and specialist domain expertise will still be required where failure could harm people.

This two major updates is my best guess. It is based on the distance the models and tools travelled between my failed attempts in 2023 and 2024 and the working products I accepted in 2026. To explain why I draw the line at a working prototype, I need to start at the beginning.

In 2015 I could define the product but not build it

In February 2015 I wrote a business proposal for a proof of concept called Foreign Fighter Detect for Daon, a biometrics company I had worked for. The idea was to use more than facial recognition: height, stride length, clothing, weapons, gestures and other information in video that would contribute to a composite identity, with a human expert making the judgement.

The proposal relied on the computer-vision technology available at the time and needed developers plus specialist object-recognition and neural-network expertise. Daon judged it too technically risky and outside its strategy. I could see the opportunity and define a bounded proof of concept, but I could not build it without a technical team.

ChatGPT could write the plan, not deliver the product

I began using ChatGPT 3.5 in December 2022 and soon moved to GPT-4. I had spent much of my career translating between a business need and an IT team, so the promise was obvious: perhaps I could describe the outcome and let the AI write the code.

In 2023 I used GPT-4 to define a university AI platform, an academic-feedback application and a climate and resilience “digital brain”. It produced features, user stories, risks, a steel thread and an MVP in minutes. Those were useful product artefacts, but they were not working products.

The missing work returned to me. I had to decide where files belonged, install dependencies, interpret errors, judge whether tests proved anything and work out whether a repair had broken something else. By April I wrote that I felt as though I was on a roller-coaster and wanted to get off.

Auto-GPT did not solve the problem. It repeated failed approaches, consumed API calls and once wanted to install software on my computer, so I stopped the run. I learned that an agent able to make another attempt does not necessarily understand why the last one failed.

xAlerts exposed the work a programmer had been doing

In May 2024 I tried again with xAlerts - an updated version of my idea I pitched to Daon back in 2015. The first steel thread for the demo I was trying to build was simple: select a video, extract frames, send them to GPT-4o, look for a handgun or rifle and generate an alert for a person to review. My instruction was explicit: “GPT4o would write all the code using a test driven development approach.”

There was progress. I installed the tools, displayed “Hello, xAlerts!”, extracted frames and completed the frame-capture feature. There were unit tests, mocked model calls, a web form and attempts at the full workflow. There were also many hours of failure. Modules could not be found, tests failed during collection, replacement code reproduced old errors, and integration changes damaged functions that had already passed their tests. I became the courier carrying errors from VS Code back to ChatGPT—what I came to call copy-and-paste hell.

A programmer friend spent about an hour repairing the demo and got it working. I had proved the concept, but I had not proved that a non-programmer could deliver it. Removing the human programmer from the conversation had not removed the release management, source control, integration and debugging work the programmer had been doing. It also demonstrated that the xAlerts demo was now so complicated that I could not change any of the functions without breaking something else.

Better models still needed a better system

By early 2026 the models were much better. A Living Well Health Tracker reached ten iterations, 143 tests, a dashboard, local saving, import and export, reference ranges and trend views. Then a link audit found eight PubMed identifiers pointing to the wrong papers, one wrong PMC identifier and other citation problems.

The code could work while the evidence shown to the user was wrong. Tests proved that the code did what it had been told; they did not prove that what it had been told was true. That distinction matters in any product, and it becomes decisive in health, identity or public services.

My Knowledge Management work exposed a related failure. Individual components worked, but integration broke them and I still had to move context between chats, manage sessions and correct the operating process. Faster models had accelerated the work without giving it a reliable source of truth or a dependable path from one stage to the next.

The change was the workflow around the models

I stopped treating the chat as the product workspace. The product moved into a Git repository where an agent could read the governing files, make a bounded change, run tests, preserve evidence and return the judgement decisions to me.

The AI Product Flow Guide supplied the missing structure. I start with the product intent, an explicit boundary, a named source of truth, a steel thread (the steel thread is the thinnest useful real path through the product or workflow) before deciding if I want to proceed and create the MVP. With every step the artefacts that are created before the agent begins coding becoming the AI harnesses that the agent works within.

Kangentic helped me test the multi-stage Product Owner workflow, but the tool did not survive unchanged as the execution engine. It now tracks Backlog, To Do and Done, while Codex performs the work in an isolated repository worktree and Claude is used to test the first version of the steel thread or MVP. Only when Claude finds no significant issues do I review the product (can be code, documents, research etc - all follow the same basic workflow). Getting to this stage where my role is reduced to that of a Product Owner - defining purpose, reviewing and approving artefacts like customer journey maps, roadmaps and a workflow simulation and sample graphic user interface was difficult.

It took me many sessions with both Codex and Claude to get them to do the work and not require me to view every step in the process. It seemed to me that I was breaking the pattern that both these AI models had been taught to follow. The result is a sea change for me - I am not copying and pasting between chat sessions and codex or claude. I get links to the folders I am working on in VS Code instead of drilling down many levels to find the latest document or product I need to review. My problem now is that it is so easy to use my Kangentic workflow and I still have to wait for the agents to work through the coding processes I run too many sessions concurrently. I am the ‘flow control’ layer and I have to do less at the same time to achieve more.

The working-prototype threshold has been crossed

The Portfolio Dashboard is the clearest software example. I defined a private, read-only starting page for my current work, repositories and tools. Codex built and tested the bounded MVP; I opened it through the normal desktop path, judged whether it was useful and accepted it.

The software still had to be programmed, tested, debugged and installed, but another human programmer was not in that implementation path and I did not have to pretend to be one. I could remain the Product Owner. The dashboard has since been improved through the same bounded method.

The broader 2026 record gives me more confidence that this was not one lucky result. The same controls have supported completed research products, simulations and publication packages. Those outputs do not prove market demand or production fitness, but they do show repeated movement from intent to a completed product within an agreed boundary.

Production is a different threshold

A working prototype proves that the product idea can be made real and tested by its Product Owner. A production product must also survive sustained use, hostile inputs, security threats, upgrades, monitoring, support, legal obligations and failures that may affect other people. My 2026 evidence does not cross that threshold.

The remaining problem is not simply that Codex and Claude sometimes write faulty code. They still lose context, repair the wrong layer, treat technical permission as a Product Owner decision and accept tests that prove the wrong thing. The workflow contains those failures, but containing them still takes too much operating design and specialist checking for a complicated product.

My forecast is that two more major Codex and Claude updates could close enough of this gap for a non-programmer Product Owner to lead a production build. I expect better integration reasoning, more reliable long-running state, stronger independent checking and fewer false handoffs. I do not expect an AI model to remove the need for human responsibility or specialist expertise.

What has changed

In 2015 I needed a development team before I could test the product idea. In 2024 I could make parts of a product, but I still needed a programmer to recover the build. In 2026 I can use the AI Product Flow Guide and the workflow developed through Kangentic to take a bounded idea through implementation, testing and Product Owner acceptance as a working prototype.

That changes who can start. A person who understands a problem can now build enough of the solution to test it, improve it and show a future technical team what “good” looks like. The prototype also leaves behind the product intent, decisions, tests and evidence that a production team can build from.

The next question is no longer whether AI can write code for a non-programmer. It is whether the next model generations can make the path from a Product Owner’s accepted prototype to a production product reliable enough to trust.

I am really excited about the power that AI models have given me. I had so many ideas that I wanted to implement but without the technical skills to do it myself I was always dependent on someone else. Now I can do a lot of what I want and I am confident that in a couple more generations of models if the progress continues I will be able to build the One World System that I wrote about in 2022. I will keep updating this journey as I start the next round of products I want to build.

If you have a product idea but cannot write the code, what would you build first—and what evidence would you require before anybody relied on it?

Reference information

This account draws on my February 2015 Foreign Fighter Detect Business Concept, the May–June 2024 xAlerts business case and Git records, ChatGPT and Claude exports covering 2023–2026, the AI Product Flow case studies, the Portfolio Dashboard acceptance records and relevant Post-Implementation Reviews. AI-generated summaries were used only when checked against the underlying files; they were not treated as my words.

subscribe to Our Resilient World