As governments worldwide grapple with artificial intelligence regulation, a growing chorus of researchers is challenging the scientific credibility of the numerical risk assessments that increasingly shape policy decisions. The debate centers on whether civilization-ending AI scenarios deserve the policy priority they have been receiving, and whether the probabilities used to justify that urgency hold up to scrutiny.
According to AI Snake Oil, a critical analysis of AI risk quantification, the field has constructed what amounts to a false sense of precision. When researchers cite specific probabilities for AI-induced extinction events, those figures often emerge from intuitive judgments rather than validated forecasting methods that have been tested against historical data. The distinction matters enormously for policymakers tasked with allocating finite resources across competing priorities.
The Problem With Numerical Authority
Numbers possess psychological power. When an AI safety researcher claims a 10 percent chance of human extinction from artificial intelligence within decades, the specificity lends weight that qualitative warnings do not receive. Yet this numerical veneer can obscure the underlying methodological weakness.
The core issue is straightforward: most AI existential risk forecasts lack the grounded methodology that makes probability estimates useful in other domains. Weather predictions, election forecasting, and sports analytics all rest on historical datasets, validated models, and repeated testing. AI extinction probabilities typically rest on neither.
"Probabilities are usually derived from some grounded method, so we have a strong cognitive bias to view quantified risk estimates as more valid than qualitative ones. But it is possible for probabilities to be nothing more than guesses."
Democratic Accountability and Costly Policies

The legitimacy concern extends beyond scientific rigor into democratic governance. When governments restrict beneficial technologies or allocate massive resources based on speculative risks, citizens deserve justification grounded in more than expert intuition. This becomes particularly acute when policies impose asymmetric costs.
Consider proposals to restrict open-source AI model releases. Such policies would benefit some stakeholders while imposing costs on researchers, companies, and developers who advocate for open access. Democratic principles require that government action limiting freedom be justified through explanation that reasonable people can evaluate and potentially reject, not merely asserted through probabilistic claims that lack methodological transparency.
Not Against Quantification, But For Rigor
Critics emphasize they are not opposing quantitative approaches to AI safety more broadly. Legitimate forecasting efforts, including structured expert elicitation projects, represent valuable methodological work. The objection targets a specific failure: the presentation of AI existential risk probabilities as though they emerged from the same rigorous processes that generate reliable forecasts in other domains.
- Most AI x-risk estimates lack validated models or historical testing
- Numerical precision can obscure underlying uncertainty and assumption
- Policy decisions based on such estimates warrant heightened democratic scrutiny
- Alternative frameworks for thinking about AI safety may deserve greater attention
The challenge now is whether the field can maintain focus on genuine safety challenges while being more transparent about the speculative nature of extinction-scale probability estimates. Doing so would preserve credibility for AI safety advocacy while grounding policy discussions in clearer reasoning about what we actually know and can validate.



