We present a plausible projection of where inadequate AI governance could lead at a societal scale. Each element is grounded in warnings already issued by named researchers, institutions, and governments – not speculation.
Systemic Financial Contagion
The financial system’s shared digital infrastructure – cloud services, payment networks, widely used software – creates conditions for correlated AI failures that no single institution can prevent unilaterally. The IMF’s May 2026 analysis explicitly identifies this as an emerging macro-financial threat: “confidence effects, payment disruptions, liquidity strains, and fire-sale dynamics could follow if multiple institutions are affected simultaneously.” Advanced AI cyber tools operating at machine speed mean that defenders must patch faster than attackers can exploit – a race that existing frameworks are not designed to win.
Without strong AI governance in financial institutions, algorithmic trading systems and automated credit decisioning tools create additional systemic fragility. When multiple institutions rely on similar AI models trained on similar data, they may behave similarly in crisis conditions – amplifying rather than dampening market stress.
Discriminatory Decisions at Industrial Scale
Without governance, AI-driven discrimination operates at a scale that no human bias could replicate. A single biased algorithm deployed in a mortgage underwriting system can affect hundreds of thousands of applications before it is identified. A biased hiring tool used across a staffing firm’s full client base can systematically exclude protected groups at national scale. The Bletchley Declaration – signed by 28 nations including the United States, China, Canada, and EU member states at the November 2023 AI Safety Summit – specifically identified “potential intentional misuse or unintended issues of control relating to alignment with human intent” as a shared global concern requiring urgent action.
Deepfake-Driven Market and Election Manipulation
Synthetic media – AI-generated audio, video, and images that are indistinguishable from authentic footage – represent a direct threat to the integrity of democratic processes and financial markets. A fabricated video of a CEO announcing fraud, or of a political candidate endorsing violence, can circulate globally within hours. The Bletchley Declaration noted “the potential for unforeseen risks stemming from the capability to manipulate content or generate deceptive content” as a critical concern requiring immediate attention. This is not theoretical: a single deepfake event in January 2024 cost Arup USD 25 million (see Chapter 5).
Autonomous Systems Beyond Human Control
As AI systems are granted greater autonomy – making and executing decisions without human authorization – the consequences of misalignment become harder to reverse. This is especially acute in defence and critical infrastructure contexts. The Bletchley Declaration explicitly names “cybersecurity and biotechnology” as domains of “especially urgent” concern from frontier AI systems. The Declaration acknowledges that “there is potential for serious, even catastrophic, harm, either deliberate or unintentional, stemming from the most significant capabilities of these AI models.”
Professor Yoshua Bengio – scientific director of Mila, a Canada CIFAR AI Chair, and co-chair of the International Scientific Report on the Safety of Advanced AI coordinated by the UK AI Security Institute – has warned repeatedly that AI systems capable of acting on long-range goals without adequate human oversight represent a category of risk requiring proactive, not reactive, governance.
Surveillance States and the Erosion of Civil Liberties
Ungoverned AI surveillance tools – facial recognition systems, predictive policing algorithms, social credit scoring – have the potential to enable levels of population monitoring previously impossible. The Dutch SyRI case (Chapter 5) demonstrates that governments will deploy such tools unless courts or laws stop them. Without international governance norms, surveillance technology that is banned in one jurisdiction can be deployed or exported elsewhere. The Stanford HAI 2026 AI Index notes that in the United States, only 31 percent of surveyed citizens trust their government to regulate AI effectively – the lowest level among surveyed countries.
The cumulative picture is one not of a single catastrophic event but of a gradual erosion: of trust in institutions, of fairness in consequential decisions, of economic stability, and of democratic accountability. That erosion is already underway. The question governance answers is whether it is reversed or accelerated.
