AI Governance and Open Problems

"Multimodal models see, hear, and read. They still cannot fold laundry, but give them time."

Sage, Laundry Patient AI Agent

1. Compute Governance

You cannot regulate an algorithm. You can barely regulate data. But you can regulate a warehouse full of GPUs that consumes as much electricity as a small town. Among the many possible levers for governing AI, compute governance has emerged as one of the most concrete and enforceable. Unlike data (which is easily copied and distributed) or algorithms (which can be published in a paper), compute is physical: it requires specific hardware, significant electricity, and substantial capital. This physicality makes it trackable and, in principle, regulable.

The Logic of Compute Governance

Heim (2024) at the Centre for the Governance of AI articulated the core argument for compute governance. Training a frontier model requires a compute cluster that costs hundreds of millions of dollars and consumes megawatts of power. These clusters are built from a small number of chip types (primarily NVIDIA H100/H200/B200 GPUs), manufactured by a small number of companies, using equipment from an even smaller number of suppliers (primarily ASML for advanced lithography). This concentrated supply chain creates natural chokepoints for governance.

The United States has already leveraged compute governance through export controls on advanced AI chips to China, implemented through the Bureau of Industry and Security (BIS) starting in October 2022 and progressively tightened. These controls restrict the sale of chips above a certain performance threshold (measured in TOPS, or tera-operations per second) and restrict the sale of the equipment needed to manufacture such chips.

Know-Your-Customer for Compute

A more targeted proposal is "know-your-customer" (KYC) requirements for cloud compute providers, analogous to KYC rules in banking. Under this framework, cloud providers offering GPU clusters above a certain size would be required to verify the identity of customers and the intended use of the compute. This would not restrict access but would create an audit trail, enabling detection of unauthorized frontier training runs.

Proponents argue that KYC for compute is minimally invasive: it does not restrict what you can compute, only requires that you identify yourself. Critics counter that it creates a surveillance infrastructure that could be abused, that it disadvantages academic researchers and startups relative to incumbents, and that determined actors can circumvent it by distributing compute across multiple providers or jurisdictions.

Limitations of Compute Governance

Compute governance has important limitations. First, it governs training, not deployment: once a model is trained, it can be copied and deployed on much less compute. Export controls on training chips do not prevent the use of models trained elsewhere. Second, the compute threshold for "frontier" models is a moving target; algorithmic improvements (better architectures, more efficient training) continuously lower the compute required to achieve a given capability level. Third, compute governance is geopolitically fraught: it is perceived by some nations as a tool of technology containment rather than safety governance, which complicates international cooperation.

2. International Regulatory Landscape

As of early 2026, the global AI regulatory landscape is fragmented, with different jurisdictions pursuing substantially different approaches. Understanding these differences is essential for any organization deploying AI internationally.

The European Union: AI Act

The EU AI Act, which entered into force in August 2024 with provisions phasing in through 2026, represents the most comprehensive AI-specific regulation globally. It classifies AI systems into risk categories:

• Unacceptable risk (banned): social scoring, real-time biometric surveillance in public spaces (with narrow exceptions), manipulation of vulnerable groups.
• High risk (heavy regulation): AI in critical infrastructure, education, employment, law enforcement, migration, and democratic processes. These systems require conformity assessments, human oversight mechanisms, detailed documentation, and post-market monitoring.
• Limited risk (transparency requirements): chatbots, deepfakes, and emotion recognition systems must disclose their AI nature to users.
• Minimal risk (no specific requirements): spam filters, AI-powered video games, and similar low-stakes applications.

The AI Act also includes specific provisions for "general-purpose AI models" (GPAIs), including frontier LLMs. Models above a compute threshold (10^25 FLOPS for training) face additional requirements: adversarial testing, incident reporting, cybersecurity protections, and energy consumption disclosure. Open-source models receive some exemptions, though the scope of these exemptions has been contentious.

The United States: Executive Orders and Sector-Specific Regulation

The US approach has been less centralized than the EU's. The Biden administration's Executive Order 14110 on AI Safety (October 2023) established reporting requirements for frontier training runs (above approximately 10^26 FLOPS) and directed federal agencies to develop sector-specific AI guidelines. Subsequent legislative proposals have varied widely, from narrow bills addressing deepfakes in elections to broad proposals for AI licensing regimes.

The regulatory direction shifted with the change in administration in January 2025. The focus moved from preemptive regulation toward innovation-friendly policies, with more emphasis on voluntary commitments by industry and less on mandatory compliance. The long-term trajectory of US AI regulation remains uncertain, as bipartisan support exists for some measures (deepfake labeling, child safety) but not others (frontier model licensing, compute reporting).

China: Targeted Regulation

China has adopted a more targeted approach, regulating specific AI applications rather than AI broadly. Regulations have been issued for recommendation algorithms (2022), deepfakes and synthetic content (2023), and generative AI services (2023). The generative AI regulation requires providers to obtain government approval before launching public-facing services, to ensure training data complies with Chinese law, and to prevent generated content from "subverting state power" or "undermining national unity."

China's approach is notable for its speed (regulations are issued and enforced quickly relative to the EU's multi-year legislative process) and its focus on content control alongside safety and reliability.

The UK: Evaluation and Soft Governance

The UK established the AI Safety Institute (AISI) in November 2023, positioning itself as a hub for frontier model evaluation rather than top-down regulation. AISI conducts pre-deployment evaluations of frontier models from leading labs and publishes safety assessments. The UK approach emphasizes voluntary cooperation with labs, technical evaluation capacity, and international coordination, while avoiding prescriptive regulation that might drive AI development to other jurisdictions.

3. The Open-Weight Debate

The regulatory frameworks above govern how models are deployed, but they do not fully address a prior question: should powerful models be publicly available at all? One of the most contentious governance questions is whether frontier AI models should be released with open weights (allowing anyone to download, modify, and deploy the model). The debate has two clearly articulated sides, and the resolution has significant implications for the structure of the AI industry.

The Case for Open Weights

• Safety through transparency. Open models can be scrutinized by independent researchers for biases, vulnerabilities, and alignment failures. Closed models must be evaluated through their API, which provides limited visibility.
• Competition and innovation. Open models prevent concentration of AI capabilities in a small number of companies. They enable startups, academics, and researchers in developing countries to participate in AI development.
• Reproducibility. Scientific claims about AI capabilities, safety properties, and alignment techniques can only be verified if the models are available for independent testing.
• Resilience. A world where AI capabilities are distributed across many actors is more resilient to capture by any single entity (corporate or governmental).

The Case Against Unrestricted Open Weights

• Proliferation risk. Once a model's weights are released, they cannot be recalled. If a model has dangerous capabilities (e.g., assisting with bioweapon synthesis), those capabilities are permanently available to all actors, including malicious ones.
• Safety guardrails are removable. Fine-tuning can remove safety training from an open-weight model with modest compute, as demonstrated by multiple research groups. Open weights make safety guardrails a suggestion, not a constraint.
• Liability gaps. When a model is released open-source and then modified by a third party, liability for harmful outputs is unclear. The original developer, the modifier, and the deployer may each claim that harm is the others' responsibility.
• Competitive pressure. If frontier labs release open models, they face pressure to release ever-more-capable models openly to maintain community goodwill, even as capabilities reach levels where open release may be genuinely risky.

The practical reality is that the industry has settled on a spectrum. Models below a certain capability level (roughly GPT-3.5 class) are widely available as open weights. Frontier models are generally closed, with some exceptions (Meta's Llama series). The question is where the threshold should be, who should set it, and whether it should be a hard legal boundary or a soft norm.

4. Frontier Model Evaluations

A growing consensus (across both open and closed model advocates) is that frontier models should undergo rigorous pre-deployment evaluation for dangerous capabilities. The question is: what should be evaluated, by whom, and with what consequences?

What to Evaluate

The UK AISI and major labs have converged on a set of "dangerous capability evaluations" (evals) that test whether a model can:

• Assist with the synthesis of biological, chemical, or radiological weapons
• Generate persuasive disinformation at scale
• Conduct autonomous cyberattacks (discovering vulnerabilities, writing exploits, establishing persistence)
• Assist with surveillance, social engineering, or manipulation
• Engage in autonomous self-replication or resource acquisition
• Deceive evaluators about its capabilities or intentions

These evaluations are distinct from standard capability benchmarks (MMLU, HumanEval, etc.) in that they specifically target misuse scenarios. They require adversarial testing methodologies, often including red-teaming by domain experts (biosecurity specialists, offensive security researchers).

ARC-AGI and Capability Benchmarks as Governance Tools

Francois Chollet's ARC-AGI benchmark has gained attention as a potential governance tool because it attempts to measure general reasoning ability rather than narrow task performance. The idea is that a model's ARC-AGI score could serve as a proxy for its "general capability level," which could then trigger regulatory requirements above certain thresholds.

The challenges with using benchmarks as governance tools are substantial. Benchmarks can be gamed (trained on directly or indirectly). They measure specific capabilities, not general dangerousness. A model could score poorly on ARC-AGI but excel at assisting with specific dangerous tasks, or vice versa. And any fixed benchmark will eventually saturate as models improve, requiring constant revision.

Who Should Evaluate

Three models for evaluation governance have emerged:

1. Self-evaluation by labs. Major labs conduct their own safety evaluations. This is the current default. The concern is obvious: the entity with the strongest incentive to deploy a model is also judging whether it is safe to deploy.
2. Government evaluation. Institutions like the UK AISI or a US equivalent evaluate models before deployment. This provides independence but requires government agencies to maintain frontier technical capabilities, which is challenging given talent competition with industry.
3. Independent third-party evaluation. Analogous to financial auditing, independent organizations evaluate models against agreed standards. This is the model proposed by several governance researchers, though no such institution yet operates at scale.

5. Open Problems in AI Governance

Several governance questions remain genuinely unresolved:

Dual-Use Capabilities

Almost all AI capabilities are dual-use: the same model that helps a doctor diagnose rare diseases can help an attacker craft convincing phishing emails. Governance regimes that restrict dangerous uses must avoid crippling beneficial ones. No jurisdiction has found a satisfactory solution to this tension. The challenge is compounded by the fact that "danger" is context-dependent: a model capability that is safe in one context (a cybersecurity professional testing defenses) is dangerous in another (a criminal exploiting vulnerabilities).

Liability for AI Outputs

When an AI system causes harm, who is liable? The model developer, the deployer, the user who crafted the prompt, or some combination? Existing legal frameworks (product liability, negligence, contract law) do not map cleanly onto AI systems, which are neither traditional products nor traditional services. The EU AI Act assigns primary responsibility to deployers (for high-risk systems), but the boundaries are contested and untested in court.

Training Data IP

The legal status of using copyrighted material for AI training remains unresolved in most jurisdictions. The US courts are hearing multiple cases (New York Times v. OpenAI, Getty Images v. Stability AI, and others). The EU AI Act requires transparency about training data but does not clearly resolve the copyright question. Japan has taken a permissive stance, while the UK has oscillated between positions. The outcome of these legal battles will significantly shape the economics of AI development.

Autonomous Systems

As AI agents become more capable (see Chapter 22), governance must address systems that take actions in the world with limited human oversight. Autonomous trading systems, self-driving vehicles, and AI-powered hiring tools all raise questions about accountability, liability, and the appropriate level of human control. The challenge intensifies as AI agents are chained together in multi-agent systems (Chapter 24), where the behavior of the system emerges from interactions between agents and may not be predictable from the behavior of any individual agent.

Exercises