Governing Generative AI Specifically – Why Generative AI Is Different

Traditional machine learning models are narrow in scope: a fraud detection model flags transactions; a credit scoring model outputs a risk score. Their failure modes, while consequential, are generally bounded and measurable.

Generative AI (GenAI) – large language models, image generators, code assistants, and multimodal systems – is different in kind. It produces open-ended outputs across almost any domain, with no fixed output space. This creates a qualitatively different risk profile: the same system that drafts a contract can also produce discriminatory content, generate plausible but false information, expose personal data from training sets, or assist in creating harmful materials. Standard model risk management frameworks were not designed for this.

NIST AI 600-1: The Definitive Risk Taxonomy for GenAI

The NIST AI 600-1 Generative AI Profile, published July 26, 2024, is the authoritative public-sector taxonomy of GenAI-specific risks. It identifies 12 categories of unique risk:

1. Confabulation – the generation of false or misleading content presented with apparent confidence (commonly called hallucination).
2. Dangerous or violent recommendations – content that provides instructions or encouragement for harmful activities.
3. Data privacy violations – exposure of personal information from training data or through inference from prompts.
4. Environmental impact – the significant energy and resource consumption of large model training and inference.
5. Human-AI configuration failures – risks arising from misunderstanding of AI capabilities, over-reliance, or inappropriate deployment contexts.
6. Information integrity failures – generation of misleading, biased, or manipulated content at scale.
7. Information security failures – vulnerabilities introduced by AI components, including prompt injection and model compromise.
8. Intellectual property infringement – generation of content that reproduces copyrighted material or trade secrets.
9. Obscene, degrading, or abusive content – generation of harmful content targeted at individuals or groups.
10. Toxicity and bias – outputs that reflect or amplify harmful stereotypes or discriminatory patterns.
11. Value chain and component integration failures – risks from dependencies on third-party models, datasets, and APIs.
12. Overreliance – users treating AI outputs as authoritative without appropriate verification.

OWASP Top 10 for LLM Applications

The OWASP Top 10 for Large Language Model Applications (2025 edition) provides the practitioner’s security risk taxonomy for GenAI deployments. The ten risks, relevant for organizations deploying LLM-based tools:

1. Prompt Injection – manipulation of model behavior through crafted inputs.
2. Sensitive Information Disclosure – exposure of confidential data through model outputs.
3. Supply Chain Vulnerabilities – risks from compromised training data, models, or dependencies.
4. Data and Model Poisoning – tampered training data leading to compromised model behavior.
5. Improper Output Handling – insufficient validation of LLM outputs before use in downstream systems.
6. Excessive Agency – LLM systems granted excessive autonomy to take actions without appropriate human oversight.
7. System Prompt Leakage – disclosure of confidential system-level instructions through model outputs.
8. Vector and Embedding Weaknesses – security vulnerabilities in retrieval-augmented generation (RAG) systems.
9. Misinformation – hallucinated or false outputs that users treat as reliable.
10. Unbounded Consumption – resource exhaustion attacks that degrade AI system availability.

Shadow AI: The Unmanaged GenAI Problem

Employees are using GenAI tools whether or not organizations have approved them. According to Cyberhaven’s Shadow AI analysis, 91 percent of AI tools used inside companies are completely unmanaged, and 75 percent of employees now bring their own AI (BYOAI) into work – a figure consistent with Microsoft’s Work Trend Index findings cited by Cyberhaven. The IBM Cost of a Data Breach Report 2025 found that 20 percent of organizations studied suffered a breach partly due to shadow AI, and that 97 percent of organizations experiencing AI-related breaches had no proper AI access controls in place.

The Samsung case is the canonical example: Samsung engineers pasted proprietary source code into ChatGPT for debugging assistance, inadvertently transmitting confidential intellectual property to OpenAI’s external systems. This incident – widely cited across security literature – illustrates that shadow AI exposure requires no malicious intent and can affect sophisticated, security-conscious organizations.

Copyright and IP Exposure

GenAI’s training data provenance creates ongoing copyright litigation globally. The most significant active case in North America is The New York Times v. OpenAI, which involves allegations that OpenAI’s models were trained on Times content without license and can reproduce that content in ways that harm the publication’s market. As of May 2025, a court-ordered preservation of all ChatGPT conversation logs – affecting over 400 million users globally – has added significant enterprise data governance complexity.

In the UK, the High Court’s November 2025 ruling in Getty Images v. Stability AI found that Stable Diffusion model weights are not themselves “infringing copies” of training images – a partial victory for AI developers – while finding that trademark infringement could arise from watermark reproduction in generated outputs. Both cases remain significant for organizations using GenAI tools that may have been trained on copyrighted material.

Hallucination Management in Practice

Confabulation (hallucination) is the most operationally pervasive GenAI risk for most organizations. Governance practices that materially reduce hallucination risk include:

Retrieval-Augmented Generation (RAG). Rather than relying solely on a model’s trained knowledge, RAG systems retrieve current, organization-specific documents at inference time and ground model outputs in verified sources. This significantly reduces confabulation in enterprise knowledge management and customer service contexts.

Citation grounding. Requiring models to cite the specific source passage supporting each claim, and displaying those citations to users, enables human verification and reduces uncritical acceptance of false outputs.

Human review thresholds. Defining which GenAI outputs require human review before action – particularly in regulated contexts such as legal advice, medical recommendations, or financial disclosures – and building those review steps into workflows as mandatory gates rather than optional steps.