For most enterprises today, the primary source of AI risk is not internal AI development. It is the AI embedded in the software they have already purchased. Microsoft 365 Copilot, Salesforce Einstein, ServiceNow’s AI features, HR platforms with automated screening, and financial tools with embedded prediction models all represent AI deployments that most organizations made by clicking “enable” in existing SaaS subscriptions – not by building systems from scratch.
This matters for governance because the organization deploying the tool – the “deployer” in EU AI Act terminology – bears obligations regardless of whether the underlying AI was built externally. Risk enters through the supply chain; governance accountability does not transfer with it.
EU AI Act: Provider vs. Deployer Obligations
The EU AI Act draws a clear legal distinction between providers (entities that develop AI systems and place them on the market) and deployers (entities that use AI systems in their operations). Under Article 26 of the EU AI Act, deployers of high-risk AI systems must:
- Assign appropriate human oversight measures when deploying the system.
- Monitor system operation and report serious incidents to the provider and relevant authorities.
- Inform individuals when they are subject to a high-risk AI system’s outputs.
- Maintain logs of the system’s operation, where technically possible.
Deployers cannot simply assert that a vendor-supplied AI system is the provider’s responsibility. Latham and Watkins’ analysis notes that if a deployer white-labels or substantially modifies a third-party AI system, it may assume provider-level obligations – a critical consideration for organizations that rebrand or customize AI tools for their own use.
NIST AI RMF on Supply Chain Risk
The NIST AI Risk Management Framework addresses AI supply chain risk through its Map and Manage functions. Organizations are expected to identify third-party AI components in their systems, assess the risk management practices of AI vendors, and maintain documentation of third-party model provenance. This is not a compliance aspiration – it is a practical operational requirement. An organization that cannot explain where its AI model’s training data came from, or what the model’s known failure modes are, cannot govern the system responsibly.
Vendor Due Diligence: What to Ask
A robust AI vendor due diligence questionnaire – adapted from elements of the Cloud Security Alliance’s AI Controls Matrix (AICM), released in July 2025 with 243 control objectives across 18 security domains – should address:
| Element | What to Request |
| Model documentation | Model card or equivalent: training data sources, intended use cases, known limitations, evaluation methodology |
| Training data provenance | How training data was sourced, whether personal data was used, consent and legal basis for collection |
| Evaluation results | Bias and fairness testing results, accuracy benchmarks across demographic groups, adversarial robustness testing |
| Incident history | Any known model failures, prior regulatory inquiries, or customer-reported harms |
| Data residency | Where data used in model operation is stored and processed; whether Canadian and provincial data residency requirements are met |
| Subprocessor disclosure | A complete list of downstream vendors with access to data processed through the AI system |
| Security certifications | SOC 2 Type II, ISO 27001, or equivalent independent security attestations |
The AICM’s companion AI Consensus Assessment Initiative Questionnaire (AI-CAIQ) provides a standardized self-assessment format that vendors can complete and organizations can require as part of procurement.
Contractual Provisions
Three contractual elements are now considered standard practice in AI vendor agreements:
IP Indemnity. Several major AI providers have made public commitments to defend customers against copyright infringement claims arising from AI-generated outputs. Microsoft’s Copilot Copyright Commitment – expanded in November 2023 to cover Azure OpenAI Service commercial customers – commits Microsoft to defend and pay for adverse judgments if customers are sued for copyright infringement when using covered Copilot services, provided customers implement required guardrails and do not attempt to generate infringing materials. Organizations should verify whether their AI vendor agreements include equivalent IP protection, and understand the conditions that must be satisfied for coverage to apply.
Audit Rights. Contracts with AI vendors providing high-risk AI systems should include the right for the organization (or its designated auditor) to audit the vendor’s AI governance practices, model validation records, and security controls. This right is increasingly expected by regulated financial institutions under OSFI E-23’s framework for third-party model risk.
Model Change Notification. AI models can change significantly when vendors release updates. A model that was validated for a specific use case may behave differently after an update. Contracts should require vendors to provide advance notice of material model changes – including retraining, architectural changes, and changes to training data – sufficient to allow the organization to re-assess the model’s suitability before the change is deployed.
Emerging Standards
The Cloud Security Alliance AI Controls Matrix – released September 2025 – represents the most comprehensive vendor-agnostic framework currently available for AI supply chain governance, mapping 243 controls to ISO 42001, ISO 27001, NIST AI RMF, and the EU AI Act. ISO/IEC standards bodies are also developing specific guidance on AI supply chain risk, building on the supply chain security work established in ISO/IEC 27036.
