AI systems, by definition, are data systems. They are trained on data, they ingest data at inference time, they produce data-derived outputs, and they frequently generate inferences about individuals that those individuals never consented to produce. Every AI deployment in an organization is simultaneously a data governance question, a privacy law compliance question, and – in regulated sectors – a model risk management question.
For Canadian organizations, this intersection is governed by at least three overlapping frameworks: PIPEDA (the federal private-sector baseline), Quebec Law 25 (Canada’s most stringent active privacy statute), and – for any organization interacting with EU residents or markets – the GDPR.
PIPEDA and the OPC Generative AI Guidance
Canada’s federal Personal Information Protection and Electronic Documents Act (PIPEDA) applies to organizations that collect, use, or disclose personal information in the course of commercial activities. It predates AI governance as a concept and does not address AI systems explicitly – but the Office of the Privacy Commissioner of Canada (OPC) has applied PIPEDA’s 10 principles to AI through interpretive guidance.
In December 2023, Canada’s federal, provincial, and territorial privacy commissioners issued joint principles for responsible, trustworthy, and privacy-protective generative AI technologies. The nine principles establish specific requirements for GenAI developers and deployers covering: legal authority and consent for data collection; appropriate and limited purposes for AI use; openness and transparency with individuals about AI use in decisions affecting them; individual access to and correction of personal data in AI systems; and safeguards against novel security threats specific to GenAI. Torys LLP’s analysis of the principles notes that both developers and deployers bear obligations – deployers cannot simply assert that a vendor’s compliance program satisfies their own obligations.
The OPC’s 2024-25 Annual Report also notes the Commissioner’s welcoming of LinkedIn’s pause of AI training on Canadian member data pending privacy compliance discussions – a signal that Canadian regulators are actively monitoring AI training data practices.
Quebec Law 25 and Automated Decision-Making
Quebec’s Law 25 (Act 25) – fully in force since September 2024 – includes binding automated decision-making (ADM) provisions in Section 12.1. These provisions apply to any organization using personal information to render a decision based exclusively on automated processing.
Where automated decision-making applies, organizations must:
- Inform the individual no later than when the decision is communicated to them.
- Explain the decision on request, including the personal information used and the principal factors that led to the decision.
- Provide a correction mechanism allowing the individual to request correction of inaccurate information used in the decision.
- Allow human review on request – the individual has the right to have a person examine the automated decision.
Critical point: a decision is only “exclusively automated” under Law 25 if no human exercises meaningful judgment in the outcome. Technical validation or automatic approval processes that do not involve real decision-making power do not constitute sufficient human intervention. This creates a design requirement for any AI-assisted decision workflow touching Quebec residents: either build in genuine human review capacity at the decision point, or comply fully with the ADM transparency and contestation requirements.
GDPR: Article 22 and the Right of Erasure
The GDPR’s automated decision-making provisions create obligations that Canadian organizations must understand when processing data of EU residents – increasingly common in financial services, technology, and multinational operations.
Article 22 provides that individuals have the right not to be subject to decisions based solely on automated processing – including profiling – that produce legal effects or similarly significant effects on them. GDPR Article 22 is broad: loan applications, insurance pricing, hiring decisions, and creditworthiness assessments all qualify. Exceptions exist where the processing is necessary for a contract, authorized by law, or based on explicit consent – but in each exception case, the organization must implement safeguards including the right to obtain human intervention, express a point of view, and contest the decision.
The right of erasure (Article 17) creates a technically complex challenge for AI. Where a model was trained on personal data and the legal basis for that training is subsequently withdrawn or found insufficient, data subjects may request deletion of their information – but current AI architectures make it technically infeasible to selectively remove specific training data from trained model weights. The EDPB’s Opinion 28/2024, adopted December 17, 2024 in response to a request from the Irish Data Protection Commission, establishes the authoritative GDPR position: AI models trained on personal data cannot, in all cases, be considered anonymous. Whether a specific model qualifies as anonymous requires case-by-case assessment, accounting for the likelihood that personal data could be extracted through model inversion or query attacks. If anonymization cannot be demonstrated, data subject rights – including erasure – apply to the model itself.
The EDPB’s December 2024 Opinion on AI Models
EDPB Opinion 28/2024 resolves four foundational questions that had created significant legal uncertainty for AI developers and deployers across Europe:
- AI models trained on personal data cannot automatically be considered anonymous – anonymity requires demonstrated evidence that personal data cannot be extracted by any means reasonably likely to be used.
- Legitimate interest can serve as a legal basis for AI model training and deployment, but only when a three-step test is satisfied: the interest is legitimate and real; the processing is necessary; and the interest is not overridden by data subjects’ rights.
- Unlawful processing in the development phase of an AI model can have downstream consequences on the model’s subsequent deployment – supervisory authorities may order corrective measures including deletion of the model or its training data.
- Technical safeguards – including output filters, access restrictions, and post-training suppression techniques – are relevant mitigating measures that should be implemented to reduce the privacy risk of deployed AI models.
Practical Questions for Canadian Organizations
Can you train on customer data? Training AI models on customer data requires a legal basis under PIPEDA and Law 25 (and GDPR if applicable). The OPC’s joint principles make clear that using data collected for one purpose to train AI for a different purpose requires either original consent broad enough to cover AI training or a fresh legal basis. The reasonable expectations of customers are a significant factor.
What consent is required? For AI deployments that make or substantially influence decisions affecting individuals, notice and – in many cases – consent or opt-out mechanisms are required. Law 25 requires automated decision-making disclosure at the time of decision; GDPR Article 22 requires explicit consent for automated profiling decisions that are not necessary for a contract or authorized by law.
How do DSARs work for AI systems? A Data Subject Access Request (DSAR) that touches an AI system raises the question of whether the AI model itself contains the individual’s personal data. Following the EDPB’s Opinion 28/2024, organizations cannot categorically assert that their AI models contain no personal data – they must assess this specifically. In practice, organizations should maintain records of what personal data was used to train each model, enabling them to respond substantively to DSARs.
Biometric and inference data. AI systems that generate inferences about individuals – including behavioral profiles, emotional state assessments, health condition predictions, or creditworthiness scores derived from indirect signals – generate sensitive data that may not have been collected directly from the individual. Under Quebec Law 25, PIPEDA, and GDPR, these inferences are personal data when they relate to an identifiable person, and they attract the full suite of data subject rights.
Privacy by Design as an AI Governance Overlay
Privacy by Design (PbD) – originally developed by former Ontario Information and Privacy Commissioner Dr. Ann Cavoukian – requires that privacy protections be built into systems from the design stage, not retrofitted as compliance measures. Applied to AI, PbD principles produce a concrete design checklist:
- Data minimization: Use the minimum personal data necessary for the AI system’s intended function. Avoid training on personal data when non-personal or synthetic data would be sufficient.
- Purpose limitation: Define the AI system’s purpose specifically and do not use training data or model outputs beyond that defined purpose without fresh legal basis.
- Accuracy and fairness: Design evaluation and monitoring processes that detect and correct bias before and after deployment.
- Transparency: Build in mechanisms for individuals to understand when AI is being used in decisions affecting them, and to access explanations.
- Security: Implement technical safeguards – including output filters, access controls, and anomaly detection – as design requirements, not add-ons.
- Accountability: Assign named accountability for each AI system’s privacy compliance, connected to the organization’s existing data governance structure.
Quebec Law 25 requires Privacy Impact Assessments (PIAs) before launching any new automated decision system that uses personal information. GDPR Article 35 requires Data Protection Impact Assessments (DPIAs) for processing likely to result in high risk to individuals, including systematic automated decision-making. These regulatory requirements operationalize PbD as a mandatory pre-deployment step.
The interaction of privacy law and AI governance is not a niche compliance consideration. It is the foundation on which every AI deployment in a regulated sector must rest – and in Canada, it is enforceable today, without waiting for federal AI legislation.
