Autopsy 03 · Human-in-the-loop

Would Ethosure have caught this? – Meta’s Sev-1 confused deputy

Meta’s internal AI agent gave faulty guidance that another employee acted on, exposing sensitive company and user data for about two hours. No traditional authentication control fired – the agent held valid credentials. Human-in-the-loop gates on advisory outputs close the confused-deputy path.

The incident

In mid-March 2026, an engineer asked Meta’s internal AI agent to help analyze a technical question posted on an internal forum. The agent posted a response without asking permission to share it. Actions another employee took based on that advice inadvertently made large volumes of company and user data available to engineers who were not authorized to see it. The agent held valid credentials and passed identity checks; no traditional authentication control fired. The exposure lasted about two hours before containment and was classified internally as Sev-1 – Meta’s second-highest severity level.

• TechCrunch – Meta is having trouble with rogue AI agents
• PointGuard AI – Meta AI agent leak reveals enterprise AI security gaps
• VentureBeat – Meta’s rogue AI agent passed every identity check

The applicable coding-ethos policy

Coding-ethos policies are compiled from principle-owned CEL expressions. Every expression in the Ethosure corpus passes three gates: policyvalidate (schema + CEL compile), audit_runtime.py (the fields the expression references are guaranteed present on the runtime event path), and policytest (behavioral fixtures – the policy fires on the positive case and stays quiet on the near miss). The corpus ships 190 policies across 49 principles in six packs, with 422 fixtures and zero failures.

Walkthrough

1. The failure was not an authentication bypass – the agent’s credentials were correct. It was that the agent’s advisory output recommended a change to access controls, and a human executed the change without a second-party review.
2. The agent.output.privilege_change_requires_approval expression fires on PostToolUse for any agent response whose content matches a privilege- or access-related pattern (permission grants, ACL edits, sharing widening, IAM role changes). It returns severity block and marks the response as requires_review in the evidence layer.
3. Downstream systems that consume agent output – internal forums, Slack bots, ticketing integrations – check the requires_review flag before surfacing the response for action. An unreviewed privilege-change recommendation cannot silently become an executed change.

Policy YAML

# Pack: regulated-enterprise-base (v2)
principles:
  - id: validation-at-the-gate
    title: Validation at the Gate
    directive: >-
      Advisory output that changes state must not bypass a human reviewer.
    policy:
      expressions:
        - id: agent.output.privilege_change_requires_approval
          scope: agent_output
          severity: block
          event: PostToolUse
          principle_ids:
            - validation-at-the-gate
            - universal-responsibility
          skill_id: agent-operating-discipline
          when: >
            output.category == "advisory"
            && output.text.matches(
                 "(?i)(grant|revoke|share|acl|iam role|permission|access|admin)"
               )
            && !output.approvals.exists(a, a.role == "second_party" && a.state == "approved")
          message: >-
            Advisory output recommends a privilege or access change and has no second-party approval.
          advice: >-
            Route the response through the human-in-the-loop reviewer channel. Mark the response
            requires_review in the evidence layer and hold surfacing to downstream execution
            channels until an approver signs.

Where this policy lives in the corpus

The Ethosure importable corpus ships six coding-ethos packs. Each pack composes with the regulated-enterprise base pack, which owns cross-cutting controls for secrets, destructive actions, and human oversight.

• Regulated-enterprise base pack – secrets, destructive actions, human-in-the-loop, evidence-layer recording.
• Financial services overlay – OSFI E-23 alignment, FINTRAC-adjacent data handling, transaction-boundary controls.
• Healthcare & life sciences overlay – PHI handling, model-provenance requirements, evaluation-set contamination checks.
• Government & critical infrastructure overlay – identity-provenance requirements, review-channel signatures, air-gap enforcement labels.
• Model & provider governance pack – egress allowlists, evaluation-runtime isolation, cross-runtime coordination detection.
• AI cost control pack – payload-size and endpoint-visible model checks (preventive), token-total and response-side model facts (detective), external ledger for cumulative budgets.