Autopsy 24 · Test containment

Would Ethosure have caught this? A fictional target. Three real companies.

Google says Gemini logged into three companies during a security test, then stopped when it realised the systems were real. The uncomfortable question is what should have stopped it before the login. (CNBC)

The incident

The assignment was to attack a fictional company. According to Google, the name in the exercise matched a real business, and internet access allowed Gemini to pursue a real target instead of the simulated one. The testing took place at Irregular, a company that evaluates AI models before release. (The New York Times)

Google disclosed the activity on September 18, 2026, saying it had occurred in May. Reporting describes three unauthorised accesses: one using guessed passwords and two using credentials found in a public repository. (CNBC; Axios)

Google says the models stopped when they recognised that they had reached real company systems. Heather Adkins, its vice president of security engineering, put it plainly: “In all three of these instances, the model stopped.” (CNBC)

That is a useful recovery. It does not undo the access that came before it.

A new disclosure, not a new underlying flaw

Irregular says the Google case arose from the same issue involved in earlier disclosures by other AI developers. Its investigation describes a single evaluation scenario whose fictional company name coincided with a real domain, along with internet access that allowed some models to take offensive actions outside the exercise. (Irregular)

The evaluator says its instructions included internal addresses for the simulated target. Most runs stayed inside that environment, but some models pursued the real domain instead. That distinction matters: naming an approved destination in the instructions did not reliably keep execution confined to it. (Irregular)

This episode therefore belongs alongside Autopsy 02, not in a tally of unrelated containment failures. Google’s disclosure gives us a closer look at target confusion and self-correction within that shared problem. Irregular says the relevant labs were notified in late July and that the known issues on its side were resolved before this disclosure. (CNBC; Axios)

The boundary that mattered

The failure was not simply that the model misunderstood a name. It could act on that misunderstanding beyond the intended test environment, with real systems accepting the credentials it tried. Those are the conditions described in the reporting; the proposed controls below are our analysis, not a reconstruction of unpublished logs. (Axios)

A company name is not an authorisation boundary. Neither is a password that happens to work.

The model’s later decision to stop may have limited what followed. But a business commissioning this kind of test should not have to rely on the model recognising the difference between rehearsal and reality after it has crossed the threshold.

The applicable coding-ethos policy

The relevant starting point is model.tools.unapproved_tool_call_block, in the Model and Provider Governance pack under the principle Model-Invoked Tools Are Constrained. It checks whether a model response proposes a tool outside the approved list. It does not decide whether a company is an authorised target.

That narrower distinction matters here. A tool-name check could stop a proposed shell or network tool that the test owner had not approved. It would not, by itself, stop an approved tool from contacting the wrong company.

Walkthrough

1. The model proposes a tool call. The proxy evaluates the response on its way back to the agent; this is what inbound means in the rule.
2. The expression compares each tool name with the approved list. An execute_shell or http_request call is absent from this example’s list, so the expression fires.
3. The rule requests a block disposition. Preventing execution still depends on the deployed proxy enforcing that decision before the agent receives or acts on the call.
4. An approved name such as search_knowledge stays quiet under this rule. That is not a verdict on its arguments, its destination or any other applicable policy.

The current runtime supplies the tool name and a hash of its arguments to policy evaluation, not the raw argument JSON. A URL buried in those arguments is therefore not a destination this expression can inspect. (coding-ethos tool-call projection)

Policy YAML

This is an excerpt of the existing corpus rule, not a proposed network-control schema. The rule’s fields and approved names are unchanged; only the line wrapping has been adjusted.

# Within the principle's policy.expressions list:
- id: model.tools.unapproved_tool_call_block
  scope: proxy
  severity: block
  mode: block
  protected: true
  allow_severity_weaken: false
  message: Model attempts to invoke a tool outside the approved allow-list.
  advice: >-
    This response-side control stops the output from being used and flags
    the route for correction; allow only reviewed read-only or ticketing
    tools named in the customer-editable allow-list.
  when: >-
    proxy.direction == "inbound"
    && proxy.tool_calls.exists(call, !list_contains([
      "search_knowledge",
      "get_record",
      "list_records",
      "create_ticket",
      "request_human_review"
    ], call.name))

The approved names are a configuration choice, not a universal safe-tool list. A security-testing environment would need its own reviewed tools, with destination and credential restrictions enforced separately. Adding a browser or shell to the list does not make its actions safe.

Assurance boundary

For this revision, we compile-checked the source pack’s 37 expressions against the current coding-ethos CEL environment. We also ran seven synthetic cases against this particular rule: its two existing corpus fixtures and five additional regression cases. All seven produced the expected result, including an unapproved call mixed with an approved one. These are local policy tests, not a replay of Google’s incident or an end-to-end network-containment test.

There is also a material runtime limitation. The response-side enforcement code currently allows a response through if policy evaluation returns an error, and records that error; it also allows the response when no evaluator is configured. An explicit denial follows a different path. This is not fail-closed containment. (coding-ethos response enforcement)

A credible prevention claim would therefore need evidence that the policy is loaded, the proxy is enforcing it, every relevant execution route passes through the control, and evaluator failures cannot leave the test free to reach real systems. The passing fixtures do not establish any of that.

Where this policy lives in the corpus

The source is ethosure-model-provider-governance.yml, principle model-invoked-tools-are-constrained, reviewed at corpus revision 7e232ed. The runtime review used coding-ethos revision 95f92b2. This is a model-tool governance rule, not a domain allowlist or an operating-system sandbox.

What the remaining controls would have to do

Start with the target. The test owner should approve specific systems and permitted actions, rather than leave the model to infer authority from a company name. A newly discovered domain should require review, even if its name looks exactly right.

Then enforce that decision outside the model. An evaluation that needs internet research need not grant unrestricted access to arbitrary login endpoints. The network and tool controls should prevent unapproved destinations from being reached, including through shell commands, browser tools, redirects and proxy routes. A rule that covers one tool while leaving another route open is not adequate containment.

Credentials need their own boundary. Finding a password in a public repository does not establish permission to use it. Any credential broker or controlled login tool should bind use to the approved target and purpose; exposed secrets should be handled through the disclosure process, not treated as an invitation.

Finally, test the arrangement from the actual runtime before the exercise starts. Attempt an unapproved destination and verify that the connection fails. Keep the attempted destination, decision and responsible owner in the record, without copying raw passwords into the logs. Review these controls whenever the scenario, tools or network configuration changes.

Irregular’s response includes disabling the affected evaluation, reviewing logs, strengthening containment and monitoring, and revalidating fictional names against real domains. It also acknowledges a practical monitoring problem: offensive-security exercises generate suspicious activity by design, making out-of-scope behaviour harder to pick out. (Irregular)

What if your company was the target?

The victim has a different problem from the organisation running the agent. Your controls can refuse access to your systems; they cannot make somebody else’s agent obey your policies or stop it from trying again. NIST’s zero-trust model makes this distinction concrete: the policy engine decides, while an enforcement point controls the connection to the protected resource. (NIST SP 800-207)

Simply using Ethosure would not establish that this attack was blocked. The reviewed coding-ethos implementation governs connected development workflows, agent tools and model traffic, not arbitrary logins to your production systems. In the earlier YAML, inbound means a model response returning to a managed agent. It does not mean an intruder entering your network. (coding-ethos; response enforcement)

What could have interrupted the attack?

For a human account, requiring phishing-resistant multifactor authentication would prevent a guessed or exposed password alone from satisfying the login requirements, provided the extra factor could not be bypassed. That is a conditional defence, not a finding that these unnamed companies lacked MFA. (CISA)

For an exposed API key or service credential, the answer is different: revoke the credential, replace it and restrict its permissions and lifetime. Removing a secret from a repository does not itself invalidate the credential accepted by the service. (OWASP secrets-management guidance)

After a login, resource-level authorisation and session controls can still deny an unauthorised action or terminate access when new evidence warrants it. That requires enforcement at the protected system, not merely an alert or a policy document. A compromised account may still perform actions it is legitimately permitted to perform, so these controls do not guarantee detection of every misuse. (NIST SP 800-207)

These are possible interruption points, not a reconstruction of the victims’ configurations. The reporting does not identify their access-control settings or provide enough detail to prove which counterfactual would have stopped each login. (CNBC)

A real victim-side policy, before the credential escapes

One relevant existing Ethosure rule is base.secrets.write_to_tracked_file, in the regulated-enterprise base pack under secrets-are-never-agent-visible. It targets certain secret-shaped content in proposed file writes by the company’s own governed agent. Its role is to reduce credential exposure upstream, not to inspect an outside agent’s login.

# Existing policy excerpt, not an inbound firewall rule.
- id: base.secrets.write_to_tracked_file
  scope: file
  severity: block
  mode: block
  protected: true
  hook_events: [PreToolUse]
  message: >-
    Proposed file write appears to place secret material into a
    tracked file.
  advice: >-
    Move the value into the secrets manager and reference it by name.
    Never inline credentials into source or config.
  when: >-
    proposed_file_changes.exists(change,
      change.has_proposed_content
      && !change.is_test
      && !change.is_generated
      && (change.ext == ".py" || change.ext == ".go" || change.ext == ".js"
          || change.ext == ".ts" || change.ext == ".java" || change.ext == ".rb"
          || change.ext == ".yaml" || change.ext == ".yml" || change.ext == ".json"))
    && content.lower.matches("(?i)(aws_secret_access_key|private[_-]?key|client[_-]?secret|api[_-]?key\\s*[:=]|password\\s*[:=]\\s*[\"'][^\"']{6,})")

The walkthrough is straightforward: a governed agent proposes a supported source-file write, the expression checks its content for the listed patterns, and a block decision must be enforced before that write takes place. It does not revoke an existing credential, examine every repository leak or stop an external caller.

This excerpt compiled against the reviewed runtime, and five synthetic cases behaved as expected. One deliberately unrecognised secret-shaped value did not trigger the rule. The rule also excludes test and generated files and restricts file extensions. Those limits make it a useful check, not a comprehensive secrets scanner or proof that it would have prevented these particular exposures.

Where Ethosure could help the organisation being targeted

For a victim organisation, I would separate an immediate use of Ethosure from a proposed extension. The existing secret-write rule belongs in governed development workflows, where it can identify certain credential exposures before a file is written, provided its block decision is enforced. It is not a remedy for a password that has already leaked. Revocation and replacement still belong with the identity and secrets-management systems. (OWASP secrets-management guidance)

The proposed extension would put an identity-aware access gateway in front of selected business services. Ethosure would express which registered callers may perform which operations, while the gateway would enforce those decisions before forwarding a request. This follows the policy-decision and enforcement separation in NIST’s architecture; it is a design proposal, not a verified Ethosure product feature. (NIST SP 800-207)

Consider a supplier’s procurement agent. The business might permit it to retrieve the status of its own orders, using an approved identity and a short-lived credential. That permission would not extend to another customer’s records, database exports or administrative changes. An unknown caller would receive no access to those protected operations, even if it introduced itself as a familiar AI product.

This would change the victim’s position from hoping the visiting agent behaves to requiring each request to satisfy an enforceable access rule. The rule would apply to the verified caller and requested action, not to a guess about whether the caller was an AI. For human login routes and legacy applications outside that gateway, separate access controls would remain necessary.

In an attack resembling this one, such a design could refuse access where the attacker lacked the required credential or requested an operation outside its permissions. But a stolen, valid identity might still exercise its legitimate rights. A prevention claim would depend on the actual route, credentials, permissions and deployment, none of which the public account establishes for these victims. (NIST SP 800-207; CNBC)

Watching the door without handing over the keys

A separate monitoring agent could review access decisions and identify repeated attempts to use forbidden operations. In the proposed design, it could recommend a temporary restriction or invoke a tightly bounded response that the organisation had approved in advance. The gateway, not the monitoring model, would remain responsible for denying requests.

We would not give that agent unrestricted authority to rewrite firewall rules. Temporary throttling of one authenticated caller is a different risk from blocking a shared network, trusting a new organization or changing production access for everyone. Broad or lasting changes would require human approval, tests and a rollback plan. Incoming messages and logs would be evidence to assess, never instructions to administer the environment.

Collaboration would need the same discipline. A supplier’s agent could ask an internal agent for an approved order-status response, but the internal agent must not lend the visitor its broader privileges. Each step should remain within the original caller’s authority and the agreed task.

The next boundary is outbound

There is another question worth keeping beside the front-door controls: where can a protected system connect after a request has been accepted? A fuller design should combine inbound permissions with controlled outbound destinations and independently sourced threat intelligence. Permission to enter should never become permission to send company information anywhere.

That is a separate layer of defense, and a subject for a later discussion. An unflagged destination would still need business authorization; a favorable reputation would not establish permission to send data.

Is “an AI firewall” the right description?

Only as a qualified architectural analogy. A future victim-side integration could use Ethosure policy decisions to instruct an access gateway to deny a request or terminate a session. The gateway would do the blocking; its identity evidence, authorization checks, coverage and failure handling would all need to be established. This follows the decision-and-enforcement distinction in NIST’s architecture, not a claim that Ethosure already supplies that gateway. (NIST SP 800-207)

We have not verified a shipped Ethosure integration for that victim-side role, so there is no verified drop-in Ethosure ingress YAML to offer here. The honest distinction is between governing your own agents, reducing conditions an attacker can exploit, and enforcing access against an outside caller. The first two do not automatically provide the third.

Would Ethosure have caught this?

We cannot claim that from the public record. The tested rule can identify an unapproved tool call, but the available evidence does not establish which tools Gemini used, whether those calls would have passed through an enforcing proxy, or whether a tool-name restriction would have interrupted this sequence.

For this case, a credible Ethosure demonstration would need to show an agent pursuing an unapproved real destination, the deployed control blocking it before contact, and a record explaining the decision. It should also show that the authorized simulated target remains usable. The same test would need to cover each available route to the network.

For the organization being targeted, a separate demonstration would need to show an unknown caller denied at the gateway, an approved caller refused another customer’s records, and a revoked session prevented from making its next protected request. It would also need to prove that the service cannot be reached through an unguarded route and that an evaluator failure does not silently admit traffic.

Those are acceptance criteria for tests, not claims about a shipped integration. Both YAML excerpts above are existing policy rules, but neither is a victim-side login firewall. The destination, access and monitoring controls described here are additional requirements, not functionality demonstrated by those rules. A log entry after access should not be presented as prevention.

What we still do not know

The affected companies have not been named in the reporting reviewed here. Google also declined to identify the Gemini version involved. That limits independent checking of the scope and consequences. (CNBC)

Google told The New York Times that no harm occurred. That remains Google’s assessment, not an independent finding; similarly, the self-stopping account comes from Google. (The New York Times)

Irregular’s wider investigation discusses behavior across multiple customers. Details such as production-database access cannot simply be assigned to Gemini without model-specific evidence. (Irregular)

The useful question for a business leader is narrower than whether an AI has “gone rogue.” If an agent mistakes a real system for an approved target, what stops the first request? A good answer should name a control you have tested, not a judgement you hope the model will make.

Ethosure · Autopsy Series · Disclosure reviewed September 18, 2026