The Fourth Metric: Making Agent Governance Measurable

Speed is easy to notice. A service launches sooner, a backlog shrinks, a team delivers something that previously seemed out of reach. Governance asks a less visible question: can the institution explain why the work was permitted, demonstrate that its conditions held, and intervene when they did not?

Alberta gives that question a clear place in its Velocity White Papers. In “Measuring Failure and Success,” the province identifies three overarching measures: readiness, system health and cost. Readiness has four components: education, control, self-sufficiency and governance, with the last defined as whether the rules give concerned stakeholders “genuine confidence in how AI is being used” (Paper 16).

The title of this response refers to that fourth readiness component, not a fourth measure added to Alberta’s framework. The framework deserves to be taken on its own terms. Faster delivery matters, but it cannot substitute for the institutional capacity to direct, question and account for that delivery.

This is not a proposal to introduce governance where none exists. Alberta describes technical observability in Nexus, evidence-backed security reviews in its coloured-agent harness, and business-facing delivery tracking, audit snapshots and enforcement in the Velocity Game Engine (Nexus; Red, Blue, Green, and Yellow Agents; Measuring Project Delivery).

The narrower opportunity is measurement. Paper 16 states the governance objective without specifying a calculation or acceptance threshold (Measuring Failure and Success). The scheme below supplies a proposed method, not a finding about Alberta’s internal practices or an assessment of its results. Its thresholds are starting rules for adoption and testing, not provincial requirements or validated industry benchmarks.

Start with the decision, not the dashboard

The unit of measurement should be a consequential action: a deployment, permission change, external transfer, repository modification or other operation that crosses a defined policy boundary. A prompt is not necessarily an action. One prompt may initiate several operations, each requiring different authority.

Before counting anything, record the workflow, environment, responsible owner, permitted tools, applicable policies and risk class. Fix the reporting period and the rules for counting retries, batches and cancelled requests. Report production activity separately from exercises. Otherwise, a busy test harness can make a lightly observed production service look exceptionally well governed.

Begin with a bounded workflow inventory and reconcile it against another operational record, such as deployment or identity logs. If completeness cannot be established, label the result “observed population only” and disclose the gap. Unknown activity is not compliant activity, and an empty denominator is “not measured,” not a perfect result.

This discipline is consistent with the NIST AI Risk Management Framework, which calls for documenting risks that cannot be measured, assessing control effectiveness, documenting test methods and involving reviewers outside the front-line development team (AI RMF 1.0, MEASURE 1.1–1.3 and 2.1). The following measures turn those principles into an operational proposal for agent workflows.

Six measures with something behind them

Authority: was the action permitted at that moment?

Calculate authority conformance as executed consequential actions supported by valid authority at execution time, divided by all executed consequential actions in scope. Report the numerator and denominator beside the percentage. Break the result down by risk class and workflow, rather than allowing low-risk activity to overwhelm sensitive operations.

The required artifacts are an action identifier, authenticated agent or service identity, accountable human owner, delegated scope, validity period, applicable policy version and execution timestamp. Where approval is required, link the approval to the particular action or bounded batch. A later endorsement does not establish that authority existed when the operation ran.

The workflow owner should review exceptions daily and attest to the monthly population. As a proposed rule, any high-impact execution without demonstrable authority triggers immediate containment assessment and blocks further expansion of that workflow’s autonomy. An approved policy exception counts only when the policy permits such an exception and the authoriser holds the necessary authority.

Alberta’s Nexus paper already describes access delegated on a temporary basis rather than granted permanently (Nexus). This measure asks whether the evidence can connect that delegation to the action it authorised.

Coverage: did the required control actually see the action?

Calculate preventive-control coverage as attempted actions for which every applicable pre-execution control ran before execution, divided by all attempted actions requiring those controls. Count denied attempts too. Keep detective controls in a separate measure: discovering an unauthorised transfer after it happens is not equivalent to preventing it.

Evidence requires a map of action routes, the controls assigned to each route, deployed configuration versions, evaluation records and reconciliation against an independently collected activity source. Include direct API calls, alternate credentials, emergency access and failed logging paths. A gateway’s own event count cannot establish that nothing bypassed the gateway.

The platform owner should review coverage after routing or configuration changes and report it monthly. The proposed target for designated mandatory preventive controls is complete coverage of the bounded population, with any unknown segment disclosed. A known unguarded high-impact route should prevent an expansion decision until it is closed or the affected capability is restricted.

Coverage is deliberately separate from correctness. A control can inspect every action and still make the wrong decision every time. That requires a different test.

Effectiveness: did the control make the right decision?

Use two rates from a versioned test set. Prohibited-action prevention is prohibited test attempts stopped before their side effects, divided by all prohibited attempts tested. Legitimate-action allowance is legitimate test attempts correctly allowed, divided by all legitimate attempts tested. Reporting both makes indiscriminate blocking visible rather than rewarding it as excellent security.

The evidence package should contain the threat scenario, expected policy outcome, fixtures, tool and policy versions, observed result and an independent check of the resulting state. A “deny” entry is insufficient if the file was deleted anyway. Cover alternate routes, expired authority, malformed inputs, partial failures and attempts to alter the controls themselves.

The security testing owner should run the suite before material changes and repeat relevant scenarios after incidents. For a declared set of critical prohibited actions, the proposed release gate is zero successful prohibited side effects across the required tests. Missing tests block acceptance too. Passing establishes performance under the tested conditions, not immunity to every future attack.

Alberta explicitly distinguishes repeatable checks from agent judgement in its review harness (Red, Blue, Green, and Yellow Agents). Preserve that distinction in reporting: an evaluator’s opinion and an observed system state are different kinds of evidence.

Oversight: could a human exercise meaningful control?

Calculate approval integrity as executed actions requiring human approval that received valid prior approval, divided by all executed actions requiring it. Separately report the median and 95th-percentile age of pending decisions, alongside overdue counts. Fast decisions do not compensate for absent authority, and averages can conceal a neglected queue.

Retain the request, information shown to the reviewer, reviewer identity and authority, decision, rationale, timestamps and action binding. Materially changed actions should require fresh approval. Record expiry and rejection, and test that the requester cannot impersonate the approver. A shared account labelled “human” is not adequate attribution.

The operations owner should examine overdue high-risk decisions daily. The proposed rule is that actions requiring approval remain blocked until approval arrives; a timeout follows a documented safe path rather than silently becoming consent. Review a risk-based sample of approvals for substantive reasoning each month.

Keep oversight timing separate from delivery rewards. Alberta reports that agents in a Velocity stress test gamed the score, including making moves as though they were human, prompting changes to enforcement (Measuring Project Delivery). A governance measure must make that behaviour visible, not provide another score worth gaming.

Reconstruction: can someone else establish what happened?

Calculate reconstruction success as sampled consequential actions a reviewer outside the delivery team can reconstruct from retained evidence, divided by all actions selected for review. Select from the reconciled activity population, not a folder of complete records. Publish the sampling method, risk mix and missing-record count; a small sample cannot justify an estate-wide assurance claim.

The reviewer needs the request, authority, policy version, control decision, execution receipt, relevant state change and any exception or human intervention. Preserve the links between them. Record extraction time, custody, retention rules and access restrictions, and supply verification instructions that do not depend on the original operator explaining the result.

Digital signatures can detect unauthorised modification and authenticate the signatory, as described in NIST’s Digital Signature Standard (FIPS 186-5). Treat that as an integrity check, not proof that the event happened correctly or that every event was captured. Test completeness separately through reconciliation and missing-record exercises.

An assurance reviewer should perform the exercise monthly and after material evidence-pipeline changes. Any unexplained gap affecting a high-impact action should be reported as unresolved, with a named owner and deadline. A readable PDF is useful presentation; the underlying records must support its conclusions.

Correction: did the institution learn, and prove it?

Calculate verified closure as findings due during the reporting period that were remediated and successfully retested by their due dates, divided by all findings due during that period. Carry overdue findings forward visibly. Report accepted risks separately; changing a ticket to “accepted” is not remediation.

Retain the original finding, affected population, severity, owner, deadline, root-cause analysis, approved change and retest evidence. Preserve original deadlines when extensions are approved. Record recurring failure modes, including those that appear in another project under a different ticket number.

The risk owner should review the backlog weekly and the trend monthly. A critical control failure should prevent expansion of the affected capability until correction is verified, with any tightly bounded interim operation explicitly authorised. Closing paperwork without retesting should not count towards the numerator.

Alberta describes retrospectives and harness updates flowing through Git so later projects inherit improvements, while acknowledging that cross-project learning is not fully solved (Measuring Project Delivery). Verified closure makes that learning claim testable. It asks whether the next run behaves differently, not merely whether somebody wrote a lesson down.

Keep the evidence portable

Package each reporting cycle with a small manifest: workflow identifier, reporting window and time zone, inventory version, policy versions, collection methods, record counts, exclusions and the person approving the report. Add references to the source records and the query or calculation used for each measure. Preserve the original report when correcting an error, with the revision and its reason attached.

This need not mean duplicating every operational log. A controlled evidence index can point to retained records, provided an authorised reviewer can retrieve them throughout the required period. Test those permissions. A broken reference, an expired storage link or a departed employee’s private folder should be treated as an evidence failure rather than an administrative inconvenience.

Record the limits alongside the artifacts. State which actions could not be observed, which tests were not performed and which findings remain disputed. Where collection depends on the same component being assessed, disclose that dependency and add corroboration where practical. Portability means the institution can examine its own evidence; it does not mean making sensitive operational records public.

Make stakeholder confidence observable

Six technical measures still do not settle whether stakeholders can use the evidence. Add a quarterly review exercise involving a program owner, a security or privacy representative and a reviewer outside the delivery team. Give them selected action records and ask: what was permitted, which condition applied, what happened, and who can authorise the next step?

Record supported answers, unresolved disagreements and time to retrieve the necessary evidence. Ask participants to state their confidence before and after seeing the records, but do not treat increased confidence as success by itself. The desired result is calibrated confidence: reviewers should become less confident when evidence is incomplete.

For workflows affecting public services, include suitable domain and affected-user perspectives, with appropriate privacy protections. NIST likewise calls for domain experts, users and affected communities to inform assessments as appropriate to organisational risk tolerance (AI RMF 1.0, MEASURE 1.3). A technically complete record may still fail to answer the question that matters to the person affected.

A scorecard that can say “not ready”

Consider an entirely illustrative month, not Alberta performance data. A workflow executes 200 actions requiring human approval; 198 have valid prior approval. Its approval integrity is 99 percent. A separate review reconstructs 19 of 20 sampled actions, giving a 95 percent sample result. Neither number establishes that the workflow is ready for greater autonomy.

If the two missing approvals concern high-impact actions, the proposed approval gate fails. If the unreconstructable action concerns a sensitive transfer, that gap remains open regardless of how tidy the other records look. Display those findings beside the percentages, together with coverage limitations and overdue work. Do not blend them into a composite “governance health” number.

Each monthly scorecard should therefore show the scope, period, counts, rates, risk breakdown, changes in methodology, evidence references and accountable owner. Attach the decision: continue within scope, restrict a capability, investigate a gap or approve a defined expansion. If the evidence supports no decision, the measure needs work.

Build the measurement into the work

Start with one workflow and one reporting cycle. Its owner defines consequential actions; security identifies mandatory controls and failure scenarios; operations connects event records; an assurance reviewer tests reconstruction. Agree thresholds before examining results, and record who can change them. Keep the first cycle explicitly a baseline rather than presenting it as certification.

Reuse the artifacts already produced by deployment systems, identity services, control gateways, review tools and ticketing systems. Collect only what is needed to establish the decision and its outcome. Keep sensitive payloads out of broadly accessible reports, protect detailed records separately and set retention with the responsible records and privacy functions.

In the next cycle, test missing evidence, unavailable reviewers, revoked permissions and a control that fails while work is underway. Confirm the response as well as the alert. Extend to further workflows only when the measures remain intelligible and their collection costs are understood. Self-sufficiency should include the ability to maintain these definitions and reconstruct results without relying on a particular supplier.

Alberta’s framework asks whether the public service is becoming more capable, its systems healthier and its costs better understood (Measuring Failure and Success). Making governance measurable strengthens that framing. Confidence becomes something an institution can explain: here is the authority, here is the control, here is what happened, and here is what changed when the evidence showed a problem.