AuthZBench+ — An Executable Authority Oracle for Dynamic Agent Authorization

Status: Pilot complete · scale-up pending · part of the agent-authorization and tool-security line of work.

The question

Most agent-authorization benchmarks fix the authority boundary for the duration of a run. Real deployments do not: grants arrive, get delegated, and are revoked while the agent is still working. AuthZBench+ asks what happens across those transitions, and builds the machinery needed to answer it — an event-sourced authorization state, ten stateful mock domain services, a declarative scenario language, per-action instrumentation, and an execution-time oracle that resolves every action into one of six outcome states rather than a binary allow/deny.

Results so far

• The headline hypothesis is not supported at pilot scale. Unauthorized executed-action rate moves from 20.1% under static authorization to 23.9% under dynamic transitions — a difference of +3.8 points [−1.8, +10.1] over 1,440 matched pairs with a family-clustered bootstrap. Directionally consistent with the hypothesis, but underpowered.
• A measurement correction changed that conclusion, and is the more useful result. Comparing whole trajectories first gave static 41.0% versus dynamic 23.9% — apparently contradicting the hypothesis outright. That comparison is invalid: in the dynamic arms the agent holds the grant until step 3 and cannot violate before then, while the static arm can violate from step 0 and mostly does. Matching on comparable steps removes the artifact.

The correction is the transferable finding: any dynamic-authorization evaluation that compares whole trajectories across arms with different grant timings will measure the timing, not the authorization.