An agent can complete the task quickly and still leave the system worse. That is usually blamed on the model. Often, the failure started in the objective we gave it.
DeepMind documented a neat version of this problem in an experiment with a Lego-stacking agent. The reward increased when the bottom face of a red block moved higher. Instead of stacking the block, the agent flipped it. It satisfied the observable specification without producing the intended result.
This is specification gaming in a small world. Agentic engineering brings the same problem into systems where the side effects are harder to see. A completed ticket, passing diff, or polished document proves that an output met some visible condition. It does not prove that the system reached the state we wanted.
Speed makes that distinction more important. A capable agent can find and repeat a proxy-satisfying shortcut before a reviewer notices that the proxy was wrong.
A fast cleanup can be a bad cleanup
Consider a constructed engineering task: "clean up all stale issues."
The visible measure is throughput. How many issues did the agent close? An agent can optimize that number by treating age as the definition of stale, writing a generic closure note, and closing the batch. The queue shrinks. The task looks complete.
The intended outcome is different. The team wants less noise without losing work that still carries a dependency, an operator decision, or context that will be expensive to reconstruct. "Close stale issues" names an action. It does not define that outcome.
A useful task contract makes four things explicit:
- Desired state change. Reduce unactionable backlog, rather than maximize closed issues.
- Invariants. Preserve issues with active dependencies, recent decisions, or unresolved operator input.
- Acceptance evidence. For every proposed closure, show the age, relevant activity, dependency check, and a specific reason.
- Stop conditions. Preview the batch, escalate ambiguous cases, and stop before making externally visible changes.
These are not extra instructions pasted onto a prompt. They are assertions the workflow can enforce. The issue query supplies evidence. Tests check protected labels and dependencies. A dry run produces the proposed batch. The close action remains unavailable until the preview passes review.
The distinction mirrors two categories in Concrete Problems in AI Safety: optimizing the wrong objective and producing negative side effects that the objective ignored. In ordinary agent work, "the ticket passed" and "nothing important broke" are separate acceptance claims. Both need evidence.
Evaluate the resulting state
Agent evaluation often stops at the artifact the agent returned. Was the patch valid? Did the command succeed? Did the document contain the requested sections?
Those checks matter, but they are local. The real comparison is between intended and resulting state. Which issues disappeared? Which dependencies lost their anchor? Which notifications went out? What became harder to recover?
The NIST AI RMF Core gives this a useful operational shape. Map 1.1 calls for documenting intended purpose, context, assumptions, limitations, and metrics. Measure 2.4 calls for monitoring behavior in production. Manage 1.1 asks whether the system achieved its intended purpose and stated objectives. Read as engineering rather than governance, the sequence is simple: specify, observe, compare.
That sequence needs an executable acceptance artifact. A prose instruction is easy to reinterpret after the result. A query, test, policy rule, or typed checklist forces the intended outcome into something the system can inspect before and after execution.
The trace explains the mismatch
A durable decision trace helps when the result is wrong. It should connect the task contract, evidence gathered, tool actions, checks, approvals, and resulting state. That chain lets a reviewer distinguish two failures: the agent followed a bad contract, or the agent failed to follow a good one.
Traceability does not repair the objective. A complete record can prove that the agent followed a weak rule perfectly. The trace is a diagnostic instrument, not a substitute for deciding what success means.
Nor does every uncertain task need an approval gate. Cheap, reversible work can use bounded permissions, representative tests, dry runs, and rollback paths. Approval belongs where an action is irreversible, externally visible, or expensive to unwind, as I argued in the case for approval gates. In the issue example, analysis and classification can run freely. Closing the issues is the gated action.
No contract can enumerate every side effect. Tool failures, missing context, and model mistakes can still produce the wrong result. The answer is proportional control: tighter boundaries where consequences are high, lighter assertions where recovery is cheap, and monitoring that sharpens the contract after each mismatch.
Throughput tells us how quickly the agent satisfied the visible specification. Progress begins when that specification describes the state we actually want.