The Agent Operations Checklist

This is a companion piece to our Governance checklist, as both are equally important parts of your posture. First, you need to decide what’s allowed. That’s covered in our Agent Governance Checklist. But then, there’s the day-to-day of keeping a running agent in check.

Agents are unlike regular software. An agent’s behavior changes without the agent itself being changed directly. Updates to models, MCP servers, and updated content entering instructions can alter your agent's behavior without a version bump.

When someone else is responsible for the changes, you shouldn’t group operations by a lifecycle stage. Instead, check for changes you did not make. Ideally, they’re calendar-based, with the name of the person responsible next to them.

On every change

The change has a version number and a diff.

• You can understand the prompt, tool list, model, and config diff between the two versions.
• The only system prompt you save is the current one.

Behavior is tested repeatedly, not once.

• You’ve seen each capability several times and are relatively certain that the non-deterministic flows are safe.
• You ran a single pass over a fixed set, ignoring run-to-run variance and broken capabilities.

The model identifier is pinned, and pinning is not trusted to be enough.

• The exact model ID is recorded per agent, and tests are re-run on a schedule.
• You assume that the model generates fixed behavior over time.

Tool contracts are pinned and diffed.

• Tool schemas, annotations, and descriptions are snapshotted, and a changed description is treated as a code change.
• Tool changes are unnoticed if the tools keep working.

The change can be rolled back.

• The previous version is recoverable, and a rollback has been done once for real.
• Rolling back is either impossible or too difficult.

The change is attributable

• You record who changed what, when, and why
• Anyone can alter the prompt without the alteration being noticed

Every week

Spending is reviewed per agent.

• You know how much each agent should cost per week, and have a threshold with an owner.
• Org-level agentic bill

Cost caps are enforced automatically.

• The agent is stopped when running over budget.
• Cost observability without controls.

Loops and retries are bounded.

• There’s a maximum step count, timeout, and a retry limit.
• The agent is dependent on the external system blocking problem calls.

Failed and abandoned runs get reviewed together with the finished ones.

• The week's errors, timeouts, and half-completed runs are examined.
• Only successful outputs are reviewed, not failures.

Every agent has a reachable owner.

• The recorded owner is still with the company and didn’t switch roles.
• Ownership isn’t reviewed after registration.

Approval queues are seen.

• There’s an SLA for agent approvals, and it’s adhered to.
• Employees gave up on asking for approval and designed workarounds.

Every quarter

Idle agents were retired.

• There’s evidence that inactive agents are decommissioned and their credentials are revoked.
• Idle agents are running quarter after quarter.

The task-case set was refreshed from real traffic.

• New cases are added from the quarter's failures and near-misses.
• A test set was written at launch.

Overlapping agents were consolidated.

• Agents that perform the same job for different teams are identified and merged.
• No one asks if the number of agents grows because of duplication.

The dependency inventory was rebuilt.

• Each agent's MCP servers, tools, models, and external services were re-enumerated, with owners and support status.
• Deprecated upstream dependencies are discovered only when the agent breaks.

Cost per unit of work is trended.

• You know the cost per task or per resolved item, quarter over quarter, alongside the total.
• Spend and usage are the only things that are tracked.

Someone owns the operation of the fleet.

• A named person or team is accountable for the items above.
• Every check is a side project.