AI Agent Governance: 7 Controls to Put in Place Before Production

The risk changes when an AI system can act

A chatbot returns an answer. An agent can also retrieve records, choose a tool, call an API, update a system or trigger the next step in a workflow. That move from output to action changes the operating model.

Model quality still matters, but it is no longer enough. A production owner also needs to know what the agent can read, what it can change, whose authority it uses and how the business can stop it.

Microsoft describes autonomous agentic systems as systems that can plan, invoke tools, access data and execute actions with limited human intervention. Its secure agentic systems guidance recommends layered, deterministic controls so one failure cannot create unacceptable harm.

Match autonomy to the consequence of the action

Reading an approved knowledge base is not equivalent to emailing a customer, changing a price or releasing a payment. Governance should follow the consequence of the action rather than apply one approval policy to every step.

A useful operating model has three modes: recommend, execute after human approval, and execute autonomously. An agent earns more autonomy only when the process is understood, controls are enforceable and recovery has been tested.

• Low impact: research, classification and draft preparation with no change to a source system.
• Moderate impact: reversible updates within deterministic rules and monitored thresholds.
• High impact: financial, external, destructive or irreversible actions that require explicit approval.

1. Enforce least privilege outside the prompt

Give the agent only the data, tools and operations required for its job. A support-triage agent may need to read tickets and propose a category. It does not need permission to delete tickets or browse every customer record.

A sentence in a system prompt is not an authorization boundary. Permissions must be enforced by identities, API scopes, database roles and tool policies that remain effective even when the model makes a bad decision.

• Start with read-only access.
• Allowlist individual tools and operations.
• Scope data by role, team, customer or geography.
• Review privileges whenever the workflow changes.

2. Isolate identities and environments

A test agent should not share production credentials. Give each agent and environment a distinct technical identity, dedicated secrets and a limited network boundary. Isolation reduces the blast radius when instructions, integrations or dependencies fail.

For an onboarding workflow, test against synthetic applications and a sandboxed CRM. Connect to live records only after normal, ambiguous and hostile scenarios have passed, using a revocable service identity.

• One traceable identity per agent and environment.
• No shared employee credentials.
• Separate test data from live customer data.
• Restrict outbound connections to required services.

3. Put human approval at the point of consequence

Human-in-the-loop does not mean asking for confirmation at every step. Repeated, low-value prompts create approval fatigue. The control belongs where a decision becomes consequential.

Before an external message, financial change or deletion, the reviewer should see the proposed action, relevant parameters and expected effect. Approval should bind to that exact action and expire when its parameters change.

• Preview the message, amount or record change.
• Assign a named approver or business role.
• Fail closed when approval is missing or stale.
• Store approval evidence with the resulting action.

4. Log decisions, tool calls and outcomes

A chat transcript is not an audit trail. Incident review requires the initial trigger, identity, sources consulted, tool calls, relevant parameters, approvals, results and errors.

The OWASP AI Agent Security Cheat Sheet recommends structured logging for decisions, tools and outcomes, alongside controls that keep credentials and sensitive data out of plain-text logs.

• Assign a unique ID to every run.
• Link actions to the user or process that initiated them.
• Record denials, retries, errors and human intervention.
• Set a retention period that serves a defined operational or legal need.

5. Cap spend, volume and action depth

Agents can loop, retry an ambiguous failure or multiply tool calls. Runtime limits protect both the budget and the underlying business process.

A lead-qualification agent might have daily caps for records processed, messages drafted, API calls and model spend. Unusual behavior should pause the run and alert an owner, not trigger another uncontrolled retry.

• Token and cost ceiling per run.
• Maximum tool calls, retries and chain depth.
• Business limits for value, volume, recipients or frequency.
• Alerts and circuit breakers for abnormal behavior.

6. Protect data and secrets across the workflow

API keys, passwords and access tokens do not belong in prompts. Keep them in a secrets manager and expose them only to the execution component that performs an authorized action.

Map the data the agent receives, every service it reaches, what appears in logs and whether memory persists between sessions. A vendor retention setting is useful, but it is not a substitute for understanding the complete data path.

• Classify data before connecting it to the agent.
• Redact fields the workflow does not need.
• Encrypt and rotate credentials.
• Review retention, residency and subprocessors.

7. Design shutdown and incident response before launch

A stop control is part of the production architecture. Operators need a reliable way to suspend the agent, revoke its access and prevent a dangerous action from resuming automatically.

The incident procedure should name the decision-maker, show how credentials are revoked, explain how completed actions are identified and define which changes can be rolled back. Test the procedure before the agent handles live work.

• An accessible manual stop control.
• Automatic circuit breakers for anomalies and limits.
• Central revocation of identities and secrets.
• A documented recovery, correction and notification path.

Production readiness checklist

This list is not a complete risk assessment. It is a practical gate for catching the most common gaps before an agent receives production access.

• A business owner and technical owner are accountable for the agent.
• Purpose, data scope and permitted actions are documented.
• The agent has a dedicated identity with minimal permissions.
• Test and production environments are separated.
• Consequential actions require explicit, parameter-bound approval.
• Tool calls, outcomes, errors and interventions are traceable.
• Cost, time, volume, retry and action limits are active.
• Secrets stay outside prompts and logs.
• Shutdown and access revocation have been tested.
• An incident owner knows what to do when the agent fails unexpectedly.

Start with a bounded process, then earn autonomy

The best first production use case is rarely the most ambitious one. Choose a frequent, bounded process with observable decisions, reversible actions and clear exceptions.

Let the agent recommend first, then approve selected actions, then automate only the paths that have earned trust. NOVENZA uses this progression when helping teams map a workflow, define a controlled first version and expand it safely.