Rolling Out AI Agents: Guardrails and Verification

Between a working prototype and a production system stands one question: what happens when the agent is wrong. Without an answer, the prototype stays a prototype.

Why agents lie about results

This is not a defect of a particular model or a consequence of a bad prompt. The model generates a continuation of text; “task complete” is a continuation like any other, and a particularly likely one, because that is how conversations usually end.

The consequence is worth accepting as given: the agent’s report of a result is not evidence of the result. It is useful as an explanation, not as confirmation.

Typical forms:

• The tool returned an error and the agent reported success.
• The agent did three steps out of five and wrote that everything was done.
• No data was found, and the report contains plausible but invented values.
• A prompt constraint was violated and the report says it was respected.

The verification layer

Simple in principle, demanding in discipline: after the agent finishes, the system checks the fact itself.

What that looks like:

• The agent says it created a ticket - confirm the ticket exists and the mandatory fields are filled.
• The agent says it sent a message - confirm by the send identifier, not by its word.
• The agent says data was updated - read it back and compare to expectation.

The key requirement: the check must be independent of the agent. A check performed by the same agent confirms nothing; it will confirm its own conclusion.

A second level checks quality rather than fact: a separate pass that looks at the result with fresh eyes and hunts for inconsistencies. For development work that technique is covered on the blog: the verification layer.

Guardrails in code, not in prompts

The difference is fundamental.

A prompt constraint is a request. Usually respected. Occasionally not, and those occasions are what become incidents.

A code constraint is an impossibility. The agent will not delete data if it has no deletion tool and no delete permission on the database.

What moves from prompt to code first:

• Permissions. A dedicated database user with access to only the required tables and only the required operations.
• The absence of dangerous tools. Not “a delete tool with a warning in its description” but no tool.
• Human confirmation for irreversible actions: payments, mailings, deletion, publication.
• Step and cost limits per run.
• Scope limits. A tool that can update one record is safer than one that can update all of them.
• Idempotency. Repeating an action does not produce a second effect - see idempotent pipelines.

The full checklist is on the blog: guardrails for agents in production.

The launch sequence

Five steps, each removing its own class of risk:

1. Observation mode. The agent proposes, a human executes. Gives the true error rate instead of the assumed one.
2. A narrow slice. One task type, low volume, reversible consequences.
3. Verification before widening. Until result checking is automated, do not expand: you simply will not learn about errors.
4. Metrics and alerts. Measurement first, volume second.
5. Gradual expansion while keeping the ability to switch it off.

Separately: there must be a way to shut the agent down in one minute, and a clear rollback. If shutting it off requires a deploy, it will not work when you need it.

What to monitor

• Completion rate by fact checks, not by reports.
• Handover rate. A rise means something changed in the input data.
• Step count, mean and maximum. A rising maximum is an early sign of loops.
• Cost per run. It rises before quality visibly drops.
• Rerun rate - a marker of trouble with errors and idempotency.

The defining operational property of agent systems: they degrade quietly. They do not crash, they start being wrong more often. Without those numbers, the difference between “working” and “half working” is invisible until complaints arrive.

How this is accounted for during the build is in agent development. The overview is in the AI agents guide.