Skip to content
PD
EN RU
AI Automation 7 min read

Run the Agent Once, Then Compile It: Turning AI Agent Runs into Deterministic Workflows

Most agent tasks repeat the same steps every day, yet we pay the model to rediscover them every time. Here is how I record a successful agent run, compile it into a deterministic workflow, and only call the agent back when the workflow breaks.

PD

Pavel Duglas

AI Automation & MVP Architect

Most of the AI agents I see in production do the same job every single day. They log into the same supplier portal, click the same filters, export the same report and push it into the same spreadsheet. Every run, the model reasons from zero about where the button is. You are paying frontier-model prices for a decision that was already made yesterday. My default now is different: let the agent figure the task out once, then compile that run into a boring deterministic workflow, and only wake the agent up when the workflow breaks.

You are paying for reasoning you already did

An agent is great at exploration. It handles a page it has never seen, recovers from a weird modal, guesses that “Export” is hidden under a three-dot menu. That is genuinely valuable the first time.

The tenth time, it is waste. And it is not just money. Each fresh reasoning pass is a fresh chance to do something different. One day the agent picks the CSV export, the next day it picks XLSX because the model update shifted its preferences. One day it finishes in 40 seconds, the next in 4 minutes because it decided to double-check every row. Your downstream pipeline does not want creativity. It wants the same output shape at the same time.

So I split the work into two modes:

  • Explore mode - the agent, with tools, solves an unknown or changed task.
  • Replay mode - a deterministic script executes known steps, with no model in the loop or only tiny model calls for narrow extraction.

The interesting engineering is the bridge between them.

When a task is ready to be compiled

Not every agent task should become a script. I look for these signals before compiling:

  1. The task ran successfully at least 3 to 5 times with similar step sequences. If every run looks different, you do not have a workflow yet, you have research.
  2. The inputs vary, but the path does not. Different date range, same clicks. Different customer ID, same API calls.
  3. The output has a fixed schema. A table, a JSON object, a file in a known format.
  4. Failure is detectable. You can tell from the result whether it worked, without a human squinting at it.

If a task hits all four, keeping it on a pure agent is a choice to pay more for less predictability.

The pipeline: explore, record, compile, replay, fall back

Step 1: Record a trace with structure, not a chat log

The chat transcript is useless for compilation. You need a structured trace of every tool call, its arguments, what the agent observed, and why it moved on. I log each step like this:

{
  "step": 4,
  "tool": "browser.click",
  "args": {"selector": "button[data-test=export]", "text": "Export"},
  "observed_before": {"url": "/reports/sales", "has_text": "Sales report"},
  "observed_after": {"download_started": true},
  "duration_ms": 820
}

The two fields people skip are observed_before and observed_after. Those become your assertions later. Without them you are recording blind clicks, and blind clicks are exactly the brittle macros everyone hated before agents existed.

Step 2: Extract the skeleton

Take several successful traces and diff them. The steps that appear in every run, in the same order, with the same tool, are the skeleton. Steps that appear only sometimes are usually recovery behavior: closing a cookie banner, dismissing a “what’s new” popup, retrying a slow load.

Recovery steps are important. Do not throw them away. Turn them into optional handlers: “if a modal with text X is visible, click close”. This is where most hand-written automations fail and where the agent already did the research for you.

I do this extraction with a model, once, offline. I give it the traces and ask for a step list in a strict schema. Then I review it by hand. This review takes ten minutes and it is the cheapest quality gate in the whole system.

Step 3: Parameterize

Anything that changed between runs and is clearly input becomes a parameter: dates, account IDs, search queries, file names. Anything that changed but should not have, like a selector that worked one way on Monday and another way on Friday, is a smell. Either the site is A/B testing you or the agent found two paths. Pick the more stable one, keep the other as a fallback selector.

Selector stability matters a lot here. I rank them roughly like this: data attributes and ARIA roles first, visible text second, structural CSS paths last. Never compile an XPath like div[3]/div[2]/span. It will die on the next frontend deploy.

Step 4: Guard every step with assertions

This is the part that makes compiled workflows safe. Every step gets a precondition and a postcondition derived from the recorded observations. Before clicking Export, assert we are on the sales report page. After clicking, assert a download started within N seconds.

A simplified runner looks like this:

for step in workflow.steps:
    if not check(step.pre, page):
        if handler := match_recovery(page, workflow.handlers):
            handler.run(page)
        if not check(step.pre, page):
            raise Drift(step=step.id, reason="precondition failed")
    execute(step, page, params)
    if not wait_for(step.post, page, timeout=step.timeout):
        raise Drift(step=step.id, reason="postcondition failed")

The key design decision: the runner never improvises. It either matches a known state or it raises Drift. That exception is your signal, not a bug to swallow.

Step 5: Fall back to the agent, then recompile

When Drift fires, hand control to the agent with full context: the workflow, the step that failed, the current page state and the goal. The agent finishes the task in explore mode, which keeps your business running. Its new trace goes into the same pipeline, and the next compilation pass produces an updated workflow.

Two rules I keep strict here:

  • The fallback agent never edits the compiled workflow directly. It produces a trace. A human, or at minimum a separate review step, approves the new version. Otherwise you get a self-modifying script that slowly drifts into nonsense.
  • Version every workflow. When something goes wrong downstream, I want to know which version ran and which trace it was compiled from.

A concrete example

One client had an agent pulling daily wholesale prices from 14 supplier portals. Pure agent mode cost a noticeable amount per day in tokens, took over an hour end to end, and produced slightly different column orders depending on the model’s mood.

We ran the agent for a week to collect traces, compiled 12 of the 14 portals into deterministic flows (some in Python with Playwright, a few as Browser Automation Studio projects because the client’s team already knew BAS), and kept 2 portals on the agent because their UI changed constantly.

The results after a month:

  • Runtime dropped from over an hour to about 9 minutes.
  • Model spend dropped by roughly 90 percent, almost all of it now going to the 2 unstable portals and to fallback runs.
  • Drift fired 6 times. Each time the agent completed the run, and we recompiled. The business never missed a day of prices.

The narrow model calls that remained were things like “extract the price table from this HTML into this schema”. Small, cheap, easy to evaluate. That is a very different risk profile from “go do the whole job”.

What not to compile

Some tasks should stay agentic, and pretending otherwise just creates fragile scripts:

  • Tasks where the judgment is the product. Triaging support tickets, drafting replies, deciding which lead is worth a call. You can compile the data gathering around them, not the decision.
  • Highly unstable targets. If a site redesigns monthly, you will spend more time recompiling than you save.
  • Rare tasks. If it runs twice a quarter, the agent is fine. Compilation pays off on frequency.
  • Anything with irreversible side effects and no clear postcondition. If you cannot assert success, you cannot replay safely. Fix observability first.

The metrics I track

Once you have both modes, track them separately, otherwise you lose the story:

  • Replay success rate per workflow version. Below 95 percent means the skeleton is wrong or the target is too unstable.
  • Drift rate and drift step. If the same step keeps drifting, its selector or assertion is weak.
  • Fallback cost per week. This tells you how much the unstable part of your system actually costs.
  • Time to recompile after drift. If it takes days, your review process is the bottleneck, not the tech.

My rule of thumb

Agents are for discovering how to do something. Code is for doing it every day. The mistake is treating every run like a discovery. Record the first good runs, compile the stable path, guard it with assertions from real observations, and keep the agent as a well-paid emergency contractor rather than a full-time employee. Your costs go down, your outputs get predictable, and when something does change, you find out from a clear Drift error instead of a quietly wrong spreadsheet.

FAQ

Isn't this just going back to old-school brittle macros?

No, because of two differences. First, the steps and recovery handlers come from real agent traces, so they already include the edge cases a human would miss. Second, every step has pre and postconditions, and failure triggers the agent instead of silently breaking. Old macros failed quietly. Compiled workflows fail loudly and heal through the fallback path.

How many successful agent runs do I need before compiling?

I usually wait for 3 to 5 successful runs with similar step sequences, ideally across different inputs. One run shows that a path exists. Several runs show which steps are the stable skeleton and which are occasional recovery behavior like closing popups. If every run looks different, the task is not ready for compilation.

Should the fallback agent automatically update the compiled workflow?

I strongly recommend against it. Let the fallback agent finish the task and save a new trace, then recompile through the same pipeline with a review step and a new version number. Auto-editing workflows leads to slow, invisible drift, and you lose the ability to say exactly which version produced which output.

Related articles