Skip to content
PD
EN RU
AI Automation 7 min read

Was the Model Nerfed? Build a Behavior Canary Instead of Guessing

Your LLM provider can change behavior under you without a version bump. Here is the canary system I run to detect model drift with data, not forum threads.

PD

Pavel Duglas

AI Automation & MVP Architect

Every few weeks someone posts that their favorite model got dumber overnight. Half the replies agree, half say it is placebo, and nobody has numbers. I stopped participating in that argument a long time ago, because for my clients it is not a philosophical question. If a lead classifier starts returning broken JSON on Tuesday, I need to know on Tuesday, not when the sales team complains on Friday. So I run a behavior canary: a small, boring system that tells me with data whether the model I depend on still behaves the way it did last week.

Why behavior changes without you touching anything

When you call a hosted model, you are renting a moving target. Even if your code has not changed in a month, the thing on the other side of the API might have. The common causes I have actually seen:

  • Aliases move. If you call a name like “latest” or a family name without a dated snapshot, the provider can repoint it to a newer checkpoint. Newer is not always better for your specific task.
  • Serving changes. Providers tune inference infrastructure constantly: batching, quantization, hardware, routing between clusters. Most of it is invisible. Some of it shifts outputs at the margins.
  • Safety and policy tuning. A model that happily summarized medical intake forms may start adding disclaimers or refusing edge cases.
  • Default parameter changes. Max tokens, reasoning effort, or tool calling behavior can change defaults if you do not set them explicitly.
  • Your own side. A library upgrade changed how the SDK formats messages. Someone edited a shared system prompt. A retrieval index got rebuilt. These are the most common and the most embarrassing.

The point is not to blame the provider. The point is that “it worked when I tested it” has an expiration date, and you need a signal that tells you when it expired.

What a behavior canary actually is

A canary is a fixed set of inputs that you send to your production model configuration on a schedule, scored by deterministic checks, compared against a stored baseline. That is it. No LLM-as-judge in the critical path, no dashboards with 40 charts. It answers one question: did behavior on known inputs move beyond normal noise?

It is different from a pre-release eval. An eval asks “is this new build good enough to ship”. A canary asks “is the thing I already shipped still the thing I shipped”. Same test cases can serve both, but the canary runs forever, against production settings, on a timer.

Step 1: Build the canary set from real traffic

Do not invent test prompts. Pull 30 to 80 real inputs from production logs, anonymized, and pick them deliberately:

  • 60% typical cases that represent your normal volume
  • 25% edge cases that previously broke something
  • 15% adversarial or ambiguous inputs where the correct answer is a refusal, a “needs review” flag, or an empty result

For each input, store the expected output or, more realistically, the expected properties of the output. For a lead classifier that means the expected category and the required JSON fields. For an extraction pipeline it means the exact values of 3 to 5 key fields. For a summarizer it means “under 120 words, mentions the invoice number, no invented dates”.

Freeze this set. Version it in the repo. When you add cases later, bump the version and re-baseline, otherwise you cannot compare runs.

Step 2: Score with code, not vibes

Every check should be a function that returns a number or a boolean. The ones I use on almost every project:

  • Schema validity rate. Does the output parse and pass the Zod or Pydantic schema? This is the first metric to break and the most expensive one to miss.
  • Field accuracy. Exact match on the fields that matter, normalized (lowercase, trimmed, dates in ISO).
  • Refusal and hedge rate. A simple regex list of phrases your pipeline should never produce, plus explicit refusal detection.
  • Output length distribution. Median and p90 tokens. A sudden 40% jump in length is often the first visible sign of a checkpoint change, even when accuracy holds.
  • Latency and cost. p50 and p95 latency, input and output tokens per call. Drift in cost is still drift.
  • Tool call shape. For agents: did it call the expected tool, with valid arguments, in a reasonable number of steps?

Here is the core of the scoring loop, stripped down:

def run_canary(cases, call_model, schema):
    results = []
    for case in cases:
        started = time.monotonic()
        raw, meta = call_model(case["input"])
        latency = time.monotonic() - started
        parsed, valid = try_parse(raw, schema)
        results.append({
            "case_id": case["id"],
            "valid": valid,
            "correct": valid and match_fields(parsed, case["expect"]),
            "refused": looks_like_refusal(raw),
            "out_tokens": meta["output_tokens"],
            "latency": latency,
            "model_id": meta["model"],
            "fingerprint": meta.get("system_fingerprint"),
        })
    return aggregate(results)

Notice the last two fields. Always record whatever model identifier and fingerprint the API returns in the response, not just what you requested. When drift shows up, the first thing you want to know is whether the served model changed.

Step 3: Run it on a schedule, with production settings

The canary must use the exact configuration production uses: same model string, same temperature, same system prompt layers, same tools, same SDK version. I run it as a separate job in the same worker infrastructure as the real pipeline, so it shares the same code path. If the canary uses a different code path, it tests a different system.

Frequency depends on stakes. For most client pipelines, every 6 hours is enough. For anything customer-facing with volume, hourly. With 50 cases on a mid-tier model, an hourly canary costs a few dollars a month. That is the cheapest insurance in the whole stack.

Run each case twice if you use non-zero temperature. Disagreement between the two runs is itself a useful metric: if self-consistency drops, something changed.

Step 4: Compare against a baseline, not against perfection

Your canary will never score 100%, and it should not have to. What matters is movement. When you first deploy, run the canary 10 to 20 times over a couple of days and store the distribution of each metric. That is your baseline, including its natural noise.

Then set alert thresholds relative to it. My defaults:

  • Schema validity drops more than 2 points below baseline: alert immediately
  • Field accuracy drops more than 5 points across two consecutive runs: alert
  • Median output length moves more than 30% in either direction: warning
  • Refusal rate on normal cases goes above zero: alert
  • Returned model ID or fingerprint changes: info message, and it annotates every later chart

Two consecutive runs matters. Single-run dips are usually noise or a transient provider issue. Two in a row is a pattern.

Passive signals: the canary’s second half

A canary tests known inputs. Production sees unknown ones. So I also track a few cheap metrics directly on live traffic, no labeling required:

  • Parse failure and retry rate per hour
  • Share of items routed to human review
  • Output length distribution on live calls
  • Downstream rejection rate, for example CRM API errors caused by malformed fields

If the canary is green but live retry rate doubles, the model is probably fine and your input distribution changed: a new traffic source, a scraper feeding garbage, a client uploading a new document format. If both move together, it is the model or your config. That split alone saves hours of debugging.

When the canary fires: the playbook

Having an alert without a plan just creates anxiety. This is the order I work through:

  1. Check your own changes first. Diff deploys, prompt layers, SDK versions and retrieval indexes since the last green run. In my experience this explains more than half of incidents.
  2. Check the returned model ID and fingerprint. If they changed, you have your answer.
  3. Rerun the failing cases manually and read the actual outputs. Numbers tell you something moved. Only reading tells you how.
  4. Pin or fall back. If you were on an alias, pin the dated snapshot that matched your baseline. If you already run a model router, shift traffic for the affected task to your fallback model and let the canary confirm the fallback is green.
  5. Fix forward if needed. Sometimes the new behavior is permanent and you adapt: tighten the schema instructions, add a few-shot example, add a repair step. Then re-baseline deliberately and write down why.

The important part is step 4. You can only fall back fast if you prepared a fallback before the incident. Keep at least one alternative model per critical task that already passes your canary, even if you never use it. I run the canary against the fallback daily for exactly this reason.

What this looks like in a real project

One client pipeline classifies inbound B2B leads into eight categories and extracts company size and budget into a CRM. After a routine week, the canary flagged median output length up 45% while accuracy stayed flat. Nothing was broken yet. Two days later, schema validity dropped from 99% to 94%, because longer outputs occasionally included an explanatory sentence before the JSON. The canary had given us a two-day head start. We pinned the snapshot, added a stricter output instruction, verified on the canary, and later moved to the new version on purpose. Nobody on the sales team ever noticed.

That is the whole value: you stop arguing about whether a model was nerfed, and you start knowing, early, whether your system still does its job.

Start small today

You do not need a platform. Pick your most important LLM task, pull 30 real inputs, write three scoring functions (schema valid, key fields correct, length), run it every six hours with a cron job, and post results to a Telegram channel. That is an afternoon of work. Add thresholds after a week of baseline data. Everything else is refinement.

FAQ

Can I use an LLM as a judge in the canary?

I keep it out of the alerting path. A judge model can drift too, and then you are measuring one moving target with another. Deterministic checks like schema validity, exact field matches, length and refusal detection catch most real regressions. If you want a judge for fuzzy quality, run it as a secondary, informational metric and pin its version.

Is pinning a dated model snapshot enough to avoid drift?

It removes the biggest source, but not all of them. Serving infrastructure, default parameters and your own code can still change behavior, and snapshots eventually get deprecated. Pin snapshots for stability, and keep the canary running anyway so you know when something moves and so you can validate the next snapshot before switching.

How many canary cases do I need?

Thirty to eighty per task is the sweet spot for most projects. Fewer than thirty and noise dominates, so a single flaky case looks like a regression. More than a hundred mostly adds cost and run time without much extra signal. Choose cases from real traffic, weighted toward typical inputs plus the edge cases that have already hurt you.

Related articles