Testing Your Agent Between Releases

Your app's UI changes constantly — a button gets renamed, a form field is added, a screen is restructured. Every one of those changes can silently break the agent, because the agent only knows your UI through the MCP document sync generates from it. appilots eval is how you catch that before it ships, instead of finding out from a support ticket.

It works like a snapshot test for agent behavior: you write a handful of scenarios ("given this screen, when the user says X, the agent should do Y"), run them in dry-run mode against your current MCP document, and compare the result to a committed baseline. No side effects — nothing is ever executed against your real app or backend; the agent only decides what it would do.

Quick start

# 1. Scaffold an example scenario
appilots eval --init

# 2. Edit .appilots/scenarios/example.json to match a real screen

# 3. Capture the current (good) behavior as your baseline
appilots eval --update-baseline

# 4. Later — after a UI change and a re-sync — check for regressions
appilots sync
appilots eval

appilots eval exits non-zero when the pass rate drops below the configured threshold or a passing scenario regresses against the baseline, so step 4 works as a CI gate exactly like appilots sync --strict-metadata does for metadata quality.

Writing a scenario

Each scenario is one *.json file in .appilots/scenarios/ (configurable — see Configuration below):

{
  "name": "agent-fills-signup-form",
  "description": "The agent should fill the visible signup form fields from a single spoken instruction.",
  "given": {
    "screen": "SignupScreen",
    "observation": {
      "route": "SignupScreen",
      "texts": ["Create your account"],
      "inputs": [
        { "id": "email", "label": "Email", "required": true },
        { "id": "name", "label": "Full name", "required": true }
      ],
      "buttons": [{ "id": "btn-submit-signup", "label": "Sign up" }]
    }
  },
  "when": "My name is Jane Doe and my email is jane@example.com",
  "expect": {
    "mode": "any",
    "actions": [
      {
        "type": "form_fill",
        "payload": { "fields": [{ "fieldId": "name" }, { "fieldId": "email" }] }
      }
    ]
  }
}

Field	Required	What it's for
name	Yes	Unique identifier for this scenario. Used to match it up in the baseline — renaming it drops its history.
description	No	Free text for humans; not evaluated.
given.screen	No	The screen name the scenario pretends the user is on.
given.observation	No	A snapshot of that screen — route, texts, inputs, buttons, sliders, lists. Only include what the scenario needs; everything is optional.
when	Yes	The user's prompt for this turn.
modelOverride	No	Test against a specific model instead of the project's configured one.
expect	Yes	What the agent should do — see below.

expect — describing the correct behavior

{
  "mode": "any",
  "actions": [{ "type": "form_fill", "payload": { "fields": [{ "fieldId": "plate" }] } }],
  "replyMustContain": ["confirm", "tem certeza"],
  "replyMustNotContain": ["1234-5678-9012"],
  "forbidActionTypes": ["delete_item"],
  "anyOf": []
}

• mode (default "any") — how the listed actions must appear in what the agent actually did:
  • any — every listed action must appear somewhere in the actual actions (the agent can also do more; useful for form-fill scenarios where the model might fill extra fields).
  • first — the first actual action must match the first listed one; the rest just need to appear somewhere.
  • exact — the actual actions must match the list one-for-one, in order, with no extras.
  • none — the agent must emit zero actions (clarification / read-only probes).
• payload constraints are partial matches — every key you list must equal the actual value, but the actual payload can have extra keys. form_fill's fields array is matched by fieldId regardless of order; if you also pin a value, that value must match exactly — otherwise any value is accepted (useful since the model may phrase a value differently than you typed it).
• replyMustContain — at least one of these strings (case/accent insensitive) must appear in the agent's reply. Use this for confirmation-gated destructive actions, where the important behavior is the agent asking, not an action being emitted.
• replyMustNotContain — none of these may be echoed back — useful for asserting the agent doesn't repeat sensitive input.
• forbidActionTypes — fail if the agent emits any action of these types, regardless of mode.
• anyOf — a list of alternative expect blocks; the scenario passes if any one of them fully matches. Use this when more than one outcome is legitimately correct (e.g. either the agent wires up a confirm action, or it asks an explicit question and emits nothing).

The baseline

.appilots/eval-baseline.json (configurable) records, per scenario, whether it passed and how many tokens the run cost. It's meant to be committed to version control, the same way qa/baselines/agentic/ baseline.json works in the Appilots monorepo's own internal eval suite.

• appilots eval --update-baseline runs the suite and, only if every scenario passes at least the configured pass-rate threshold, overwrites the baseline with the current results. It refuses to update (and exits non-zero) if the pass rate is below threshold — you fix the failure first, you don't bless it into the baseline.
• appilots eval (no flag) runs the suite and diffs it against the existing baseline, flagging two kinds of regression on top of plain pass/fail:
  • newly-failing — a scenario that passed on the baseline now fails.
  • token-regression — a scenario still passes, but now costs meaningfully more tokens than it used to (see maxTokenRegression below). A model call that suddenly takes 3x the tokens is often a sign something about the prompt or MCP document grew unexpectedly, even when the outcome still happens to be correct.

A scenario with no baseline entry yet (new scenario, or first run) never regresses — there's nothing to compare it against.

Configuration

All of this is overridable via flags or in .appilotsrc:

{
  "apiKey": "ak_...",
  "eval": {
    "scenariosDir": ".appilots/scenarios",
    "baselinePath": ".appilots/eval-baseline.json",
    "minPassRate": 0.95,
    "maxTokenRegression": 0.25
  }
}

.appilotsrc field	Flag	Default	Meaning
eval.scenariosDir	--dir <path>	<outputDir>/scenarios	Directory of *.json scenario files.
eval.baselinePath	--baseline <path>	<outputDir>/eval-baseline.json	Where the committed baseline lives.
eval.minPassRate	--min-pass-rate <n>	0.95	Minimum fraction (0–1) of scenarios that must pass.
eval.maxTokenRegression	--max-token-regression <n>	0.25	Max fractional token increase (0.25 = +25%) before a passing scenario is flagged.

--model <id> applies a model override to every scenario that doesn't set its own modelOverride. --api-key, --project-id, and --server follow the same precedence as every other command — see Configuration.

Cost & limits

Each scenario run is one dry-run LLM call — same cost profile as the dashboard's sandbox. Three guards keep that spend bounded:

• Monthly budget. Eval spend counts toward the project's monthly budget (Dashboard → Project → Budget), together with production and sandbox usage. If the project is over its cap with mode block, the run is refused with BUDGET_EXCEEDED before any LLM call; with mode degrade, scenarios run on the same cheaper fallback model production traffic would get (check modelUsed in the results if a degraded run surprises you).
• Per-request cap. A single POST /cli/eval/run request is capped at 20 scenarios; split a larger suite across multiple runs (e.g. by feature area) if you need more than that.
• Hourly rate limit. Across all requests and API keys of a project, at most 200 scenarios per sliding hour run by default. Beyond that the API answers 429 with a Retry-After header and a message saying how long to wait — a misconfigured CI retry loop stops costing money instead of fanning out unbounded LLM calls. Self-hosting? Tune the ceiling with the APPILOTS_EVAL_SCENARIOS_PER_HOUR env var on the API server.

CI

- name: Run Appilots agent eval
  run: npx --yes @appilots/cli eval --json
  env:
    APPILOTS_API_KEY: ${{ secrets.APPILOTS_API_KEY }}
    APPILOTS_SERVER_URL: https://api.appilots.com

Run it after sync in the same job, so the eval always checks the document you just uploaded. See CI Integration for the general pattern (env-var-only config, --json for a parseable result, non-zero exit on failure).