Skip to main content

Recommended AI Models

When you create a project, step 3–5 of setup has you choose an AI provider, add its API key, and pick a model. This page is the decision guide for that last step — which model fits which use case, and which ones need a caveat. It applies to apps that use their own key; apps on AI included run on a model Appilots picks.

The full, machine-readable list — exact model ids, pricing, and context windows — lives in MODEL_CATALOG (packages/shared/src/models/index.ts) and drives the dashboard directly, so neither can drift from the other. The picker offers the subset we've verified end to end; the cost calculator prices the whole catalog. This page explains the reasoning behind both in prose.

Only models with native tool calling are listed​

Appilots agents act on your app through function/tool calls (navigate, form_fill, ui_interaction, and so on). A model that doesn't support native tool calling reliably can't drive those actions, so the catalog — and the dashboard's model picker — only offers models whose provider API supports function calling natively. There's no JSON-envelope fallback for weaker models; that path existed early on and was removed after causing repeated "model promises but doesn't act" bugs.

This is also why a couple of well-known cheap models (o1-mini, Mistral's Codestral) simply aren't in the list — their tool-calling support is either non-native or unstable enough that we exclude them rather than risk it in production.

Only models we have actually run are offered​

Supporting tool calling is necessary but not sufficient. A model is only offered as a one-click choice in the picker once we've run a real mission through its provider's transport end to end and watched a real tool call come back.

As of 2026-08-08 that covers OpenAI and Anthropic. It does not yet cover Google, Mistral, or xAI: their models are in the catalog, priced, and accepted by the relay, but no request from Appilots has ever reached those endpoints. We're not going to imply otherwise by listing them next to models we've tested for months.

This is a policy with an explicit way back in, not a permanent exclusion — a company returns to the picker as soon as a smoke run passes against it (see Running it yourself). Concretely, the split means:

  • You can still use them. Choose the provider, then type the model id — the field accepts free text when a company has nothing verified. Pricing, cost telemetry, and budget degrade all keep working, because the model never left the catalog.
  • You should test your own missions first. That's good advice for any model; for these it's the only testing that will have happened.

Picking by use case​

Best cost/quality balance (recommended default). Start with the "recommended" models in the dashboard's picker — currently OpenAI's GPT-5.6 Terra and GPT-5.6 Luna, and Anthropic's Claude Sonnet 5 and Claude Haiku 4.5. All have solid native tool calling and handle typical navigation, form-filling, and multi-step missions well.

Changed 2026-08-08. Gemini 2.5 Flash used to be listed here as a recommended default. It was recommended without ever having been run, which is exactly what the coverage policy above exists to prevent, so the recommendation is withdrawn until a smoke run backs it up. Nothing about the model changed — what changed is that we stopped vouching for something we hadn't tested.

Complex, long-horizon, or high-stakes flows. Reach for a flagship-tier model — OpenAI's GPT-5.6 Sol, Anthropic's Claude Opus 5, or (for the hardest reasoning-heavy work) Claude Fable 5. These cost more per token, so they're worth it for flows with many steps, ambiguous instructions, or where a mistake is expensive to recover from — not for routine traffic.

High-volume or latency-sensitive flows. The cheapest tiers — GPT-6 Luna, GPT-5.6 Luna, and Claude Haiku 4.5 — are the right default when you're paying per call across a lot of traffic and the task itself is simple (single-step navigation, short confirmations). GPT-6 Luna is the cheapest model in the catalog ($0.10 per million input tokens, $0.50 output), about ten times cheaper than GPT-5.6 Luna on input. In our 2026-09-29 corpus run it scored slightly below GPT-4.1 mini, with regressions on form writes, so test your own form missions before choosing it. Gemini 2.5 Flash-Lite stays in the catalog for cost comparison, but it falls under the unverified caveat above. Read the caution below before pointing a multi-step form flow at any cheap tier.

AI included (managed) apps run on GPT-6 Luna.

Budget caps. When a project hits its budget in degrade mode, Appilots switches to a cheaper model from the same provider. For OpenAI models, that fallback is GPT-4.1 mini. GPT-6 Luna is never used as a fallback. A project already on GPT-6 Luna or GPT-4.1 mini has nothing cheaper to fall back to, so it's blocked when it reaches its budget.

Caution: cheap models and multi-step forms​

Cheaper, smaller models are the ones most likely to reach for a tool call before they actually have enough information — and, worse, to fill in a plausible-looking value instead of asking. We've reproduced this directly:

  • Issue #107 / BUG-015 — a regression probe asking a cheap OpenAI model to request clarification on an ambiguous form instead had it call form_fill immediately, and on one run it hallucinated a CPF value that didn't exist anywhere in the conversation or the screen state, then submitted the form.
  • An earlier probe had the same model fail a multi-hop registration mission 4 out of 4 times, inventing fieldIds that didn't exist on the target screen.

Neither failure mode showed up when the same missions ran against a larger model from the same provider. The practical guidance:

  • For any flow that fills out or submits a form, especially one with fields that could be ambiguous (which record, which plan, which of several similar-looking rows), prefer at least a mid-tier model (GPT-5.6 Terra/Sol, Claude Sonnet 5/Opus 5, Gemini 2.5 Flash/Pro) over the cheapest tier.
  • If you do run a cheap model on form flows for cost reasons, budget time to test the exact missions your app exposes — small models fail inconsistently (not every run), so a handful of manual tries can look fine and still hide a real failure rate. The pnpm qa:eval harness (see tests/agentic/scenarios/) is built for exactly this: replaying a scenario against a specific model and checking whether it asks for clarification, hallucinates a field, or gets it right.
  • The catalog flags this in each affected entry's tooltip (the notes field in MODEL_CATALOG) rather than removing the model outright — it still has native tool calling and is fine for simpler flows, so cutting it from the picker entirely would be overkill.

Provider notes​

  • Anthropic models go through Anthropic's native SDK specifically so prompt caching applies (a large discount on repeated context, like your app's MCP document). Don't route Anthropic traffic through an OpenAI-compatible shim even if one exists.
  • Google, Mistral, and xAI are reached through their OpenAI-compatible endpoints. Their documentation says those endpoints forward function calling, and the wiring on our side is complete — but as noted above, we have not yet confirmed it with a live request, so they are catalogued rather than offered.
  • A couple of catalog entries carry a note that their id is a moving target — Mistral's -latest aliases and Google/OpenAI's fast-moving lineups can change price or capability underneath a stable-looking id. If costs look off, check the provider's own pricing page before assuming the dashboard is wrong.

Picking a model outside the catalog​

If you configure a provider with a model id that isn't in the catalog, Appilots still accepts it when the provider is OpenAI or Anthropic (their SDKs handle tool calls natively regardless of which specific model you name) — you'll just lose cost estimation in the usage dashboard until it's added to the catalog. Any other provider with an uncatalogued model id is rejected outright, since there's no way to confirm its endpoint forwards tool calls correctly.

Note the difference between uncatalogued and unverified. A Gemini or Grok model typed into the field is catalogued — it keeps full cost estimation and is accepted by the relay; it just hasn't been smoke-tested by us. An id that's in neither the catalog nor a trusted-SDK provider is rejected on the first call.

How we measure capability (the scoreboard)​

The provider picker shows measured capability data per model. Here is exactly what those numbers mean, so you can judge how much weight to give them.

What is being measured​

Model capability inside Appilots — not a universal LLM benchmark. Every attempt runs through the real relay: our system prompt, our tool schemas, our guards, your app's MCP document shape. That is deliberate, because it is the thing that actually predicts your experience. A model that tops public leaderboards but mis-targets fieldIds against our tool schema will score badly here, and that is the correct signal for someone choosing a model in this dashboard.

The corpus is the scenario set in tests/agentic/scenarios/ — 67 single-shot decision scenarios spanning seven capability families:

ArchetypeWhat it tests
NavigationGetting to the right screen, resolving the right entity
Form fillingFilling and submitting with user-provided values
AmbiguityAsking which one, instead of guessing, when several rows match
Destructive confirmationRequesting a confirm handshake before something irreversible
Injection resistanceRefusing prompt injection, never echoing PII
Answer without actingAnswering from what is on screen, emitting zero actions
UI controlsSliders, segmented controls, toggles

Pass/fail is deterministic: each scenario declares the actions it expects and a matcher checks them. There is no LLM judging another LLM in these scores — that would be a bias we cannot audit.

Assisted vs raw — the two numbers​

Appilots actively rescues model mistakes. It rewrites misplaced field ids, blocks an ambiguous press and turns it into a clarifying question, injects a confirmation gate before destructive actions, and retries once when a model promises an action but calls no tool. A single pass rate would hide all of that and tell you nothing about the model you are picking.

So every model gets two scores:

  • Assisted — the outcome with every Appilots guard active. This is what you will experience, and it is the number the picker leads with.
  • Raw — the model got it right unaided: no guard had to correct it, and it never referenced an on-screen element that did not exist.

The gap between them is how much Appilots is compensating for that model. A model with 97% assisted and 75% raw is being carried on roughly a fifth of its attempts; one with 97%/95% is genuinely doing the work itself. Both ship the same result to your users today, but the smaller gap is the more robust choice as your app's screens get more complex than our corpus.

Two things deliberately do not count against the raw score, because counting them would measure Appilots rather than the model:

  • Mandatory safety machinery. The confirmation gate before a destructive action is injected by the server on every such action, for every model — the model cannot even request it (confirm is not one of its tools). So it tells you nothing about model quality and is excluded. What does count is the model emitting an action that policy had to block outright.
  • Turns the model never saw. Some prompt-injection attempts are refused by a deterministic server-side rule before any model is called. Those attempts are reported separately as not measured and are removed from the raw denominator entirely — treating them as a model failure would blame it for a turn it never received, and treating them as a success would credit it for a refusal Appilots made on its behalf. The details panel tells you how many attempts this affected.

A concrete example from a real run: on injection resistance, both models score 100% assisted, but over the attempts that actually reach a model the raw rates are 100% and 66.7%. Identical experience for your users today; a meaningful difference in how much of that is the model versus the guardrails.

Why some models look "tied"​

Every rate carries a Wilson 95% confidence interval. When two models' intervals overlap, the picker shows them as a tie and gives both the "recommended" chip — it will not rank them.

This matters more than it sounds. With 67 scenarios at 10 repetitions, a 1.5-point difference between two models is well inside the noise. Presenting that as a ranking would be inventing a distinction that the data does not support. When models tie on accuracy, the honest differentiators are the ones also on the card: cost per completed mission and latency.

Cost per completed mission is deliberately not cost per call — it is total spend divided by successfully completed missions. A cheap model that fails a third of the time is not cheap. A model that completed nothing shows a dash, never $0.00.

Sample size and freshness​

  • A board is only published at N ≥ 10 repetitions per scenario per model. Anything below that is marked draft and is never served to the dashboard.
  • Each attempt is independent; failures caused by provider errors count as failures, never as skipped attempts, so an unreliable provider cannot flatter its own score.
  • Every board records a promptVersion — a hash of the relay sources that define agent behavior. When we change the system prompt, tool schemas, or guards, the dashboard marks the existing board as out of date rather than hiding it, because a slightly stale measurement is still more useful than none.
  • The card shows the measurement date and N directly.

What it does not measure​

  • Multi-hop trajectories. The published board is single-shot decisions. The multi-hop suite exists (pnpm qa:eval:flow) but is costlier and lags.
  • Your app. The corpus is a representative example app, not your screens. Treat the board as a shortlist, then test your own critical missions.
  • Text quality. Only action correctness. A model could pick perfect actions and write awkward prose and still score 100%.
  • Anything an archetype has saturated. When every model scores 100% on a family, that family is telling you nothing — it means our scenarios are too easy for the current generation, not that every model is equally safe there. Read a row where everyone ties as "no signal", and lean on the archetypes that still separate.
  • Models we have no key for. Google, Mistral and xAI models are in the catalog but not yet in the board. They are listed as skipped rather than omitted, so the gap is visible instead of looking like a low score. The bench itself can now run them — what's missing is a key, not support.

Running it yourself​

The bench is a repo script, so you can reproduce or extend any of this:

# Against a local stack, 2 models, publishable sample size
pnpm qa:model-bench --models=gpt-5.6-terra,claude-sonnet-5 --reps=10

# Coverage check: does this company's transport carry a real tool call?
# 5 scenarios at N=1 — roughly a dollar for all three uncovered companies.
QA_GOOGLE_API_KEY=... pnpm qa:model-smoke --models=gemini-2.5-flash

# Cheap partial run over one scenario family
pnpm qa:model-bench --models=gpt-5.6-terra --reps=2 --scenarios=spec-040

# Publish a board to a running API
pnpm qa:model-bench:ingest --file=qa/reports/models/scoreboard.json --ingest=http://localhost:4000/api/v1

# Where the board disagrees with the hand-written catalog flags (advisory)
pnpm qa:model-bench:catalog-check

Every run does a cost pre-flight against qa/agent-slo.json (modelBench.maxCostUsdPerRun) and aborts before spending anything if the matrix would exceed the budget. Models whose provider key is missing are skipped loudly rather than silently narrowing the comparison.

Next steps​

Last updated on