Decisions

Not every step needs a coding agent. Which team owns a ticket, whether a comment reports a bug, how urgent an issue is: these are closed questions about a piece of text. A decision model answers them in about 200 ms with a probability for every possible answer. No agent starts, nothing reads your repository, and there is no turn to wait on.

awf has one decision model built in, jev, and it needs an OPENROUTER_API_KEY (below). Everything else comes from agentswf/workflow:

import { typed } from "agentswf/typebox";
import {
  choice,
  DecisionError,
  defineWorkflow,
  score,
  yesNo,
} from "agentswf/workflow";

Write the questions once, ask the model, keep what it is sure of, and send the rest to an agent:

export const QUESTIONS = {
  team: choice("Which team owns `ticket`?", {
    payments: "Checkout, billing and refunds",
    accounts: "Sign-in, sign-up and permissions",
    frontend: null,
  }),
  bug: yesNo("Does `ticket` report something broken?", {
    yes: "something that worked no longer does",
    no: "a request for something new or different",
  }),
  urgency: score("How soon does `ticket` need a fix?", URGENCY),
};

const decided = await wf.decisions
  .decide({
    key: "route",
    model: "jev",
    state: { ticket },
    questions: QUESTIONS,
  })
  .catch((error: unknown) => {
    if (error instanceof DecisionError) return undefined; // the agent decides instead
    throw error;
  });

if (decided) {
  const { team, bug, urgency } = decided.answers;
  const sure =
    team.probabilities[team.choice] >= 0.9 &&
    Math.abs(bug.yes - 0.5) >= 0.4;
  if (sure) {
    // `expected` is the probability-weighted level, 0 to 2 here.
    const level =
      urgency.expected >= 1.5 ? 2 : urgency.expected >= 0.5 ? 1 : 0;
    return {
      team: team.choice,
      bug: bug.yes >= 0.5,
      urgency: URGENCY[level],
      by: "decision",
    };
  }
}

const router = await wf.agents.open({ key: "router", runtime: "codex" });
const { outcome } = await router.run({
  prompt:
    "Find the code this ticket is about. " +
    `Which team owns it, is it a bug, how urgent?\n\n${ticket}`,
  schema: ROUTE,
});
if (!outcome.answered) return wf.stop(`${outcome.kind}: ${outcome.reason}`);
return { ...outcome.value, by: "agent" };

The ticket goes to the agent when the model is unsure of the team or of bug-or-not, or when the decision fails. The whole sample, with its test, is examples/site/guide/decisions.ts.

A decision or an agent

Use a decision	Use an agent
The answer is one of a few you can list	The answer is open, or is work
Everything it needs is in the text you pass	It has to read code, run a command, look something up
You ask the same question many times	It's asked once, and getting it right is worth minutes

A decision model judges the text you give it, nothing else. Measured on awf's own questions, it was reliable at matching text to a list of options, and not at anything that needed reading code: the findings have the numbers. So the shape above is the usual one: a threshold on the probability, and an agent behind it.

The call

wf.decisions.decide takes one state and any number of questions about it, and returns an answer for each:

• key names the call in the run's record, and its usage when the run has no stages. A key is a name, as an agent's is, with no colon. Keys need not be unique.
• model is the alias of a decision model: jev, the built-in one, while OPENROUTER_API_KEY is set.
• state is what every question is about: a string, an object or an array. The model reads it once per call however many questions you ask, so ask them together rather than one call each.
• questions are at least one, keyed by the name you read the answer by. Each is answered on its own and doesn't see the others' answers.
• timeout, optional, a duration or a Date, bounds this call, within the deadline around it.

The sample at the top uses all three kinds of question:

Question	Gives	Its answer
choice(instructions, options)	options by name, each with a description, or null when the name says enough	choice, the most likely name, typed from your options; probabilities by name
score(instructions, levels)	ordered levels, lowest first	level, the most likely index; expected, the probability-weighted level; probabilities, one per level
yesNo(instructions, criteria?)	optionally what counts as yes and as no	yes, the probability of yes

On a tie, choice is the first option and level the lowest. Every answer keeps the whole distribution as the model gave it, not renormalised: the top pick is a convenience, and the probabilities are what you decide on.

Instructions and descriptions can be structured text, an object, as well as a string.

Choosing a threshold

Set your own threshold on the probabilities. It depends on the question and what a wrong answer costs you:

• A choice: the top probability, team.probabilities[team.choice]; a score's is urgency.probabilities[urgency.level].
• A yes/no: how far yes is from 0.5. A confident no, yes at 0.05, is as sure as a confident yes.
• A score read as a number: expected and cut points of your own, as the sample does for urgency, often say more than level.

The model is close to deterministic but not quite: the same call can move a probability by a few hundredths. On matching review comments to a list of issues, answers at 0.9 or above were right 117 times in 118 in the measurements above, but the issues' descriptions had been drafted from those same comments, so text written independently will match less well.

The model and its key

jev is Jev through OpenRouter. It needs OPENROUTER_API_KEY, set in the environment or in a .env in the folder where you run awf. That is the only name awf reads from .env.

The engine makes the calls and holds the key; it is removed from every agent's environment, so no agent can read it. A run without the key starts as usual; only asking for jev is refused.

Each call's record goes into the run's out.json: the key, the model and the versioned snapshot that answered, the outcome, the attempts, and tokens as far as the provider reported them. The request and its answers are kept beside it, in ops/{key}.{n}/decision.json in the run's directory, the nth asking of the key, so you can fit a threshold on real answers later. A threshold holds for the snapshot it was fitted on.

When a decision fails

• Refused before it is sent. A call that can't be asked rejects with an Error that says what to fix: no key, no questions, a question of unknown type, a state that isn't text, an object or an array, an unknown alias, a bad timeout, or jev without a key, for example decision model "jev" is unavailable: OPENROUTER_API_KEY is not set. Nothing is sent or recorded. A choice with no options or a score with no levels throws sooner, where you build it.
• Retried. When OpenRouter can't be reached, is rate limited or has a server error, awf tries again, up to three requests in all, waiting longer each time, and not past the deadline. Each request may be billed, so the record counts them as attempts.
• Failed. Any other error, or an answer that skips a question or whose probabilities don't add up, rejects with DecisionError. You never get a partial answer. The error's record is the call's record, which the run keeps too.
• Out of time. A decision runs under the deadline around it, the run's --timeout or a parallel's, and rejects with DeadlineExceededError when it passes. See Timeouts.

Uncaught, any of these fails the run like any other thrown error. To hand the item to an agent instead, catch DecisionError, as the sample at the top does, and let anything else through: a missing key then fails the run rather than quietly sending every item to an agent.

The provider sets its own limits on the number of options and levels and the size of the state, and refuses past them with an error that says which; awf passes it on as a DecisionError.

Testing

awf test never calls the model. You script each decision's answers by key, as a probability per option, a level's probabilities, or just the answer you want:

const run = await testWorkflow(route, {
  args: { ticket },
  decisions: {
    route: {
      team: { payments: 0.97, accounts: 0.02, frontend: 0.01 },
      bug: 0.95,
      urgency: [0.1, 0.3, 0.6],
    },
  },
});

A decision with no script fails the test. Testing covers the rest, and the testing API every form a scripted answer takes.

See also

• Decisions in the workflow API.
• examples/triage: tickets triaged in parallel, three questions per call, the unsure answers flagged, with its test.
• Parallel: decisions are fast enough to ask for every item of a list at once.