agents.wf

Appearance

Testing ​

A workflow is code that branches on what agents say: an answer, a refusal, silence. A test gives it each of those on purpose. awf test runs your workflow through the real engine with every agent replaced by a script of answers, so no agent starts, nothing is metered, and a test takes milliseconds. It checks your workflow's logic: its branches, its loops, what it does with each outcome. How well a real agent answers is a different question, and a test doesn't ask it.

A workflow and its test ​

The workflow asks a writer for release notes and, when the writer says the change is breaking, asks a checker on another harness to confirm:

ts
import { typed } from "agentswf/typebox";
import { defineWorkflow } from "agentswf/workflow";
import Type from "typebox";

export const NOTES = typed(
  Type.Object(
    { notes: Type.String(), breaking: Type.Boolean() },
    { additionalProperties: false },
  ),
);
export const CHECK = typed(
  Type.Object(
    { breaking: Type.Boolean(), why: Type.String() },
    { additionalProperties: false },
  ),
);

export type ReleaseNotes = {
  notes: string;
  breaking: boolean;
  checked: boolean;
};

export default defineWorkflow({
  meta: {
    name: "release-notes",
    description:
      "Draft release notes; a second agent checks a claimed breaking change.",
  },
  prepare: ({ argv }) => ({ range: argv[0] ?? "origin/main...HEAD" }),
  async run(wf, args: { range: string }): Promise<ReleaseNotes> {
    const writer = await wf.agents.open({ key: "writer", runtime: "claude" });
    const draft = await writer.run({
      prompt: `Write release notes for the commits in ${args.range}.`,
      schema: NOTES,
    });
    if (!draft.outcome.answered)
      return wf.stop(`no notes: ${draft.outcome.reason}`);
    const { notes, breaking } = draft.outcome.value;
    if (!breaking) return { notes, breaking, checked: false };

    const checker = await wf.agents.open({ key: "checker", runtime: "codex" });
    const check = await checker.run({
      prompt:
        `These notes call ${args.range} a breaking change. ` +
        `Read its diff and say whether it is.\n\n${notes}`,
      schema: CHECK,
    });
    if (!check.outcome.answered) return { notes, breaking, checked: false };
    return { notes, breaking: check.outcome.value.breaking, checked: true };
  },
});

Its test sits beside it:

ts
import { expect, test } from "bun:test";
import { answer, noAnswer, testWorkflow } from "agentswf/testing";
import releaseNotes, { CHECK, NOTES } from "./release-notes";

const args = { range: "v1.2.0...HEAD" };

test("notes with no breaking change take one agent and one turn", async () => {
  const run = await testWorkflow(releaseNotes, {
    args,
    agents: {
      writer: answer(NOTES, { notes: "Faster startup.", breaking: false }),
    },
  });
  expect(run.value).toEqual({
    notes: "Faster startup.",
    breaking: false,
    checked: false,
  });
  expect(run.agents.map((agent) => agent.key)).toEqual(["writer"]);
  expect(run.turnsOf("writer")[0]!.prompt).toContain("v1.2.0...HEAD");
});
sh
awf test
  • testWorkflow(workflow, options) runs the workflow and returns what it did. Its args are an option, which a workflow that takes them must be given.
  • agents maps each agent's key to its script. answer(NOTES, value) answers a turn that asks for NOTES with value, checked against the schema as a real answer is. The script names the same schema the workflow asks with; a different one fails the test with it asks for … its script answers ….
  • run.value is what the workflow returned; run.agents the agents it opened; run.turnsOf(key) one agent's turns, each with the prompt it was given.

The tests are Bun's: test and expect come from bun:test.

Scripting answers ​

An answer can be computed from the turn. The checker here reads its prompt, and answers differently by what it says:

ts
test("a claimed breaking change goes to a checker on codex, which can overrule it", async () => {
  const run = await testWorkflow(releaseNotes, {
    args,
    agents: {
      writer: answer(NOTES, { notes: "Removed --legacy.", breaking: true }),
      checker: answer(CHECK, (turn) => ({
        breaking: !turn.prompt.includes("--legacy"),
        why: "--legacy was deprecated two releases ago",
      })),
    },
  });
  expect(run.value).toEqual({
    notes: "Removed --legacy.",
    breaking: false,
    checked: true,
  });
  expect(run.agentOf("checker").execution.alias).toBe("codex");
});

The function gets the turn: turn.prompt, turn.n, turn.agent, turn.nudge and more, and it may be async. answer("text") answers a turn that asks for no schema.

A script is one entry, which meets every turn of that agent, or a list, one entry per turn: reviewer: [answer(VERDICT, revise), answer(VERDICT, approve)] answers the first review with revise and the second with approve. A list is strict: a turn past its end, or an entry never reached, fails the test. A key can be a pattern, "reviewer-*", for agents a workflow opens in a loop; an exact key wins over a pattern.

Scripting outcomes ​

A test earns its keep on the turns that don't go to plan. noAnswer ends a turn without an answer, the way a real agent can:

ts
test("a writer that never answers is nudged once, then the run stops with why", async () => {
  const run = await testWorkflow(releaseNotes, {
    args,
    agents: { writer: noAnswer.silent() },
  });
  expect(run.turnsOf("writer").map((turn) => turn.nudge)).toEqual([
    false,
    true,
  ]);
  expect(run.ending).toMatchObject({
    kind: "stopped",
    reason: expect.stringMatching(/^no notes: /),
  });
  expect(() => run.value).toThrow();
});

test("a checker whose harness fails leaves the writer's word, marked unchecked", async () => {
  const run = await testWorkflow(releaseNotes, {
    args,
    agents: {
      writer: answer(NOTES, { notes: "Removed --legacy.", breaking: true }),
      checker: noAnswer.failed("harness crashed"),
    },
  });
  expect(run.value).toMatchObject({ breaking: true, checked: false });
  expect(run.turnsOf("checker")[0]!.outcome).toBe("failed");
});
  • noAnswer.silent() ends the turn with no answer. awf sends one nudge, as it would on a real run, and the same script entry meets the nudge with turn.nudge true; still silent, the turn ends unanswered.
  • noAnswer.blocked(), noAnswer.failed() and noAnswer.timedOut() end it blocked, failed and timed-out, each with a reason you can pass. noAnswer.hang() holds it until the engine cancels it, to test a deadline or a sibling's failure cancelling it under wf.parallel.
  • run.ending is how the attempt ended, the Ending a workflow's report gets: kindstopped with the reason and stage when the workflow stopped with wf.stop, and reading run.value then throws, as it does when the workflow threw.

noAnswer.needsLogin() and every other script are in Scripts.

Asserting what the run did ​

run holds more than the value. The ones you reach for first:

  • run.turnsOf(key): each turn's prompt, schema, nudge, model, outcome and stage, nudges included. run.turns has every agent's.
  • run.agentOf(key): what the agent was opened with, such as execution.alias, its instructions, skills and sandbox. run.agents lists them all.
  • run.stages: each stage the attempt entered, in order, by its name, outcome, attempt and value. A reused stage's is from the earlier attempt that ran it.
  • run.logs and run.decisions: the log lines and decision calls.

Under wf.parallel, read agents by key, not by index. The full list is What the run did.

Arguments ​

testWorkflow passes its args option to run as given; it doesn't call prepare for them. Test prepare by calling it with the argv it would get:

ts
test("the range is the first argument, origin/main...HEAD without one", () => {
  expect(releaseNotes.prepare({ argv: [], cwd: "/repo" })).toEqual({
    range: "origin/main...HEAD",
  });
  expect(
    releaseNotes.prepare({ argv: ["v1.2.0...HEAD"], cwd: "/repo" }),
  ).toEqual(args);
});

Running tests ​

sh
awf test                                # every *.test.ts and *.spec.ts under here
awf test release-notes.test.ts          # one file
awf test -t "breaking"                  # the tests whose names match
awf test --watch                        # rerun on a change
text
usage: awf test [paths...] [-t <pattern>] [--watch] [--timeout <duration>]

Runs a workflow's tests (*.test.ts and *.spec.ts files) under the current directory, or under
each file or directory given, with agentswf/workflow, agentswf/testing and typebox served as
awf run serves them: nothing is installed. -t runs the tests whose names match the pattern;
--watch reruns them on a change; --timeout bounds each test, 5s by default.

Exits 0 when every test passed, 1 when one failed or none was found, 2 on a usage error. What
it prints is for reading, not parsing.

awf test serves agentswf/workflow, agentswf/typebox, agentswf/client, agentswf/testing and typebox itself, so a folder of workflows that declares no agentswf needs no bun install to be tested; one that declares agentswf runs on the one it installs. The runtime aliases claude and codex resolve as they do on awf run; runtimes in the options adds or replaces others.

Catches ​

  • A missing script fails the test, not the workflow. An agent with no script, an answer its turn's schema refuses, or a script that throws is the test's mistake, and the test fails saying so.
  • A stuck run fails rather than hangs. Nothing starting or ending for 2 s fails the test; stall in testWorkflow's options, a duration such as "5s", changes it. A loop whose turns keep answering never stalls, so more than 20 turns started fails the test; maxTurns changes that.
  • A misspelled option fails the test. An option testWorkflow doesn't take is refused, named, rather than ignored.
  • Deadlines are real time. A turn's timeout runs on the clock, so keep them short in a test, or end the turn at once with noAnswer.timedOut().
  • Sandboxes are fakes. They confine nothing; a test checks what each agent was given, not what a real sandbox enforces.
  • Messages, decisions, events and stages have scripts too. An agent messaging another, a decision model's answers, a workflow parked on an event, a continue over what an earlier attempt left: each has its own option, on the reference page.

Next ​