Agent Skills: Skill: Write a Spec

Skill: Write a Spec

Every evals/core/*.e2e.test.ts or evals/proofs/*.proof.test.ts a PR adds or changes runs in CI on the PR head and is published as that PR's proof (private report + one PR comment). Write the spec for the person who will read that report in thirty seconds, not for the machine that runs it.

Map: from change to reviewed proof

Which lane runs your spec (and when to tag)

CI picks the lane from the file name and tags alone. Pick the lane first: it decides the world you can build.

Tag checkpoints only when a still image cannot show what the reviewer has to check: an open stream, a populated workspace to explore, a state that costs minutes to reach. The tag moves the spec to a protected lane that uses a secret, where fork PRs cannot run. It saves each tagged test's world as launched for the commit (before the body acts) and its end state, plus every user.checkpoint("caption") and every step(..., { checkpoint: true }). Plain user.screenshot() never saves one. Say why in a comment beside the tag. Checkpoints cost VM snapshots, so capture the one or two moments a reviewer would open, not every step. Run it locally with pnpm evals:e2e <slug> --local --checkpoints (needs FREESTYLE_API_KEY).

Do not write one when…

• A core journey covers the behaviour and the change broke or extends it: add a step there instead.
• The change is a pure function or a server rule (permissions, isolation, validation): write a colocated unit or API test (ee/apps/den-api/test/api for Den security rules), not an e2e proof.
• You would import ../../apps|packages|ee, read source files, or spawn another runner. That is a unit test in disguise; pnpm --dir evals lint:specs rejects it and names the core journey to extend instead.

No UI does not mean no spec. A server, proxy, or protocol bug gets a browser-less world whose steps are the requests a real client makes; the steps still read as one person's before → after.

Core journey or proof? There are nine core journeys (evals/journeys.ts); adding a tenth is a review decision. Almost every PR writes a one-off proof in evals/proofs/<name>.proof.test.ts: it runs on your PR and is not maintained.

The proof shape

A proof answers: who can now do what they could not before, and who still cannot. Structure every spec as that story. Each beat is one step() with a claim-sized name; each visual beat ends in user.screenshot().

persona  → the test title names a person: "an owner", "a teammate", "a member without access"
before   → what that person sees or cannot do today (features only)
action   → the person does the thing, through the UI, as they would
after    → what they now see; screenshot at that moment
boundary → who else is affected, and the negative half: who is not

The review app is generated from the spec. It shows exactly four strings, and each comes from one place in your code; write those strings for the reviewer:

Each test is one section, marked with a status icon. Its checks collapse to "N of M checks passed" and open only when one fails. Its screenshots form a numbered gallery in capture order, so the captions have to tell the story on their own. A section with no recordAssertionEvidence shows as Incomplete.

A screenshot taken outside any step() is captioned "<title> artifact N", which tells the reviewer nothing. No verbs like "assert", no selectors, no internal names anywhere in those strings.

Example title and steps:

test("an owner enables Code Mode and a teammate turns one chat into a shared Workflow", async ({ user, probe, step }) => {
  await step("before: the teammate's agent has no script tool", …);        // screenshot
  await step("the owner enables Code Mode with one switch", …);             // screenshot
  await step("after: the teammate's request runs as one script", …);        // screenshot
  await step("the result is saved as a Workflow the team can open", …);    // screenshot
  await step("a member outside the team cannot see it", …);                // screenshot
});

Assertions still live inside each step; they just hang off user-visible moments. A spec whose assertions are RPC responses collected in the world and compared in one block is a mechanism check, not a proof. It is still allowed; it just tells the reviewer nothing.

Same story in three places

The spec, the report, and the PR body tell one story in the same words. Write the report you want to read first, then make the code produce those lines. A bug fix with no screen still gets this shape; the "screenshot" is one recordAssertionEvidence line per step whose evidence is the observed fact.

Report the reviewer reads (every line comes from the spec):

▶ a member's conversation stays readable while OpenWork is still checking who owns it
  ✔ given an engine where the ownership check is slow and the messages read is fast
      GET /session/:id answers in 150 ms; GET /session/:id/message in 2 ms
  ✔ when the member opens the conversation and memory is reclaimed mid-check
      three reads sent as the app sends them; gc() ran three times
  ✔ then the runtime really reclaimed the engine's response
      WeakRef → undefined  (witness; without it a pass means "nothing happened")
  ✔ after: all three reads return the conversation, none say internal_error
      200 / 200 / 200; body {"id":"ses_x", …}

Spec that produces it:

test("a member's conversation stays readable while OpenWork is still checking who owns it", async ({ world, step, evidence }) => {
  await step("given an engine where the ownership check is slow and the messages read is fast", async () => {
    evidence.recordAssertionEvidence("engine timings", `GET /session/:id ${world.ownershipMs} ms; GET /session/:id/message ${world.messagesMs} ms`, true);
  });
  await step("when the member opens the conversation and memory is reclaimed mid-check", async () => { /* act; gc(); */ });
  await step("then the runtime really reclaimed the engine's response", async () => {
    const collected = world.engineResponse.deref() === undefined;
    evidence.recordAssertionEvidence("engine response reclaimed", collected ? "WeakRef → undefined" : "WeakRef still alive: window not exercised", collected);
    expect(collected).toBe(true);
  });
  await step("after: all three reads return the conversation, none say internal_error", async () => {
    evidence.recordAssertionEvidence("reads", `${statuses.join(" / ")}; body ${body.slice(0, 40)}`, statuses.every((s) => s === 200));
    expect(bodies.join()).not.toContain("internal_error");
  });
});

PR body line that names it (open-a-pr):

## Evidence
`evals/specs/conversation-stays-readable.e2e.test.ts` — before: the read fails
with "Response body object should not be disturbed or locked"; after: all
three reads return the conversation and the reclaim witness is true.

Rules that make this parseable:

• Step names are given / when / then / after: (or before: / after: with screenshots). One clause each, the person's words, no function names.
• Every step records exactly one evidence line: the observed fact, with the numbers in it. The reviewer never opens the code to learn what happened.
• A witness step proves the risky condition actually occurred. If it did not, that step fails; a pass without the witness is not a proof.
• The quoted "before:" string in the PR body is the exact error the user or Sentry saw. CI shows it red on dev and green on the head; never narrate it.

Rules of thumb:

• At least three steps. Zero steps renders as "no claims declared".
• Every step has at least one user.* act or user.see. probe.* observes; it does not carry a step alone.
• One screenshot per visual beat. Non-visual work records recordAssertionEvidence with the command or response excerpt as the evidence text.
• For features, show the before state in the same world. Base-vs-head runs are expensive; a step that starts with the switch off is not.
• For permissions, sharing, or scopes, always include the negative persona.
• The world runs the code on the runtime it ships on. A spec that passes on the wrong runtime proves nothing (run-tests → Match the runtime).
• Prefer seed.appWeb (headless Chrome, real app). Use seed.desktop only for a native capability a browser cannot show, and say why in nativeReason: the titlebar and window controls, native menus and dialogs, openwork:// links and the OAuth hand-off, the built-in browser pane, the updater, quit and relaunch, the desktop runtime's own settings, or packaged builds. A desktop world opens a window on whoever runs it locally.
• Signed in to a Den: denForAppWeb then signedInAppWeb (evals/worlds/signed-in-app-web.ts) replace seed.desktop({ den, as }).
• Extending a journey whose world uses seed.desktop for none of those reasons? Move that world to seed.appWeb rather than adding to it. The app in a browser routes by path, not hash: use go() and probe.hash(), which handle both.
• No seed.evalIn / probe.eval in new specs. If you need one, comment why.

Use the testkit channels

Import spec (and expect) from @openwork/testkit; bind the world:

const test = spec.world(myWorld, {
  resources: { surfaces: ["appWeb"], services: ["den"] },
});

Well-known targets: "composer" for the editor; otherwise role, label, text, placeholder, or test id. Bound every wait; declare external requirements in needs() so a missing dependency skips loudly.

Claims and witnesses

• Every claim is machine-checkable with an observable assertion and its negative half: what must not happen to another identity, account, or state.
• Prose is never proof. Screenshots explain an assertion; they cannot replace it.
• Describe product behaviour, not incidental layout.
• If product and claim diverge, change one explicitly; never bend the claim.
• Never smuggle the answer into the prompt: assert the user-facing request carries no connector or resource IDs.
• Use mcpMock() witnesses under evals/packages/labs/src/; never call real providers. Witnesses are deterministic, identity-scoped, and queryable.

Evidence contract

• Evidence is ambient: user.screenshot() records an artifact, user.looks() records a visual validation (judged later; pending until then), recordAssertionEvidence() records a witness assertion. Never create or pass recorder handles.

• Run it once before pushing and read your own record:

  pnpm evals:e2e <slug> --local
  ls evals/results/test-runs/<latest>/   # index.html, test-run.json, NN-*.png
  

  If the screenshots would not convince you, they will not convince the reviewer. Read the captions in index.html top to bottom: they should tell the before → after story on their own.

• Your own eye passes what you designed. Run the design review on the same record before pushing and fix its medium notes, or say in the PR why the screen is right (evals/design-review/README.md; on a PR, the fix-design-notes skill reads the notes from the Evidence preview check):

  pnpm --dir evals design:review -- --test-run latest
  

• In CI the spec runs on PR change proof, one job per spec; the trusted publisher aggregates every changed spec's records into one report. Failed, skipped, and cancelled runs stay visible as such; nothing substitutes for them.