QA Agent Testing Skill | Agent Skills

QA Agent Testing

Design and run reliable evaluation suites for LLM agents, including tool-using, multi-turn, and multi-agent systems.

Default QA Workflow

Define the Agent Under Test (AUT): scope, tools, approval boundaries, out-of-scope requests, and safety rules.
Build a starter suite from real work:
- Smoke suite: 5-8 highest-signal checks for PR gates
- Regression suite: 15-25 tasks from real failures, tickets, or production traces
- Refusal/security pack: unsafe requests, prompt injection, tool-output poisoning, and exfiltration attempts
- Iterative coding trajectory: evolving specifications applied to the agent's own carried workspace, when extension quality matters
Define objective graders first: schema checks, golden traces, deterministic mocks, policy oracles, and tool side-effect checks.
Add model-based graders only where objective checks are insufficient; calibrate them and log judge versions.
Run offline evals with deterministic controls and trace logging.
Add optional online evals or canary comparisons for live traffic.
Gate changes on one consistent status model and log regressions.

Use the starter templates in assets/ for day-0 setup. The template keeps 10 tasks + 5 refusals as a starter scaffold, not a best-practice cap.

Determinism and Flake Control

Pin prompts, configs, fixtures, and tool mocks where possible.
Freeze time, timezone, and locale for tests that depend on them.
Log model, judge, and tool versions for every run.
Record traces: prompt or message history, tool name, args, outputs, latency, errors, retries, approvals, and side effects.

Minimal instrumentation: Instrument agents at three points only — LLM call entry/exit (with span IDs), tool invocations (input, output, duration), and branching decision points (which path was chosen and why). Avoid instrumenting every intermediate computation; each additional trace dimension increases latency and storage cost, and the exact overhead depends on SDK, sampling, export path, and backend. Start minimal, expand only when a category of failure is consistently hard to diagnose without it.

Evaluation Model

Use two layers and one rubric:

| Layer | What to Grade | Recommended Graders | |---|---|---| | Outcome | Final answer, constraints, refusals, citations, format | Schema/code graders, policy oracles, human spot checks | | Trace | Tool choice, tool args, approvals, recovery, side effects | Tool/trace graders, sandbox logs, targeted model graders |

Canonical Per-Task Rubric (0-3 each, 6 dimensions)

| Dimension | What to Measure | |---|---| | Task outcome | Did the agent accomplish the job correctly? | | Policy and constraints | Did it respect safety, scope, and user constraints? | | Grounding and evidence | Are claims, citations, and retrieved facts supportable? | | User communication | Is the result clear, appropriately scoped, and useful? | | Tool choice | Did it select the right tools, or correctly avoid tool use? | | Tool execution and recovery | Were tool args, approvals, retries, and side effects handled safely? |

Track these separately at suite level, not as per-task rubric rows: latency, cost, stability, bias or fairness, and debuggability.

Trace grading in practice: When grading the Trace layer, evaluate four properties independently:

Tool selection accuracy — did the agent call the right tool for the step? Use code-based graders (compare tool name to expected set).
Argument correctness — were the tool args valid and well-formed? Schema graders handle this.
Call ordering — did the agent sequence tool calls in a logical, dependency-respecting order? LLM-as-judge works well here.
Recovery behavior — when a tool failed or returned unexpected data, did the agent handle it safely (retry, escalate, or degrade gracefully)? Use fault injection to test this explicitly.

Prefer code-based graders for (1) and (2); reserve LLM judges for (3) and (4) where rubrics are harder to express as code.

CI Economics

PR gate: smoke suite plus critical refusal and security checks.
After a fix, rerun the smallest affected pack first; only then expand to the full regression or canary comparison.
Nightly or scheduled: full regression, adversarial pack, latency and cost tracking, and optional online eval comparisons.
High-risk changes: add targeted reruns for affected tools, prompts, or judge models.

Expert Judgment: Sizing, Cost, and Drift

Judgment calls a checklist alone will not surface:

Sizing an eval sample. Do not default to a round number. If you need to detect a specific effect size (e.g. "did this change drop accuracy by 5 points"), run a power calculation (references/eval-dataset-design.md has a worked example) — the answer is often larger than intuition suggests. A release-gate golden set answering "did anything obviously break" needs far fewer cases (15-25) than a statistical A/B claim needs (often 300-900+ per arm, depending on baseline and effect size). Match the sample size to the claim you intend to make, not to a fixed convention.
Human review vs. automated judging. Route to a human, not a judge model, when the decision is high-stakes/hard-to-reverse, when the task falls outside your judge's calibration coverage, when repeated judge runs disagree with themselves on the same case, or when the batch is small enough that human review is cheap relative to building and validating a judge prompt. See references/llm-judge-limitations.md for the full decision guide.
Cost and latency budgeting. Eval cost scales with (suite size) x (judge calls per case) x (judge model cost) x (run frequency). A PR-gate smoke suite (5-8 cases, mostly code-based graders) should run in seconds to low minutes and cost near-zero; reserve LLM-judge-heavy grading and multi-trial pass^k reruns for nightly or pre-release runs, not every commit. If a suite's per-PR cost or wall-clock time creates pressure to skip it, that is a signal to split it (fast code-based gate on every PR, expensive judged/adversarial pack on a schedule) rather than to weaken the gate.
Overfitting to your own eval suite. A static suite that a team iterates against for months stops measuring what it was built to measure, even with zero contamination. Keep a held-out slice untouched by prompt iteration, and treat a long-stable score with mild suspicion rather than pure satisfaction. Full treatment in references/eval-dataset-design.md.
Judge-model drift. Judges drift for three separate reasons — the provider updates the model behind a stable-looking name, the grader prompt gets edited for one fix and quietly changes other scores, or the agent's real failure modes shift — and each has a different fix. Re-check judge-vs-human agreement on a fixed cadence, not only when something looks wrong.

Security and Robustness Tests (Required for Tool Agents)

Prompt injection: retrieved text, tool outputs, and user files must be treated as untrusted.
Tool-output poisoning: tool returns malicious instructions; the agent must ignore them.
Tool argument smuggling: unsafe parameters must be blocked by validation or approval layers.
Secret exfiltration: verify the agent refuses and does not leak environment or file secrets.
Tool faults: timeouts, partial data, retries, malformed payloads, and permission failures.
Approval-boundary checks: verify the agent does not silently cross sandbox or approval limits.
Differential tests: compare model or config changes on the same suite for regressions.

Regression Prevention for Coding Agents

When testing agents that modify code (SWE agents, coding assistants, CI agents), use graph-based impact analysis to surface which tests cover the files being changed.

TDAD (arXiv:2603.17973) is a high-signal reference here: it found that targeted source-to-test context outperformed generic procedural TDD prompting for coding-agent regression control.

Working rule:

give the agent targeted test context, not only test instructions
require affected-test evidence before marking a task complete
track regression rate separately from task completion

Use references/coding-agent-regression-testing.md for the benchmark details, protocol, and how to combine this with classic TDD.

Iterative Extension Evaluation

A coding agent can pass independent tasks yet degrade when it repeatedly extends its own earlier work. For edit, refactor, and migration agents, add an evolving-spec trajectory when this compounding risk is in scope:

start from an empty workspace, then carry the agent-produced workspace from checkpoint to checkpoint
start each checkpoint with fresh conversation and runtime state so only the workspace preserves earlier decisions
grade observable behavior through an external black-box contract, including held-out tests
report strict, isolated, core, and regression correctness separately at every checkpoint
track structural erosion, verbosity, cost, and duration as trajectory signals; do not treat the quality signals as correctness predictors
aggregate unequal-length trajectories into Start, Early, Mid, Late, and Final phases

Benchmark-mode hidden tests and production-mode targeted test context serve different goals. Keep held-out tests hidden from the agent when estimating unbiased benchmark performance. In production regression control, expose the relevant TDAD source-to-test map and targeted test context so the agent can protect known behavior; retain a separate held-out evaluation slice for measurement.

Use references/iterative-coding-agent-evals.md for the protocol, formulas, interpretation limits, and trajectory record.

Do / Avoid

Do:

Prefer code-based or schema-based graders over model judges.
Keep task cases tied to real failures and live usage patterns.
Calibrate judge models on a small human-labeled set before trusting them.
Quarantine flaky evals with an owner and expiry date.
Include regression rate as a metric for coding agents, not just task success.
Provide targeted test context (source→test maps) instead of generic "write tests" instructions.

Avoid:

Treating happy-path prompt checks as sufficient coverage.
Letting one generic rubric stand in for tool traces, approvals, or side effects.
Using unsourced numeric claims in guidance or thresholds.
Treating LLM-as-judge as the sole source of truth for high-stakes tasks.
Adding procedural TDD instructions without pairing them with targeted test context — this can increase regressions.

Quick Reference

| Need | Use | Location | |---|---|---| | Build the starter suite | Task patterns + starter scaffold | references/test-case-design.md | | Control regressions in coding agents | TDAD pattern + source-to-test context | references/coding-agent-regression-testing.md | | Test iterative coding robustness | Carried-workspace checkpoints + trajectory scoring | references/iterative-coding-agent-evals.md | | Design refusals | Refusal categories + templates | references/refusal-patterns.md | | Score runs consistently | Canonical rubric + thresholds | references/scoring-rubric.md | | Compute suite math | CLI utility script | scripts/score_suite.py | | Manage regressions | Rerun scopes + baseline policy | references/regression-protocol.md | | Sandbox tool execution | Isolation tiers + MCP/tool hardening | references/tool-sandboxing.md | | Choose an eval toolchain | Tooling comparison for regression, traces, and policy gates | references/eval-tooling-patterns.md | | Test multi-agent systems | Coordination patterns + suite template | references/multi-agent-testing.md | | Use LLM-as-judge safely | Biases + mitigations | references/llm-judge-limitations.md | | Test prompt injection attacks | Injection taxonomy + defense checks | references/prompt-injection-testing.md | | Detect hallucinations | Claim extraction + citation checks | references/hallucination-detection.md | | Design eval datasets | Dataset construction + maintenance | references/eval-dataset-design.md | | Choose or critique an agent benchmark | τ²-bench usage + ABC checklist methodology | references/agentic-benchmarks.md | | Red-team with automated scanners | garak (batch probes) + PyRIT (multi-turn adversarial) | references/prompt-injection-testing.md | | Start from templates | Harness + scoring + regression log | assets/ |

Eval Tooling Patterns

Key tools mapped to QA jobs (see references/eval-tooling-patterns.md for the full table and references/eval-platform-selection.md for platform comparison with code examples):

Promptfoo for config-driven regression suites, refusal packs, and red-team attack sets. Supports trajectory assertions and major agent SDKs. Use it when the team needs fast iteration and diffable eval configs.
DeepEval (v4.0+) for pytest-style unit evals with agent-native metrics (Task Completion, Tool Correctness, Step Efficiency, Plan Quality). Use when evaluation should live next to CI tests.
lmnr for trace-native evaluation and execution-graph visibility. Use when regressions are about workflow shape, latency, or tool sequencing.
Langfuse for production tracing and online evals on live traffic. Supports native OpenTelemetry, code evaluators, and MCP in 2026.
Agent Governance Toolkit when policy boundaries, approvals, and authorization rules need explicit middleware-level tests instead of prompt-only checks.

Platform deprecation note (June 2026): The OpenAI Platform Evals UI (hosted at platform.openai.com) is being shut down — read-only October 31 2026, full shutdown November 30 2026. The open-source openai/evals package and API remain available. Teams using the Platform UI should migrate to Promptfoo (OpenAI-recommended), DeepEval, or Braintrust.

Rule of thumb:

Start with your eval design and grader model first.
Then choose tooling based on the failure mode you need to observe.
Do not let the tool pick the eval rubric for you.

Decision Tree

Testing an agent?
  - New agent?
    - Create starter harness -> Run smoke suite -> Establish baseline
  - Prompt or tool changed?
    - Re-run smoke suite + affected regression cases -> Compare to baseline
  - Model or judge changed?
    - Re-run smoke + refusal/security pack + targeted regression cases
  - Multi-agent or tool workflow?
    - Add trace graders, fault injection, and approval-boundary tests
  - Preparing production rollout?
    - Add optional online evals or canary comparisons

Scoring and Gates

Score each task with the canonical 6-dimension agent rubric (0-3 each, max 18).
Score refusals separately on a 0-3 refusal rubric.
Use one consistent status model everywhere:
- FAIL: any task <9, any refusal =0, or any objective policy hard fail
- PASS: all tasks >=12 and all refusals >=2
- CONDITIONAL: everything else
If you also track normalized score bands, treat them as informational quality bands unless your suite explicitly adopts them as gate criteria.

ASCII Flow

Agent QA request
  -> Define AUT scope, tools, approvals, and forbidden behavior
  -> Build smoke + regression + refusal/security packs from real work
  -> Prefer objective graders: schema, policy, golden traces, side effects
  -> Add calibrated model judges only for hard-to-code judgments
  -> Run offline evals with traces and deterministic controls
  -> Compare against baseline and classify FAIL/PASS/CONDITIONAL
  -> Expand to canary or online evals only for production rollout evidence

Navigation

Resources

references/scoring-rubric.md - canonical scoring model, thresholds, and variance notes
references/regression-protocol.md - rerun scopes, baselines, online evals, and recovery
references/tool-sandboxing.md - sandbox tiers, MCP and tool hardening, approval checks
references/eval-tooling-patterns.md - Promptfoo, trace tooling, and governance-tool selection rules
references/eval-platform-selection.md - platform comparison and decision tree for DeepEval, Inspect AI, Braintrust, Ragas, Promptfoo, OpenAI Evals, and Langfuse
references/multi-agent-testing.md - coordination testing patterns and handoff checks
references/llm-judge-limitations.md - judge biases, calibration, and escalation rules
references/agentic-benchmarks.md - τ²-bench usage, ABC checklist, and anti-patterns for reading benchmark results
references/iterative-coding-agent-evals.md - evolving-spec, carried-workspace protocol and longitudinal quality signals

Templates

assets/qa-harness-template.md - starter harness
assets/scoring-sheet.md - per-run scoring tracker
assets/regression-log.md - versioned regression log

External Resources

See data/sources.json for current primary sources, including OpenAI eval and grader docs (note: Platform Evals UI deprecated Nov 2026), Anthropic agent eval guidance, LangSmith evaluation docs, Promptfoo, DeepEval v4.0, Langfuse, Agent Governance Toolkit, OWASP LLM Top 10, OWASP Agentic Top 10 (2026), and UK AISI Inspect sandboxing docs.

Related Skills

| Skill | Purpose | |-------|---------| | qa-testing-strategy | Test strategy and risk prioritization | | ai-prompt-engineering | Prompt and guardrail design |

Quick Start

Copy assets/qa-harness-template.md
Fill in AUT scope, tools, and approval boundaries
Define the starter 10 tasks + 5 refusals
Add smoke, regression, and security packs from real work
Set objective graders and refusal oracles
Run baseline tests and record traces
Log results in assets/regression-log.md

Fact-Checking

Known bugs, regressions, framework/compiler/runtime footguns, and version-specific crash or workaround guidance must be verified against current primary web sources before being treated as current fact.
Use web search or web fetch to verify current external facts, versions, pricing, deadlines, regulations, or platform behavior before final answers.
Prefer primary sources; report source links and dates for volatile information.
If web access is unavailable, state the limitation and mark guidance as unverified.

Learnings Loop

Before applying this skill on a non-trivial task, read learnings.consolidated.md in this directory (and learnings.md if present).

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.

Agent Skills: QA Agent Testing

Install this agent skill to your local

Skill Files