Agent-Skills.md

Agent Skills: Eval Harness

Build evaluation harnesses for AI-assisted implementations — capability evals, regression tests, graders (exact-match, LLM-judge, code-exec, rubric), and standardized metrics (pass@k, accuracy, latency, cost). Use when the user wants to set up or run evals, benchmark agents/skills, create regression test suites for prompts or AI features, compare model/prompt variants, or measure implementation quality. Triggers on "set up evals", "create eval harness", "benchmark this skill", "regression test the prompt", "run evals", "/eval-harness", or any request to systematically measure AI output quality.

UncategorizedID: mhylle/claude-skills-collection/eval-harness
171

Install this agent skill to your local

pnpm dlx add-skill https://github.com/mhylle/claude-skills-collection/tree/HEAD/skills/eval-harness

Skill Files

Browse the full folder contents for eval-harness.

Download Skill

Loading file tree…

skills/eval-harness/SKILL.md

Skill Metadata

Name
eval-harness
Description
>-

Eval Harness

<!-- contents: generated by tests/skill_structure.py --write; edit headings, not this block -->

Contents

Reference files (open one when a step points to it):

  • references/eval-types.md: Eval types
  • references/file-format.md: Eval file format
  • references/graders.md: Graders
  • references/metrics.md: Metrics — pass@k, pass^k, latency, cost
<!-- /contents -->

Build and run evaluation suites for AI features in the user's application — a prompt, an LLM call, a RAG pipeline, an agent. Covers two eval types (capability, regression), three grader families (code, model, human), and the metrics that summarise them (pass@k, pass^k, latency, cost).

Not this skill:

  • Evaluating or benchmarking a Claude Code skill or its triggering → the skill-creator skill.
  • A Claude API feature where the user wants a full eval set with measured cost, a report viewer, and iterative prompt tuning → offer the claude-api skill's build-eval and hillclimb subcommands, which go further than this skill. If claude-api is not in your skill list, carry on here.

Modes

| Mode | Does | Writes | |---|---|---| | create | Defines the suite: success criteria, eval files, graders, thresholds | evals/eval-suite.yaml + eval files | | run | Generates or reuses the runner script and executes the suite | <output_dir>/<run-id>/results.jsonl, summary.json | | report | Reads a run's results, computes metrics, compares to thresholds and the previous run | <output_dir>/<run-id>/report.md |

Without an argument, infer the mode: no suite yet → create; a suite but no results, or the code changed since the last run → run; results present and the user asks how it went → report.

Evaluation-driven development (EDD)

Evals are defined before or alongside implementation, not after, so "working" is explicit and testable from day one.

  1. Define success first — write the evals that say what "working correctly" means before building.
  2. Measure continuously — run evals during development, not just at the end.
  3. Automate where possible — code graders for anything codifiable; model graders for semantic quality; human graders for nuance and for calibrating the model graders.
  4. Track regressions — every capability added becomes a regression eval.
  5. Don't modify evals to pass — unless the requirements changed, and then update baselines explicitly.

Eval types (summary)

| Type | Purpose | When to use | |---|---|---| | Capability | Verify new functionality works | Adding features, improving functionality, testing edge cases | | Regression | Protect against unintended breakage | After code changes, before PR merge, after dependency or model updates |

Full structures and YAML examples → references/eval-types.md.

Grader types (summary)

| Type | Strength | When to use | |---|---|---| | Code (exact match, regex, function output) | Fast, deterministic, cheap | Any codifiable correctness check | | Model (LLM judge with rubric, structured verdict) | Handles semantic variation, tone, quality | Natural-language outputs, multiple valid answers | | Human (Likert, binary, ranking) | Catches what machines miss | Creative content, safety-critical judgments, calibrating model graders |

Configs, implementation sketches, judge call and rubric templates → references/graders.md.

Metrics (summary)

| Metric | Question answered | Use case | |---|---|---| | pass@1 | "Can it solve this at all?" | Single-shot capability | | pass@k | "Can it ever solve this?" | Best-of-k capability, retries cheap | | pass^k | "Can it always solve this?" | Consistency, reliability, production | | latency p50 / p95 | "How long does a case take?" | Interactive features, latency budgets | | tokens and cost per case | "What does a case cost?" | Model or prompt comparisons, budget checks |

Formulas, what to record per case, and the picking guide → references/metrics.md.

Mode: create

Write the suite file and the eval files it references. Eval file format → references/file-format.md.

# evals/eval-suite.yaml
eval_suite:
  name: "Feature X evaluation suite"
  version: "1.0.0"
  description: "Capability and regression evals for Feature X"

  target:                                         # how the runner calls the system under test
    entry_point: "src/feature_x/answer.py:answer"  # the real code path, not a rebuilt API call

  config:
    reps: 3                     # samples per case; the k available to pass@k and pass^k
    max_workers: 4
    timeout_per_case_seconds: 120

  thresholds:
    pass_at_1: 0.85
    pass_hat_3: 0.70
    latency_p95_seconds: 8

  evals:
    - $ref: "./capability/cap-001.yaml"
    - $ref: "./capability/cap-002.yaml"
    - $ref: "./regression/reg-001.yaml"

  output_dir: "./results"

Checklist:

  • Success criteria written as concrete, checkable statements
  • Capability evals for the new behaviour, regression evals for what must not break
  • A grader per case, code wherever the output can be checked programmatically
  • For model graders, the judge model agreed with the user (references/graders.md)
  • Thresholds that are realistic for the stage (see Common thresholds below)
  • Edge cases (empty input, very long input, adversarial input) covered

While implementing the feature, run the relevant subset often (run with a filter) and use failures to guide the work.

Mode: run

There is no eval-harness CLI. The runner is a small script in the user's project, written in the project's language and idiom, that you generate once and reuse.

  1. Look for an existing runner (evals/run_evals.*, an eval script in package.json or the Makefile, a pytest marker). Reuse it if it meets the contract below; extend it rather than writing a second one.
  2. Otherwise generate evals/run_evals.<ext> — a pytest module or Click CLI for Python, an npm script for Node, and so on.
  3. Run it on two or three cases first and read one written row in full. Fix the runner until every field is present and plausible, then run the whole suite.

Runner contract:

  • Input: the suite file path; optional --filter <category-or-tag>, --reps <k>, --run-id <id>.
  • Load: parse the suite (YAML or JSON), resolve each $ref, and validate that every test case has input, grader, and expected where its grader needs one.
  • Execute: for each case × rep, call the system under test through target, mocking only external side effects (emails, payments, production writes), with the per-case timeout. Record the output, latency_s, and — when the call surfaces them — model and token usage from the response.
  • Grade: code graders in-process; model graders call the judge with a structured-output schema and record judge_model and judge_usage; human-graded cases go to review.jsonl with status pending. Details → references/graders.md.
  • Write: append one JSON line per (case, rep) to <output_dir>/<run-id>/results.jsonl as each finishes, with case_id, rep, category, tags, status (ok | error | timeout | pending), passed, score, details, latency_s, model, usage, judge_model, judge_usage. Infrastructure failures get status: error, not passed: false, so they aren't counted as model failures. On restart, skip (case, rep) pairs already written.
  • Finish: write summary.json (pass rate per category and per grader type, pass@k / pass^k, latency and token summaries per references/metrics.md), print one PASS/FAIL line per threshold, and exit non-zero if any threshold fails, so the run can gate a merge.

Mode: report

Read results.jsonl (plus any grades filled into review.jsonl), compute the metrics in references/metrics.md, and write report.md from a summary like this:

eval_report:
  metadata:
    suite_name: "Feature X evaluation suite"
    run_id: "eval-20261006-143052"
    timestamp: "2026-10-06T14:30:52Z"
    duration_seconds: 127
    environment:
      model: "claude-opus-5-5"      # model under test, as reported by the responses
      app_version: "1.2.3"

  summary:
    total_cases: 25
    passed: 22
    failed: 3
    errors: 0
    pending_human: 0
    pass_rate: 0.88

    by_category:
      capability:   { total: 15, passed: 13, pass_rate: 0.867 }
      regression:   { total: 10, passed: 9,  pass_rate: 0.90 }

    by_grader:
      code:  { total: 18, passed: 17 }
      model: { total: 5,  passed: 4 }
      human: { total: 2,  passed: 1 }

  metrics:
    pass_at_1: 0.88
    pass_at_3: 0.96
    pass_hat_3: 0.68
    latency_s: { p50: 2.1, p95: 4.8, max: 7.2 }
    tokens_per_case: { input_median: 1850, output_median: 420 }
    cost_usd: { per_case_median: 0.016, run_total: 1.21, judge_total: 0.41 }   # derived from usage at report time

  thresholds:
    - { metric: pass_at_1, value: 0.88, threshold: 0.85, status: PASS }
    - { metric: pass_hat_3, value: 0.68, threshold: 0.70, status: FAIL }

  failures:
    - eval_id: "cap-003"
      name: "Complex nested extraction"
      reason: "Missed nested array handling"
    - eval_id: "cap-011"
      name: "Edge case: empty input"
      reason: "Threw exception instead of returning empty"
    - eval_id: "reg-007"
      name: "Code review comment quality"
      reason: "Specificity score 3.6, baseline 4.2 (tolerance -0.3)"

  regressions_since_previous_run:
    - "reg-007: passed in eval-20261005-091200, fails now"

  recommendations:
    - "Improve nested data structure handling"
    - "Add empty input validation"

Compare with the previous run in the same output_dir when there is one, and list the cases that flipped from pass to fail.

Evals can be run as a manual gate before marking an implementation phase complete: run the suite and check the threshold lines.

Quick reference

Grader selection guide

| Output type | Recommended grader | Rationale | |---|---|---| | Exact values | Code (exact match) | Fast, deterministic | | Formatted text | Code (regex) | Validates structure | | Code output | Code (function) | Tests correctness | | Natural language | Model | Handles variation | | Creative content | Human | Subjective judgment | | Safety-critical | Human + Model | Double verification |

Metric selection guide

| Scenario | Recommended metric | |---|---| | Code generation (any solution works) | pass@k | | Production system reliability | pass^k | | Single-shot capability | pass@1 | | Best-effort with retries | pass@5 or pass@10 | | Consistency requirements | pass^3 or pass^5 | | Interactive feature | latency p95 alongside pass@1 |

Common thresholds

| Context | pass@1 | pass@k | pass^k | |---|---|---|---| | Development | 0.70 | 0.90 | 0.50 | | Staging | 0.80 | 0.95 | 0.65 | | Production | 0.90 | 0.99 | 0.80 | | Critical | 0.95 | 0.999 | 0.90 |

References (loaded on demand)

  • references/eval-types.md — capability and regression eval structures with YAML examples
  • references/graders.md — code / model / human grader configs, the structured-output judge call, rubric examples, calibration
  • references/metrics.md — pass@k and pass^k formulas, latency and cost recording and summaries, picking guide
  • references/file-format.md — eval file schemas (YAML/JSON) and the recommended directory layout