Regression Test: Golden Dataset Management Skill

Regression Test: Golden Dataset Management

Manage golden test datasets for LLM prompt regression testing using Promptfoo. Ensures prompt changes don't degrade output quality.

One eval tells you today's score. A kept set tells you what your change did. There is a recorded three-run walkthrough of exactly that, runnable with no API key, at https://github.com/7alexhale5-rgb/alexhale-skills/tree/main/example

Prerequisite: npm install -g promptfoo (check with which promptfoo)

Commands

`/regression-test init`

Scaffold regression testing infrastructure in the current project.

Create directory structure:
```
mkdir -p .promptfoo/golden
```

Generate .promptfoo/promptfooconfig.yaml (the path matters: tests: "golden/*.yaml" below resolves relative to this file, and /regression-test run looks for it here):

description: "Golden dataset regression tests"
# Without this, promptfoo has no prompt to send and every case scores an empty string.
# The vars come from the golden files that `/regression-test add` writes.
prompts:
  - |
    [{"role":"system","content":"{{system_prompt}}"},{"role":"user","content":"{{user_input}}"}]
providers:
  - id: anthropic:messages:claude-sonnet-5
    config:
      temperature: 0
tests: "golden/*.yaml"
defaultTest:
  assert:
    - type: llm-rubric
      value: "Output should match the expected behavior described in the test case"

Add .promptfoo/results/ to .gitignore if not already present. Ignore the results, commit the goldens: a golden dataset nobody else can see is not a shared baseline, and the whole point is that your teammates and your CI run the same cases you do. If a prompt in a golden case is genuinely sensitive, keep that one case out of the repo rather than hiding the dataset.
Report: "Initialized .promptfoo/golden/ — run /regression-test add <name> to capture test cases."

`/regression-test add <name>`

Capture a prompt+response pair as a golden test case.

Parse <name> from user input (slug format, e.g., auth-flow, error-handling)
Prompt user for:
- System prompt (or detect from current project's CLAUDE.md/skill)
- User input (the test prompt)
- Expected behavior (natural language description of good output)
- Vars (optional template variables)

Write golden test case:

# .promptfoo/golden/{name}.yaml
- description: "{name}"
  vars:
    system_prompt: |
      {system prompt content}
    user_input: |
      {user input}
  assert:
    - type: llm-rubric
      value: |
        {expected behavior description}
    - type: not-contains
      value: "error"

Report: "Added golden test {name} — {N} total test cases."

`/regression-test run`

Execute Promptfoo evaluation against all golden datasets.

Check prerequisites:

which promptfoo 2>&1  # must return path
ls .promptfoo/golden/*.yaml 2>/dev/null | wc -l  # must be > 0

Run evaluation:

cd {project_root}
promptfoo eval --config .promptfoo/promptfooconfig.yaml \
  --output .promptfoo/results/eval-$(date +%Y%m%d-%H%M%S).json \
  --no-cache 2>&1

Parse results JSON:
- Count PASS/FAIL per test case
- Extract failure details (expected vs actual)

Report:

Regression Test Results
├─ Total: {N} test cases
├─ Pass: {N} ({%})
├─ Fail: {N} ({%})
└─ Duration: {N}s

Failures:
- {test_name}: {assertion_type} — expected "{expected}", got "{actual}"

If all pass: "All golden tests pass — safe to ship." If failures: "Regressions detected — review before shipping."

`/regression-test report`

Show regression trends over the last 5 evaluations.

Read result files:

ls -t .promptfoo/results/eval-*.json | head -5

For each result file, extract:
- Date (from filename)
- Pass/fail counts
- Any new failures vs previous run

Report:

Regression Trends (last 5 evals)
├─ 2026-03-31 14:30: 12/12 PASS ✓
├─ 2026-03-30 09:15: 11/12 PASS (1 fail: auth-flow)
├─ 2026-03-29 16:45: 12/12 PASS ✓
├─ 2026-03-28 11:00: 10/12 PASS (2 fail)
└─ 2026-03-27 08:30: 12/12 PASS ✓

Trend: STABLE (4/5 clean)

Where to wire it in

The value shows up when the run is automatic, not when you remember to type it.

| Moment | What to do | |--------|------------| | After changing any prompt | /regression-test run before you commit. This is the one that catches most regressions. | | Before a release | Run it in CI and fail the build on a new failure. promptfoo eval returns non-zero when an assertion fails. | | After a model or version change | Re-run the whole set. A provider upgrade is a silent prompt change. | | Weekly, on a schedule | /regression-test report to see whether quality is drifting rather than breaking. |

If your agent harness supports hooks or a pre-commit step, put the run there. A check that depends on discipline is a check you will lose.

Graceful Degradation

| Component | If Missing | Fallback | |-----------|-----------|----------| | promptfoo CLI | Not installed | Prompt user: npm install -g promptfoo | | .promptfoo/ directory | Not initialized | Prompt: /regression-test init | | Golden datasets | No .yaml files | Skip with note: "No golden tests — add with /regression-test add" | | Results directory | Missing | Create on first run |

Agent Skills: Regression Test: Golden Dataset Management

Install this agent skill to your local

Skill Files