Mutation Testing Skill | Agent Skills

Mutation Testing

Expert knowledge for mutation testing - validating that your tests actually catch bugs by introducing deliberate code mutations.

When to Use This Skill

| Use this skill when... | Use another skill instead when... | |------------------------|----------------------------------| | Validating test effectiveness | Writing unit tests (use vitest-testing) | | Finding weak/insufficient tests | Analyzing test smells (use test-quality-analysis) | | Setting up Stryker or mutmut | Writing E2E tests (use playwright-testing) | | Improving mutation score | Generating test data (use property-based-testing) | | Checking if tests catch real bugs | Setting up code coverage only |

Core Expertise

Mutation Testing Concept

Mutants: Small code changes (mutations) introduced automatically
Killed: Test fails with mutation (good - test caught the bug)
Survived: Test passes with mutation (bad - weak test)
Coverage: Tests execute mutated code but don't catch it
Score: Percentage of mutants killed (aim for 80%+)

What Mutation Testing Reveals

Tests that don't actually verify behavior
Missing assertions or edge cases
Overly permissive assertions
Dead code or unnecessary logic
Areas needing stronger tests

TypeScript/JavaScript (Stryker)

Installation

# Using Bun
bun add -d @stryker-mutator/core

# For Vitest
bun add -d @stryker-mutator/vitest-runner

# For Jest
bun add -d @stryker-mutator/jest-runner

Running Stryker

npx stryker run                                    # Run mutation testing
npx stryker run --incremental                      # Only changed files
npx stryker run --mutate "src/utils/**/*.ts"       # Specific files
npx stryker run --reporters html,clear-text        # HTML report
open reports/mutation/html/index.html              # View report

Understanding Results

Mutation score: 82.5%
- Killed: 66 (tests caught the mutation)
- Survived: 14 (tests passed despite mutation - weak tests!)
- No Coverage: 0 (mutated code not executed)
- Timeout: 0 (tests took too long)

Example: Weak vs Strong Test

// Source code
function calculateDiscount(price: number, percentage: number): number {
  return price - (price * percentage / 100)
}

// WEAK: Test passes even if we mutate the calculation
test('applies discount', () => {
  const result = calculateDiscount(100, 10)
  expect(result).toBeDefined() // Too weak!
})

// STRONG: Test catches mutation
test('applies discount correctly', () => {
  expect(calculateDiscount(100, 10)).toBe(90)
  expect(calculateDiscount(100, 20)).toBe(80)
  expect(calculateDiscount(50, 10)).toBe(45)
})

Python (mutmut)

Installation

uv add --dev mutmut                    # Using uv
pip install mutmut                     # Using pip

Running mutmut

uv run mutmut run                                          # Run mutation testing
uv run mutmut run --paths-to-mutate=src/calculator.py      # Specific files
uv run mutmut results                                      # Show results
uv run mutmut summary                                      # Summary
uv run mutmut show 1                                       # Show specific mutant
uv run mutmut apply 1                                      # Apply mutant manually
uv run mutmut html                                         # HTML report

Understanding Results

Status: 45/50 mutants killed (90%)
- Killed: 45 (tests caught the mutation)
- Survived: 5 (tests passed despite mutation)

Hand-rolled harnesses report LESS than Stryker and mutmut do

Everything above assumes a framework. Plenty of real mutation testing is a hand-rolled loop instead — apply a mutation, run one assertion, catch the failure — typically because the thing under test is a build-time check in a generator or builder rather than a unit test suite.

That loop is worth writing. But it drops the one piece of bookkeeping the frameworks give you for free: Stryker and mutmut tell you which test killed each mutant. A hand-rolled harness usually reports only that something failed, and "something failed" is indistinguishable from "the check I am testing failed". Three ways that goes wrong, all observed in one session:

1. An earlier check masks the one under test

run(mutate_frame_count, "check P: off-grid length")
  -> CAUGHT: "beat 'x' asks for 20 words in 5.42 s (3.69 words/s, ceiling 3.0)"

Reported as caught; the message is from check N, a words-per-second rule that fires before the grid check ever runs. Check P was never exercised. The mutation tripped a different assertion on the way past.

Always print and read the failure message, never just the pass/fail. If the message does not name the check you are testing, the mutation did not reach it.

2. The mutation has to be one ONLY the target check can see

Fixing the above is not "mutate harder" — it is choosing a mutation that no earlier check can intercept:

| Testing | Bad mutation | Works | |---|---|---| | an off-grid frame count | any beat (a talky one trips the words/sec check first) | a wordless beat | | a cast-shrink rule | a beat whose prose also names the removed character (trips the alias check) | a beat where only the count changes |

This is the same discipline as isolating a variable in an A/B: the mutation is the independent variable, and anything else it perturbs is a confound.

3. Mutating a table leaves import-time derived state stale

The subtlest one, and it caused two of the three maskings. Modules commonly build lookup dicts from a table at import:

SEGMENTS = (...)
_SEG_OF = {beat: name for name, beats, _ in SEGMENTS for beat in beats}

Monkeypatching SEGMENTS in the harness leaves _SEG_OF describing the old table, so the first check that consults it fails with a stale-lookup error — masking everything downstream:

mod.SEGMENTS = new_table
mod._SEG_OF = {b: n for n, ids, _ in mod.SEGMENTS for b in ids}   # REQUIRED

Rebuild every derived structure you can find, or reload the module. Grep for comprehensions over the table you mutated.

The consequence for a green table

A harness that prints CAUGHT for every mutation is often quoted as proof the suite is sound. It proves something weaker:

An all-CAUGHT table proves each mutation was caught by some assertion. It never proves the assertion you meant was the one that caught it — nor that any individual assertion is capable of failing.

Two cheap additions close most of the gap:

A deliberate no-op mutation the harness should miss. A table where everything is CAUGHT is indistinguishable from a broken harness; one expected MISS tells them apart.
Assert on the message, not just the exception. Match the mutation to an expected substring of the failure, so a masked result is a harness failure rather than a silent pass.

Mutation Score Targets

| Score | Quality | Action | |-------|---------|--------| | 90%+ | Excellent | Maintain quality | | 80-89% | Good | Small improvements | | 70-79% | Acceptable | Focus on weak areas | | 60-69% | Needs work | Add missing tests | | < 60% | Poor | Major test improvements needed |

Agentic Optimizations

| Context | Command | |---------|---------| | Quick TS mutation | npx stryker run --incremental --reporters clear-text | | Targeted TS mutation | npx stryker run --mutate "src/core/**/*.ts" | | Quick Python mutation | uv run mutmut run --paths-to-mutate=src/core/ | | View survived | uv run mutmut results \| grep Survived | | CI mode | npx stryker run --reporters json |

For detailed examples, advanced patterns, and best practices, see REFERENCE.md.

References

Stryker: https://stryker-mutator.io/
mutmut: https://github.com/boxed/mutmut
Mutation Testing Intro: https://en.wikipedia.org/wiki/Mutation_testing

Agent Skills: Mutation Testing

Install this agent skill to your local

Skill Files