Mutation Testing
Overview
This skill sets up mutation testing to evaluate test suite quality by introducing deliberate code changes (mutants) and verifying tests catch them. A high mutation score indicates tests are effective at catching real bugs.
Version: 0.1.0 Status: Initial Release
What This Skill Does
Mutation Testing Workflow:
- Analyze codebase to identify mutation targets
- Generate mutants (deliberate code changes)
- Run test suite against each mutant
- Report mutation score and surviving mutants
- Recommend test improvements for weak spots
When to Use This Skill
Use when:
- Coverage is high but confidence in tests is low
- Validating test suite effectiveness
- Finding tests that pass but don't catch real bugs
- Preparing for production release
- Improving test quality beyond coverage metrics
Don't use if:
- No existing test suite
- Test coverage below 50% (improve coverage first)
- CI pipeline can't afford extra runtime
Trigger Phrases
- "Set up mutation testing"
- "Are my tests catching bugs?"
- "Check test effectiveness" / "Test quality analysis"
- "Run mutation analysis" / "Mutation score"
- "Find weak tests"
Supported Frameworks
| Language | Framework | Command |
|----------|-----------|---------|
| JavaScript/TypeScript | Stryker | npx stryker run |
| Java | PIT | mvn org.pitest:pitest-maven:mutationCoverage |
| Python | mutmut | mutmut run |
Mutation Score
| Score | Interpretation | |-------|----------------| | 90%+ | Excellent - tests catch most bugs | | 75-89% | Good - some gaps to address | | 50-74% | Fair - significant testing gaps | | <50% | Poor - tests need major improvement |
Common Mutations
| Type | Example | Tests Should Catch |
|------|---------|-------------------|
| Arithmetic | + → - | Math logic errors |
| Conditional | > → >= | Boundary conditions |
| Boolean | true → false | Logic inversions |
| Return | return x → return null | Null handling |
| Remove call | validate() → removed | Missing validations |
Installation Summary
For JavaScript/TypeScript (Stryker):
npm install --save-dev @stryker-mutator/core
npx stryker init
For Python (mutmut):
pip install mutmut
mutmut run
Quick Start
- Ensure test suite exists and passes
- Install mutation testing framework
- Configure mutation targets (critical code paths)
- Run mutation analysis
- Review surviving mutants
- Add/improve tests for uncaught mutations
Performance Considerations
- Mutation testing is compute-intensive
- Start with critical modules only
- Use incremental mode for CI
- Cache results between runs
- Consider parallel execution
Success Criteria
- Mutation testing framework installed
- Configuration targets critical code paths
- Initial mutation run completes
- Mutation score reported
- Action plan for surviving mutants
Additional Resources
- Workflow guide:
workflow/mutation-analysis.md - Configuration reference:
reference/stryker-config.md - Example report:
examples/mutation-report.md
Version History
- 0.1.0 - Initial release with Stryker, PIT, mutmut support