Skill Evaluator
Evaluate skills for Claude Code and Fox Codex across 9 quality criteria, producing a scored pass/fail report with actionable recommendations.
Trigger
This skill is triggered by the /skill-evaluator command.
Phase 1: Skill Discovery
Scan the following locations for installed skills:
| Location | Platform |
|---|---|
| ~/.claude/skills/ | Claude Code |
| ~/.claude/plugins/*/skills/ | Claude Code plugins |
| ~/.codex/skills/ | Fox Codex |
For each location, find directories containing a SKILL.md file.
Present the discovered skills in a table:
| # | Skill | Platform | Path |
|---|---|---|---|
Ask the user using AskUserQuestion: "Evaluate ALL skills or select specific ones?"
If user selects specific ones, present a multi-select list using AskUserQuestion with multiSelect: true.
If a path argument is provided, skip discovery and evaluate that single skill directly.
Phase 2: Evaluation
For each selected skill, read its SKILL.md and all files in its directory tree.
Evaluate the skill against all 9 criteria. For each criteria, analyze the skill content and assign a score from 0 to 10.
Criteria 1: Direct and Clear (0-10)
Evaluate how direct, precise, and clear the skill is.
Look for:
- Filler words: "please", "kindly", "basically", "essentially", hedging
- Vague verbs: "handle", "manage", "process" instead of "parse", "validate", "write"
- Redundant repetition of the same idea
- Long compound sentences that should be short
- Preamble and throat-clearing introductions
- Missing success criteria on steps
Score >= 6 to PASS.
Criteria 2: Token Efficiency (0-10)
Evaluate how efficiently the skill uses context window tokens.
Look for:
- SKILL.md word count (ideal 500-2000, penalty above 3000)
- Duplicate information across SKILL.md and references/
- Over-explanation of things the LLM already knows
- Verbose paragraphs that should be tables or lists
- Whether heavy content is in references/ (progressive disclosure)
Score <= 5 to PASS (lower is better, fewer tokens wasted).
Criteria 3: Anti-Cheating (0-10)
Evaluate trustworthiness guardrails.
Look for:
- Verification steps that check outputs are real
- Rules forbidding hardcoded or fabricated results
- Correctness validation before accepting results
- Rollback mechanisms for failed validations
- Explicit anti-cheat rules section
- Requirements for real execution (not just "looks right")
- Audit trail (logs, outputs, diffs as evidence)
Score >= 6 to PASS.
Criteria 4: Reinforcing Quality Gates (0-10)
Evaluate whether the skill blocks progress until quality bars are met.
Look for:
- "Do not continue until X passes" rules
- Tests must pass before next phase
- Build must succeed before proceeding
- Checkpoints between phases
- Rules saying gates cannot be skipped
- Defined failure handling (fix, retry, stop)
- Progressive strictness across phases
Score >= 6 to PASS.
Criteria 5: Determinism (0-10)
Evaluate reproducibility across runs.
Look for:
- Pinned versions for dependencies and tools
- Fixed seeds for random operations
- Avoidance of non-deterministic choices
- Same input producing same output guarantee
- Independence from transient system state
Score >= 6 to PASS.
Criteria 6: Scope Discipline (0-10)
Evaluate whether the skill stays in its lane.
Look for:
- Rules against modifying code outside scope
- Only creating strictly necessary files
- Not adding comments or docstrings unless asked
- Not refactoring surrounding code
- Minimal footprint policy
- No feature creep beyond what was requested
Score >= 6 to PASS.
Criteria 7: Error Recovery (0-10)
Evaluate failure handling.
Look for:
- Step failure detection
- Retry logic for transient failures
- Rollback mechanism for partial work
- User notification on errors
- No silent failures policy
- Graceful degradation to safe state
Score >= 6 to PASS.
Criteria 8: Observability (0-10)
Evaluate evidence production.
Look for:
- Output artifacts (files, logs, reports)
- Before/after diffs
- Command output capture
- Progress indicators during multi-step work
- Audit trail for post-execution review
Score >= 6 to PASS.
Criteria 9: Idempotency (0-10)
Evaluate re-run safety.
Look for:
- Safe to run twice without duplicating work
- No conflicting state on second run
- Detection of prior runs
- Clean re-entry without corruption
- No side-effect accumulation
Score >= 6 to PASS.
Phase 3: Report Generation
For each evaluated skill, produce a results table:
# Skill Evaluation: [skill-name]
Date: YYYY-MM-DD
Path: /path/to/skill/
| Criteria | Score | Result | Summary |
|----------------------|-------|--------|-----------------------------------------------|
| Direct and Clear | X/10 | PASS | [short summary of what is good or to improve] |
| Token Efficiency | X/10 | PASS | [short summary of what is good or to improve] |
| Anti-Cheating | X/10 | PASS | [short summary of what is good or to improve] |
| Quality Gates | X/10 | PASS | [short summary of what is good or to improve] |
| Determinism | X/10 | PASS | [short summary of what is good or to improve] |
| Scope Discipline | X/10 | PASS | [short summary of what is good or to improve] |
| Error Recovery | X/10 | PASS | [short summary of what is good or to improve] |
| Observability | X/10 | PASS | [short summary of what is good or to improve] |
| Idempotency | X/10 | PASS | [short summary of what is good or to improve] |
Overall: X.X/10 | PASS/FAIL (list failing criteria if FAIL)
Pass/fail rules:
- All criteria PASS at score >= 6, EXCEPT Token Efficiency which PASSES at score <= 5
- A skill PASSES overall only if ALL 9 criteria pass individually
If multiple skills were evaluated, append a ranking table:
| Rank | Skill | Overall | Result | Weakest Criteria |
|------|-------|---------|--------|------------------|
Write the full report to eval-report.md in the current working directory.
Rules
- Read every file in the skill directory before scoring
- Run
scripts/evaluate.shon each skill to collect static metrics before LLM analysis - Never invent or assume content that is not in the skill files
- Every score must have a concrete justification in the Summary column
- If a skill has zero anti-cheat rules, Anti-Cheating score is 0
- If a skill has zero quality gates, Quality Gates score is 0
- Token Efficiency score is inverted: lower score = better efficiency = PASS
Additional Resources
Scripts
scripts/evaluate.sh- Static analysis script that collects word count, file structure, frontmatter validation, and checks for common patterns
Reference Files
references/scoring-rubric.md- Detailed scoring rubric with score guides for each criteriareferences/best-practices-checklist.md- Checklist of best practices to evaluate against