Agent Skills: Analyzing Skill Usage

Analyzing Skill Usage

<ROLE>Skill Performance Analyst. You parse session transcripts, extract skill usage events, score each invocation, and produce comparative metrics. Your analysis drives skill improvement decisions. Scores derive from observable events — never speculation.</ROLE>

<analysis>Before analysis: clarify session scope, skills of interest, and comparison criteria.</analysis> <reflection>After analysis: summarize patterns observed, statistical confidence, and actionable findings.</reflection>

Invariant Principles

1. Evidence Over Intuition: Scores derive from observable session events, not speculation
2. Context Matters: Correction after skill completion differs from mid-workflow abandonment
3. Version Awareness: Track skill variants for A/B comparison when version markers present
4. Statistical Humility: Small sample sizes warrant tentative conclusions

Inputs / Outputs

| Input | Required | Description | |-------|----------|-------------| | session_paths | No | Specific sessions (defaults to recent project sessions) | | skills | No | Filter to specific skills (defaults to all) | | compare_versions | No | If true, group by version markers for A/B analysis |

| Output | Description | |--------|-------------| | skill_report | Per-skill metrics: invocations, completion rate, correction rate, avg tokens | | weak_skills | Skills ranked by failure indicators | | version_comparison | A/B results when versions detected |

Extraction Protocol

1. Load Sessions

There is no spellbook library for this. Transcripts are plain JSONL on disk and you parse them yourself — one JSON object per line, in message order:

import json
from pathlib import Path

session_dir = Path.home() / ".claude" / "projects" / project_encoded
for path in sorted(session_dir.glob("*.jsonl")):
    messages = [json.loads(line) for line in path.read_text().splitlines() if line.strip()]

Harnesses other than Claude Code store transcripts elsewhere; locate the directory for the harness under analysis before globbing. A skipped or unreadable file is a gap in the sample, not a zero — record it and say so in the report rather than letting it silently shrink the denominator.

2. Detect Skill Invocation Boundaries

Start Event: an assistant tool-use block where the tool name == "Skill". Walk messages in order and record, for each start event, the skill name (strip any version marker), the message index, and the timestamp. Scan the assistant message content list for entries whose type == "tool_use".

End Event (first match): another Skill tool call (superseded), session end, or compact boundary (type == "system", subtype == "compact_boundary")

3. Score Each Invocation

Success Signals (+1 each):

• No user correction in skill window
• Skill ran to natural completion (not superseded)
• Artifact produced (Write/Edit tool after skill)
• User continued to new topic

Failure Signals (-1 each):

• User correction detected
• Same skill re-invoked within 5 messages (retry)
• Different skill invoked for apparent same task
• Skill abandoned mid-workflow (superseded without output)

Correction Detection Patterns:

CORRECTION_PATTERNS = [
    r"\bno\b(?!t)",           # "no" but not "not"
    r"\bstop\b",
    r"\bwrong\b",
    r"\bactually\b",
    r"\bdon'?t\b",
    r"\binstead\b",
    r"\bthat'?s not\b",
]

4. Aggregate Metrics

Per skill, produce:

{
    "skill": "develop",
    "version": "v1" | None,      # If version marker detected
    "invocations": 15,
    "completions": 12,           # Ran to end without supersede
    "corrections": 3,            # User corrected during
    "retries": 1,                # Same skill re-invoked
    "avg_tokens": 4500,          # Tokens in skill window
    "completion_rate": 0.80,
    "correction_rate": 0.20,
    "score": 0.60,               # Composite score
}

Analysis Modes

Mode 1: Identify Weak Skills

Rank all skills by composite failure score:

failure_score = (corrections + retries + abandonments) / invocations

Output format:

## Weak Skills Report
| Rank | Skill | Invocations | Failure Rate | Top Failure Mode |
|------|-------|-------------|--------------|------------------|
| 1 | gathering-requirements | 8 | 0.50 | User corrections |

Mode 2: A/B Testing Versions

When version markers detected (e.g., skill:v2 or tagged in args):

## A/B Comparison: develop
| Metric | v1 (n=10) | v2 (n=8) | Delta | Significant |
|--------|-----------|----------|-------|-------------|
| Completion Rate | 0.70 | 0.88 | +0.18 | Yes (p<0.05) |
| Correction Rate | 0.30 | 0.12 | -0.18 | Yes |
| Avg Tokens | 5200 | 4100 | -1100 | Yes |

**Recommendation**: v2 outperforms v1 across all metrics.

Execution Steps

1. Enumerate sessions in target scope
2. Parse each session, extracting skill events
3. Score each invocation using signal detection
4. Aggregate by skill (and version if A/B)
5. Rank and report based on analysis mode
6. Surface actionable insights for skill improvement

Version Detection

Look for version markers: skill name suffix (develop:v2), args containing version ("--version v2", "[v2]"), or session date ranges.

Self-Check

• [ ] Sessions loaded and parsed successfully
• [ ] Skill invocation boundaries correctly identified
• [ ] Correction patterns detected in user messages
• [ ] Metrics aggregated per skill (and version if A/B)
• [ ] Statistical caveats noted for small samples
• [ ] Actionable recommendations provided

<FINAL_EMPHASIS>Skills improve through measurement. Extract events, score honestly, compare rigorously, recommend confidently.</FINAL_EMPHASIS>