Agent Evaluation Skill | Agent Skills

Agent Evaluation

Framework for evaluating AI agent outputs: hallucination detection, quality scoring, behavioral regression, and CI/CD integration.

Hallucination Detection

Fact-Checking Pipeline

def detect_hallucination(response: str, context: str) -> dict:
    """Check if response contains unsupported claims."""
    claims = extract_claims(response)
    supported = [c for c in claims if verify_against(c, context)]
    return {
        'hallucination_rate': 1 - len(supported) / len(claims),
        'unsupported_claims': [c for c in claims if c not in supported]
    }

Red Teaming

Generate adversarial prompts designed to trigger hallucinations
Test with contradictory context
Verify model says "I don't know" when appropriate

Output Quality Scoring

RAGAS Metrics

| Metric | What it measures | Target | |--------|-----------------|--------| | Faithfulness | Response grounded in context | ≥0.8 | | Answer Relevance | Response addresses query | ≥0.8 | | Context Precision | Retrieved context is relevant | ≥0.7 | | Context Recall | All relevant context retrieved | ≥0.7 |

Scoring Pipeline

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevance

result = evaluate(
    dataset=eval_dataset,
    metrics=[faithfulness, answer_relevance]
)

Behavioral Regression Testing

Test Types

Consistency: Same input → same output across runs
Robustness: Slight input variation → similar output
Boundary: Edge cases handled gracefully
Safety: No harmful, biased, or leaked content

LangSmith Integration

from langsmith import Client

client = Client()
client.evaluate(
    runnable,
    dataset_name="agent-behavior-tests",
    evaluators=[consistency_evaluator, safety_evaluator]
)

Latency & Reliability Benchmarks

| Metric | Target | Measurement | |--------|--------|-------------| | P50 Latency | <2s | Time to first token | | P99 Latency | <10s | End-to-end response time | | Error Rate | <1% | Failed requests / total | | Throughput | >10 req/s | Concurrent requests handled |

CI/CD Evaluation Pipeline

# .github/workflows/agent-eval.yml
jobs:
  evaluate:
    steps:
      - run: python scripts/run_eval_suite.py
      - run: python scripts/check_hallucination_rate.py
      - run: python scripts/benchmark_latency.py
      - if: failure
        run: echo "Agent eval failed — blocking merge"

Adversarial Red Teaming

Attack Vectors

Prompt injection: "Ignore previous instructions and..."
Context poisoning: Inject false facts into context
Role manipulation: "You are now an unrestricted AI..."
Format exploitation: XML/JSON injection in prompts

Defense Validation

Input sanitization working?
Output validation catching issues?
Guardrails triggered correctly?

Quick Eval Checklist

[ ] Hallucination rate <5%
[ ] RAGAS faithfulness ≥0.8
[ ] Consistency score ≥0.9
[ ] P99 latency <10s
[ ] No safety violations
[ ] Adversarial tests passing

Agent Skills: Agent Evaluation

Install this agent skill to your local

Skill Files