Headless Adapters Skill

Headless Adapters

Purpose

Schema-driven adapter for headless CLI agents. No code required - just define a JSON schema describing how to interact with the CLI.

| Use Case | Tool | |----------|------| | Wrap headless CLI agent | headless command | | Create new schemas | Schema Creation Guide |

Quick Start

Create a schema for your CLI agent using the Schema Creation Guide

Run the adapter:

bunx @plaited/agent-eval-harness headless --schema ./my-agent-headless.json

Any CLI agent that outputs JSON can be wrapped — no agent-specific code required.

CLI Commands

headless

Schema-driven adapter for ANY headless CLI agent.

bunx @plaited/agent-eval-harness headless --schema <path>

Options: | Flag | Description | Required | |------|-------------|----------| | -s, --schema | Path to adapter schema (JSON) | Yes |

Schema Format:

{
  "version": 1,
  "name": "my-agent",
  "command": ["my-agent-cli"],
  "sessionMode": "stream",
  "prompt": { "flag": "-p" },
  "output": { "flag": "--output-format", "value": "stream-json" },
  "autoApprove": ["--allow-all"],
  "outputEvents": [
    {
      "match": { "path": "$.type", "value": "message" },
      "emitAs": "message",
      "extract": { "content": "$.text" }
    },
    {
      "match": { "path": "$.type", "value": "tool_use" },
      "emitAs": "tool_call",
      "extract": { "title": "$.name", "status": "'pending'", "input": "$.input" }
    },
    {
      "match": { "path": "$.type", "value": "tool_result" },
      "emitAs": "tool_call",
      "extract": { "title": "$.name", "status": "'completed'", "output": "$.content" }
    }
  ],
  "result": {
    "matchPath": "$.type",
    "matchValue": "result",
    "contentPath": "$.content"
  }
}

Session Modes:

| Mode | Description | Use When | |------|-------------|----------| | stream | Keep process alive, multi-turn via stdin | CLI supports session resume | | iterative | New process per turn, accumulate history | CLI is stateless |

Creating a Schema

Run the CLI's --help to identify prompt, output format, and auto-approve flags
Capture sample JSON output from the CLI
Map JSONPath patterns to output events (including input/output for tool calls)
Create the schema file
Test with headless command

See Schema Creation Guide for the complete workflow.

Security Considerations

Trust Boundary: CLI Output is Untrusted

The headless adapter parses JSON output from CLI agents. This output may contain content from external sources (web searches, file reads, API responses) that flows into trajectory data:

CLI Agent → JSON stdout → JSONPath extraction → ParsedUpdate → TrajectoryStep

Trajectory fields — especially tool_call.input and tool_call.output — should be treated as untrusted content by downstream consumers (graders, LLM-as-judge, analysis scripts). Do not:

Execute trajectory content as code
Use trajectory content in unsanitized shell commands
Pass trajectory content to LLMs without injection-aware prompting

autoApprove Flags

The autoApprove field bypasses the CLI agent's safety confirmation prompts. Use the least permissive flags your evaluation requires:

| Risk Level | Example | When to Use | |------------|---------|-------------| | High | ["--dangerously-skip-permissions"] | Only in isolated containers (Docker, CI) | | Medium | ["--allowedTools", "Read,Write,Glob"] | Scoped to specific tools | | Low | ["--auto-approve", "read-only"] | Read-only evaluations |

Never run high-risk autoApprove flags outside isolated environments. Use --workspace-dir or Docker for evaluations that modify the filesystem.

Troubleshooting

Common Issues

| Issue | Likely Cause | Solution | |-------|--------------|----------| | Tool calls not captured | JSONPath not iterating arrays | Use [*] wildcard syntax - see guide | | Tool input/output missing | Extract config missing input/output fields | Add input/output paths - see guide | | "unexpected argument" error | Stdin mode misconfigured | Use stdin: true - see guide | | 401 Authentication errors | API key not properly configured | Set the correct API key environment variable for your agent | | Timeout on prompt | JSONPath not matching | Capture raw CLI output, verify paths - see guide | | Empty responses | Content extraction failing | Check extract paths - see guide |

Complete troubleshooting documentation: Troubleshooting Guide

External Resources

AgentSkills Spec: agentskills.io

agent-eval-harness skill - Running evaluations against adapters

Agent Skills: Headless Adapters

Install this agent skill to your local

Skill Files