Human-Eval Review Skill Skill

Human-Eval Review Skill

Turn analysis output into a single self-contained HTML review file. A human opens it in any browser (no server, no build step, no internet), works through the items, and clicks Download Results to export their judgments as JSON. Progress is autosaved in the browser, and a previous results file can be re-loaded to resume.

This generalizes a family of "did the model get it right?" review tasks into one builder.

When to Use

You ran a matching / retrieval / classification / extraction / translation step and a human needs to confirm, correct, score, or rank the output before it's trusted.
You want a clean review UI you can hand to a collaborator who has only a browser.
You're building a labeled gold set from model proposals.

Critical: Things Claude Won't Know Without This Skill

The output is ONE file, rendered from embedded data

scripts/review_builder.py substitutes your data into scripts/template.html and writes a complete HTML file with the CSS, the renderer JS, and the data all inline. There is no CDN and no dependency. The reviewer can be fully offline.

You shape the data; the page renders it

The renderer is data-driven. You produce a {"config": {...}, "items": [...]} structure (directly, or via the ReviewBuilder API), and the page builds itself. To support a new review task you change the data, never the template.

Only `excerpt` (and `source` with `source_html`) is rendered as HTML — use `` for highlights

In a candidate, excerpt is injected as HTML so you can wrap matched spans in … (rendered as a yellow highlight, exactly like the examples). An item's source is also HTML, but only when you set source_html: true. All other fields (title, reason, heading, prompt, labels, notes, and source by default) are inserted as plain text and are safe. Only put trusted, builder-generated HTML in excerpt / HTML source; escape any text that comes from outside data before you wrap it in .

Five question types cover most eval shapes

| type | Reviewer does | "Reviewed" when | |--------|---------------|-----------------| | match | checks any matching candidates (multi-select) | ≥1 candidate or None checked | | single | picks one candidate (radio) | one candidate or None chosen | | rating | clicks a value on a scale (e.g. 1–5) | a value is chosen | | boolean | clicks Yes / No | either chosen | | text | types a correction | the box is non-empty |

allow_none: true adds a "None of the above" option (mutually exclusive with candidates). notes: true adds an optional free-text notes box to any question. An item can hold several questions (e.g. one per sub-entry) — progress counts questions, not items.

Excluded candidates stay selectable

Candidates you put in excluded render inside a collapsed "Show N excluded candidates" <details> block. They're still selectable, so a reviewer can rescue one the system wrongly dropped — useful for catching false negatives.

CJK text needs `cjk: true`

Set cjk: true (and an appropriate lang, e.g. zh-Hant) to enable word-break: break-all so long unbroken Chinese strings wrap. The default theme is a serif "manuscript" look; override any CSS variable via theme (see references/review_format.md).

Workflow

1. Point at the analysis output and shape it

Read whatever the upstream step produced (CSV, JSON, JSONL, markdown table) and map it onto items + questions. The fluent builder is usually the most direct:

from scripts.review_builder import ReviewBuilder

rb = ReviewBuilder(
    title="Deed Matching Review",
    subtitle="Fufang Will vs. Jitou Collection",
    lang="zh-Hant", cjk=True,
    result_filename="deed_review_results.json",
)

for row in analysis_rows:                       # however your data is shaped
    item = rb.add_item(source=row["deed_text"], heading=f"Entry {row['n']}")
    q = item.add_question(type="match", allow_none=True, notes=True)
    for cand in row["candidates"][:5]:          # e.g. top-5
        q.add_candidate(
            id=cand["doc_id"], label=cand["doc_id"], title=cand["title"],
            excerpt=cand.get("excerpt"),        # may contain <mark>…</mark>
            score=cand.get("score"), reason=cand.get("reason"),
            preselected=cand.get("score", 0) >= 90,   # optional: pre-check strong hits
        )
    for cand in row["candidates"][5:]:          # the rest, collapsed
        q.add_excluded(id=cand["doc_id"], label=cand["doc_id"], title=cand["title"],
                       score=cand.get("score"))

rb.save_html("deed_review.html")

2. Generate the file

rb.save_html("review.html") — or, if you assembled a data dict / JSON file instead:

python scripts/review_builder.py data.json -o review.html

where data.json is {"config": {...}, "items": [...]} (get this shape from rb.to_dict() / rb.to_json()). Then open it:

open review.html        # macOS
xdg-open review.html    # Linux

3. Review, export, resume

The reviewer works top to bottom; the header bar shows live progress and a green Reviewed badge appears on each completed card.
Keyboard: j/k (or ↑/↓) move between questions, 1–9 select the Nth option / rating, n toggles None, ? shows help.
Work is autosaved to the browser continuously; reopening the same file restores it and shows a "Restored your in-progress review" banner (Start over clears it).
Download Results writes the JSON. Load re-imports a downloaded results file to resume on another machine.

4. Read the results JSON back

{
  "title": "Deed Matching Review",
  "exported_at": "2026-05-27T12:00:00.000Z",
  "reviewed_count": 18, "total": 20,
  "results": [
    {"item_id": "1", "question_id": "q1", "type": "match", "reviewed": true,
     "source": "毛陳氏賣契壹紙…", "selected": ["7-2-15-2"], "none": false, "notes": ""},
    {"item_id": "3", "question_id": "q1", "type": "single", "reviewed": true,
     "selected": [], "none": true},
    {"item_id": "5", "question_id": "q1", "type": "rating", "reviewed": true, "rating": 4},
    {"item_id": "6", "question_id": "q1", "type": "boolean", "reviewed": true, "value": false},
    {"item_id": "7", "question_id": "q1", "type": "text", "reviewed": true, "text": "…"}
  ]
}

Each result carries item_id + question_id (the join keys), type, reviewed, and the type-specific answer. Use it to build gold labels, compute agreement, or feed the next step.

Best Practices

Pre-select high-confidence proposals (preselected=True) so the reviewer confirms by exception rather than selecting everything from scratch — but only when scores are reliable.
Show the model's reasoning. Populate reason and  the matched span in excerpt; it dramatically speeds up adjudication.
Keep the visible candidate list short (top ~5) and push the long tail into excluded.
One question per decision. For sub-entry tasks, add multiple questions to one item rather than cramming choices together — progress and results stay clean.
Set a stable storage_key in config if you regenerate the file (new title or changed items) and want the browser autosave to carry over. Without it, autosave keys off the title plus a hash of the items, so two different review files never share saved answers.

Related Skills

cbdb-local / tgaz-sqlite / wikidata-search — produce the candidate matches this skill reviews (entity linking, placename resolution).
sync-html — push the generated review file to the central HTML vault so a collaborator can open it from the browser.

Resources

scripts/review_builder.py — zero-dependency Python builder + CLI.
scripts/template.html — the HTML/CSS/JS template (edit only to change look or behavior for all reviews; per-task changes go in the data).
references/review_format.md — complete data schema, every field, theming variables, and the results JSON spec.

Agent Skills: Human-Eval Review Skill

Install this agent skill to your local

Skill Files