review-scoring
v1's review rubric was a single scalar score (0-10, threshold 9.9) computed by an automated runtime daemon. That daemon is cut for v2 (no standing usage justified keeping a background process for it) and the scalar-score design itself didn't survive review: a single number hides which specific claims were checked and which weren't. v2's rubric is a defect list with severity, produced by a human-in-the-loop review (the refutation-reviewer agent, or you reviewing your own or another agent's work) — this skill is that rubric.
The rubric
A review is not "does this look right" — it's "what would prove this wrong, and did I check for it". For the change under review:
- List the claims. What does the implementer say the change does? (Handles the empty-input case. Covered by a test. Matches the existing error-handling pattern. Doesn't leak the new field to unauthorized users.)
- Try to falsify each one. For each claim, find the concrete input, state, or code path that would make it false — actually trace it or run it, don't just judge plausibility.
- Record what survived and what didn't, each with:
- File/line — where the defect is, precisely.
- Failure scenario — the concrete input/state that triggers wrong output, a crash, or
data loss. "This looks fragile" is not a failure scenario; "calling this with an empty
array throws because
arr[0]is accessed unconditionally" is. - Severity —
critical(data loss/security/crash on common paths),high(wrong behavior on a real path),medium(wrong behavior on an edge case),low(style, maintainability, non-functional). - Verdict —
CONFIRMED(you reproduced it or traced it definitively) orPLAUSIBLE(strong reasoning, not independently verified). Never present aPLAUSIBLEfinding as if it wereCONFIRMED.
Grounded checks come first
Before opinion-based review, run what can actually be run: typecheck, the relevant tests, a manual exercise of the changed behavior. A defect a test or typecheck catches outranks a defect found by reading — cite the tool output, don't restate an opinion a machine already settled.
The merge gate
There is no fixed numeric threshold. The gate is: zero CONFIRMED critical/high findings
remain unaddressed. PLAUSIBLE findings and low/medium findings are judgment calls for the
orchestrator/user — surface them, don't silently drop them, but don't block on them
unconditionally either.
Fix loop discipline
When a review finds defects and the engineer fixes them, the FIX gets reviewed again before confirming — the file version of the "final fix ships unreviewed" off-by-one bug is a known trap: always review the version of the code that's actually about to ship, not the version that was reviewed two edits ago.