Agent Skills: bot-finding-arbiter · *Praetor*

bot-finding-arbiter · Praetor

Praetor — the Roman magistrate who adjudicated disputes AND issued the edictum praetoris, an annually-refined statement of the rules he would apply. This skill is the software praetor: it judges a reviewer-bot's finding (route to one of 7 dispositions) and, when the bot is verifiably wrong, issues an edict — a teaching edit to the bot's own repo config so the bot works to our governance from now on. It composes existing convergence primitives; it does not re-implement judging, verifying, commenting, or merging.

The one non-negotiable (read first)

⛔ A config edit MUST NEVER suppress a valid security or logic finding. Editing a bot's config to make it stop complaining is only legitimate for a verifiably-wrong finding (false-positive / governance-misalignment / style-only). If the finding is a real secret · injection · auth flaw · CVE · correctness bug → the answer is fix or HITL, never "teach the bot to be quiet." Silencing a real finding = gaming the scanner (Goodhart) = forbidden. The "bot is wrong" verdict is the gate to a config edit, and that verdict is independently verified (verifier > generator) before any file is touched.

When to use / not use

• Use: a build/CI/pipeline/PR is failing or blocked on a bot-code-reviewer finding and you must decide what to do; OR a reviewer bot emits a recurring false-positive worth teaching away.
• Not use: driving a whole session to green (→ quiesce); merely watching PR state (→ gh pr checks / bin/bb-pipeline-watch.sh); a finding you will simply fix inline with no arbitration needed (just fix it, per pr-governance-unified Step 4).

Inputs

<pr> (number/url) · optional <finding-selector> (bot name / check context / comment id). If omitted → ingest ALL open findings on the PR and arbitrate each.

The OODA loop (per finding)

1 — OBSERVE (intake — reuse, don't rebuild)

• Pull the finding(s) via the existing intake, per host: GitHub → gh pr view <pr> --json statusCheckRollup,reviews,comments + gh api repos/<owner>/<repo>/commits/<sha>/statuses (reviews + checks + commit-statuses); Bitbucket → bin/bb-pipeline-watch.sh (emits each bot's verdict + comment as a parseable envelope).
• GitHub WATCH parity (native — resolved, no wrapper built): to wait on a red/running check the way bb-pipeline-watch.sh does on Bitbucket, use the native primitive: gh pr checks <pr> --watch --fail-fast --json name,bucket,description,link (the bucket field is a parseable {pass|fail|pending|skipping|cancel} envelope; exit code 8 = still pending) + on failure gh run view <run-id> --log-failed for the error-relevant log tail — the <run-id> comes from the failing check's link field in the same gh pr checks --json output (…/actions/runs/<run-id>/job/…), or from gh run list --branch <branch> --status failure --json databaseId. A custom bin/gh-checks-watch.sh is intentionally NOT built — the native flags already return a structured failure diagnosis (Gordian: native-primitive-over-custom-machinery).
• For each finding capture: bot · context (check name) · state (error/failure/pending) · description · target_url · the code hunk it points at · comment.commit_id (the ref the bot actually reviewed — NOT the PR HEAD) · comment.line (null is GitHub's OUTDATED marker). Without those last two a staleness check has no data to run on, and every verdict silently assumes the bot reviewed HEAD.
• Recon-before-assume (Skopos): read the actual finding + the code it cites before forming any verdict.

2 — ORIENT (classify + INDEPENDENTLY verify)

Classify the finding into exactly one bucket — and the "bot-wrong" verdict is verified by an independent lens (not the same reasoning that proposed it):

| Bucket | Meaning | Verify gate (verifier > generator) | |---|---|---| | valid-actionable | the bot is right; the code should change | a deterministic oracle where one exists (test / compile / re-run the scanner on the hunk) OR a 2nd-lens read | | bot-wrong | false-positive · governance-misalignment · style-only · infra/account error (e.g. quota, not code) | MANDATORY independent verify via perspective-trio / cascade-resolver / convergence-engine (REFINE/SELECT) OR a deterministic proof that names the ref it was proven at (see the anchoring rule below) | | ambiguous | can't ground either way after recon | → DEFER + comment (never guess a config edit) |

• ⛔ Anchor the proof to the ref the bot reviewed — a review comment is anchored to comment.commit_id, not the PR HEAD. A "deterministic proof" MUST name the ref it was taken at, and on a mutable branch that ref MUST equal comment.commit_id. Verifying against HEAD after the code was fixed shows clean code and yields "the bot misread it" — when the bot was right and the finding is merely stale. That is a bot-wrong verdict reached with full confidence and zero grounding, and it is worse than no verdict: it teaches the next reader to distrust a correct bot. comment.line == null ⇒ OUTDATED ⇒ staleness-recon BEFORE any verdict — the correct disposition is stale-but-correct, never bot-wrong. (Empirical, session e528b822 / PR #250: both CodeRabbit findings were verified at HEAD 10c54b0 post-fix and declared FALSE; anchored at comment.commit_id a02d70b the bot was right on both. An independent arbiter reached truth only by disobeying the stale-premised brief.) This generalizes the Security-class AND below — the hardening was always the general rule, applied to one class.

• Staleness-recon reads metadata ONLY — and each command may claim only what it reaches:

  • git log --format='%H %ci' <ref>..HEAD -- <file> reaches file level: it establishes only that the file changed since <ref>. It is indirect evidence — a commit touching another region of the same file satisfies it. It can rule staleness OUT (no commits ⇒ nothing moved), never IN.
  • git diff --unified=0 <ref>..HEAD -- <file> reaches line level: read the @@ ranges and check whether comment.line falls inside one. THIS is what decides whether the finding's own region moved — and it is exactly why comment.line is in the capture list above.
  • ⛔ Never %B: reading the fixing commit's message hands the verifier the conclusion and destroys the independence the verifier > generator gate exists to guarantee.

  A file-level command reported as a hunk-level proof is the same defect this whole section exists to prevent, one layer up: an instrument whose reach is narrower than the claim it is cited for.

• ⛔ Security class (secret/injection/auth/CVE/data-loss): a "bot-wrong" verdict here requires a deterministic proof AND is HITL-gated before any suppression-style config edit — no autonomous silence.

• Account/infra error (the bot's platform failed, not the code — e.g. Snyk "test limit reached", a scanner timeout): classify bot-wrong → but NOT repo-fixable (see registry repo-fixable? column) → HITL playbook, no config edit (a repo file cannot fix an account-side quota).

• Deterministic pre-filter (run FIRST — the ECE deterministic skeleton): bin/classify.sh (requires jq) reads the finding envelope {bot, context, state, description} and code-enforces the two safety cues before the probabilistic verify runs — (a) state=error + platform keywords → account-error · NOT-repo-fixable · HITL; (b) security substance (secret/injection/auth/CVE/data-loss) → security-class · HITL-gated · never-suppress; (c) everything else → content · defer-to-verify (hand to the probabilistic ORIENT above). It only gates, never suppresses — a content verdict always defers to the verifier. The gates are proven by tests/run.sh (7/7 fixtures, incl. gitleaks-secret → never-suppress + Snyk-quota → not-repo-fixable).

3 — DECIDE (the 7 dispositions)

Route the finding to exactly ONE (elevates the 5-path pr-governance-unified Step-8 menu to per-finding + 7-way):

| # | Disposition | When | |---|---|---| | 1 | accept-total | valid; adopt the bot's suggestion as-is | | 2 | accept-partial+adapt | valid core, but adapt the suggestion to our context | | 3 | fix | valid; correct the code (may differ from the bot's exact suggestion) | | 4 | improve | valid + opportunity to go beyond the minimal fix | | 5 | expand | valid + the finding reveals a broader gap worth addressing | | 6 | reject-total | bot-wrong; the finding does not apply | | 7 | comment/justify | ALWAYS — every disposition posts a rationale comment on the PR (reuse pr-reviewer-communication / vek-pr-commentator) | Plus (only on a verified bot-wrong that IS repo-fixable): teach-the-bot (§4) — an edict.

4 — ACT

• Code dispositions (1-5): apply in the worktree; converge + merge via existing gates (pr-governance-unified Steps 4/9 · auto-merge-standing-authorization). Do NOT rebuild convergence.
• reject-total (6): post a rationale comment; if the bot supports thread-resolve, resolve it.
• teach-the-bot (edict): write the minimal teaching edit to the bot's config file per the registry (bot-config-registry.md), as a reviewed PR (worktree → review → merge), NEVER a silent commit. Prefer a narrow rule (path-scoped instruction / documented-convention note) over a broad ignore. Record it on the Bot Scorecard (vek-pr-commentator) as a config-taught disposition so accuracy is tracked (the compounding win: each edict permanently shrinks future false-positives).
  • Best-practices grounding (MANDATORY, before authoring the edict): consult BOTH (a) the bot's official, current config documentation (the find-docs skill, a docs-research MCP such as mcp__ref-tools-mcp__* when available in the host, or WebSearch — config surfaces drift between releases; a stale/wrong key is a silent no-op that lets the false-positive recur, per the registry's recon-before-assume discipline) AND (b) the governance anchor the edict encodes (pr-governance-unified § Bot-Config Correction Discipline / the repo's documented convention). The edict PR body MUST cite both — the official doc proves the key/syntax is real; the governance anchor proves the teaching is our standard, not self-serving convenience.
• ALWAYS (7): the rationale comment names the disposition + the evidence (the audit trail).

Teach-the-bot registry

The bot → config-file → what-it-teaches → repo-fixable? map is the SSOT in bot-config-registry.md. Snyk quota / account entitlements = NOT repo-fixable (dashboard) → HITL, no edit.

Composition (reuse map — build nothing new here)

| Need | Reused primitive | |---|---| | intake findings | gh (GitHub: pr view --json + api …/commits/<sha>/statuses) · bin/bb-pipeline-watch.sh (Bitbucket) | | deterministic ORIENT pre-filter (account-error / security-class gate) | bin/classify.sh + tests/ fixtures (the only net-new code — a thin gate, not new convergence) | | independent verify (verifier > generator) | skills/convergence-engine · agents/perspective-trio · agents/cascade-resolver · agents/persona-pipeline · bin/convergence-guard | | disposition menu (elevated 5→7) | rules/pr-governance-unified.md Step 8 | | post comment / scorecard | .claude/rules/pr-reviewer-communication.md · vek-ai-toolkit:vek-pr-commentator (Bot Scorecard) | | merge gate | auto-merge-standing-authorization · bin/convergence-guard | | loop discipline / OODA / no-silent-drop | loose-end-triage-queue (Taxis) |

Bounds · Skip · Guardrails

• Bounds: ≤ the existing PDCA cap (6 iterations/PR) · verify time-boxed · one config-edit PR per bot per convergence · never touch .github/workflows/ (critical-infra) autonomously → HITL.
• Skip: trivial single fix (just fix it inline) · read-only inspection · a finding already resolved · mid-orchestration under a parent already arbitrating.
• Guardrails: ⛔ never suppress a valid security/logic finding · config-edit = reviewed PR never silent · account/infra errors are not repo-fixable · shared-repo edits go in a worktree (concurrency).

Anti-patterns (do NOT)

1. ❌ Teach-the-bot to be quiet about a REAL finding (the central forbidden move — Goodhart/security).
2. ❌ Config edit without independent verify (self-declared "false positive" → self-serving edit).
3. ❌ Silent config commit (must be a reviewed PR).
4. ❌ Broad ignore when a narrow path-instruction suffices (over-suppression).
5. ❌ Treat an account/infra error as repo-fixable (Snyk quota ≠ .snyk edit).
6. ❌ Rebuild convergence / commenting / merging (compose the existing primitives).
7. ❌ Skip the always-comment (7) — every disposition leaves an audit-trail rationale.

Quality Tests (6/6 §11 self-validity + 7/7 fixtures) + grounding

1 Self-Application ✅ (built under worktree→PR→converge→merge, the very loop it arbitrates) · 2 Non-Contradiction ✅ (composes pr-governance-unified/convergence-engine/auto-merge-standing; zero duplication) · 3 Survival ✅ · 4 Bounded ✅ (PDCA cap · skips · one-edit-PR · never-critical-infra) · 5 Explicit-Exception ✅ (skips + HITL gates + §0 SER) · 6 Utility-Sunset ✅ (DUED below). Anti-theater 8/8 (the teach-the-bot lever is real config-as-code, empirically grounded in .pr_agent.toml precedent; the ⛔never-suppress-valid-security gate is now code-proven by tests/run.sh (7/7), not prose-only).

DUED Sunset (qualitative)

Deprecate when ANY: a host ships native "arbitrate + teach reviewer bot" (E1) · absorbed into convergence-engine/pr-governance-unified as one entry (E6) · operator retraction (E4) · ≥3 false-positive suppressions slip through (E5 → tighten the verify gate, not deprecate).

Refs

rules/pr-governance-unified.md (§ Bot-Config Correction Discipline — the governance this skill executes) · bot-config-registry.md (the SSOT map) · skills/convergence-engine · agents/{perspective-trio,cascade-resolver,persona-pipeline} · bin/classify.sh (deterministic ORIENT pre-filter) + tests/run.sh · bin/{convergence-guard,bb-pipeline-watch.sh} · gh CLI · vek-ai-toolkit:vek-pr-commentator (Bot Scorecard) · akasha pr-review-protocol.md / auto-merge-standing-authorization.md / loose-end-triage-queue.md. Named by anima (soul-name Praetor, per [[naming-authority]] [C-naming]).

Changelog

| Version | Date | Change | |---|---|---| | 1.4.0 | 2026-08-16 | Anchor a bot-wrong verdict to the ref the bot reviewed (docs-only; classify.sh + tests/ untouched). A review comment is anchored to comment.commit_id, not the PR HEAD — but the bot-wrong gate let a "deterministic proof" substitute the MANDATORY independent verify with no constraint on which ref the proof was taken at. Verifying at HEAD after a fix shows clean code and yields "the bot misread it" when the bot was right and the finding was merely stale: a verdict with full confidence and zero grounding, which teaches the next reader to distrust a correct bot. Compounding it, the capture list never collected the anchor — grep -ciE "commit_id\|outdated" over v1.3.0 = 0, so a staleness check had no data to run on. Three edits: (a) capture list takes comment.commit_id + comment.line; (b) a deterministic proof MUST name its ref, on a mutable branch that ref MUST equal comment.commit_id, and line == null (GitHub's OUTDATED marker) routes to staleness-recon before any verdict → stale-but-correct, never bot-wrong — this generalizes the ⛔ Security-class AND, which was always the general rule applied to one class; (c) staleness-recon reads metadata only, ⛔ never %B (the fixing commit's message hands the verifier its conclusion and destroys verifier > generator independence). Empirical (session e528b822 / PR #250 · ADR-011): both CodeRabbit findings verified at HEAD 10c54b0 post-fix and declared FALSE; anchored at comment.commit_id a02d70b the bot was right on both — an independent arbiter reached truth only by disobeying the stale-premised brief. PDCA (self-dogfood — Praetor arbitrated qodo's review of its own v1.4.0 PR #351; 2 findings, both valid-actionable, both anchored at head b5bdd50 with non-null line ⇒ live, not stale, by this very rule): (1) frontmatter bumped to 1.4.0 without a matching row here → this row; (2) the (c) command was cited beyond its reach — git log … -- <file> was documented as establishing "whether the hunk moved", but it reaches file level and returns zero line information (measured). Split into file-level (git log, indirect — can rule staleness OUT, never IN) and line-level (git diff --unified=0, read the @@ ranges against comment.line), which is precisely why comment.line is captured. An instrument cited beyond its reach is the same defect this section exists to prevent, one layer up. Surfaced by praxis-audit v0.1.0 dogfood cycle 2 (ledger praxis-audit/002); closes #262. | | 1.3.0 | 2026-07-01 | Residual /enhance Round #(N+1) — directive-triage verified the operator directive ~85-90% already satisfied by v1.2.x; this MINOR closes the 3 verified residual gaps (docs-only; classify.sh + tests/ untouched, still 7/7). (a) G-bp — explicit best-practices grounding: the teach-the-bot edict now MANDATES consulting the bot's official current config docs (find-docs skill / a docs-research MCP e.g. mcp__ref-tools-mcp__* / WebSearch) AND the governance anchor before authoring, and citing BOTH in the edict PR (was implicit in narrowest-form; directive asks "best practices + our governance" explicitly). (b) G-gh — GitHub watch parity resolved native: documented gh pr checks <pr> --watch --fail-fast --json name,bucket,… (bucket envelope, exit 8 = pending) + gh run view --log-failed as the GitHub sibling of bb-pipeline-watch.sh; probe confirmed the native flags return a structured failure diagnosis → NO custom wrapper built (Gordian). (c) G-fire — fire-point effectivation (cross-file): skills/quiesce v0.2.0 + pr-governance-unified v1.2.1 now ROUTE bot-blocked PRs / per-finding review analysis to this skill as the default handler — "toda vez que" now has an active trigger instead of a passive skill (Skopos PHASE-2 lesson: sharpen a fire-point > add passive prose). | | 1.2.1 | 2026-07-01 | PDCA (self-dogfood — Praetor arbitrated qodo's review of its own v1.2.0 PR #193; 3 valid findings → all accept-partial+adapt, docs-only, classify.sh + tests/ still 7/7). (1) Semgrep \|\| true: reframed from a neutral "teaching form" to an explicit suppression anti-pattern (HITL-only + alerting + expiry, never silent) — a security-class row must not read as endorsing gate-disabling (aligns with the row's own never blanket-disable + ⛔never-suppress). (2) nosemgrep syntax: # nosemgrep is a no-op in //-comment languages (JS/TS/Java) → generalized to "a comment in the flagged file's own language syntax containing nosemgrep" + JS/TS + Python examples (correctness). (3) consistency: rules/pr-governance-unified.md bot→config map reframed as candidate surfaces (recon-before-assume) so it no longer lists .amazonq/rules/ plainly as SSOT without the confirm-first caveat the registry now carries. qodo's recurring "missing Jira key" sub-claim = known FP (repo uses Linear; already taught in .pr_agent.toml) → rejected, no edit. | | 1.2.0 | 2026-07-01 | Registry /enhance (docs-only — classify.sh + tests/ untouched, still 7/7). (a) corrija (anti-theater R4): the Amazon-Q row claimed .amazonq/rules/*.md as a teaching surface, but that is not the observed convention — .amazonq/ dirs are commonly OpenSpec/prompt hosts, not reviewer-config → row marked ⚠️ confirm-first, default HITL when absent (never invent the file). (b) melhore: added a Grounding discipline (recon-before-assume) subsection — the registry entries are candidate surfaces; several documented config files are frequently NOT adopted (Qodo best_practices.md → real surface is extra_instructions; Amazon-Q .amazonq/rules/), so OBSERVE must ls-verify the file exists before an edict (a written-but-wrong path is a silent no-op that lets the FP recur). (c) expande: added a Semgrep row — commonly CI-registry-driven (--config p/ci) with no committed rule file; teach = inline # nosemgrep: <rule-id> + rationale (narrowest) or tune the CI invocation; ⚠️ security-class, verify-gated. Vendor-neutral (Layer-Purity clean — no repo-specific inventory hardcoded into the community skill; the per-repo map stays operator-side). | | 1.1.0 | 2026-07-01 | Harden — (a) corrija: fixed the dangling bin/pr-review-watch GitHub-intake reference (cited 5× but the file never existed) → repointed to the real gh CLI (pr view --json + api …/statuses), Bitbucket stays bb-pipeline-watch.sh; a symmetric bin/pr-review-watch wrapper is a deferred nicety (gh already does it). (b) melhore: added bin/classify.sh — a deterministic ORIENT pre-filter (ECE skeleton) that code-enforces the two safety cues (account/platform-error → NOT-repo-fixable · security-class → HITL-gated never-suppress; content → defer-to-verify) + tests/ (7 fixtures + run.sh, converge-style) that prove the ⛔never-suppress-valid-security gate (7/7 pass). The gate was prose-only in v1.0.0. PDCA (self-dogfood — Praetor arbitrated its own PR #192): (i) amazon-q :stop_sign: → tightened bare limit to quota-specific phrases (accept-partial+adapt; their test.?limit suggestion would have broken the real "used your limit of private tests" variant → case-06); (ii) qodo → removed the ambiguous import cue (a code import-error is repo-fixable → defer, not dismiss → case-07), word-boundaried HTTP codes, surfaced classifier stderr on test failure, documented the jq dependency. jq required by classify.sh. | | 1.0.0 | 2026-07-01 | Bootstrap — per-finding 7-way OODA disposition arbiter + teach-the-bot config-as-code edict (the greenfield capability: no prior tool wrote back to a bot's config; elevates the manual .pr_agent.toml + pr-governance-unified Known-FP precedent into a governed, multi-bot, verified loop). ⛔ never-suppress-valid-security hard gate + verifier>generator verify before any edit. Composes existing convergence/intake/comment/merge primitives (zero duplication). Soul-name Praetor via anima. |