{{!--
  COMPARISON Template - A/B Testing

  Purpose: Generate side-by-side prompt variant comparisons.

  Use Cases:
  - A/B testing prompt variations
  - Comparing model outputs
  - Parameter sweep analysis
  - Prompt optimization

  Key Principles (from research):
  - Position swapping to mitigate bias
  - Same test cases for both variants
  - Statistical significance testing
  - Multi-judge panels for reliability

  Input Data Structure:
  ```yaml
  comparison:
    name: string
    hypothesis: string
    variants:
      a:
        name: string
        description: string
        prompt: string (or path)
      b:
        name: string
        description: string
        prompt: string (or path)
    test_cases: [string] (test case IDs)
    judges:
      - name: string
        model: string
        focus: string
    settings:
      position_swap: boolean
      num_runs: number
      confidence_level: number
  ```

  Usage:
  bun run RenderTemplate.ts -t Evals/Comparison.hbs -d Data/PromptABTest.yaml
--}}

# {{comparison.name}}

## Hypothesis

{{comparison.hypothesis}}

---

## Variants

### Variant A: {{comparison.variants.a.name}}

{{comparison.variants.a.description}}

{{#if comparison.variants.a.prompt}}
**Prompt:**
```
{{comparison.variants.a.prompt}}
```
{{/if}}

### Variant B: {{comparison.variants.b.name}}

{{comparison.variants.b.description}}

{{#if comparison.variants.b.prompt}}
**Prompt:**
```
{{comparison.variants.b.prompt}}
```
{{/if}}

---

## Evaluation Setup

### Test Cases

{{#if comparison.test_cases}}
Running against {{comparison.test_cases.length}} test cases:
{{#each comparison.test_cases}}
- `{{this}}`
{{/each}}
{{else}}
*Test cases not specified - will use all available*
{{/if}}

### Judges

| Judge | Model | Focus |
|-------|-------|-------|
{{#each comparison.judges}}
| {{name}} | `{{model}}` | {{focus}} |
{{/each}}

### Settings

| Setting | Value |
|---------|-------|
| Position Swapping | {{#if comparison.settings.position_swap}}Yes (bias mitigation){{else}}No{{/if}} |
| Runs per Test Case | {{default comparison.settings.num_runs 1}} |
| Confidence Level | {{default comparison.settings.confidence_level 0.95}} |

---

## Execution Protocol

1. **For each test case:**
   {{#if comparison.settings.position_swap}}
   - Run with Variant A as "Option 1", Variant B as "Option 2"
   - Run with Variant B as "Option 1", Variant A as "Option 2"
   - Average scores to mitigate position bias
   {{else}}
   - Generate output with both variants
   - Evaluate with all judges
   {{/if}}

2. **For each judge:**
   - Evaluate both outputs independently
   - Record scores and reasoning
   - Calculate winner for each dimension

3. **Aggregate results:**
   - Calculate win rates per variant
   - Compute statistical significance
   - Determine overall winner

---

## Results Template

### Summary

| Metric | Variant A | Variant B |
|--------|-----------|-----------|
| Win Rate | __%      | __%       |
| Avg Score | __       | __        |
| Std Dev | __        | __        |

### Statistical Significance

- **p-value:** __
- **Confidence Interval:** [__, __]
- **Significant Difference:** Yes/No

### Per-Dimension Breakdown

| Dimension | A Wins | B Wins | Tie |
|-----------|--------|--------|-----|
{{#each comparison.judges.[0].dimensions}}
| {{this}} | __ | __ | __ |
{{/each}}

### Conclusion

Based on the results:
- **Winner:** [Variant A / Variant B / No Clear Winner]
- **Confidence:** [High / Medium / Low]
- **Recommendation:** [Accept hypothesis / Reject hypothesis / Inconclusive]

---

## Run Command

```bash
bun run ~/.opencode/Skills/Evals/EvalServer/Lib/compare.ts \
  --config {{comparison.name}}.yaml \
  --output results/{{comparison.name}}_$(date +%Y%m%d).json
```

---

{{#if comparison.notes}}
## Notes

{{comparison.notes}}
{{/if}}
