{{!--
  REPORT Template - Eval Results Summary

  Purpose: Generate statistical reporting with confidence intervals.

  Use Cases:
  - Summarizing eval run results
  - Regression reports (version-to-version)
  - Performance dashboards
  - Stakeholder reporting

  Key Principles (from research):
  - Report Standard Error of Mean (SEM)
  - Include confidence intervals
  - Show pass/fail rates
  - Track version comparisons

  Input Data Structure:
  ```yaml
  report:
    title: string
    eval_run_id: string
    timestamp: string
    use_case: string
    version: string (optional - for regression)
  summary:
    total_cases: number
    passed: number
    failed: number
    pass_rate: number
    mean_score: number
    std_dev: number
    sem: number
    confidence_interval: [number, number]
  dimensions:
    - name: string
      mean: number
      std_dev: number
      weight: number
  failures:
    - test_id: string
      score: number
      reason: string
  comparison:
    previous_version: string
    previous_score: number
    delta: number
    significant: boolean
  ```

  Usage:
  bun run RenderTemplate.ts -t Evals/Report.hbs -d Data/EvalResults.yaml
--}}

# {{report.title}}

**Run ID:** `{{report.eval_run_id}}`
**Timestamp:** {{report.timestamp}}
**Use Case:** {{report.use_case}}
{{#if report.version}}
**Version:** {{report.version}}
{{/if}}

---

## Executive Summary

{{#gt summary.pass_rate 0.9}}
✅ **EXCELLENT** - {{percent summary.pass_rate 1}}% pass rate
{{else}}
{{#gt summary.pass_rate 0.75}}
✓ **GOOD** - {{percent summary.pass_rate 1}}% pass rate
{{else}}
{{#gt summary.pass_rate 0.5}}
⚠️ **NEEDS IMPROVEMENT** - {{percent summary.pass_rate 1}}% pass rate
{{else}}
❌ **CRITICAL** - {{percent summary.pass_rate 1}}% pass rate
{{/gt}}
{{/gt}}
{{/gt}}

| Metric | Value |
|--------|-------|
| Total Test Cases | {{summary.total_cases}} |
| Passed | {{summary.passed}} |
| Failed | {{summary.failed}} |
| **Pass Rate** | **{{percent summary.pass_rate 1}}%** |

---

## Score Statistics

| Metric | Value |
|--------|-------|
| Mean Score | {{summary.mean_score}} |
| Standard Deviation | {{summary.std_dev}} |
| Standard Error (SEM) | {{summary.sem}} |
| 95% Confidence Interval | [{{summary.confidence_interval.[0]}}, {{summary.confidence_interval.[1]}}] |

{{#if comparison}}
### Version Comparison

| Metric | Previous ({{comparison.previous_version}}) | Current ({{report.version}}) | Delta |
|--------|----------------------------------------------|------------------------------|-------|
| Mean Score | {{comparison.previous_score}} | {{summary.mean_score}} | {{comparison.delta}} |

{{#if comparison.significant}}
⚠️ **Statistically significant change detected**
{{else}}
✓ No statistically significant change
{{/if}}
{{/if}}

---

## Dimension Breakdown

| Dimension | Weight | Mean | Std Dev |
|-----------|--------|------|---------|
{{#each dimensions}}
| {{name}} | {{percent weight 1}}% | {{mean}} | {{std_dev}} |
{{/each}}

---

{{#if failures}}
## Failure Analysis

{{failures.length}} test cases failed:

| Test ID | Score | Reason |
|---------|-------|--------|
{{#each failures}}
| `{{test_id}}` | {{score}} | {{truncate reason 50}} |
{{/each}}

### Common Failure Patterns

{{!-- This would be calculated by the renderer --}}
*Analyze the failure reasons to identify patterns*
{{/if}}

---

## Recommendations

{{#lt summary.pass_rate 0.75}}
### Immediate Actions Required

1. Review failing test cases for common patterns
2. Analyze dimension scores to identify weak areas
3. Consider prompt adjustments for low-scoring dimensions
{{/lt}}

{{#if comparison.significant}}
### Regression Alert

{{#lt comparison.delta 0}}
Performance has **decreased** since {{comparison.previous_version}}. Investigate recent changes.
{{else}}
Performance has **improved** since {{comparison.previous_version}}. Document successful changes.
{{/lt}}
{{/if}}

---

## Appendix: Raw Data

```yaml
summary:
  total_cases: {{summary.total_cases}}
  passed: {{summary.passed}}
  failed: {{summary.failed}}
  pass_rate: {{summary.pass_rate}}
  mean_score: {{summary.mean_score}}
  std_dev: {{summary.std_dev}}
  sem: {{summary.sem}}
  confidence_interval: [{{summary.confidence_interval.[0]}}, {{summary.confidence_interval.[1]}}]

dimensions:
{{#each dimensions}}
  - name: {{name}}
    mean: {{mean}}
    std_dev: {{std_dev}}
    weight: {{weight}}
{{/each}}
```

---

*Report generated by PAI Evals System*
