Agent-Skills.md

Agent Skills: Schema E2E Validation

Earthly E2E validation for YAML schema contracts. TRIGGERS - schema validation, YAML schema, schema contracts, regenerate types.

UncategorizedID: terrylica/cc-skills/schema-e2e-validation

Repository

terrylica/cc-skills
terrylicaLicense: MIT
264

Install this agent skill to your local

pnpm dlx add-skill https://github.com/terrylica/cc-skills/tree/HEAD/plugins/quality-tools/skills/schema-e2e-validation

Skill Files

Browse the full folder contents for schema-e2e-validation.

Download Skill

Loading file tree…

plugins/quality-tools/skills/schema-e2e-validation/SKILL.md

Skill Metadata

Name
schema-e2e-validation
Description
Earthly E2E validation for YAML schema contracts. TRIGGERS - schema validation, YAML schema, schema contracts, regenerate types.

Schema E2E Validation

When to Use This Skill

Use this skill when:

  • Validating schema changes before commit
  • Verifying YAML schema matches live ClickHouse Cloud
  • Regenerating Python types, DDL, or docs
  • Running full schema workflow validation

Prerequisites

Docker Runtime (Required)

Earthly requires Docker. Start Colima before running:

colima start

Check if running:

docker ps  # Should not error

Doppler Access (For validation targets)

Required for +test-schema-validate and +test-schema-e2e:

doppler configure set token <token_from_1password>
doppler setup --project gapless-network-data --config prd

Earthly Installation

brew install earthly

Quick Commands

Generation only (no secrets)

cd ~/eon/gapless-network-data
colima start  # If not already running
earthly +test-schema-generate

Full E2E with validation (requires Doppler)

cd ~/eon/gapless-network-data
colima start  # If not already running
./scripts/earthly-with-doppler.sh +test-schema-e2e

All non-secret targets

cd ~/eon/gapless-network-data
earthly +all

Artifacts

After running +test-schema-generate or +test-schema-e2e, check ./earthly-artifacts/:

| Path | Contents | | -------------------------- | --------------------------- | | types/blocks.py | Pydantic + TypedDict models | | types/__init__.py | Package init | | ddl/ethereum_mainnet.sql | ClickHouse DDL | | docs/ethereum_mainnet.md | Markdown documentation |

For E2E, artifacts are under e2e/types/, e2e/ddl/, e2e/docs/.

Earthfile Targets Reference

| Target | Secrets | Purpose | | ----------------------- | ------- | -------------------------- | | +deps | No | Install uv + dependencies | | +build | No | Copy source files | | +test-unit | No | Run pytest | | +test-schema-generate | No | Generate types/DDL/docs | | +test-schema-validate | Yes | Validate vs ClickHouse | | +test-schema-e2e | Yes | Full workflow + artifacts | | +all | No | Run all non-secret targets |

Troubleshooting

"could not determine buildkit address - is Docker or Podman running?"

Cause: Docker/Colima not running

Fix:

colima start
# Wait for "done" message, then retry
earthly +test-schema-generate

"unable to parse --secret-file argument"

Cause: Wrong flag name or malformed secrets file

Fix: The correct flag is --secret-file-path (NOT --secret-file). The wrapper script handles this, but if running manually:

# WRONG
earthly --secret-file=/path/to/secrets +target

# CORRECT
earthly --secret-file-path=/path/to/secrets +target

Also ensure secrets file has no quotes around values:

# WRONG format
CLICKHOUSE_HOST="host.cloud"

# CORRECT format
CLICKHOUSE_HOST=host.cloud

"OSError: Readme file does not exist: README.md"

Cause: hatchling build backend requires README.md in container

Fix: Ensure Earthfile copies README.md in deps target:

deps:
    COPY pyproject.toml uv.lock README.md ./  # README.md required!

"missing secret" during validation

Cause: Doppler not configured or secrets not passed

Fix:

# Verify Doppler has the secrets
doppler secrets --project gapless-network-data --config prd | grep CLICKHOUSE

# Use the wrapper script (handles secret injection)
./scripts/earthly-with-doppler.sh +test-schema-validate

Cache Issues

Force rebuild without cache:

earthly --no-cache +test-schema-e2e

Implementation Details

Doppler Secret Injection

The wrapper script scripts/earthly-with-doppler.sh:

  1. Downloads secrets from Doppler
  2. Filters for CLICKHOUSE_* variables
  3. Strips quotes (Doppler outputs KEY="value", Earthly needs KEY=value)
  4. Passes via --secret-file-path flag
  5. Cleans up temp file on exit

Secrets Required

| Secret | Purpose | | ------------------------------ | --------------------- | | CLICKHOUSE_HOST_READONLY | ClickHouse Cloud host | | CLICKHOUSE_USER_READONLY | Read-only user | | CLICKHOUSE_PASSWORD_READONLY | Read-only password |

Related Files

| File | Purpose | | ---------------------------------------------------------------------- | ------------------------ | | ~/eon/gapless-network-data/Earthfile | Main build file | | ~/eon/gapless-network-data/scripts/earthly-with-doppler.sh | Secret injection wrapper | | ~/eon/gapless-network-data/schema/clickhouse/ethereum_mainnet.yaml | SSoT schema | | ~/eon/gapless-network-data/docs/adr/2025-12-03-earthly-schema-e2e.md | ADR |

Validation History

  • 2025-12-03: Created and validated with full E2E run against ClickHouse Cloud
  • Lessons Learned:
    • --secret-file-path not --secret-file (Earthly v0.8.16)
    • Doppler --format env outputs quotes, must strip with sed 's/"//g'
    • README.md must be copied for hatchling build backend
    • Colima must be started before Earthly runs

Design Authority

<!-- ADR: 2025-12-10-clickhouse-skill-delegation -->

This skill validates schemas but does not design them. For schema design guidance (ORDER BY, compression, partitioning), invoke quality-tools:clickhouse-architect first.

Related Skills

| Skill | Purpose | | ------------------------------------------ | ------------------------------- | | quality-tools:clickhouse-architect | Schema design before validation | | devops-tools:clickhouse-cloud-management | Cloud credentials for E2E tests | | devops-tools:clickhouse-pydantic-config | Client configuration |