Agent-Skills.md

Agent Skills: ClickHouse Observability

|

UncategorizedID: jeremylongshore/claude-code-plugins-plus-skills/clickhouse-observability

Repository

jeremylongshore/claude-code-plugins-plus-skills
jeremylongshoreLicense: NOASSERTION
1,723221

Install this agent skill to your local

pnpm dlx add-skill https://github.com/jeremylongshore/claude-code-plugins-plus-skills/tree/HEAD/plugins/saas-packs/clickhouse-pack/skills/clickhouse-observability

Skill Files

Browse the full folder contents for clickhouse-observability.

Download Skill

Loading file tree…

plugins/saas-packs/clickhouse-pack/skills/clickhouse-observability/SKILL.md

Skill Metadata

Name
clickhouse-observability
Description
|

ClickHouse Observability

Overview

Set up comprehensive monitoring for ClickHouse using built-in system tables, Prometheus integration, Grafana dashboards, and alerting rules. The workflow layers four signal sources: system.* tables (always available, zero dependencies), a Prometheus scrape endpoint, application-level client instrumentation, and alert rules that fire on the failure modes that actually page an on-call — high error rate, latency creep, merge backlog, and resource exhaustion.

Deep configs live in references/ so this file stays a fast, followable map.

Prerequisites

  • ClickHouse instance with system.* table access
  • Prometheus (or compatible: Grafana Alloy, Victoria Metrics)
  • Grafana for dashboards
  • AlertManager or PagerDuty for alerts

Instructions

Step 1: Query system tables for a health snapshot

Start with zero dependencies — the system.* tables already hold everything. Run this for an instant server-health read:

SELECT
    (SELECT count() FROM system.processes) AS running_queries,
    (SELECT value FROM system.metrics WHERE metric = 'MemoryTracking') AS memory_bytes,
    (SELECT count() FROM system.merges) AS active_merges;

Query throughput, insert rates, and per-table part counts (the merge-health signal), plus a full table of which system.* table to poll at what frequency: system table queries & reference.

Step 2: Wire up Prometheus scraping

ClickHouse Cloud exposes a managed Prometheus endpoint (Basic auth with a Cloud API key); self-hosted uses the built-in :9363 /metrics endpoint enabled in config.xml. Write the scrape config to your prometheus.yml. Full Cloud + self-hosted scrape configs and the config.xml block: Prometheus scrape config & Grafana dashboards.

Step 3: Instrument the application client

Server metrics show what ClickHouse does; client metrics attribute latency, error codes, and insert volume to your own code. Wrap queries in a prom-client histogram/counter and expose /metrics. Full instrumentation + structured logging: application-level instrumentation.

Step 4: Build Grafana dashboard panels

Panels for QPS, P50/P95/P99 latency, error rate, and insert throughput are in the Grafana dashboards reference. Or import the official community dashboard: https://grafana.com/grafana/dashboards/23415.

Step 5: Load alert rules

Write Prometheus alert rules for the five production failure modes (error rate, latency, part count, memory, disk) to a rules file loaded by AlertManager. Full rule set plus per-alert tuning notes: Prometheus alert rules.

Output

Applying this skill produces a set of monitoring config artifacts you write to your infrastructure repo:

  • prometheus.yml — scrape config targeting your ClickHouse endpoint
  • clickhouse-alerts.yml — the five-rule alert group loaded by AlertManager
  • A Grafana dashboard (imported ID 23415 or the custom JSON panels)
  • Client instrumentation exposing clickhouse_query_duration_seconds, clickhouse_query_errors_total, and clickhouse_insert_rows_total
  • Ad-hoc system.* queries for on-demand health snapshots

Error Handling

| Issue | Cause | Solution | |-------|-------|----------| | Metrics endpoint empty | Prometheus not configured | Enable /metrics in config | | High cardinality alerts | Too many label values | Reduce label cardinality | | Missing query_log data | Logging disabled | Set log_queries = 1 in config | | Dashboard gaps | Scrape interval too long | Use 10-15s scrape interval |

Examples

Snapshot server health right now — run the Step 1 query against any instance with system.* access; no exporter or scrape needed. See system-tables.md for throughput and merge-health variants.

Alert when merges fall behind — the ClickHouseTooManyParts rule fires when a table exceeds 300 active parts for 10 minutes, the classic inserts-outpacing-merges signal. Full rule + tuning guidance: alerting.md.

Attribute slow queries to your service — wrap calls in instrumentedQuery() so P95 latency and error codes land in Prometheus labeled by query type. See instrumentation.md.

Resources