Groq Observability Skill

Groq Observability

Overview

Monitor Groq LPU inference for latency, token throughput, rate limit utilization, and cost. Groq's defining advantage is speed (280-560 tok/s), so latency degradation is the highest-priority signal. The API returns rich timing metadata (queue_time, prompt_time, completion_time) and rate limit headers on every response.

Prerequisites

A Groq account with an API key exported as the GROQ_API_KEY environment variable — the groq-sdk client reads it automatically (new Groq()).
Node.js with groq-sdk and prom-client installed (npm install groq-sdk prom-client).
A Prometheus scrape target and (optionally) Grafana for the dashboard panels.

Key Metrics to Track

| Metric | Type | Source | Why | |--------|------|--------|-----| | TTFT (time to first token) | Histogram | Client-side timing | Groq's main value prop | | Tokens/second | Gauge | usage.completion_time | Throughput degradation | | Total latency | Histogram | Client-side timing | End-to-end performance | | Rate limit remaining | Gauge | x-ratelimit-remaining-* headers | Prevent 429s | | Token usage | Counter | usage.total_tokens | Cost attribution | | Error rate by code | Counter | Error handler | Availability | | Estimated cost | Counter | Tokens * model price | Budget tracking |

Instructions

Apply these six steps in order. Steps 1-2 are the core instrumentation loop — wrap the client, then feed a Prometheus instrument set from each call. Steps 3-6 add rate-limit tracking, alerting, structured logs, and dashboards on top. The lean client skeleton is below; the full code for every step lives in references/implementation.md.

Instrumented client — wrap groq.chat.completions.create so latency, tokens, queue time, and estimated cost are captured on the same path as the request (trackedCompletion).
Prometheus metrics — register a histogram (latency), counters (tokens, cost, errors), and gauges (throughput, rate-limit remaining), then feed them from emitMetrics.
Rate limit header tracking — parse x-ratelimit-remaining-* off every response into a gauge so you alert before a 429, not after.
Prometheus alert rules — ship latency/rate-limit/throughput/error/cost alerts tuned to Groq's sub-200ms, 280+ tok/s baseline.
Structured request logging — emit one JSON line per request for log aggregation, preserving per-request detail metrics roll up.
Dashboard panels — TTFT distribution, tokens/sec, rate-limit utilization, request volume, error rate, cost, and queue time.

import Groq from "groq-sdk";

const groq = new Groq(); // reads GROQ_API_KEY

async function trackedCompletion(model: string, messages: any[]) {
  const start = performance.now();
  const result = await groq.chat.completions.create({ model, messages });
  const latencyMs = performance.now() - start;
  const usage = result.usage!;
  const metrics = {
    model,
    latencyMs: Math.round(latencyMs),
    tokensPerSec: Math.round(usage.completion_tokens / ((usage as any).completion_time || latencyMs / 1000)),
    totalTokens: usage.total_tokens,
  };
  emitMetrics(metrics); // -> Prometheus (Step 2)
  return { result, metrics };
}

See references/implementation.md for the complete GroqMetrics shape, pricing table, Prometheus instruments, rate-limit tracking, alert rules, structured logging, and dashboard panel list.

Output

Applying the workflow produces:

A trackedCompletion wrapper that returns { result, metrics }, where metrics is a GroqMetrics object (latency, TTFT, tokens/sec, token counts, queue time, estimated cost).
A Prometheus metric set — groq_latency_ms (histogram), groq_tokens_total / groq_cost_usd / groq_errors_total (counters), and groq_tokens_per_second / groq_ratelimit_remaining (gauges).
Five alert rules (GroqLatencyHigh, GroqRateLimitCritical, GroqThroughputDrop, GroqErrorRateHigh, GroqCostSpike).
A structured JSON log line per request and a 7-panel dashboard spec.

Examples

Instrument a single completion and emit a structured log line:

const { result, metrics } = await trackedCompletion(
  "llama-3.3-70b-versatile",
  [{ role: "user", content: "Summarize this incident report in two sentences." }]
);
logGroqRequest(metrics, result.id);
// metrics.tokensPerSec -> 310, metrics.estimatedCostUsd -> 0.000404

For a 429-guard using rate-limit headers and a dashboard health-reading table, see references/examples.md.

Error Handling

| Issue | Cause | Solution | |-------|-------|----------| | 429 with high retry-after | RPM or TPM exhausted | Implement request queuing | | Latency spike > 2s | Model overloaded or large prompt | Reduce prompt size or switch to lighter model | | 503 Service Unavailable | Groq capacity issue | Enable fallback to alternative provider | | Tokens/sec drop | Streaming disabled or large prompts | Enable streaming for better perceived performance |

Resources

references/implementation.md — full code for all six observability steps.
references/examples.md — worked instrumentation, 429-guard, and dashboard-reading examples.
Groq API Reference (usage fields)
Groq Rate Limits
prom-client on npm
For incident response procedures, see the groq-incident-runbook skill.

Agent Skills: Groq Observability

Install this agent skill to your local

Skill Files