OpenTelemetry Instrumentation with Kopai
Emit wide events, then prove they are good.
A span is a wide event: one flat record carrying the full context of a unit of work. Kopai runs on localhost, so you never ship instrumentation on faith — you drive traffic through the app yourself and assert on what arrived. The loop is green when every assertion in validate-traces passes. Nothing here is finished while an assertion is red.
Branches
| You are… | Start at | | ------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | Setting up a service that has no telemetry | Workflow step 1 below | | Retrofitting OTel into an existing codebase | context-propagation first — it is ~60% of the work — then the workflow | | Chasing data that isn't arriving | troubleshoot-no-data, then step 4 of the workflow |
Workflow
| # | Step | Done when |
| --- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1 | Start the backend — npx @kopai/app start | curl -s -o /dev/null -w '%{http_code}' -X POST http://localhost:4318/v1/traces -H 'Content-Type: application/json' -d '{"resourceSpans":[]}' returns 2xx |
| 2 | Configure the environment — setup-environment | OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_SERVICE_NAME, and a validation.run_id run tag are all exported in the shell that launches the app |
| 3 | Instrument — the matching lang-<language> rule under Language SDKs below, then the judgment rules | The app boots with the SDK loaded and prints no OTel export errors |
| 4 | Drive traffic yourself — drive-traffic | Every discovered entry point hit at least once and at least one deliberate failure driven |
| 5 | Assert — validate-traces, then validate-logs, validate-metrics, validate-shutdown | Every assertion passes. The loop is green |
| 6 | Fix — the matching troubleshoot-* rule under Fix it below | Return to step 4 and re-drive. Never stop on a red assertion |
Steps 4–6 are a loop, not a sequence. Report the instrumentation complete only when step 5 is green against traffic you drove in step 4 of the same run.
What earns a span
Two questions decide it:
- Interesting? — does this work meaningfully move latency or failure for the request?
- Aggregable? — grouped by name and attributes, does it produce a useful trend?
Both yes → create a span. Otherwise → put an attribute on the span you already have.
When in doubt, prefer attributes over child spans. An attribute on the parent needs no JOIN, costs nothing extra to query, and is immediately groupable. Full decision table, plus the three ways this goes wrong, in instrument-spans.
The core pattern
One span, rich attributes, explicit error status. Node.js shown; every language follows the same shape — see your language's rule under Language SDKs below:
import { trace, SpanStatusCode } from "@opentelemetry/api";
const tracer = trace.getTracer("checkout");
export async function processPayment(order: Order) {
return tracer.startActiveSpan("process-payment", async (span) => {
try {
span.setAttributes({
"order.id": order.id, // IDs are attributes, never span names
"order.total": order.total,
"payment.provider": order.provider,
});
const receipt = await charge(order);
span.setAttribute("payment.status", receipt.status);
return receipt;
} catch (err) {
span.recordException(err as Error);
span.setStatus({
code: SpanStatusCode.ERROR,
message: (err as Error).message,
});
throw err;
} finally {
span.end();
}
});
}
Full code for every pattern — nested spans, async fan-out, queues, manual context — in custom-instrumentation.
Naming
- Span names describe the operation:
GET /api/users,db.query SELECT,process-payment - Attribute names are dot-namespaced:
user.id,order.total,cache.hit - Follow OTel semantic conventions for anything standard:
http.route,db.system,rpc.service - Namespace your own additions:
app.,checkout.,<company>.
Span names must be low-cardinality. Never interpolate an ID into a span name — that belongs in an attribute.
Picking instrumentation packages
Detect the project's package manager first: the packageManager field in package.json
is authoritative when present; otherwise the lockfile decides — pnpm-lock.yaml →
pnpm, yarn.lock → yarn, bun.lock → bun, package-lock.json → npm. Run every
install with that manager — a second manager writes a second lockfile and diverges
silently. The npx @kopai/cli validation commands are manager-neutral; npx ships with
Node and never touches the project's dependency tree.
Check deprecation against the registry, not install output — only npm prints full
deprecation warnings at install time; pnpm truncates them and Yarn 4 prints none at
all. When npm does print them, read them: never filter an install through tail,
grep -v, or --quiet.
npm view @opentelemetry/instrumentation-<lib> deprecated # empty output = not deprecated
npm view is a pure registry read that works inside pnpm/yarn projects too (npm ships
with Node); native equivalents are pnpm view <pkg> deprecated and
yarn npm info <pkg> --fields deprecated --json (Yarn 4).
A deprecation message usually names its replacement — use it, never the deprecated
package; when none is named, follow the package's own migration guidance instead of
guessing. When a framework team ships its own plugin (Fastify → @fastify/otel), prefer
it over the contrib package: it registers through the framework's plugin system instead
of intercepting require/import — interception that native-ESM imports bypass unless
an experimental loader hook is added.
Rules
Task-scoped rule files. Load one only when the workflow points at it.
Setup — setup-backend, setup-environment
Language SDKs — lang-nodejs, lang-fastify, lang-nextjs, lang-python, lang-go, lang-java, lang-dotnet, lang-ruby, lang-php, lang-rust, lang-erlang, lang-cpp
What to instrument — instrument-spans (what earns a span),
instrument-attributes (which attributes, timing attributes, async summaries),
instrument-errors (error, span status, slug),
context-propagation (threading context; framework accessor gotchas),
layered-telemetry (traces vs metrics vs logs),
sampling
Prove it — drive-traffic (generate the traffic yourself), validate-traces, validate-logs, validate-metrics, validate-shutdown
Fix it — troubleshoot-no-data, troubleshoot-missing-spans, troubleshoot-missing-attrs, troubleshoot-wrong-port
References
Deep dives — load only when the task needs them:
- attributes — the canonical attribute catalog, by category
- custom-instrumentation — full code for every pattern, per language
- architectural-patterns — queues, async fan-out, ETL, serverless
- cli-reference — Kopai CLI commands
- nextjs-examples — Next.js instrumentation examples
- otel-docs — OpenTelemetry documentation links
Related skills
- otel-genai-instrumentation — LLM, agent, and tool-call instrumentation
- root-cause-analysis — investigating telemetry once it is flowing. Instrument for the questions that skill asks: who was affected, what changed, where the time went
- create-dashboard — visualising the signals you emit here