OpenTelemetry Implementation Guide Skill

OpenTelemetry Implementation Guide

Overview

OpenTelemetry (OTel) is a vendor-neutral observability framework for instrumenting, generating, collecting, and exporting telemetry data (traces, metrics, logs). This skill provides guidance for implementing OTEL in Kubernetes environments.

Quick Start

Deploy OTEL Collector on Kubernetes

# Add Helm repo
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

# Install with basic config
helm install otel-collector open-telemetry/opentelemetry-collector \
  --namespace monitoring --create-namespace \
  --set mode=daemonset

Send Test Data via OTLP

# gRPC endpoint: 4317, HTTP endpoint: 4318
curl -X POST http://otel-collector:4318/v1/traces \
  -H "Content-Type: application/json" \
  -d '{"resourceSpans":[]}'

Core Concepts

Signals: Three types of telemetry data:

Traces: Distributed request flows across services
Metrics: Numerical measurements (counters, gauges, histograms)
Logs: Event records with structured/unstructured data

Collector Components:

Receivers: Accept data (OTLP, Prometheus, Jaeger, Zipkin)
Processors: Transform data (batch, memory_limiter, k8sattributes)
Exporters: Send data (prometheusremotewrite, loki, otlp)
Extensions: Add capabilities (health_check, pprof, zpages)

Collector Configuration

Basic Pipeline Structure

config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: ${env:MY_POD_IP}:4317
        http:
          endpoint: ${env:MY_POD_IP}:4318

  processors:
    batch:
      timeout: 10s
      send_batch_size: 1024
    memory_limiter:
      check_interval: 5s
      limit_percentage: 80
      spike_limit_percentage: 25

  exporters:
    prometheusremotewrite:
      endpoint: "http://prometheus:9090/api/v1/write"
    loki:
      endpoint: "http://loki:3100/loki/api/v1/push"

  service:
    pipelines:
      metrics:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [prometheusremotewrite]
      logs:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [loki]
      traces:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [otlp/tempo]

Kubernetes Attributes Enrichment

processors:
  k8sattributes:
    auth_type: "serviceAccount"
    passthrough: false
    filter:
      node_from_env_var: ${env:K8S_NODE_NAME}
    extract:
      metadata:
        - k8s.pod.name
        - k8s.namespace.name
        - k8s.deployment.name
        - k8s.node.name

Deployment Modes

| Mode | Use Case | Pros | Cons | |------|----------|------|------| | DaemonSet | Node-level collection | Full coverage, host metrics | Higher resource usage | | Deployment | Centralized gateway | Scalable, easier management | Single point of failure | | Sidecar | Per-pod collection | Isolated, fine-grained | Resource overhead per pod |

Common Patterns

Development Environment

Enable debug exporter for visibility
Lower resource limits (250m CPU, 512Mi memory)
Include spot instance tolerations for cost savings

Production Environment

Implement sampling (10-50% for traces)
Higher batch sizes (2048-4096)
Enable autoscaling and PodDisruptionBudget
Use TLS for all endpoints

Detailed References

For in-depth guidance, see:

Collector Configuration: COLLECTOR.md
Kubernetes Deployment: KUBERNETES.md
Troubleshooting: TROUBLESHOOTING.md
Instrumentation: INSTRUMENTATION.md

Validation Commands

# Check collector pods
kubectl get pods -n monitoring -l app.kubernetes.io/name=otel-collector

# View collector logs
kubectl logs -n monitoring -l app.kubernetes.io/name=otel-collector --tail=100

# Test OTLP endpoint
kubectl run test-otlp --image=curlimages/curl:latest --rm -it -- \
  curl -v http://otel-collector.monitoring:4318/v1/traces

# Validate config syntax
otelcol validate --config=config.yaml

Key Helm Chart Values

mode: "daemonset"  # or "deployment"
presets:
  logsCollection:
    enabled: true
  hostMetrics:
    enabled: true
  kubernetesAttributes:
    enabled: true
  kubeletMetrics:
    enabled: true
useGOMEMLIMIT: true
resources:
  limits:
    cpu: 500m
    memory: 1Gi
  requests:
    cpu: 100m
    memory: 256Mi

Gotchas

Collector tail_sampling at end of pipeline doesn't release inflight buffer — drop decisions made late still hold memory; OOM under load.
Auto-instrumentation + manual instrumentation can double-count traces — pick one strategy per service or de-dupe explicitly via sampler.
Context propagation: HTTP B3 vs W3C traceparent headers don't auto-convert; mixed environments silently break trace continuity.
OTel SDK vs Collector protocol versions: minor version mismatches can drop attributes silently (especially semantic-convention attributes).
Resource detection adds platform attributes (cloud.account.id, host.id) that bloat traces — disable when not used for routing.
Batch processor + memory limiter ordering: limiter must precede batch in the pipeline or memory pressure causes batch drops without backpressure.

Agent Skills: OpenTelemetry Implementation Guide

Install this agent skill to your local

Skill Files