Observability Skill
Bonded to: backend-observability-agent (Primary)
Quick Start
# Invoke observability skill
"Set up structured logging for my application"
"Configure Prometheus metrics"
"Implement distributed tracing with OpenTelemetry"
Three Pillars
| Pillar | Purpose | Tools | |--------|---------|-------| | Logs | What happened | ELK, Loki | | Metrics | System state | Prometheus, Grafana | | Traces | Request flow | Jaeger, OpenTelemetry |
Examples
Structured Logging
import structlog
structlog.configure(
processors=[
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.JSONRenderer()
]
)
logger = structlog.get_logger()
def process_request(request):
log = logger.bind(
correlation_id=request.headers.get("X-Correlation-ID"),
user_id=request.user.id
)
log.info("request_started", method=request.method)
Prometheus Metrics
from prometheus_client import Counter, Histogram
REQUEST_COUNT = Counter('http_requests_total', 'Total requests', ['method', 'status'])
REQUEST_LATENCY = Histogram('http_request_duration_seconds', 'Request latency')
@REQUEST_LATENCY.time()
async def handle_request(request):
response = await process(request)
REQUEST_COUNT.labels(method=request.method, status=response.status_code).inc()
return response
OpenTelemetry Tracing
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
@tracer.start_as_current_span("process_order")
def process_order(order_id: str):
span = trace.get_current_span()
span.set_attribute("order.id", order_id)
with tracer.start_as_current_span("validate"):
validate(order_id)
with tracer.start_as_current_span("charge"):
charge(order_id)
Key Metrics (RED Method)
| Metric | Description | Alert Threshold | |--------|-------------|-----------------| | Rate | Requests/sec | Drop > 50% | | Errors | Error rate % | > 1% for 5 min | | Duration | P99 latency | > 500ms for 5 min |
Troubleshooting
| Issue | Cause | Solution | |-------|-------|----------| | Missing logs | Log level too high | Adjust log level | | High cardinality | Too many labels | Reduce label values | | Broken traces | Context not propagated | Forward headers |