Agent Skills: Performance Testing

Performance Testing

Overview

Validate that a system meets latency, throughput, and stability targets under load, and locate the bottleneck when it doesn't. This file assumes you can write the test code; it focuses on the measurement traps that make load-test numbers lie and the decision criteria for tool/shape selection.

Core Concepts (read first — these are where numbers go wrong)

Open vs. closed workload models

The single most consequential choice. It determines what your numbers mean.

| Model | Load driver | Throughput is… | Overload behavior | Use for | | ---------- | ------------------------------------ | ------------------------ | ------------------------------------------------ | -------------------------------- | | Closed | Fixed VUs, each loops request→wait | Emergent (backpressure) | Self-throttles: slow server → fewer requests sent | Modeling a fixed client pool | | Open | Fixed arrival rate (req/s) | Controlled (you set it) | Queue grows unbounded; latency explodes | Web traffic, finding breaking pt |

⚠ Closed models hide overload. With fixed VUs, when the server slows down each VU sends fewer requests, so offered load silently drops. You can't overwhelm the server past what its own latency allows — you measure a moving target, not capacity. Real internet traffic is open (users arrive independently of server health), so use an arrival-rate executor to find true breaking points.

• k6: closed = constant-vus/ramping-vus; open = constant-arrival-rate/ramping-arrival-rate.
• Locust is closed-model (users); JMeter thread groups are closed; Gatling injectOpen/injectClosed; artillery arrivalRate is open.
• Open executors need preAllocatedVUs/maxVUs headroom; if k6 warns "insufficient VUs," it dropped iterations and your rate was never achieved.

Coordinated omission — why naive latency numbers lie

A closed-model generator that waits for each response before sending the next stops the clock during a stall. One 10s stall that should have blocked 10,000 scheduled 1ms requests gets recorded as one 10s sample instead of 10,000 samples ≥ their overdue time. Result: p99 looks great while users are timing out.

• Detect: suspiciously clean tail (p99 ≈ p50) under load that clearly stuttered; throughput dips that don't show in latency percentiles.
• Fix: measure against an intended schedule, not send-time. Use tools that correct it:
  • wrk2 -R<rate> --latency (constant throughput + HdrHistogram correction) — the reference implementation.
  • k6 arrival-rate executors record each iteration against its scheduled start, so iteration_duration is corrected; still prefer open model for tail truth.
  • HdrHistogram recordValueWithExpectedInterval(v, interval) back-fills omitted samples.
• Never report tail latency from a closed-loop ab/single-thread wrk run at saturation.

Percentiles, not averages

• Report p50 / p95 / p99 / max — never mean alone. Averages hide the tail; the mean can be 40ms while p99 is 3s.
• Latency distributions are right-skewed and multi-modal (cache hit vs miss, GC pause). Mean sits in an empty valley between modes.
• p99 is a per-request user probability, not a rare edge case: a page issuing 100 requests hits its p99 latency on ~63% of loads (1−0.99¹⁰⁰). Tail latency is the typical experience for fan-out.
• Percentiles don't average across shards — never mean the p99s of two load-gen workers. Merge the raw histograms (HdrHistogram) or you get garbage.

Little's Law — sanity-check every run

L = λ × W → concurrency = throughput × latency.

• Required VUs ≈ target_RPS × avg_response_time_s. Wanting 1000 RPS at 200ms needs ≥200 concurrent in flight.
• If achieved RPS ≈ VUs / (latency + think_time) doesn't hold, your generator is the limit, not the server.

Saturation: knee vs. collapse

As load rises: throughput climbs, then plateaus (the knee) while latency turns hockey-stick. Past the knee, more load buys only more latency and eventually errors/collapse.

• Report the knee (max sustainable throughput at acceptable latency), not the collapse point.
• USE method for the SUT: Utilization, Saturation (queue depth/run-queue), Errors — per resource (CPU, mem, disk, net, connection pool).

Warm-up, JIT, GC — discard the transient

• Always discard a warm-up window. Cold caches, empty connection pools, unfilled DB buffer pool, and JIT (JVM/V8) compilation make the first seconds meaningless. In k6 use a ramp stage; in JMH warm-up iterations; in Gatling a nothingFor/ramp.
• GC/JIT cause tail latency, not throughput loss — stop-the-world pauses show up as p99/max spikes. A soak test surfaces them; a 30s run won't.
• JVM needs thousands of iterations to reach steady state (C2 compiler). Benchmark JVM services after warm-up or numbers are 2–10× pessimistic.

Measure the server, not the client

• Run the generator on a separate host from the SUT — co-located, they fight for CPU/net and you measure interference.
• Load-generator-is-the-bottleneck is the most common false result. Detect: latency rises but server CPU is low while generator CPU pegs, or you hit ephemeral-port/FD exhaustion. Symptoms:
  • Single-threaded generators (ab, artillery/Node, locust single-worker) cap ~1 core.
  • Ephemeral port exhaustion (netstat full of TIME_WAIT) → looks like server refusing connections.
  • FD limit (ulimit -n) caps concurrent connections.
• Fix: multi-core generators (k6, Gatling, wrk, JMeter), distributed mode (locust master/worker, k6 cloud/operator), raise ulimit -n, reuse connections. Always correlate client numbers with server-side metrics (APM/Prometheus) — divergence means you're measuring the wrong thing.

Connection reuse & think time

• Keep-alive dominates: new-connection-per-request adds TCP + TLS handshake (often > the request itself). Most tools reuse per VU by default (k6 does). Test both if clients differ (mobile cold-start vs. pooled service). k6: noConnectionReuse: true / noVUConnectionReuse.
• Think time models real users (pauses between actions) and, in closed models, sets effective concurrency (Little's Law). Zero think time in a closed model = an unrealistic hammer that mostly measures your keep-alive path. Use randomized think time (not fixed — fixed sleeps create synchronized request waves).

Realistic data & cardinality

• Hitting one key lies: repeating GET /users/1 measures a cache/DB-buffer hit path nobody experiences. Use realistic key distribution — Zipfian/Pareto (80/20), not uniform, not constant.
• Dataset must exceed cache size, or you benchmark RAM.
• Unique-per-request params defeat query-plan/result caches (good for worst case); constant params inflate results.
• Match production payload sizes, cardinality, and content (unicode, large records).

Load Shapes

One canonical k6 config per shape. Swap constant-vus→constant-arrival-rate (and ramping-vus→ramping-arrival-rate, target = req/s) for the open model when finding limits.

| Shape | Goal | Pattern | | -------------- | -------------------------- | ------------------------------- | | Smoke | Validate script works | 1–5 VUs, 1 min | | Load | Baseline at expected load | Ramp → hold steady → ramp down | | Stress | Find breaking point | Step up past expected until fail | | Spike | Sudden burst recovery | Jump to N×, hold, drop, recover | | Soak | Leaks/degradation | Moderate load for hours | | Breakpoint | Max capacity | Linear ramp with no cap |

// Load: warm-up ramp is mandatory; thresholds abort/mark-fail the run.
export const options = {
  stages: [
    { duration: "2m", target: 100 }, // ramp (discard for analysis)
    { duration: "10m", target: 100 }, // steady state = the measurement
    { duration: "2m", target: 0 }, // ramp down
  ],
  thresholds: {
    http_req_duration: ["p(95)<500", "p(99)<1000"],
    http_req_failed: ["rate<0.01"],
  },
};

// Stress (open model — reveals true limit): step arrival rate until errors/latency break.
export const options = {
  scenarios: {
    stress: {
      executor: "ramping-arrival-rate",
      startRate: 50, timeUnit: "1s",
      preAllocatedVUs: 100, maxVUs: 2000, // headroom or iterations drop silently
      stages: [
        { duration: "2m", target: 200 },
        { duration: "2m", target: 400 },
        { duration: "2m", target: 800 }, // watch for the knee here
        { duration: "2m", target: 1600 },
      ],
    },
  },
};

// Spike: validate autoscaling/cache/recovery, not steady throughput.
export const options = {
  stages: [
    { duration: "1m", target: 50 }, // baseline
    { duration: "10s", target: 1000 }, // spike
    { duration: "3m", target: 1000 }, // hold — does it shed load or fall over?
    { duration: "10s", target: 50 }, // drop
    { duration: "3m", target: 50 }, // recovery — latency must return to baseline
  ],
};

// Soak: surfaces leaks, FD/connection exhaustion, GC creep, disk fill. Hours, not minutes.
export const options = {
  vus: 100, duration: "4h",
  thresholds: { http_req_duration: ["p(99)<1000"], http_req_failed: ["rate<0.01"] },
};
// Watch: RSS growth, GC frequency, pool saturation, p99 drift over time (not the average).

Tools

k6 (default for API/protocol load testing)

Go-based, multi-core, JS scripting, open+closed executors, built-in thresholds. Best default.

import http from "k6/http";
import { check, sleep } from "k6";
import { Rate, Trend } from "k6/metrics";

const errorRate = new Rate("errors");

export const options = {
  scenarios: {
    // open model: request rate is what you control, independent of server health
    api: {
      executor: "ramping-arrival-rate",
      startRate: 100, timeUnit: "1s",
      preAllocatedVUs: 200, maxVUs: 1000,
      stages: [{ duration: "2m", target: 500 }, { duration: "5m", target: 500 }],
    },
  },
  thresholds: {
    "http_req_duration{expected_response:true}": ["p(95)<500", "p(99)<1000"],
    http_req_failed: ["rate<0.01"],
  },
};

export function setup() {
  const r = http.post(`${__ENV.API_URL}/auth/login`,
    JSON.stringify({ email: "load@test.com", password: "x" }),
    { headers: { "Content-Type": "application/json" } });
  return { token: r.json("token") }; // runs once; returned data passed to default fn
}

export default function (data) {
  const res = http.get(`${__ENV.API_URL}/api/users/${__VU}`, {
    headers: { Authorization: `Bearer ${data.token}` },
    tags: { name: "GetUser" }, // aggregate by name, NOT by dynamic URL (avoids metric cardinality blowup)
  });
  errorRate.add(res.status !== 200);
  check(res, { "200": (r) => r.status === 200 });
  sleep(1);
}

k6 gotchas:

• Tag dynamic URLs with a static tags.name, or per-URL metrics explode and dashboards choke.
• discardResponseBodies: true cuts generator memory/CPU when you don't inspect bodies — raises max achievable load.
• check() failures do not fail the run; only thresholds do (and --fail-on-check / abortOnFail on a threshold aborts early).
• CLI: k6 run -e API_URL=... script.js; --out experimental-prometheus-rw or --out influxdb=... for Grafana; k6 run --vus 50 --duration 30s overrides options.
• Extensions (xk6) add gRPC, SQL, browser, Kafka — not in the core binary.

Locust (Python, closed-model, distributed)

Use when scenarios need Python logic/libraries or complex stateful user journeys.

from locust import HttpUser, task, between

class WebsiteUser(HttpUser):
    wait_time = between(1, 3)  # think time → sets effective concurrency (closed model)

    def on_start(self):
        r = self.client.post("/login", json={"username": "u", "password": "p"})
        self.token = r.json()["token"]

    @task(3)  # weight: runs 3× as often as weight-1 tasks
    def browse(self):
        # name= groups metrics for parameterized URLs (same role as k6 tags.name)
        self.client.get(f"/api/products/{randint(1,1000)}", name="/api/products/[id]",
                        headers={"Authorization": f"Bearer {self.token}"})

locust -f lf.py --host=https://api.example.com                       # web UI
locust -f lf.py --headless -u 500 -r 50 -t 5m --host=...             # -u users, -r spawn/s
locust -f lf.py --master   &   locust -f lf.py --worker --master-host=IP  # distributed: 1 worker/core

⚠ Single Locust process is one core — it is the bottleneck past a few hundred RPS. Always run master + workers (≥1 worker per CPU) for real load. FastHttpUser (geventhttpclient) is ~5–6× faster than the default HttpUser (requests-based) — use it for high RPS.

JMeter (Java, GUI + CLI, protocol breadth)

Mature, huge protocol/plugin ecosystem (JDBC, JMS, LDAP). Verbose XML .jmx.

• Author in GUI, run headless: jmeter -n -t plan.jmx -l results.jtl -e -o report/ (-n non-GUI, -e -o HTML report). GUI mode adds huge overhead — never load-test in it.
• Thread Group = closed model (threads, ramp-up, loops). "Concurrency Thread Group" / "Throughput Shaping Timer" plugins add open-model + arbitrary shapes.
• Distributed: jmeter -n -t plan.jmx -R host1,host2 (remote servers).
• Disable "View Results Tree" listeners under load (memory hog); write JTL and post-process.
• ⚠ JMeter's default aggregate report can suffer coordinated omission; prefer the "Response Times Percentiles" / backend-listener → InfluxDB path and treat timers correctly.

Gatling (Scala/Java/Kotlin DSL, efficient, code-as-test)

Async (Netty), high load per core, versioned code, good HTML reports. injectOpen/injectClosed make the model explicit.

setUp(
  scn.injectOpen(                       // open model: arrivals independent of responses
    rampUsersPerSec(10).to(200).during(5.minutes),
    constantUsersPerSec(200).during(10.minutes)
  )
).protocols(http.baseUrl("https://api.example.com"))
 .assertions(global.responseTime.percentile4.lt(1000)) // percentile4 = p99 by default

artillery (YAML, quick, Node)

Fast to author; good for smoke/CI and moderate load. ⚠ Node = effectively single-core → limited generator throughput; distribute (AWS Lambda/Fargate mode) or switch to k6/Gatling for heavy load.

config:
  target: "https://api.example.com"
  phases:
    - { duration: 60, arrivalRate: 5, name: warmup }      # open model
    - { duration: 300, arrivalRate: 20, rampTo: 50, name: ramp }
  ensure: { p95: 500, maxErrorRate: 1 }                    # fails CI on breach
scenarios:
  - flow:
      - post: { url: /auth/login, json: { u: "{{ $randomString() }}" }, capture: { json: "$.token", as: token } }
      - think: 2
      - get: { url: /api/products, headers: { Authorization: "Bearer {{ token }}" } }

HTTP micro-benchmark one-liners

Fast single-endpoint checks — not user journeys:

wrk2 -t4 -c400 -R2000 --latency http://host/ep   # PREFER: constant 2000 RPS, corrects coordinated omission
wrk  -t12 -c400 -d30s --latency http://host/ep    # high throughput but closed-loop → CO at saturation
oha  -z 30s -c 100 http://host/ep                 # Rust, live TUI, HdrHistogram
hey  -n 100000 -c 100 http://host/ep              # simple; single generator core-bound

⚠ ab (ApacheBench) is single-threaded, no keep-alive by default (-k to enable), and severely subject to coordinated omission — avoid for anything but a trivial reachability check.

API Scenario Patterns (k6)

Mixed read/write with per-scenario thresholds — model real traffic ratios:

export const options = {
  scenarios: {
    reads:  { executor: "constant-arrival-rate", rate: 700, timeUnit: "1s",
              preAllocatedVUs: 50, maxVUs: 200, exec: "read" },
    writes: { executor: "constant-arrival-rate", rate: 300, timeUnit: "1s",
              preAllocatedVUs: 30, maxVUs: 100, exec: "write" },
  },
  thresholds: {
    "http_req_duration{scenario:reads}":  ["p(95)<200"],
    "http_req_duration{scenario:writes}": ["p(95)<500"],
    http_req_failed: ["rate<0.01"],
  },
};
export function read()  { /* GET with Zipfian id, not id=1 */ }
export function write() { /* POST unique payload */ }

GraphQL: POST the query, and assert on the body — a GraphQL error returns HTTP 200, so http_req_failed won't catch it:

const res = http.post(url, JSON.stringify({ query, variables }), { headers });
check(res, { "no gql errors": (r) => !r.json("errors") }); // 200 ≠ success here

WebSocket: import ws from "k6/ws"; open a socket, subscribe, assert on messages, socket.setTimeout(() => socket.close(), 60000); check res.status === 101. For long-lived connections, concurrency ≠ RPS — measure connections held + message latency.

Database Performance

Single benchmark harness (percentiles, warm-up, sorted) — reuse for any async op, don't re-implement per query:

import { performance } from "perf_hooks";

async function bench(name: string, fn: () => Promise<unknown>, iterations = 100) {
  for (let i = 0; i < 10; i++) await fn(); // WARM-UP: JIT, pool fill, buffer cache (discard)
  const t: number[] = [];
  for (let i = 0; i < iterations; i++) {
    const s = performance.now();
    await fn();
    t.push(performance.now() - s);
  }
  t.sort((a, b) => a - b);
  const at = (p: number) => t[Math.min(t.length - 1, Math.floor(t.length * p))];
  return { name, min: t[0], p50: at(0.5), p95: at(0.95), p99: at(0.99), max: t.at(-1) };
}
// Usage: await bench("findById", () => db.users.findById(zipfianId()));

⚠ Serial-loop benchmarks measure single-connection latency, not concurrent throughput — they miss lock contention, pool exhaustion, and buffer thrash. For throughput, drive N concurrent workers for a fixed duration and report QPS + p95 together (increase N until QPS plateaus = pool/DB knee).

EXPLAIN / slow-query hunting (Postgres):

EXPLAIN (ANALYZE, BUFFERS) SELECT ...;   -- ANALYZE = real timings; BUFFERS = cache hits vs disk reads

-- pg_stat_statements: rank by TOTAL time (calls × mean) — the real load driver, not slowest single query
SELECT query, calls, mean_exec_time, total_exec_time, rows
FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 20;

-- Seq scans that should be index scans (large seq_tup_read on hot tables)
SELECT relname, seq_scan, seq_tup_read, idx_scan
FROM pg_stat_user_tables WHERE seq_scan > idx_scan ORDER BY seq_tup_read DESC;

• Read EXPLAIN bottom-up; watch for Seq Scan on big tables, row-estimate vs actual rows divergence (stale stats → ANALYZE), and Nested Loop over large sets (want Hash/Merge join).
• N+1 detection: normalize SQL (strip literals) and count executions of each shape per request; >~10 identical shapes = N+1 → batch/eager-load. ORMs hide these behind lazy relations.
• Benchmark with production-scale rows and realistic key distribution — a query that's instant on 10k rows table-scans on 10M.

Profiling

Profile after a load test localizes the hot endpoint — don't profile blind.

| Target | Tool / command | Reads as | | ------------- | ------------------------------------------------------- | --------------------------------- | | Node CPU | node --prof app.js → --prof-process; or --cpu-prof (.cpuprofile → Chrome DevTools) | flame graph of self time | | Node heap | v8.writeHeapSnapshot(); compare 2 snapshots for growth | retained objects → leak source | | Node loop lag | measure delay of setImmediate; >100ms = blocked loop | sync work starving the event loop | | Python CPU | python -m cProfile -o out.prof; view with snakeviz; py-spy top --pid (no code change, prod-safe) | cumulative time | | Rust | cargo flamegraph; perf record/perf report | flame graph | | JVM | async-profiler (-agentpath), JFR | flame graph, alloc profiling | | Any/prod | perf, bpftrace, continuous profiling (Pyroscope/Parca) | on-CPU/off-CPU |

• On-CPU vs off-CPU: high latency + low CPU = you're blocked on I/O/locks; profile off-CPU (wall-clock), not just on-CPU, or you'll stare at an idle flame graph.
• Memory leak signal: RSS/heap grows monotonically across a soak and never returns after load stops. Two heap snapshots under steady load → diff retained set. GC frequency climbing is an early tell.

Metrics, Budgets, Bottlenecks

Target metrics

| Metric | Meaning | Typical target | Alert | | ------------- | -------------------- | -------------- | ------ | | Latency p50 | median | <100ms | >200ms | | Latency p95 | tail | <500ms | >1s | | Latency p99 | far tail (fan-out!) | <1s | >2s | | Throughput | req/s at the knee | per SLO | — | | Error rate | failed % | <0.1% | >1% | | Saturation | resource util | <70% | >85% |

Define budgets per endpoint (not global) and enforce in CI — k6 thresholds, artillery ensure, Gatling assertions, Lighthouse CI assert all fail the build on breach. Web budgets: LCP <2.5s, INP <200ms, CLS <0.1, JS bundle <300KB.

Instrument the server with a Histogram (Prometheus histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))), not a gauge/summary you can't aggregate across instances. Pick buckets around your SLO (e.g. [0.01,0.05,0.1,0.25,0.5,1,2.5,5]) — percentiles are only as precise as bucket edges.

Common bottlenecks → fix

| Bottleneck | Symptom | Diagnose | Fix | | ------------------ | ------------------------------ | ---------------------- | -------------------------------- | | N+1 queries | latency ∝ result count | query count/request | batch / eager load | | Missing index | seq scan on big table | EXPLAIN ANALYZE | add index; check selectivity | | Connection pool | timeouts at load, low CPU | pool saturation metric | size pool (≈ cores×2..4 for DB), queue | | Sync I/O on loop | high event-loop lag | loop-lag probe | async / worker threads | | Memory leak | heap grows over soak | heap snapshot diff | fix retention (caches, listeners) | | Chatty serialization | high CPU in JSON | CPU profile | stream / faster codec / smaller payload | | Large payloads | network-bound, high TTFB | response-size metric | pagination, compression, fields | | Lock contention | throughput plateaus, cores idle | mutex/off-CPU profile | shard locks, reduce critical section |

Realistic Load Modeling

• Weighted user mix, not one journey: e.g. 60% browsers, 30% shoppers, 10% power users; pick per-VU and follow a session (login → browse N pages → maybe cart → maybe checkout) with think time between steps.
• Key distribution: draw IDs Zipfian/Pareto (hot keys + long tail), never constant, never pure-uniform over a tiny range.
• Randomized think time (jittered), not fixed — fixed sleeps synchronize VUs into artificial request waves.
• Include error/edge scenarios in the mix (invalid input→400, not-found→404, large payloads, timeouts) so you measure error-path cost, which is often worse than the happy path.
• Data generation: @faker-js/faker / Python faker; apply 80/20 (20% of users generate 80% of activity) so cache/index behavior matches production.

Best Practices (condensed)

• Environment: production-like hardware, data volume, and config (cache/CDN/LB). Generator on a separate host from the SUT.
• Baseline first: record current p50/p95/p99 + throughput before optimizing; track over time to catch regressions. Nightly full loads, smoke on every commit.
• Change one variable at a time, measure before/after with the same script, keep results in version control.
• Correlate load-gen output with server-side APM/Prometheus every run — divergence = you're measuring the generator or the network, not the service.
• Analyze percentiles + max, over the steady-state window only (drop warm-up).
• Load tests complement, never replace, production monitoring (real user cardinality/geography).

Verification checklists

Before trusting a load-test result:

• [ ] Generator ran on a separate host; generator CPU/net/FDs were not the limit (server-side metrics corroborate)
• [ ] Open model (arrival rate) used for any breaking-point/capacity claim; VU headroom sufficient (no dropped-iteration warnings)
• [ ] Coordinated omission ruled out (open executor or wrk2/HdrHistogram); tail isn't suspiciously flat
• [ ] Warm-up window discarded; steady-state window is what's reported
• [ ] Reported p50/p95/p99/max, not just mean; percentiles merged from raw histograms, not averaged
• [ ] Realistic key distribution (not id=1) and dataset > cache; payload sizes production-like
• [ ] Keep-alive setting matches the real client; think time randomized
• [ ] The knee (max sustainable throughput at acceptable latency) identified, not just the collapse point
• [ ] Little's Law sanity check passes (achieved RPS ≈ concurrency / (latency+think))

Before shipping a perf test into CI:

• [ ] Thresholds/assertions defined per endpoint and fail the build on breach
• [ ] Smoke variant (1–5 VUs) runs on every commit; full load nightly/on-PR
• [ ] Dynamic URLs tagged with static names (no metric-cardinality blowup)
• [ ] Secrets/tokens injected via env, not hardcoded; test data cleaned up in teardown

Before calling a bottleneck fixed:

• [ ] Root cause identified via profile/EXPLAIN, not guessed
• [ ] Re-ran the same script; before/after deltas on p95/p99 + throughput
• [ ] Fix didn't just move the bottleneck (next resource now saturates at a higher load)
• [ ] Regression guard in CI (budget threshold) so it can't silently return