Performance Testing

Concept of the skill

Performance testing is the discipline of measuring a system's non-functional properties — latency, throughput, error rate, resource utilization, saturation — by running the system under controlled load and observing the resulting metrics. Where functional tests answer "does the system produce the right output?", performance tests answer "does the system produce the right output quickly enough and at sufficient scale, while staying within resource budgets and error tolerances?" The unit of judgment is whether measured metrics meet defined acceptance thresholds — typically Service-Level Objectives (SLOs) expressed as percentiles (e.g., "p95 latency below 200ms at 1,000 RPS sustained for 30 minutes").

Replaces "is it fast enough?" answered by intuition or user complaints with empirical answers measured against an explicit SLO. Solves the problem that performance is multi-dimensional — latency distribution, throughput ceiling, error rate under load, resource utilization, saturation point are each separate properties; each requires its own measurement; each can be acceptable while others fail. Reducing performance to a single number (especially average latency) is the most common discipline failure. Sub-purpose: verify that declared performance budgets (per performance-budgets) hold under realistic load — without performance tests, a budget is an aspirational threshold without empirical evidence; without budgets, a performance test produces measurements without a verdict. A complete pre-launch performance test suite runs all six load shapes because each verifies a different property; an ongoing suite typically runs smoke on every PR, load on every merge, with stress/spike/soak/breakpoint on cadence.

Distinct from performance-engineering, which owns the activity of profiling and optimizing a specific slow path once it has been identified — this skill owns the discipline of exercising the system under controlled load to discover and quantify performance behavior; the two compose (this skill produces measurements that locate bottlenecks and verifies optimizations afterward). Distinct from performance-budgets, which owns the declaration of the threshold-and-consequence contract (metric, threshold, percentile, consequence) as a quality property — this skill owns the test mechanism that exercises the system under load and verifies the budgets hold. Distinct from observability, which owns real-time runtime measurement of the deployed system — this skill is offline controlled measurement; the two compose (one for pre-deploy verification, the other for production runtime validation). Distinct from chaos-engineering (fault injection in deployed systems), microbenchmarks (single-function isolation via language benchmark tools), testing-strategy (level/scope), integration-test-design and e2e-test-design (correctness across seams or user journeys — this skill exercises those same paths under load), and mutation-testing (test-suite quality measurement). Performance testing is to a software system what a load-bearing inspection is to a bridge — you do not certify a bridge by walking across it (functional test) and concluding it works; you drive trucks of known weight across at increasing volumes, with strain gauges on every beam, and verify the deflection stays within spec under expected traffic, that the failure mode is graceful when overloaded (cracks before collapse), that nothing creeps over a long soak. A bridge whose 'average' load it can carry is 50 tonnes but whose p99 stressor reveals harmonic resonance at 80 tonnes is the bridge that fails on a windy day. The wrong mental model is that average latency is a meaningful performance metric. It is not. A system whose mean is 50ms and p99 is 5 seconds has a user-experience problem the mean hides — 1 in 100 users sees a 5-second response time. Acceptance criteria should always be percentiles (or distributions); averages are easily skewed by outliers in both directions. Adjacent misconceptions: that performance tests in a stripped/non-production environment are meaningful (they are not — the test produces a measurement of that environment, not the system; production hardware, network, data volumes, dependency versions, and configuration are the load-bearing investment); that one-time pre-launch testing is enough (it is not — regressions accumulate; the ongoing value is continuous testing in CI); that load tests alone characterize the system (they do not — a complete pre-launch suite runs all six shapes, because each verifies a different property: load verifies SLO at design load, stress verifies graceful failure, spike verifies elasticity, soak verifies no leaks or memory growth, breakpoint quantifies the capacity ceiling); that performance testing replaces observability (it does not — performance testing is offline controlled measurement; observability is online real-traffic measurement; the two compose); that coordinated omission in the load tool doesn't matter (it does — Tene's canonical talk shows naive percentile reporting silently drops slow responses; honest tools account for it); and that a performance test without an SLO is a verification (it is not — without an SLO it is a measurement without a verdict; the SLO is what makes the test a verification rather than an information report).

Coverage

The discipline of measuring non-functional system properties — latency, throughput, error rate, resource utilization, saturation — by running the system under controlled load and verifying against explicit SLO thresholds. Covers the five primitives (load profile, workload, latency metric, throughput metric, SLO target), the six load-shape types (smoke, load, stress, spike, soak, breakpoint) that each test a different property, the latency-percentile discipline (p50/p95/p99/p99.9) that replaces the misleading average, the environment-fidelity requirement that makes results meaningful, the modern tool landscape (k6, JMeter, Locust, Gatling, Vegeta, Wrk), and the distinction from observability (offline controlled vs online real-traffic) and benchmarking (system vs single-function).

Philosophy of the skill

Performance testing is the discipline of adding the time and scale dimensions to verification. Functional tests verify "the system produces the right output"; performance tests verify "the system produces the right output quickly enough and at sufficient scale, within resource budgets and error tolerances." Without performance testing, "is it fast enough" is answered by intuition or by users complaining; with performance testing, it is answered by measurement against an explicit SLO.

The central insight is that performance is multi-dimensional. Latency distribution, throughput ceiling, error rate under load, resource utilization, saturation point — each is a separate property; each requires its own measurement; each can be acceptable while others fail. Reducing performance to a single number (especially average latency) is the most common discipline failure.

The complementary insight is environment fidelity. A performance test in a non-production-like environment produces a measurement of that environment, not the system. Production hardware, network, data volumes, dependency versions, and configuration are the load-bearing investment for performance testing; the load tool itself is increasingly cheap.