Benchmark Authoring Guidelines

BenchmarkDotNet (BDN) is a .NET library for writing and running microbenchmarks. Throughout this skill, "BDN" refers to BenchmarkDotNet.

Note: Evaluations of LLMs writing BenchmarkDotNet benchmarks have revealed common failure patterns caused by outdated assumptions about BDN's behavior — particularly around runtime comparison, job configuration, and execution defaults that have changed in recent versions. The reference files in this skill contain verified, current information. You MUST read the reference files relevant to the task before writing any code — your training data likely contains outdated or incorrect BDN patterns.

Key concepts

• Job — describes how to run a benchmark: runtime, iteration counts, launch count, run strategy, and environment settings. Multiple jobs can be configured to run the same benchmarks under different conditions.
• Benchmark case — one method × one parameter combination × one job. The atomic unit BDN measures.
• Operation — the logical unit of work being measured. All BDN output columns (Mean, Error, etc.) report time per operation.
• Invocation — a single call to the benchmark method. By default, 1 invocation = 1 operation. With OperationsPerInvoke=N, each invocation counts as N operations.
• Iteration — a timed batch of invocations. BDN measures the total time for all invocations in an iteration, then divides by the total operation count to get per-operation time.

Benchmarks are comparative instruments

A single benchmark number has limited value — it can confirm the order of magnitude of a measurement, but the exact value changes across machines, operating systems, and runtime configurations. Benchmarks produce the most useful information when compared against something. Before writing benchmarks, identify the comparison axis for the current task:

• Approaches (A vs B): comparing alternative implementations side-by-side in the same run.