If you have used most benchmarking tools, you are likely used to the "iteration count" setting. You tell the tool to run the code 100 or 1,000 times, and it gives you the average.

The problem with fixed counts is that they are a guess. If your code is extremely stable, 1,000 samples is a waste of time. If your code is volatile, 1,000 samples might still leave you with a confidence interval so wide that the result is meaningless.

NBenchmark replaces the fixed count with an adaptive sampling loop. The engine doesn't ask "how many times should I run this?" but rather "how much data do I need before I am confident in the answer?"

The path to a precise number

Every NBenchmark run follows a three-stage lifecycle designed to eliminate bias and maximize precision.

1. Warmup and Plateau Detection

Before a single measurement is recorded, the engine enters the warmup phase. This isn't just about "heating up" the CPU; it is about reaching JIT-quiescence.

The engine monitors the timings of the warmup samples. It looks for a plateau - the point where the timings stop dropping sharply (as the JIT promotes code to Tier 1) and begin to stabilize. Once the timings plateau and stay within a specific tolerance for a set period, the engine declares the system "warm" and moves to calibration.

2. Calibration

Once warm, the engine determines the Ops-per-sample (K). If your operation is too fast for the system clock to measure accurately, the engine batches multiple calls into a single timed sample. This ensures that the measurement window is large enough to overcome timer resolution issues (covered in detail in post 10).

3. The Adaptive Loop

This is the core of the engine. NBenchmark starts collecting samples and, after every batch, recalculates the Confidence Interval (CI).

The loop continues until one of three things happens:

The Target is Met: The half-width of the confidence interval falls below the target precision (e.g., 0.5% of the mean).
The Ceiling is Hit: The number of samples reaches the MaxSamples limit.
The Budget expires: The total time spent sampling exceeds the tuning-time budget.

This means that for a simple, stable function, NBenchmark might stop at 40 samples. For a complex, volatile operation, it might push to 5,000. In both cases, you get the same level of statistical confidence.

Controlling the Loop

While the adaptive loop is the default, you have full control over the measurement process.

Pinning for Reproducibility

If you need to ensure that every run uses the exact same amount of data (for example, when comparing across different machines in a very strict environment), you can pin the counts:

WithSamples(n) or Samples = n (in [Benchmark])
WithWarmupSamples(n) or WarmupSamples = n

Note: Pinning the warmup is a conscious choice. By default, NBenchmark's plateau detection is more reliable than a fixed number because it adapts to the complexity of the method.

Cold-start Measurement

Sometimes, the "lie" you want to measure is the cold start. To measure the very first execution - including the JIT overhead - you can skip warmup entirely:

WithWarmupSamples(0)

The Dry Run

To verify that your benchmark is wired correctly without actually spending minutes on measurements, use the --dry-run flag or set Samples = 0. The engine will perform the discovery, setup, and calibration, but it will skip the measurement loop.

The `autoTune` Diagnostic

If you are curious about how the engine arrived at your result, you can inspect the autoTune diagnostic data (available in JSON reports). This data reveals:

Resolved Samples: The final number of samples collected.
Stop Reason: Whether the loop stopped because the target was met, a ceiling was hit, or the budget ran out.
Achieved CI: The actual precision reached at the moment of stopping.

Example: Adaptive vs Pinned

Consider a method with high variance (e.g., a network-bound call).

Default (Adaptive) Run: The engine sees the wide variance $\rightarrow$ the CI stays wide $\rightarrow$ it keeps sampling $\rightarrow$ it stops at 1,200 samples once the Error is $\pm 0.5%$.

Pinned Run (WithSamples(100)): The engine stops at 100 samples $\rightarrow$ the Error is $\pm 4.2%$.

In the pinned run, you have a number, but you don't have a precise number. The adaptive loop protects you from accidentally trusting a result that hasn't converged.

For a deeper dive into how we handle the resolution of the clock when your code is faster than a nanosecond, see When your code outruns the clock.

Measured until it is precise enough