One run cannot tell you how much it moves

There is a dangerous assumption that most developers make when reading a benchmark report: they assume the confidence interval describes the next run.

If a report says the median is 100ns ± 2ns, the reader assumes that if they run the code again, it will almost certainly land between 98ns and 102ns. But that's not what the interval is telling you. That interval describes the variance within a single run - the noise of the samples in that specific process.

It does not describe the variance between runs.

If you run the same benchmark on a different machine, or even just restart your current machine, you might find a new median of 110ns ± 2ns. Both runs are internally precise, but they disagree by 10%. This is "run-to-run variance," and it is the primary reason why single-run benchmarks are untrustworthy for detecting regressions.

The power of multiple launches

To solve this, NBenchmark introduces the concept of a launch.

A launch is not just a repeat of the sampling loop. It is a complete restart of the measurement cycle: the engine spawns a fresh worker process, performs a fresh warmup, and collects a fresh set of samples.

By performing multiple launches, you measure the variance of the environment itself.

In Harness mode, NBenchmark defaults to five launches. For Single and Suite modes, the default is one, though you can increase it using MeasurementOptions.LaunchCount or the --launch-count CLI flag.

Why five? Because in the world of performance engineering, two or three launches are often not enough to clear the noise of a background OS task. Five launches provide enough data to build a statistically significant distribution of the "median" itself.

Reading the aggregated report

When you perform multiple launches, the report changes. The primary fields - Median, Mean, and Error - are no longer from a single run. They are averages calculated across all launches.

Below the main results, you'll see a Launch Aggregation table. This table shows the median of every single launch. If the numbers are tightly clustered (e.g., 101, 102, 99, 100, 101), you have a stable environment. If they are wild (e.g., 100, 150, 90, 200, 110), you know that your results are being driven by external noise, and no amount of additional sampling within a single run will fix it.

One technical detail: while the median is averaged across launches, the RawSamples and trimmed marks shown in the advanced detail are taken from the launch nearest to the averaged median. This ensures you are looking at the most representative distribution of the run.

Paired ratios: The "Is it actually faster?" test

The most powerful feature of multiple launches is the paired ratio.

When you compare Implementation A to Implementation B in a suite, NBenchmark doesn't just compare the final averaged medians. It calculates the ratio for every single launch.

• Launch 1: A is 1.1x slower than B
• Launch 2: A is 1.05x slower than B
• Launch 3: A is 1.15x slower than B
• ... and so on.

These per-launch ratios are then combined on a log scale to produce a paired-interval ratio. This tells you not just "A is slower," but "Across five different process restarts, A was consistently 1.1x slower than B."

This is the only way to distinguish between a "lucky" run and a real performance difference. If the significance test (Sig) says "Yes" but the ratio interval is wide and overlaps 1.0, you have a result that is statistically significant but practically unstable. If the interval is tight and far from 1.0, you have an honest regression.

Practical constraints

There are two important interactions to keep in mind when configuring launches:

1. The Dry Run: The --dry-run flag performs exactly one launch. It verifies that your benchmarks compile and run without actually spending the time to gather a full distribution.
2. In-Process Limitation: If you use --in-process (which we'll discuss in post 18), you cannot perform multiple launches. Since the measurement happens in the current process, there is no "fresh worker" to spawn. In-process runs are for smoke tests; real regression testing requires isolation and multiple launches.

For those using [Performance] tests in a test suite, the default is one launch to keep the test suite fast. However, raising the launch count on these tests is the best way to turn a "flaky" performance test into a reliable CI gate.

---

Go deeper: Multiple launches tell you if your environment is noisy, but they don't fix the noise. To get the most precise numbers possible, you need to stop the OS from interfering in the first place.

Cutting the noise at its source →