In a performance review, a common pattern emerges: a developer presents a table showing a new implementation with a median of 420ns and the old one at 450ns. They conclude that the new version is 7% faster.

But a lower number is not evidence. In the world of high-resolution timing, 30 nanoseconds can be the result of a slightly different CPU frequency state or a lucky alignment of OS interrupts. The question isn't "Which number is smaller?" but "Is this difference real, and if so, does it actually matter?"

NBenchmark answers these two distinct questions using the Sig and Magnitude columns.

Is the difference real? The Sig column

The Sig (Significance) column tells you if the observed difference between two implementations is statistically significant - meaning it is unlikely to have occurred by chance.

How it works: Non-parametric testing

Most naive tools assume that timing data follows a "Normal" (bell curve) distribution. This is almost always false for software benchmarks, which are typically right-skewed by occasional spikes.

To avoid the errors that come with assuming a bell curve, NBenchmark uses non-parametric tests. These tests look at the ranks of the samples rather than their raw values, making them robust against outliers.

Two-group comparisons: NBenchmark uses the Mann-Whitney U test. It compares the distributions of the baseline and the candidate to see if one is stochastically smaller than the other.
Three or more groups: When you have multiple candidates, the engine first runs a Kruskal-Wallis omnibus test to see if any of the groups differ. If that test passes, it performs post-hoc pairwise Mann-Whitney U tests using the Holm-Bonferroni correction to prevent "p-hacking" (the increased likelihood of finding a false positive when running many tests).

Reading the symbols

The Sig column uses three primary states:

✓ (Checkmark): The difference is statistically significant. You can be confident the change is real.
✗ (Cross): The difference is not significant. Any gap in the medians is likely just noise.
Blank: No comparison was possible (e.g., the method is the baseline).
NotTested: Occurs when a group has fewer than two samples, making a distribution test impossible.

Tuning the evidence

The threshold for significance is the significance level ($\alpha$). By default, NBenchmark uses $0.05$. This means there is a 5% chance that a "significant" result is actually a false positive.

If you are making a high-stakes architectural decision, you may want stricter evidence. You can lower this threshold using --significance-level 0.01 or WithSignificanceLevel(0.01).

Does it matter? Magnitude and the practical-effect gate

A difference can be "significant" without being "important." If you have enough samples, a test can detect a 0.1% difference as statistically real, but in the real world, a 0.1% gain is negligible.

This is where Magnitude comes in. Magnitude uses effect size to classify the difference as Negligible, Small, Medium, or Large.

To prevent "statistically significant but practically useless" results from cluttering your report, you can use the WithMinimumPracticalEffect gate. When this is set, NBenchmark will not mark a result with a ✓ unless the effect size also exceeds your specified threshold.

Significance is not importance

The most critical lesson in reading a benchmark table is to read Sig next to Ratio.

Scenario	Sig	Ratio	Conclusion
The Real Gain	✓	0.70	Real and substantial. Deploy it.
The Noise	✗	0.90	The 10% gain is likely a fluke. Ignore it.
The Triviality	✓	0.99	The 1% gain is real, but it's a waste of a PR.

Scope and Constraints

Where the comparison happens

By default, significance is calculated within a specific scope:

Suite mode: All candidates in the suite are compared against the baseline.
Harness mode: Benchmarks within the same class are compared.

If your implementations live in separate classes, you can enable --cross-class or WithCrossClassSignificance to allow the engine to compare them.

The Runtime Profile Rule

There is one hard constraint: results measured under different runtime profiles are never compared.

If you run one benchmark with RuntimeProfile.SteadyState and another with RuntimeProfile.Production, the engine will refuse to calculate a Ratio or Sig. This is because the underlying environment (JIT tiering, GC behavior) is fundamentally different; any difference in timing would be a product of the environment, not the code.

Example: The Refactor Trap

Imagine you refactored a sorting algorithm. You're seeing a slightly lower median, but you're not sure if it's a real win.

var suite = new BenchmarkSuite()
    .Add("LegacySort", () => LegacySort(data))
    .Add("OptimizedSort", () => OptimizedSort(data))
    .Add("ExperimentalSort", () => ExperimentalSort(data))
    .WithBaseline("LegacySort");

await suite.RunAsync();

Results Table:

Method	Median	Ratio	Sig	Mag
LegacySort	1200ns	1.00	-	-
OptimizedSort	1180ns	0.98	✓	Negligible
ExperimentalSort	1100ns	0.91	✗	Small

The Analysis:

OptimizedSort is a trap. The difference is statistically real (✓), but the magnitude is Negligible. You've spent three days on a refactor that provides no practical value.
ExperimentalSort looks faster (Ratio 0.91), but the Sig is ✗. The difference is not statistically significant. You cannot claim this version is faster; you likely just had a lucky run.

For a deeper dive into how we handle the volatile samples that can confuse these tests, see The tail is a distribution, not a number.

Significant is not the same as important