If you have ever looked at a performance trace and seen a "spike" - a single sample that is 10x slower than the median - your first instinct is probably to call it an outlier and ignore it.

But not all spikes are created equal. Some are "noise" (the OS stealing your CPU core for a millisecond), while others are "signal" (a rare but real path in your code, like a cache miss or a GC trigger). If you blindly discard every spike, you aren't measuring your code - you're measuring a fantasy.

NBenchmark treats samples as a distribution, using a two-stage process to separate the noise from the signal.

Stage one: Evidence-based interference rejection

Before any statistical rules are applied, NBenchmark performs a hard check for OS interference.

When the engine runs a sample, it doesn't just time the body; it monitors the execution environment. If the OS explicitly preempts the worker process - for example, by triggering a context switch or a hard interrupt - the engine knows that the resulting timing is a lie. This isn't a statistical guess; it is evidence.

Samples that are provably interfered with are discarded immediately. This ensures that the data entering the statistical pipeline is as clean as possible. You can toggle this behavior with --no-interference-filter or WithInterferenceFilter, but in 99% of cases, you want it on.

Stage two: The adaptive fence

Even without OS interference, you will still see spikes. These could be thermal throttling, page faults, or internal engine overhead. To handle these, NBenchmark uses a statistical fence.

The IQR Fence (Default)

By default, the engine uses the Interquartile Range (IQR) method. It calculates the spread between the 25th and 75th percentiles and creates a "fence" around the data. Any sample falling outside this fence is marked as an outlier.

The beauty of the IQR fence is that it is adaptive. If your code is naturally volatile, the fence widens. If your code is extremely stable, the fence tightens. This prevents the engine from over-trimming naturally noisy code or under-trimming stable code.

Alternative Outlier Modes

Depending on what you are measuring, the IQR fence might be too aggressive or too lenient. You can change this via WithOutlierMode or --outlier-mode:

RemoveTop5Percent: A blunt instrument that simply chops off the worst 5% of samples.
RemoveTopAndBottom5Percent: Symmetrical trimming for distributions with both low and high spikes.
None: Disables trimming entirely. Use this for tail analysis where every single sample, no matter how slow, is part of the signal.

The bimodal warning: Noise or a second path?

One of the most powerful features of the NBenchmark engine is the bimodal warning.

When the engine trims outliers, it doesn't just throw them away - it analyzes them. If the "discarded" samples aren't just random spikes but actually form their own tight cluster, the engine warns you.

This is a critical signal. It suggests that your code isn't suffering from noise, but from a split execution profile. This often happens when you have a "cold path" and a "warm path" (for example, the first few calls to a method that trigger a lazy initialization). If you see a bimodal warning, stop looking at the median and start investigating why your code has two different performance identities.

Shape statistics and the tail

To understand the "shape" of your performance, NBenchmark provides several advanced metrics in the Advanced detail block:

Skewness: Tells you if your spikes are only on the high end (right-skewed) or if you have unusual "fast" spikes (left-skewed).
Kurtosis: Measures how "fat" the tails are. High kurtosis means you have frequent, extreme outliers.
MAD (Median Absolute Deviation): A robust measure of spread that is less sensitive to outliers than Standard Deviation.
CV (Coefficient of Variation): The ratio of the standard deviation to the mean, providing a normalized measure of volatility.

Example: Reading the Noise

Consider a benchmark for a cache lookup. Most hits are fast, but a rare miss is slow.

var suite = new BenchmarkSuite()
    .Add("CacheLookup", () => cache.Get(key))
    .WithOutlierMode(OutlierMode.None); // We want to see every miss

await suite.RunAsync();

Results:

Median: 40ns
P95: 120ns
P99: 1,500ns
Warnings: [Bimodal distribution detected in tail]

If OutlierMode were set to the default (IQR), those 1,500ns samples would be trimmed and the Median would look great. But by setting it to None and seeing the Bimodal warning, we realize the 1,500ns isn't OS noise - it's the cost of a cache miss. The "noise" was actually the most important part of the measurement.

For a deeper dive into how we ensure these numbers are precise enough to trust, see Measured until it is precise enough.

The tail is a distribution, not a number