You can have a benchmark that reports a lightning-fast median execution time, but your users are still complaining about intermittent latency spikes. When you look at the numbers, everything seems fine - until you realize you've only been looking at half the story.

In .NET, time is not the only resource that matters. Every byte you allocate is a debt that must eventually be paid by the Garbage Collector (GC). If your code is "fast" but allocates aggressively, you aren't actually measuring the cost of your algorithm; you're just deferring the cost to a future GC pause that will eventually stop the world and spike your p99 latency.

The hidden cost of allocations

NBenchmark treats allocation measurement as a first-class citizen. By default, every run captures the number of bytes allocated per operation. This appears in your results as the Alloc/op column.

This number represents the mean bytes allocated per single operation. Because NBenchmark captures these counters outside the timed window, allocation tracking is lightweight enough to leave on by default. If you have a specific reason to disable it - such as reducing overhead in an extremely tight loop - you can use the --no-allocations flag or the corresponding configuration setting.

While the mean is the primary metric, the reality of the heap is often found in the extremes. If you enable Advanced detail in your report, you'll see allocation percentiles:

Median: The typical allocation cost.
P95: The cost for the slowest 5% of operations.
Max: The worst-case allocation spike.

If your Median is 32 bytes but your Max is 4,000 bytes, you've found a "hidden" allocation path - perhaps a rare boxing event or a cache miss - that will cause exactly the kind of latency spikes your users are feeling.

Natural vs. Scrubbed heaps

The most critical decision you make when measuring allocations is not whether to measure them, but how the heap is managed between samples. This is controlled by GcBehavior.

GcBehavior.Natural

GcBehavior.Natural is the default. In this mode, NBenchmark leaves the warmup heap in place and only performs collections between different benchmarks.

This is the honest answer for most latency work. Because the heap is not scrubbed before every sample, a real GC pause can land inside your timed window. This will show up as a spike in your MaxNs and increase your variance. It's noisy, but it's an accurate reflection of how your code behaves in a production environment where the GC runs non-deterministically.

GcBehavior.PerSampleCollect

If you need to isolate the cost of your code from the noise of the heap, you can use GcBehavior.PerSampleCollect.

In this mode, the engine forces a Generation 0 collection before every single sample and performs a full Generation 2 collection immediately after warmup. This effectively gives every sample a "clean" heap. The result is significantly lower variance and a much tighter confidence interval.

The trade-off is that you are now measuring a machine your users do not have. By scrubbing the heap, you are removing the very GC pauses that define the real-world performance of your application. Use this mode when you want to compare the raw efficiency of two algorithms without the interference of a fragmented heap.

You can set this behavior globally via the --gc CLI flag or programmatically using WithGcBehavior. For more granular control, you can use overrides like ForceGcBeforeEachSample or ForceGcBetweenBenchmarks. Note that ForceGcBetweenBenchmarks is enabled by default under both behaviors, because allowing one benchmark to contaminate the heap of the next is never acceptable.

The K-batch interaction

There is one important interaction to watch for when using PerSampleCollect.

As we saw in the previous post, NBenchmark often batches multiple calls into a single sample (using a factor called K) to beat the system clock's resolution. If your body allocates memory, those allocations accumulate across the entire batch.

Under PerSampleCollect, if a batch is large enough, it can trigger a GC collection inside the timed window of a single sample. When this happens, the engine will surface a warning. The fix is to force the batch size to one using --ops-per-sample 1, ensuring that each timed window contains exactly one operation and its associated allocation cost.

Corroborating with diagnostics

To get the full picture, you can enable runtime diagnostics using WithDiagnostics. This adds a block of metadata to your results that reveals what was actually happening under the hood:

Collection Counts: Exactly how many Gen0, Gen1, and Gen2 collections occurred.
Heap State: The size of the managed heap at the end of the run.
CPU Time: The actual processor time spent executing, which helps distinguish between "waiting" and "working."

using NBenchmark;
using NBenchmark.Reporters.Console;

public class SerializerBenchmark
{
    // Compare two different serialization strategies
    [Benchmark]
    public byte[] JsonSerialize() => System.Text.Json.JsonSerializer.SerializeToUtf8Bytes(new { Id = 1, Name = "Test" });

    [Benchmark]
    public byte[] MsgPackSerialize() => MessagePack.MessagePackSerializer.Serialize(new { Id = 1, Name = "Test" });

    public static void Main(string[] args)
    {
        var suite = new BenchmarkSuite()
            .Add(new SerializerBenchmark())
            .WithDiagnostics()
            .WithReporter(new ConsoleReporter());

        suite.Run();
    }
}

Example Output (Natural):

Benchmark: JsonSerialize
  Median: 450 ns
  Mean:   462 ns
  Error:  ± 12 ns
  Alloc:  128 B/op
  
  Diagnostics:
    Gen0 Collections: 12
    Gen1 Collections: 0
    Gen2 Collections: 0
    Total CPU Time: 1.2 ms

Example Output (PerSampleCollect):

Benchmark: JsonSerialize
  Median: 410 ns
  Mean:   412 ns
  Error:  ± 2 ns
  Alloc:  128 B/op
  
  Diagnostics:
    Gen0 Collections: 100
    Gen1 Collections: 0
    Gen2 Collections: 0
    Total CPU Time: 1.1 ms

In this example, switching to PerSampleCollect reduced the mean and tightened the error margin significantly, but it also increased the number of collections by an order of magnitude. The Alloc/op remains the same, but the execution noise is gone.

From measurement to enforcement

Measuring allocations is the first step; enforcing a budget is the second. In a professional CI pipeline, you shouldn't just monitor allocations - you should gate them.

If a refactor increases the bytes per operation from 128 to 1024, it might not slow down the median execution time today, but it will increase the frequency of GC pauses in production. By setting an allocation threshold in your performance tests, you can fail the build the moment your code becomes "trashy," long before it hits a user's machine. We'll cover how to set up these gates in post 16.

Go deeper: Now that you know how to measure what's happening, let's look at how to get those results out of your terminal and into a format that your team can actually use.

Output that travels →

Counting what the GC sees