In this final post, we're opening the hood. We'll explore the mechanics of the worker protocol and the mathematical corrections the engine applies to turn a raw stopwatch reading into a statistical fact.

The Worker Protocol: Orchestrating Isolation

As we established in post 18, the only way to ensure a clean measurement is to move the work to a separate process. But a separate process is, by definition, a separate memory space. The "Worker Protocol" is the bridge that makes this possible.

Discovery and Addressing

When you call Run(), the harness doesn't just start a process; it establishes a high-speed communication channel. The harness acts as the orchestrator, while the worker acts as the executor. The protocol handles the discovery of the worker's address and the negotiation of the measurement budget.

The Wire Format and State Transfer

To minimize the impact of the communication itself, the protocol uses a compact binary wire format. When the engine moves a prepare block or a [BenchmarkPlan] to the worker, it doesn't just send code; it transfers the captured state.

The protocol enforces a strict SampleReservoir cap. The worker collects thousands of samples, but it only sends the statistically significant subset back to the harness. This prevents the communication channel from becoming a bottleneck and ensures that the harness doesn't run out of memory when analyzing massive suites.

The Clock: Probing for Truth

Every system clock is a liar. Some have high resolution but drift over time; others are stable but "quantized," meaning they only tick in large jumps.

NBenchmark does not assume the clock is perfect. Before a single sample is taken, the engine performs clock probing. It runs a series of empty loops to determine the minimum detectable increment of the system timer.

If the probe reveals that the clock is too coarse for the operation being measured, the engine automatically switches strategies. This is where the "Ops-per-Sample" (K) calibration from post 10 comes from. The engine calculates the exact K-factor needed to push the measurement window above the clock's resolution floor, ensuring that the resulting number is a measurement, not a guess.

Jitter Calibration and the Detector Auto-Switch

Beyond resolution is the problem of jitter - the tiny, high-frequency fluctuations in timing that happen even on a perfectly stable machine.

The engine uses a jitter calibration phase to establish the "noise floor" of the current environment. It then employs a detector auto-switch. Depending on the distribution of the jitter, the engine chooses between different outlier detection algorithms.

If the jitter is Gaussian (normal), the engine uses standard deviation-based trimming. If the jitter is skewed or heavy-tailed, it switches to the IQR (Interquartile Range) fence. This ensures that the engine isn't accidentally trimming real performance data or leaving in systemic noise.

The Host Drift Canary

Even a perfectly calibrated run can be ruined by a "drift event" - a sudden change in the host's environment, such as a CPU frequency shift or a background system task.

To detect this, NBenchmark runs a drift canary. This is a hidden, ultra-stable benchmark that runs periodically alongside your actual code. The canary's timing is known and constant. If the canary's result shifts beyond the --drift-tolerance, the engine knows that the host is no longer stable.

Rather than reporting a corrupted number, the engine triggers a restart. It flushes the current samples and starts the run over. It is better to take an extra ten seconds to restart than to report a "confident" number that was skewed by a background process.

Optional Stopping and the Unbiasing Correction

One of the most subtle challenges in statistics is the "Optional Stopping" problem.

In NBenchmark, we sample adaptively: we keep taking samples until the confidence interval is tight enough to stop. However, in statistics, stopping a test precisely when it hits a target can introduce a subtle bias. You are more likely to stop a run when a "lucky" string of fast samples narrows the interval, potentially underestimating the true mean.

To solve this, NBenchmark applies an unbiasing correction. It uses a mathematical adjustment based on the total number of samples and the achieved interval width to remove this bias. This ensures that the reported mean is an unbiased estimate of the population, not just the result of a lucky stopping point.

Validation: Pinned against the Giants

Math is only useful if it's correct. To ensure that NBenchmark's statistical implementations - from the Mann-Whitney U test to the IQR fence - are accurate, they are not developed in a vacuum.

Every core numerical function in the engine is pinned against SciPy 1.17.1 and NumPy 2.4.6. We run the same raw sample sets through both NBenchmark and the industry-standard Python libraries. If the results diverge by more than a specified tolerance (typically $10^{-9}$), the build fails.

When you see a Sig checkmark or a Median in an NBenchmark report, you aren't trusting a custom implementation - you are trusting math that has been verified against the same libraries used by data scientists worldwide.

The Final Word

Benchmarking is a battle against noise. From the process boundary and the clock probe to the drift canary and the unbiasing correction, every feature in NBenchmark exists to remove one specific way that a number can lie to you.

By combining rigorous isolation with verified statistics, NBenchmark turns the "vibe" of performance into an engineering discipline. You no longer have to guess if your code is fast; you can prove it.

Go deeper: You've reached the end of the series. If you want to dive even deeper into the internals, the full technical specifications for the worker protocol and the measurement engine are available in the Deep Dives section of the docs.

Maybe you need to stream your benchmark results into a corporate Grafana dashboard. Maybe you're working with specialized hardware that produces a very specific type of timing outlier that the IQR fence doesn't catch. Or maybe you want to correlate your benchmark samples with BCL instrumentation to see exactly which internal method is stealing your cycles.

NBenchmark is designed as a platform. Through a set of extension points, you can customize how results are reported, how outliers are detected, and how samples are observed in real-time.

Custom Reporters

If the standard formats don't fit your needs, you can implement your own IReporter. A custom reporter is responsible for taking the BenchmarkResult and turning it into your desired output.

The key to building a great reporter is the BenchmarkTable.Build utility. You don't need to re-derive the complex logic for ratios and significance tests; BenchmarkTable does that for you. Your reporter simply decides how to render that table.

To make your reporter discoverable via the --reporter CLI flag, you register it with the ReporterRegistry using a [ModuleInitializer].

public class MyCorporateReporter : IReporter
{
    public void Report(BenchmarkResult result, ReportDetailLevel detail)
    {
        var table = BenchmarkTable.Build(result);
        // Send table to corporate API...
    }
}

public static class ReporterExtensions
{
    [ModuleInitializer]
    public static void Initialize()
    {
        ReporterRegistry.Register("corp", new MyCorporateReporter());
    }
}

Specialized Outlier Detection

The default IQR (Interquartile Range) fence is a great general-purpose tool, but it isn't perfect. If you're dealing with bimodal distributions or hardware-specific jitters, you can implement a custom IOutlierDetector.

By using WithOutlierDetector, you can define a rule that examines the entire sample set and decides exactly which timings are "noise" and which are "signal." This is essential for tail-latency analysis, where a single unexpected spike might be a critical bug rather than an OS interrupt.

Real-time Observability with Observers

Sometimes you need to know what's happening during the run, not just after it. IMeasurementObserver allows you to hook into the engine's lifecycle.

Observers provide events for:

• OnPhase: When the engine moves from warmup to sampling.
• OnSample: Every time a new K-batch is completed.
• OnDetector: When the outlier detector trims a sample.
• OnResult: When the final statistics are calculated.

There is one strict rule for observers: they must be invisible. They should return immediately, never allocate on the hot path, and never throw. If an observer adds 10ns of overhead to a 50ns operation, it has just invalidated the benchmark.

BCL Instrumentation and OpenTelemetry

The most powerful way to observe a benchmark is to use the BCL's own instrumentation. NBenchmark provides a set of nbenchmark.* instruments and a span hierarchy that allow you to track execution without adding manual timing code to your body.

For those who need enterprise-grade observability, the NBenchmark.Exporters.OpenTelemetry package allows you to stream every single sample as an OTLP signal.

By using WithOpenTelemetry, your benchmarks stop being static reports and start being live telemetry. You can see your samples move in real-time in Jaeger or Honeycomb, allowing you to correlate a timing spike with a specific internal event or resource contention.

Go deeper: We've covered the extensions, the hardware, and the runtimes. To close the series, we're going to open the engine itself and look at the math and the protocols that make an honest number possible.

The engine, opened up →

For most teams, the answer to this question is a guess based on a blog post or a release note. But for a performance-critical application, "it should be faster" is not a technical justification for an upgrade. You need an empirical answer.

Multi-runtime support allows you to run the exact same benchmark suite across multiple target frameworks - such as .net8.0, .net9.0, and .net10.0 - and compare the results side-by-side.

Setting up a multi-runtime project

To compare runtimes, you first need a project that can target all of them. In your .csproj file, replace the single <TargetFramework> tag with <TargetFrameworks>, listing the versions you want to compare:

<PropertyGroup>
  <TargetFrameworks>net8.0;net9.0;net10.0</TargetFrameworks>
</PropertyGroup>

Once your project is multi-targeted, NBenchmark can orchestrate the execution across the different installed runtimes on your machine.

Configuring the runtime sweep

Depending on your usage mode, there are different ways to trigger a runtime comparison.

In Harness mode

The simplest way is via the CLI. You can specify the runtimes you want to target using the --runtimes flag:

dotnet benchmark --runtimes net8,net9,net10

Alternatively, you can use the [Runtimes] attribute on your benchmark class to pin the target versions directly in the code.

In Suite mode

When building a suite programmatically, you use the WithRuntimes(...) method. However, because each runtime is its own process by definition, you cannot use a simple inline suite. Instead, you must use a [BenchmarkPlan] factory and RunPlanAsync.

This ensures that the engine can spawn a worker process for each specified runtime, apply the correct RuntimeProfile, and collect the results in a unified report.

Reading the cross-runtime report

The output of a multi-runtime run is grouped by runtime. You will see a separate results table for .net8.0, .net9.0, and .net10.0.

The most important rule of cross-runtime reporting is this: significance and ratios are only calculated within a single runtime.

You will see a Sig column for the comparisons within .net9.0, but you will never see a significance test that spans from .net8.0 to .net9.0. Why? Because the baseline shifted. The hardware might be the same, but the JIT compiler, the GC implementation, and the BCL internals have changed. Comparing a median from one runtime to another is like comparing temperatures in Celsius and Fahrenheit - the numbers are different, but they are measuring different things.

To make a decision, you look at the Relative Magnitude. If Implementation A is 20% faster than B in .net8.0, but only 5% faster in .net10.0, the new runtime has effectively erased your optimization.

A data-driven upgrade workflow

The most effective way to use multi-runtime support is as a three-step workflow:

1. Pin the Suite: Select a representative set of benchmarks that cover your most critical hot paths.
2. Sweep Runtimes: Run the suite across your current version and the target upgrade version.
3. Analyze the Delta: Use the Sig and Magnitude columns to identify where the new runtime provides a win and where it introduces a regression.

This turns the upgrade conversation from "should we?" to "here is exactly how much faster we will be."

Go deeper: Most of the time, the built-in reporters and statistical tests are enough. But what if you need to stream your results to OpenTelemetry or implement a custom outlier detector for a specific hardware glitch?

When the built-ins are not enough →

Imagine you have two benchmarks: one that populates a static cache and another that reads from it. If you run them in that order, the second benchmark will report a blazing fast execution time. If you run them in reverse, it will be significantly slower.

This is state contamination. When one benchmark alters the environment in a way that benefits the next, your results are no longer independent. Your significance tests, which rely on the assumption that every sample is an independent event, are now lying to you. They will report "statistically significant" differences that are actually just the side effects of a shared cache.

Understanding instance lifetimes

NBenchmark gives you precise control over how your benchmark classes are instantiated. This allows you to balance the cost of construction against the need for isolation.

Per-Method (The Default)

By default, NBenchmark creates a fresh instance of your benchmark class for every single method. If you have three [Benchmark] methods in a class, the engine instantiates the class three times. This is the safest approach, as it ensures that instance fields are reset between benchmarks.

Per-Class

When construction is expensive - for example, when you need to establish a database connection or load a 1GB dataset into memory - you can use [InstanceLifetime(InstanceLifetime.PerClass)].

This tells the engine to reuse the same instance for every benchmark method in the class. This amortizes the construction cost, but it opens the door to contamination. If Method A modifies a private field that Method B relies on, you have introduced a hidden dependency.

Remedies for shared state

If you must use PerClass for performance reasons, you need a way to scrub the state between runs.

IStateReset

The most robust solution is the IStateReset interface. By implementing ResetAsync, you can define exactly how to return your class to a clean state.

public class DatabaseBenchmark : IStateReset
{
    private readonly MyDbContext _context = new();

    [Benchmark]
    public void QueryData() => _context.Users.ToList();

    public async Task ResetAsync()
    {
        // Clear the cache or truncate tables between benchmarks
        await _context.Database.EnsureDeletedAsync();
        await _context.Database.EnsureCreatedAsync();
    }
}

NBenchmark calls ResetAsync between benchmarks, immediately after the inter-benchmark Garbage Collection. This ensures that each method starts with a predictable state without the overhead of full class re-instantiation.

Explicit Declarations

If the carry-over is deliberate - for example, if you are specifically measuring how a cache performs over time - you can mark the state with [SharedState]. This tells the engine (and other developers) that the contamination is an intended part of the measurement.

Integrating Dependency Injection

In real-world applications, your code doesn't just use new(); it uses a dependency injection (DI) container. NBenchmark supports this via the NBenchmark.DependencyInjection package.

Instead of a parameterless constructor, you can use WithServices or WithScopedServices to provide a IServiceProvider to the worker process. This allows your benchmarks to resolve real services, configuration, and logging exactly as they would in production.

var services = new ServiceCollection()
    .AddSingleton<ICache, RedisCache>()
    .BuildServiceProvider();

var suite = new BenchmarkSuite()
    .Add(new CacheBenchmark())
    .WithServices(services);

The ASP.NET and EF Pitfall

A common mistake when combining DI and PerClass lifetimes is the "Scoped Service Trap."

If you resolve a scoped service (like an Entity Framework DbContext) in the constructor of a PerClass benchmark, that context lives for the entire duration of the suite. In a real application, a DbContext is short-lived. By keeping it alive across multiple benchmarks, you are measuring a bloated, long-lived context that behaves differently than a production request.

NBenchmark's analyzers catch this at compile time. NB0011 warns you when a scoped service is used under a PerClass lifetime, and NB0013 flags mutable instance fields that could lead to contamination. These warnings are your first line of defense against "confident nonsense."

By managing your lifetimes and scrubbing your state, you ensure that your results are a product of your code's efficiency, not the order in which you wrote your methods.

Go deeper: Now that we've secured the state within a single runtime, let's look at the bigger picture. What happens when you need to know if a new version of the .NET runtime itself provides a performance win?

One codebase, three runtimes →

In .NET, the most important decisions that shape your performance aren't made while your code is running - they are made while the process is starting. JIT tiering, Profile-Guided Optimization (PGO), ReadyToRun (R2R) state, and the Garbage Collection flavor are all decided at the process boundary. If you measure your code in the same process that launched the benchmark, you aren't measuring your code; you're measuring the baggage of the harness.

The 21x Demonstration

To see why this matters, consider a simple, tight loop. When measured in a dedicated, isolated worker process, the results are stable and fast: 320ns, 322ns, 329ns.

Now, run the same code using the --in-process flag. The results shift dramatically: 7,009ns, 6,733ns... and then, on the third attempt, 329ns.

What happened? In the first two attempts, the JIT compiler was still in "Tier 0" (the fast-to-compile, slow-to-run version). By the third attempt, the JIT finally promoted the method to "Tier 1" (optimized). Because the confidence interval stayed tight during the slow runs, a naive benchmark would tell you that the code is 21 times slower than it actually is, and it would do so with absolute confidence.

By spawning a fresh worker process for every measurement, NBenchmark ensures that JIT state and heap contamination from previous runs cannot leak into your current sample.

Runtime Profiles: Tuning the Machine

Isolation isn't just about cleaning the slate; it's about choosing the right slate. NBenchmark uses RuntimeProfile to control exactly how the worker process is configured.

SteadyState (The Default)

The SteadyState profile is designed for pure throughput measurement. It turns off JIT tiering, PGO, and R2R, and disables background GC. This removes the non-deterministic "noise" of the runtime, giving you a number that represents the fully-optimized steady state of your algorithm.

Production

The Production profile reproduces a shipping configuration. It leaves tiering and background GC on. This is deliberately more imprecise, but it tells you how the code will actually behave in a live environment.

ServerGc

This profile forces the worker to start with Server GC enabled, regardless of the host's settings. This is critical for server-side libraries where the GC behavior differs fundamentally from a client-side console app.

Host

The Host profile inherits whatever settings the harness process is currently using.

Because these knobs cannot be applied to a process that is already running, an in-process run silently gets none of them. This is why results from different runtime profiles are never placed in the same comparison group; a "SteadyState" number and a "Production" number are fundamentally different measurements.

Crossing the Wire

Since the code runs in a separate process, NBenchmark has to move data across a process boundary.

When you use a prepare block in Benchmark.Run, that code executes inside the worker process. This is why the prepare/body split is so critical: you build your state in the worker, on the worker's heap, using the worker's JIT state.

This boundary is also why the NBenchmark analyzer NB0014 warns you when a lambda captures local state. That state would have to be serialized and moved across the process boundary, which can introduce hidden costs and change the measurement.

Advanced State Management

For complex scenarios, NBenchmark provides two escape hatches:

• [BenchmarkState]: Used for state that is explicitly designed to cross the boundary.
• [BenchmarkPlan]: Used for suites that must build their own fixtures (like custom containers or live databases) directly inside the worker process via RunPlanAsync.

When to Opt Out

Despite the benefits, there are times when isolation is too expensive. You can disable it using [Isolation(Isolation.Off)], WithIsolation(...), or the --in-process flag.

In-process runs are excellent for "smoke tests" - verifying that your benchmark compiles and runs without spending the time to spawn workers. But for any result you intend to publish or use as a regression gate, isolation is non-negotiable.

If you're still not convinced, try the --verify-isolation flag. The engine will run the same benchmark isolated and in-process, then show you the divergence table. Usually, the 21x gap is enough to convince anyone.

Go deeper: Now that we've seen how a fresh process protects your numbers, let's look at the one thing that can survive between benchmarks: state.

State that survives between benchmarks →

Maybe the median is exactly 0 ns. Maybe the error margin is so wide that the result is useless. Or maybe the numbers change every time you shuffle the order of your benchmarks.

When the output looks wrong, the engine is usually trying to tell you something. This is the field guide to the nine most common symptoms in NBenchmark and how to fix them.

Symptom 1: MarginOfError ±0

The Problem: Your result is too precise to be real. The median and mean are identical, and the error margin is zero. The Cause: The "Clock Resolution Problem." Your operation is finishing faster than the system clock can tick. If a method takes 5ns but the clock only updates every 100ns, the engine just sees a sequence of zeros. The Fix: Increase the batch size. Use --ops-per-sample <n> (or WithOpsPerSample) to run more iterations per sample. This pushes the total sample time above the clock's resolution floor.

Symptom 2: The Interval Won't Narrow

The Problem: You've increased your sample count from 100 to 10,000, but the Error margin remains stubbornly wide. The Cause: Environmental interference. Adding more samples only helps if the noise is random. If the OS is preempting your thread every few milliseconds, you aren't measuring a distribution; you're measuring the OS scheduler. The Fix: Check your host. Use --host-quality-warnings to see if the machine is overloaded. Close background apps or move the run to a dedicated performance server.

Symptom 3: The Bimodal Warning

The Problem: The report contains a warning about a "bimodal distribution" or a "split cluster." The Cause: You aren't measuring one code path; you're measuring two. This typically happens when a method has a "cold" path (e.g., first-time cache miss) and a "warm" path. The engine sees two distinct clusters of timings and warns you that a simple median is misleading. The Fix: Separate the paths. Create two different benchmarks - one for the cold start and one for the steady state - to see exactly where the divergence happens.

Symptom 4: Order-Dependent Results

The Problem: Benchmark A is fast when it runs first, but slow when it runs after Benchmark B. The Cause: State contamination. Benchmark B is likely filling a cache, allocating a massive amount of memory, or changing a static flag that affects Benchmark A. The Fix: Enforce isolation. Ensure your benchmarks are not sharing mutable state. If you are using InstanceLifetime.PerClass, implement IStateReset to scrub the state between runs.

Symptom 5: The 20x Stability Gap

The Problem: The results are perfectly stable (tiny error margin), but the number is 20 times slower than you expect. The Cause: You are measuring in-process. When you use --in-process, you bypass the worker process and the RuntimeProfile optimizations. You are measuring the code with JIT tiering, background GC, and other "production" noise that the engine usually strips away. The Fix: Remove the --in-process flag. Let the engine spawn a dedicated worker process to get a clean, steady-state number.

Symptom 6: NotTested in the Sig Column

The Problem: The Significance column for a comparison is blank or says NotTested. The Cause: Insufficient data or incompatible profiles. Either the group has fewer than two valid samples (due to extreme outlier trimming), or you are trying to compare results measured under different RuntimeProfile settings. The Fix: If it's a data issue, check for extreme noise. If it's a profile issue, ensure all benchmarks in the suite use the same RuntimeProfile.

Symptom 7: Drift Restarts in the Log

The Problem: The run logs show that the engine restarted the measurement several times due to "Clock Drift." The Cause: The host machine's clock shifted or the load changed drastically mid-run. NBenchmark uses a "canary" benchmark to monitor the host; if the canary's timing drifts beyond the tolerance, the engine knows the current environment is compromised. The Fix: Stabilize the host. Disable Turbo Boost or switch to a dedicated machine with a stable power profile. You can tune the sensitivity with --drift-tolerance.

Symptom 8: The Errored Benchmark

The Problem: A benchmark is marked as Errored with an ErrorMessage. The Cause: The body threw an exception. NBenchmark doesn't abort the whole suite when one benchmark fails; it captures the error and moves to the next. Note that an OperationCanceledException also counts as an error. The Fix: Fix the bug in the benchmark body. Check the ErrorMessage to see exactly where the failure occurred.

Symptom 9: "Fast here, Slow in CI"

The Problem: The absolute numbers are wildly different between your laptop and the CI runner, but the ratios remain the same. The Cause: Hardware variance. This is normal. A Xeon processor in a data center and an M3 chip in a laptop will never produce the same nanosecond count. The Fix: Stop looking at absolute numbers. Use a ReferenceMethod to measure the ratio. If the ratio is stable across both machines, your performance characteristic is real.

Summary Table: Symptom to Fix

Go deeper: You've now finished the "Running it for real" section. You know how to measure, report, and gate your performance. Now, it's time to open the hood and see how the engine actually works.

Why your numbers come from another process →

It's great to have a table in a pull request showing that a refactor is 10% faster, but that's a manual check. The moment a developer forgets to look at the report, or a reviewer assumes the numbers are "close enough," performance begins to drift. To stop the slow bleed of performance degradation, you have to move from observation to enforcement.

You need to turn your numbers into consequences.

Gate one: The CLI regression limit

For dedicated benchmark projects, the simplest way to enforce performance is the --max-regression-percent flag.

This flag allows you to define a hard limit on how much slower a benchmark can be compared to its baseline. If any median result exceeds this percentage, the process exits with a non-zero code, failing your CI pipeline.

# Fail the build if any benchmark is more than 5% slower than the baseline
dotnet benchmark --max-regression-percent 5

The beauty of this approach is that it doesn't block the evidence. Even when the build fails, NBenchmark still flushes the reporters. Your GitHub Action or Azure DevOps pipeline will fail, but the Markdown table will still be there, showing exactly which implementation crossed the line and by how much.

Gate two: Test-framework integration

While CLI flags are great for standalone projects, most engineers want their performance gates to live where their other tests live: in xUnit, NUnit, or MSTest.

NBenchmark provides integration packages (e.g., NBenchmark.Integration.xUnit) that allow you to write performance tests as first-class citizens. Instead of a [Fact], you use [PerformanceFact].

using NBenchmark.Integration.xUnit;

public class PaymentProcessingTests
{
    [PerformanceFact]
    public void ProcessPayment_ShouldBeFast()
    {
        // The entire method body is the benchmark
        var processor = new PaymentProcessor();
        processor.Process(new PaymentRequest(100));
    }
}

Absolute vs. Relative thresholds

Once you've integrated benchmarks into your tests, you have two ways to define "failure."

Absolute thresholds set a hard limit on the cost of an operation:

• MaxMeanNs: Fail if the mean exceeds this value.
• MaxP95Ns: Fail if the 95th percentile exceeds this value.
• MaxAllocatedBytes: Fail if the allocation cost is too high.

Absolute thresholds are useful for strict SLAs (e.g., "this API must respond in under 50ms"), but they are brittle. They fail when you move from a powerful build server to a slower runner.

Relative thresholds are the professional choice. By specifying a ReferenceMethod and a MaxSlowdownRatio, you compare the current implementation against a known-good reference measured in the same run.

[PerformanceFact(
    ReferenceMethod = nameof(LegacyProcessPayment), 
    MaxSlowdownRatio = 1.2)] // Fail if 20% slower than legacy
public void NewProcessPayment_ShouldNotRegress() 
{
    // ...
}

Because both the reference and the target are measured on the same machine in matching worker processes, the absolute speed of the hardware doesn't matter. A fast laptop and a slow CI runner will produce the same ratio. You no longer have to check in "baseline files" that go stale the moment you update your compiler.

Precision in the gate

By default, [Performance] tests run with a single launch to keep the test suite fast. While this is fine for catching 2x regressions, it's not enough for high-precision gates.

If you need to enforce a tight 5% limit, you should increase the LaunchCount. Raising the launch count provides a paired-interval ratio, which ensures that a "flaky" run on a noisy CI server doesn't trigger a false positive.

For those who need to perform assertions manually within a test, PerformanceAssert.Run (NUnit/MSTest) or BenchmarkAssert.Validate (xUnit) allows you to trigger a measurement and validate the result against your constraints in a single line.

Choosing your gate

Which enforcement strategy should you use?

By turning your benchmarks into gates, you stop guessing if your code is fast enough. You define the limit, and the build ensures you never cross it.

Go deeper: You've set up the gates, but occasionally, the gates will fail and the numbers will look impossible. When that happens, you need a field guide to diagnose the symptoms.

When the numbers look wrong →

Your code isn't running in a vacuum. It's running on a machine where the OS is constantly stealing cycles to handle network interrupts, background telemetry, and a dozen open browser tabs. While NBenchmark's outlier machinery treats the symptoms of this noise by trimming spikes, the most precise benchmarks start by preventing the noise from reaching the sample stream in the first place.

The shared-runner confession

The first rule of performance engineering is that your laptop is not a measurement instrument.

Laptops are designed for bursty, interactive workloads, not steady-state measurement. They have aggressive thermal throttling, variable clock speeds, and background processes that wake up without warning. This is exacerbated in CI environments, where "shared runners" mean your benchmark is competing for L3 cache with three other unrelated build jobs on the same physical core.

If you want a number you can trust, you have to take control of the environment.

CPU Affinity: Pinning your work

The OS scheduler frequently moves threads between different CPU cores to balance heat and load. This "core hopping" is a disaster for benchmarks because it flushes the L1 and L2 caches, creating artificial spikes in your timing data.

CPU affinity allows you to pin the worker process to a specific core. When NBenchmark uses affinity, it ensures that the measurement happens on one dedicated core, keeping the caches hot and the timing stable.

You can enable this via the --cpu-affinity CLI flag or the WithHardwareAffinity method. On a machine with 16 cores, pinning your benchmark to core 7 removes the variance caused by the scheduler shifting your workload across the die.

Process Priority and Thread Control

Even with a pinned core, a high-priority system process can still preempt your benchmark. To minimize this, NBenchmark can elevate the worker process's priority.

Using the --priority flag (or WithProcessPriority), the engine tells the OS that the worker process is a high-priority task. This reduces the likelihood of the OS pausing your measurement to handle a background update or a telemetry ping.

Alongside priority, NBenchmark employs Thread Control. By default, the engine optimizes how threads are placed and managed to ensure that the timed window is as undisturbed as possible. While you can disable this via --no-thread-control, it is almost always beneficial to leave it on.

Note for macOS users: Apple's kernel restricts many of these operations. While NBenchmark attempts to apply these optimizations, the macOS platform may ignore affinity and priority requests, meaning you will naturally see more variance on Mac hardware than on Linux or Windows.

Knowing when the machine is beyond saving

There are times when no amount of pinning or priority can save a run. If you are running on a severely overloaded machine, the noise floor becomes so high that the results are meaningless.

NBenchmark includes a host-quality check via the --host-quality-warnings flag (or WithHostQualityWarnings). This feature monitors the system for extreme interference during the run. If it detects that the host is too noisy to provide an honest number, it will surface a warning in your report.

When you see a host-quality warning, don't try to "fix" the benchmark code. Instead, look at the machine:

• Close your browser and Slack.
• Move the run to a dedicated performance machine.
• In CI, switch from a shared runner to a dedicated, "bare metal" instance.

Reproducibility and the seed

Noise control isn't just about the OS; it's also about the order of execution. If you have ten benchmarks and the first one warms up the CPU's turbo boost, the tenth one will look faster simply because it ran last.

To combat this, NBenchmark uses --order random by default. By shuffling the order of benchmarks in every run, the engine ensures that any systemic bias (like thermal throttling or cache warming) is distributed across all candidates rather than favoring the end of the list.

If you need to reproduce a specific "weird" run for debugging, you can use the --seed flag. Providing a seed ensures that the "random" order is identical across runs, allowing you to isolate whether a spike is caused by the code or by its position in the suite.

The "in-process" trap

As a final word of caution: none of these environmental controls work if you use --in-process.

In-process measuring runs your code in the same process as the harness. This means you share the heap, the JIT state, and the priority of the main application. As we'll see in post 18, this can lead to results that are off by a factor of 20x or more. In-process runs are excellent for smoke tests, but they are the opposite of noise control.

Go deeper: You've now learned how to control the environment and the hardware. But what happens when you've done everything right and the numbers still look impossible?

When the numbers look wrong →

If a report says the median is 100ns ± 2ns, the reader assumes that if they run the code again, it will almost certainly land between 98ns and 102ns. But that's not what the interval is telling you. That interval describes the variance within a single run - the noise of the samples in that specific process.

It does not describe the variance between runs.

If you run the same benchmark on a different machine, or even just restart your current machine, you might find a new median of 110ns ± 2ns. Both runs are internally precise, but they disagree by 10%. This is "run-to-run variance," and it is the primary reason why single-run benchmarks are untrustworthy for detecting regressions.

The power of multiple launches

To solve this, NBenchmark introduces the concept of a launch.

A launch is not just a repeat of the sampling loop. It is a complete restart of the measurement cycle: the engine spawns a fresh worker process, performs a fresh warmup, and collects a fresh set of samples.

By performing multiple launches, you measure the variance of the environment itself.

In Harness mode, NBenchmark defaults to five launches. For Single and Suite modes, the default is one, though you can increase it using MeasurementOptions.LaunchCount or the --launch-count CLI flag.

Why five? Because in the world of performance engineering, two or three launches are often not enough to clear the noise of a background OS task. Five launches provide enough data to build a statistically significant distribution of the "median" itself.

Reading the aggregated report

When you perform multiple launches, the report changes. The primary fields - Median, Mean, and Error - are no longer from a single run. They are averages calculated across all launches.

Below the main results, you'll see a Launch Aggregation table. This table shows the median of every single launch. If the numbers are tightly clustered (e.g., 101, 102, 99, 100, 101), you have a stable environment. If they are wild (e.g., 100, 150, 90, 200, 110), you know that your results are being driven by external noise, and no amount of additional sampling within a single run will fix it.

One technical detail: while the median is averaged across launches, the RawSamples and trimmed marks shown in the advanced detail are taken from the launch nearest to the averaged median. This ensures you are looking at the most representative distribution of the run.

Paired ratios: The "Is it actually faster?" test

The most powerful feature of multiple launches is the paired ratio.

When you compare Implementation A to Implementation B in a suite, NBenchmark doesn't just compare the final averaged medians. It calculates the ratio for every single launch.

• Launch 1: A is 1.1x slower than B
• Launch 2: A is 1.05x slower than B
• Launch 3: A is 1.15x slower than B
• ... and so on.

These per-launch ratios are then combined on a log scale to produce a paired-interval ratio. This tells you not just "A is slower," but "Across five different process restarts, A was consistently 1.1x slower than B."

This is the only way to distinguish between a "lucky" run and a real performance difference. If the significance test (Sig) says "Yes" but the ratio interval is wide and overlaps 1.0, you have a result that is statistically significant but practically unstable. If the interval is tight and far from 1.0, you have an honest regression.

Practical constraints

There are two important interactions to keep in mind when configuring launches:

1. The Dry Run: The --dry-run flag performs exactly one launch. It verifies that your benchmarks compile and run without actually spending the time to gather a full distribution.
2. In-Process Limitation: If you use --in-process (which we'll discuss in post 18), you cannot perform multiple launches. Since the measurement happens in the current process, there is no "fresh worker" to spawn. In-process runs are for smoke tests; real regression testing requires isolation and multiple launches.

For those using [Performance] tests in a test suite, the default is one launch to keep the test suite fast. However, raising the launch count on these tests is the best way to turn a "flaky" performance test into a reliable CI gate.

Go deeper: Multiple launches tell you if your environment is noisy, but they don't fix the noise. To get the most precise numbers possible, you need to stop the OS from interfering in the first place.

Cutting the noise at its source →

Performance is not a single value; it is a function of its inputs. To understand how your code actually behaves, you need to measure it across a spectrum of sizes, shapes, and types of data.

NBenchmark handles this through parameterized benchmarks, allowing you to run a single piece of logic across many different inputs without writing multiple benchmark methods.

Parameterization in Harness mode

When using a dedicated benchmark class (Harness mode), you can define inputs using the [Arguments] and [ArgumentsSource] attributes.

Inline literals with [Arguments]

For simple sweeps - like testing a few specific array sizes - [Arguments] is the fastest way. You can apply the attribute multiple times to the same method, with each attribute representing one test case.

public class ScalingBenchmark
{
    [Benchmark]
    [Arguments(10)]
    [Arguments(100)]
    [Arguments(1000)]
    public void SortData(int size)
    {
        var data = GenerateRandomData(size);
        data.Sort();
    }
}

Dynamic sweeps with [ArgumentsSource]

When your inputs are complex - such as a generated range of values or a set of files from a directory - use [ArgumentsSource]. This attribute points to a static method or property that returns an IEnumerable of arguments.

This is particularly powerful for creating logarithmic sweeps (e.g., 10, 100, 1000, 10000), which are essential for identifying where an algorithm's complexity (like $O(n \log n)$ vs $O(n^2)$) starts to dominate the execution time.

Note: The NBenchmark analyzer NB0003 ensures that the number of arguments provided by the source matches the method's parameter count, and NB0012 prevents you from accidentally combining [Arguments] and [ArgumentsSource] on the same method.

Parameterization in Suite mode

If you are building a comparison suite programmatically, you can use WithParameter. This tells the engine to run every benchmark in the suite against the provided value.

Alternatively, you can use the typed Add overloads to define exactly how the parameters are injected into your lambdas. This allows you to create a matrix of results where you can compare Implementation A vs Implementation B across five different input sizes in a single table.

When you read the resulting table, look for the scaling trend. A linear increase in time as input grows is expected; an exponential spike is a warning that your code will collapse under production loads.

Organizing the explosion with Categories

Parameterization is powerful, but it creates a "combinatorial explosion." If you have 10 benchmarks and each has 10 parameter sets, you now have 100 benchmarks to run. In a CI pipeline, running all 100 on every commit is a waste of time.

NBenchmark solves this with categories. You can mark a method or an entire class with [BenchmarkCategory("Slow")] or [BenchmarkCategory("SmokeTest")].

Categories allow you to tier your performance testing:

1. The Smoke Set: A tiny subset of parameters (e.g., $n=10$) that runs on every PR to catch catastrophic regressions.
2. The Deep Set: The full scaling sweep that runs nightly to monitor long-term performance trends.

You can filter these at the command line using --include-category and --exclude-category, or programmatically via WithCategories.

# Run only the fast smoke tests for a quick check
dotnet benchmark --include-category SmokeTest

# Run everything except the heavy nightly sweeps
dotnet benchmark --exclude-category Slow

Reading the scaling table

When you run a parameterized benchmark, NBenchmark names the resulting entries using the pattern MethodName(arg1, arg2, ...).

In your results table, this allows you to see the scaling curve directly. If you see that SortData(10) takes 100ns and SortData(100) takes 1,000ns, your code is scaling linearly. If SortData(100) suddenly jumps to 10,000ns, you've found a non-linear performance cliff.

By combining parameters with significance tests, you can answer high-precision questions: "Is Implementation B significantly faster than A for small inputs, but significantly slower for large ones?" This is the only way to make an informed decision about which algorithm to ship.

Go deeper: Now that you can measure how your code scales with input, there's one more variable to consider: the machine itself. A single run can be a fluke. To know if a change is truly a regression, you need to see how it moves across multiple launches.

One run cannot tell you how much it moves →

When you're arguing for a refactor, "it felt faster on my machine" isn't a technical argument; it's a vibe. To make performance data a part of your engineering process, that data needs to move. It needs to be a file that can be attached to a ticket, a table that can be pasted into a PR, or a JSON record that can be fed into a dashboard.

NBenchmark provides a system of stackable reporters that turn raw measurement results into durable assets.

More than just a print statement

Most people start with result.Print(), which is great for a quick check. But for professional work, you want the ConsoleReporter. While Print gives you a static snapshot, the ConsoleReporter provides a rich, formatted table with live progress updates as the engine samples your code.

It's important to note the packaging: the JSON, Markdown, and CSV reporters are built into the core package. The ConsoleReporter, however, lives in NBenchmark.Reporters.Console. This keeps the core library zero-dependency while allowing the terminal UI to use specialized formatting libraries.

Stacking reporters for different audiences

You rarely have only one audience for your results. You need a human-readable table for your teammates, and a machine-readable file for your archives. NBenchmark handles this by allowing you to stack multiple reporters in a single run.

When you configure a suite, you can add as many reporters as you need. The engine will pipe the results to all of them simultaneously.

using NBenchmark;
using NBenchmark.Reporters.Console;
using NBenchmark.Reporters;

var suite = new BenchmarkSuite()
    .Add(() => { /* implementation A */ })
    .Add(() => { /* implementation B */ })
    .WithReporter(new ConsoleReporter()) // For the dev running the test
    .WithReporter(new MarkdownReporter("results/benchmark")) // For the PR
    .WithReporter(new JsonReporter("results/archive")); // For the history log

suite.Run();

Choosing the right format

Each reporter serves a specific purpose in the development lifecycle.

Markdown: The PR standard

The MarkdownReporter generates a GitHub-flavored markdown table. This is the gold standard for pull requests. Instead of a screenshot of a terminal (which is unsearchable and hard to read), you paste a real table. Your reviewers can see the Median, Ratio, and Significance columns clearly, making the "why" of your refactor immediately obvious.

CSV: The spreadsheet power-user

If you're doing a massive sweep of parameters and need to generate a trend line or a scatter plot, the CSVReporter is your best tool. It exports the results into a format that Excel or Google Sheets can ingest instantly, allowing you to perform your own secondary analysis on the data.

JSON: The source of truth

The JsonReporter is the most important reporter in the set. While Markdown and CSV are for humans, JSON is for the system. It carries the full record of the run, including:

• Every raw sample collected.
• The full histogram of the distribution.
• The exact runtime knobs and profile applied.
• The auto-tune diagnostic data (warmup stop reasons, CI half-width).

If you ever need to re-analyze an old run or build a custom performance dashboard, you start with the JSON file.

Controlling the detail

Not every report needs to be a data dump. NBenchmark provides several ways to control the level of detail in your output.

You can set the detail level per reporter, per suite, or via the --detail CLI flag. The Simple level gives you the core ten columns (Median, Mean, Error, etc.), while Advanced adds the deep-dive statistics: quartiles, fences, shape statistics (skewness, kurtosis), and the exact number of samples trimmed by the outlier fence.

One exception: JSON always carries the full record. Because JSON is intended as the archive, it ignores detail settings and saves everything.

Managing the payload

Because the raw sample sets can become quite large - especially in long-running suites - NBenchmark provides flags to trim the JSON payload without losing the primary metrics.

• --no-raw-samples: Removes the individual sample timings but keeps the calculated statistics.
• --full-raw-samples: Ensures every single timing is captured, even if the engine usually caps the reservoir.
• --no-histogram: Removes the distribution buckets to save space.

Single-result extensions

If you aren't using a suite and just have a single BenchmarkResult from a Benchmark.Run call, you don't need to instantiate a full reporter. NBenchmark provides a set of async extension methods for quick exports:

var result = Benchmark.Run(() => { /* ... */ });

await result.ToJsonAsync("result.json");
await result.ToMarkdownAsync("result.md");
await result.ToCsvAsync("result.csv");

These extensions are perfect for small scripts or integration tests where you just need a quick file on disk to verify a result.

Go deeper: Now that your results can travel, let's look at how to make those results meaningful across different inputs. Most code doesn't have a single "speed" - it has a scaling curve.

Benchmarks that take arguments →

In .NET, time is not the only resource that matters. Every byte you allocate is a debt that must eventually be paid by the Garbage Collector (GC). If your code is "fast" but allocates aggressively, you aren't actually measuring the cost of your algorithm; you're just deferring the cost to a future GC pause that will eventually stop the world and spike your p99 latency.

The hidden cost of allocations

NBenchmark treats allocation measurement as a first-class citizen. By default, every run captures the number of bytes allocated per operation. This appears in your results as the Alloc/op column.

This number represents the mean bytes allocated per single operation. Because NBenchmark captures these counters outside the timed window, allocation tracking is lightweight enough to leave on by default. If you have a specific reason to disable it - such as reducing overhead in an extremely tight loop - you can use the --no-allocations flag or the corresponding configuration setting.

While the mean is the primary metric, the reality of the heap is often found in the extremes. If you enable Advanced detail in your report, you'll see allocation percentiles:

• Median: The typical allocation cost.
• P95: The cost for the slowest 5% of operations.
• Max: The worst-case allocation spike.

If your Median is 32 bytes but your Max is 4,000 bytes, you've found a "hidden" allocation path - perhaps a rare boxing event or a cache miss - that will cause exactly the kind of latency spikes your users are feeling.

Natural vs. Scrubbed heaps

The most critical decision you make when measuring allocations is not whether to measure them, but how the heap is managed between samples. This is controlled by GcBehavior.

GcBehavior.Natural

GcBehavior.Natural is the default. In this mode, NBenchmark leaves the warmup heap in place and only performs collections between different benchmarks.

This is the honest answer for most latency work. Because the heap is not scrubbed before every sample, a real GC pause can land inside your timed window. This will show up as a spike in your MaxNs and increase your variance. It's noisy, but it's an accurate reflection of how your code behaves in a production environment where the GC runs non-deterministically.

GcBehavior.PerSampleCollect

If you need to isolate the cost of your code from the noise of the heap, you can use GcBehavior.PerSampleCollect.

In this mode, the engine forces a Generation 0 collection before every single sample and performs a full Generation 2 collection immediately after warmup. This effectively gives every sample a "clean" heap. The result is significantly lower variance and a much tighter confidence interval.

The trade-off is that you are now measuring a machine your users do not have. By scrubbing the heap, you are removing the very GC pauses that define the real-world performance of your application. Use this mode when you want to compare the raw efficiency of two algorithms without the interference of a fragmented heap.

You can set this behavior globally via the --gc CLI flag or programmatically using WithGcBehavior. For more granular control, you can use overrides like ForceGcBeforeEachSample or ForceGcBetweenBenchmarks. Note that ForceGcBetweenBenchmarks is enabled by default under both behaviors, because allowing one benchmark to contaminate the heap of the next is never acceptable.

The K-batch interaction

There is one important interaction to watch for when using PerSampleCollect.

As we saw in the previous post, NBenchmark often batches multiple calls into a single sample (using a factor called K) to beat the system clock's resolution. If your body allocates memory, those allocations accumulate across the entire batch.

Under PerSampleCollect, if a batch is large enough, it can trigger a GC collection inside the timed window of a single sample. When this happens, the engine will surface a warning. The fix is to force the batch size to one using --ops-per-sample 1, ensuring that each timed window contains exactly one operation and its associated allocation cost.

Corroborating with diagnostics

To get the full picture, you can enable runtime diagnostics using WithDiagnostics. This adds a block of metadata to your results that reveals what was actually happening under the hood:

• Collection Counts: Exactly how many Gen0, Gen1, and Gen2 collections occurred.
• Heap State: The size of the managed heap at the end of the run.
• CPU Time: The actual processor time spent executing, which helps distinguish between "waiting" and "working."

using NBenchmark;
using NBenchmark.Reporters.Console;

public class SerializerBenchmark
{
    // Compare two different serialization strategies
    [Benchmark]
    public byte[] JsonSerialize() => System.Text.Json.JsonSerializer.SerializeToUtf8Bytes(new { Id = 1, Name = "Test" });

    [Benchmark]
    public byte[] MsgPackSerialize() => MessagePack.MessagePackSerializer.Serialize(new { Id = 1, Name = "Test" });

    public static void Main(string[] args)
    {
        var suite = new BenchmarkSuite()
            .Add(new SerializerBenchmark())
            .WithDiagnostics()
            .WithReporter(new ConsoleReporter());

        suite.Run();
    }
}

Example Output (Natural):

Benchmark: JsonSerialize
  Median: 450 ns
  Mean:   462 ns
  Error:  ± 12 ns
  Alloc:  128 B/op
  
  Diagnostics:
    Gen0 Collections: 12
    Gen1 Collections: 0
    Gen2 Collections: 0
    Total CPU Time: 1.2 ms

Example Output (PerSampleCollect):

Benchmark: JsonSerialize
  Median: 410 ns
  Mean:   412 ns
  Error:  ± 2 ns
  Alloc:  128 B/op
  
  Diagnostics:
    Gen0 Collections: 100
    Gen1 Collections: 0
    Gen2 Collections: 0
    Total CPU Time: 1.1 ms

In this example, switching to PerSampleCollect reduced the mean and tightened the error margin significantly, but it also increased the number of collections by an order of magnitude. The Alloc/op remains the same, but the execution noise is gone.

From measurement to enforcement

Measuring allocations is the first step; enforcing a budget is the second. In a professional CI pipeline, you shouldn't just monitor allocations - you should gate them.

If a refactor increases the bytes per operation from 128 to 1024, it might not slow down the median execution time today, but it will increase the frequency of GC pauses in production. By setting an allocation threshold in your performance tests, you can fail the build the moment your code becomes "trashy," long before it hits a user's machine. We'll cover how to set up these gates in post 16.

Go deeper: Now that you know how to measure what's happening, let's look at how to get those results out of your terminal and into a format that your team can actually use.

Output that travels →

This is the resolution problem.

On most modern systems, reading the high-resolution timer itself has a cost - often in the tens of nanoseconds. If your method finishes in 50 nanoseconds, and the act of checking the clock takes 30 nanoseconds, a significant portion of your "measurement" is actually just the cost of the measurement tool. At this scale, the clock is too coarse; you aren't measuring your code, you're measuring the jitter of the system timer.

NBenchmark solves this by using ops-per-sample calibration.

The Batching Strategy: Ops-per-Sample (K)

Rather than trying to time a single execution of a nanosecond-scale operation, NBenchmark times a batch of back-to-back calls as a single "sample."

The number of calls in this batch is known as K (or Ops-per-sample).

The logic is simple: if one call is too fast to measure, 1,000 calls will be plenty fast to measure. By grouping them, the engine ensures that the total duration of the sample is large enough to dwarf the cost of the timer reads and the overhead of the loop.

Once the batch is timed, the engine divides the total duration by K to derive the per-operation time.

$$\text{Per-op Time} = \frac{\text{Sample Duration}}{\text{Ops-per-sample (K)}}$$

How Calibration Works

In most cases, you don't need to worry about K because NBenchmark handles it automatically.

During the calibration phase (which happens after warmup but before sampling), the engine probes the operation. It starts with a small K and incrementally increases it until the total sample duration is consistently above a safe threshold - one that is significantly larger than the system's timer resolution.

This auto-calibration ensures that whether your method takes 10 nanoseconds or 10 microseconds, the resulting measurement is honest and the overhead is minimized.

When Auto-Calibration is Skipped

There is one specific scenario where the engine declines to calibrate and forces $K = 1$: when you use SampleSetup or SampleTeardown hooks.

The ops-per-sample batching works by running the method body in a tight, repeated loop. If you have configured a setup or teardown method to run around every sample, the engine cannot batch the body without also batching the setup/teardown. Since setup and teardown logic often involves state changes that must happen exactly once per operation, batching them would produce incorrect results.

In these cases, NBenchmark defaults to $K = 1$. If your method is extremely fast and you are using setup hooks, you may see the "clock resolution" symptoms described below.

Pinning K for Reproducibility

While auto-calibration is usually correct, there are times when you want to pin the batch size. You can do this using --ops-per-sample <n> in the CLI or WithOpsPerSample(n) in the API.

Pinning K is useful when:

1. Comparing across environments: You want to ensure two different machines are using the exact same batching strategy to eliminate any potential (though rare) variance in calibration logic.
2. Stability testing: You suspect that a very large K is masking a periodic spike that would be visible at a smaller K.

Note that OpsPerSample is a host-wide setting. It applies to every benchmark in the suite or harness, as it defines the fundamental measurement resolution for the run.

The Warning Sign: MarginOfError ±0

If you see a result where the Error (the confidence interval) is exactly ±0ns (0.0%), it is rarely a sign of perfect stability. Instead, it is a sign of quantization.

A $\pm 0$ error means that every single sample recorded exactly the same value. When this happens with sub-microsecond operations, it almost always means the operation is faster than the clock's ability to distinguish between two different values. The clock is simply rounding every sample to the same "tick."

When you see this, the solution is to raise K. By increasing the ops-per-sample, you push the total sample duration into a range where the clock can actually see the variance in your code's performance.

Beyond Batching: Jitter and Quantization

Batching solves the resolution problem, but it doesn't solve everything. There are deeper issues - like the way the CPU's TSC (Time Stamp Counter) can drift or how the OS handles power states - that can introduce "jitter" into your numbers.

NBenchmark includes an advanced jitter auto-switch and quantization correction logic that operates beneath the batching layer to further refine the numbers. The deep dive into how the engine manages these hardware-level quirks is covered in the final post of the series.

Example: Fast Pure Function

Consider a simple function that calculates a hash.

[Benchmark]
public int CalculateHash(string input) => string.GetHashCode(input);

Run 1: Auto-Calibration (Default) The engine detects the operation is $\sim 15\text{ns}$. It calibrates $K$ to $1,000$. It times $1,000$ hashes $\approx 15,000\text{ns}$. Result: $15\text{ns} \pm 0.2\text{ns}$.

Run 2: Pinned K=1 (Forced) The engine times a single hash. The timer resolution is $30\text{ns}$. The result is either $30\text{ns}$ or $60\text{ns}$ based on where the clock tick falls. Result: $30\text{ns} \pm 0\text{ns}$ (Quantization Error).

By batching, NBenchmark turns a coin-flip measurement into a defensible number.

For a deeper dive into the cost of memory and the state of the heap during these measurements, see Counting what the GC sees.

The problem with fixed counts is that they are a guess. If your code is extremely stable, 1,000 samples is a waste of time. If your code is volatile, 1,000 samples might still leave you with a confidence interval so wide that the result is meaningless.

NBenchmark replaces the fixed count with an adaptive sampling loop. The engine doesn't ask "how many times should I run this?" but rather "how much data do I need before I am confident in the answer?"

The path to a precise number

Every NBenchmark run follows a three-stage lifecycle designed to eliminate bias and maximize precision.

1. Warmup and Plateau Detection

Before a single measurement is recorded, the engine enters the warmup phase. This isn't just about "heating up" the CPU; it is about reaching JIT-quiescence.

The engine monitors the timings of the warmup samples. It looks for a plateau - the point where the timings stop dropping sharply (as the JIT promotes code to Tier 1) and begin to stabilize. Once the timings plateau and stay within a specific tolerance for a set period, the engine declares the system "warm" and moves to calibration.

2. Calibration

Once warm, the engine determines the Ops-per-sample (K). If your operation is too fast for the system clock to measure accurately, the engine batches multiple calls into a single timed sample. This ensures that the measurement window is large enough to overcome timer resolution issues (covered in detail in post 10).

3. The Adaptive Loop

This is the core of the engine. NBenchmark starts collecting samples and, after every batch, recalculates the Confidence Interval (CI).

The loop continues until one of three things happens:

1. The Target is Met: The half-width of the confidence interval falls below the target precision (e.g., 0.5% of the mean).
2. The Ceiling is Hit: The number of samples reaches the MaxSamples limit.
3. The Budget expires: The total time spent sampling exceeds the tuning-time budget.

This means that for a simple, stable function, NBenchmark might stop at 40 samples. For a complex, volatile operation, it might push to 5,000. In both cases, you get the same level of statistical confidence.

Controlling the Loop

While the adaptive loop is the default, you have full control over the measurement process.

Pinning for Reproducibility

If you need to ensure that every run uses the exact same amount of data (for example, when comparing across different machines in a very strict environment), you can pin the counts:

• WithSamples(n) or Samples = n (in [Benchmark])
• WithWarmupSamples(n) or WarmupSamples = n

Note: Pinning the warmup is a conscious choice. By default, NBenchmark's plateau detection is more reliable than a fixed number because it adapts to the complexity of the method.

Cold-start Measurement

Sometimes, the "lie" you want to measure is the cold start. To measure the very first execution - including the JIT overhead - you can skip warmup entirely:

WithWarmupSamples(0)

The Dry Run

To verify that your benchmark is wired correctly without actually spending minutes on measurements, use the --dry-run flag or set Samples = 0. The engine will perform the discovery, setup, and calibration, but it will skip the measurement loop.

The autoTune Diagnostic

If you are curious about how the engine arrived at your result, you can inspect the autoTune diagnostic data (available in JSON reports). This data reveals:

• Resolved Samples: The final number of samples collected.
• Stop Reason: Whether the loop stopped because the target was met, a ceiling was hit, or the budget ran out.
• Achieved CI: The actual precision reached at the moment of stopping.

Example: Adaptive vs Pinned

Consider a method with high variance (e.g., a network-bound call).

Default (Adaptive) Run: The engine sees the wide variance $\rightarrow$ the CI stays wide $\rightarrow$ it keeps sampling $\rightarrow$ it stops at 1,200 samples once the Error is $\pm 0.5%$.

Pinned Run (WithSamples(100)): The engine stops at 100 samples $\rightarrow$ the Error is $\pm 4.2%$.

In the pinned run, you have a number, but you don't have a precise number. The adaptive loop protects you from accidentally trusting a result that hasn't converged.

For a deeper dive into how we handle the resolution of the clock when your code is faster than a nanosecond, see When your code outruns the clock.

But not all spikes are created equal. Some are "noise" (the OS stealing your CPU core for a millisecond), while others are "signal" (a rare but real path in your code, like a cache miss or a GC trigger). If you blindly discard every spike, you aren't measuring your code - you're measuring a fantasy.

NBenchmark treats samples as a distribution, using a two-stage process to separate the noise from the signal.

Stage one: Evidence-based interference rejection

Before any statistical rules are applied, NBenchmark performs a hard check for OS interference.

When the engine runs a sample, it doesn't just time the body; it monitors the execution environment. If the OS explicitly preempts the worker process - for example, by triggering a context switch or a hard interrupt - the engine knows that the resulting timing is a lie. This isn't a statistical guess; it is evidence.

Samples that are provably interfered with are discarded immediately. This ensures that the data entering the statistical pipeline is as clean as possible. You can toggle this behavior with --no-interference-filter or WithInterferenceFilter, but in 99% of cases, you want it on.

Stage two: The adaptive fence

Even without OS interference, you will still see spikes. These could be thermal throttling, page faults, or internal engine overhead. To handle these, NBenchmark uses a statistical fence.

The IQR Fence (Default)

By default, the engine uses the Interquartile Range (IQR) method. It calculates the spread between the 25th and 75th percentiles and creates a "fence" around the data. Any sample falling outside this fence is marked as an outlier.

The beauty of the IQR fence is that it is adaptive. If your code is naturally volatile, the fence widens. If your code is extremely stable, the fence tightens. This prevents the engine from over-trimming naturally noisy code or under-trimming stable code.

Alternative Outlier Modes

Depending on what you are measuring, the IQR fence might be too aggressive or too lenient. You can change this via WithOutlierMode or --outlier-mode:

• RemoveTop5Percent: A blunt instrument that simply chops off the worst 5% of samples.
• RemoveTopAndBottom5Percent: Symmetrical trimming for distributions with both low and high spikes.
• None: Disables trimming entirely. Use this for tail analysis where every single sample, no matter how slow, is part of the signal.

The bimodal warning: Noise or a second path?

One of the most powerful features of the NBenchmark engine is the bimodal warning.

When the engine trims outliers, it doesn't just throw them away - it analyzes them. If the "discarded" samples aren't just random spikes but actually form their own tight cluster, the engine warns you.

This is a critical signal. It suggests that your code isn't suffering from noise, but from a split execution profile. This often happens when you have a "cold path" and a "warm path" (for example, the first few calls to a method that trigger a lazy initialization). If you see a bimodal warning, stop looking at the median and start investigating why your code has two different performance identities.

Shape statistics and the tail

To understand the "shape" of your performance, NBenchmark provides several advanced metrics in the Advanced detail block:

• Skewness: Tells you if your spikes are only on the high end (right-skewed) or if you have unusual "fast" spikes (left-skewed).
• Kurtosis: Measures how "fat" the tails are. High kurtosis means you have frequent, extreme outliers.
• MAD (Median Absolute Deviation): A robust measure of spread that is less sensitive to outliers than Standard Deviation.
• CV (Coefficient of Variation): The ratio of the standard deviation to the mean, providing a normalized measure of volatility.

Example: Reading the Noise

Consider a benchmark for a cache lookup. Most hits are fast, but a rare miss is slow.

var suite = new BenchmarkSuite()
    .Add("CacheLookup", () => cache.Get(key))
    .WithOutlierMode(OutlierMode.None); // We want to see every miss

await suite.RunAsync();

Results:

Median: 40ns
P95: 120ns
P99: 1,500ns
Warnings: [Bimodal distribution detected in tail]

If OutlierMode were set to the default (IQR), those 1,500ns samples would be trimmed and the Median would look great. But by setting it to None and seeing the Bimodal warning, we realize the 1,500ns isn't OS noise - it's the cost of a cache miss. The "noise" was actually the most important part of the measurement.

For a deeper dive into how we ensure these numbers are precise enough to trust, see Measured until it is precise enough.

But a lower number is not evidence. In the world of high-resolution timing, 30 nanoseconds can be the result of a slightly different CPU frequency state or a lucky alignment of OS interrupts. The question isn't "Which number is smaller?" but "Is this difference real, and if so, does it actually matter?"

NBenchmark answers these two distinct questions using the Sig and Magnitude columns.

Is the difference real? The Sig column

The Sig (Significance) column tells you if the observed difference between two implementations is statistically significant - meaning it is unlikely to have occurred by chance.

How it works: Non-parametric testing

Most naive tools assume that timing data follows a "Normal" (bell curve) distribution. This is almost always false for software benchmarks, which are typically right-skewed by occasional spikes.

To avoid the errors that come with assuming a bell curve, NBenchmark uses non-parametric tests. These tests look at the ranks of the samples rather than their raw values, making them robust against outliers.

• Two-group comparisons: NBenchmark uses the Mann-Whitney U test. It compares the distributions of the baseline and the candidate to see if one is stochastically smaller than the other.
• Three or more groups: When you have multiple candidates, the engine first runs a Kruskal-Wallis omnibus test to see if any of the groups differ. If that test passes, it performs post-hoc pairwise Mann-Whitney U tests using the Holm-Bonferroni correction to prevent "p-hacking" (the increased likelihood of finding a false positive when running many tests).

Reading the symbols

The Sig column uses three primary states:

• ✓ (Checkmark): The difference is statistically significant. You can be confident the change is real.
• ✗ (Cross): The difference is not significant. Any gap in the medians is likely just noise.
• Blank: No comparison was possible (e.g., the method is the baseline).
• NotTested: Occurs when a group has fewer than two samples, making a distribution test impossible.

Tuning the evidence

The threshold for significance is the significance level ($\alpha$). By default, NBenchmark uses $0.05$. This means there is a 5% chance that a "significant" result is actually a false positive.

If you are making a high-stakes architectural decision, you may want stricter evidence. You can lower this threshold using --significance-level 0.01 or WithSignificanceLevel(0.01).

Does it matter? Magnitude and the practical-effect gate

A difference can be "significant" without being "important." If you have enough samples, a test can detect a 0.1% difference as statistically real, but in the real world, a 0.1% gain is negligible.

This is where Magnitude comes in. Magnitude uses effect size to classify the difference as Negligible, Small, Medium, or Large.

To prevent "statistically significant but practically useless" results from cluttering your report, you can use the WithMinimumPracticalEffect gate. When this is set, NBenchmark will not mark a result with a ✓ unless the effect size also exceeds your specified threshold.

Significance is not importance

The most critical lesson in reading a benchmark table is to read Sig next to Ratio.

Scope and Constraints

Where the comparison happens

By default, significance is calculated within a specific scope:

• Suite mode: All candidates in the suite are compared against the baseline.
• Harness mode: Benchmarks within the same class are compared.

If your implementations live in separate classes, you can enable --cross-class or WithCrossClassSignificance to allow the engine to compare them.

The Runtime Profile Rule

There is one hard constraint: results measured under different runtime profiles are never compared.

If you run one benchmark with RuntimeProfile.SteadyState and another with RuntimeProfile.Production, the engine will refuse to calculate a Ratio or Sig. This is because the underlying environment (JIT tiering, GC behavior) is fundamentally different; any difference in timing would be a product of the environment, not the code.

Example: The Refactor Trap

Imagine you refactored a sorting algorithm. You're seeing a slightly lower median, but you're not sure if it's a real win.

var suite = new BenchmarkSuite()
    .Add("LegacySort", () => LegacySort(data))
    .Add("OptimizedSort", () => OptimizedSort(data))
    .Add("ExperimentalSort", () => ExperimentalSort(data))
    .WithBaseline("LegacySort");

await suite.RunAsync();

Results Table:

The Analysis:

• OptimizedSort is a trap. The difference is statistically real (✓), but the magnitude is Negligible. You've spent three days on a refactor that provides no practical value.
• ExperimentalSort looks faster (Ratio 0.91), but the Sig is ✗. The difference is not statistically significant. You cannot claim this version is faster; you likely just had a lucky run.

For a deeper dive into how we handle the volatile samples that can confuse these tests, see The tail is a distribution, not a number.

Understanding these columns is the difference between seeing a "faster number" and knowing you have a "better implementation."

The primary metrics: Median, Mean, and Error

The first few columns provide the central tendency of your samples.

Median is your primary metric. It represents the middle value of your sample set. We prioritize the median because it is robust; a single massive spike caused by an OS context switch won't pull the median away from the typical execution time.

Mean is the arithmetic average. While less robust than the median, the mean is mathematically necessary to build the confidence interval.

Error is the ± margin of the confidence interval around the mean. It is usually displayed as a time value and a percentage (e.g., ±1.2ns (0.4%)).

A wide error margin does not necessarily mean you need more samples. The NBenchmark adaptive loop already handles sample counts to reach a target precision. A stubbornly wide interval usually points to external interference - such as a background process stealing your CPU cycles - rather than a lack of data.

The two sample sets: Trimmed vs Raw

This is the most important distinction in the NBenchmark engine. To give you an honest number, the engine does not treat all samples equally.

The trimmed set

The engine applies a statistical fence to identify and remove extreme outliers (spikes). The resulting trimmed set is used to calculate:

• Mean
• Standard Deviation (StdDev)
• Coefficient of Variation (CV)
• The Confidence Interval (Error)

By calculating these on the trimmed set, we ensure that the "typical" performance isn't skewed by a random thermal throttle event.

The raw set

The raw pre-trim set contains every single sample collected, including the spikes. This set is used for the tail metrics:

• P95 and P99 (the 95th and 99th percentiles)
• Min and Max
• The histogram

We use the raw set here because the "tail" is exactly where the spikes live. If you are building a low-latency system, you care deeply about the P99. You want to know the worst-case scenario, and that includes the samples the fence would have removed. By default, NBenchmark uses TailMetricsBasis.Raw to ensure your tail metrics are honest.

Comparison and Cost: Ratio, Sig, and Magnitude

When you define a baseline in a suite, NBenchmark adds three columns that turn raw numbers into a comparison story.

Ratio tells you how much slower or faster a method is compared to the baseline. A ratio of 1.50 means the method is 50% slower than the baseline.

Sig (Significance) answers: "Is this difference real, or is it just noise?" It uses a non-parametric test to determine if the two distributions are statistically distinct. A checkmark (✓) means the difference is significant; a cross (✗) means it is not.

Magnitude classifies the effect size as Negligible, Small, Medium, or Large.

Crucially, Ratio, Sig, and Magnitude only mean something when read together. A result that is "Significant" but "Negligible" is a difference that is mathematically real but practically useless.

Alloc/op shows the average bytes allocated per operation. This is the cost the Garbage Collector sees. If your median time is low but your allocations are high, you are borrowing time from the future; eventually, the GC will trigger a pause that lands in your P99.

The Advanced detail block

If you run with ReportDetail.Advanced, NBenchmark appends a detailed diagnostic block below the table. This block exposes the "why" behind the numbers:

• Quartiles and Fences: The exact boundaries used to identify outliers.
• Shape Statistics: Skewness and Kurtosis, which tell you if your distribution is symmetrical or leaning heavily toward the tail.
• Samples Trimmed: Exactly how many samples were discarded as noise.
• Auto-tuned Line: The history of how the engine decided when to stop sampling.

Example: Simple vs Advanced

Consider a comparison between two JSON serializers:

var suite = new BenchmarkSuite()
    .Add("System.Text.Json", () => JsonSerializer.Serialize(data))
    .Add("FastJson", () => FastJson.Serialize(data))
    .WithBaseline("System.Text.Json");

await suite.RunAsync();

Simple Output:

Advanced Detail for FastJson:

Samples: 100 (Trimmed: 97, Outliers: 3)
Fence: [280ns, 340ns]
P95: 335ns | P99: 410ns
Skewness: 1.2 (Right-skewed)
AutoTune: Target 0.5% reached at sample 82.

In this example, FastJson is significantly faster (Ratio 0.69) with a large magnitude of improvement. However, the Advanced detail shows the distribution is right-skewed, and the P99 (410ns) is much closer to the System.Text.Json median than the FastJson median is.

For a deeper dive into how we handle the spikes that create those P99s, see The tail is a distribution, not a number.

The most common mistakes - throwaway bodies, static methods, or shared state - don't trigger exceptions. They just silently invalidate your results. NBenchmark solves this by shipping a set of Roslyn analyzers (NB0001-NB0014) that act as a continuous peer review for your performance tests.

The honesty family: Preventing silent lies

The most dangerous benchmarks are those that measure nothing. If the JIT compiler realizes a method's result isn't used, it may eliminate the body entirely. You'll see an incredibly fast median, but you're measuring the cost of a no-op.

NB0004 and NB0005 catch these "empty" benchmarks. NB0004 flags methods with no observable side effects, while NB0005 flags literally empty bodies.

Then there is the lambda trap. In Single mode, it is tempting to pass an Action lambda to Benchmark.Run. However, if that lambda doesn't return a value, the runner cannot easily sink the result to defeat dead-code elimination. NB0010 warns you about this.

The fix for all three is the same: return a value.

// ❌ NB0005: Empty body
[Benchmark]
public void DoNothing() { }

// ❌ NB0004: No observable side effects (result ignored)
[Benchmark]
public void SumIgnored() 
{
    var x = 1 + 1;
}

// ✅ Fixed: Return the value so NBenchmark can sink it
[Benchmark]
public int SumHonest() => 1 + 1;

The discovery family: Ensuring the engine can run

NBenchmark's Harness mode uses reflection to discover and instantiate your benchmark classes. If the class doesn't follow the required contract, the run will fail at runtime. The analyzers move these failures to build time.

NB0001 ensures your class has a public parameterless constructor. NB0002 prevents [Benchmark] from being placed on static methods, as the engine expects instance-based execution for state management.

If you use [Arguments] or [ArgumentsSource], NB0003 verifies that the number of arguments provided matches the method's parameter count (arity). Other guards include NB0006 (preventing multiple baselines in one class) and NB0007 (detecting duplicate lifecycle attributes).

public class MyBenchmarks
{
    // ❌ NB0002: Benchmarks cannot be static
    [Benchmark]
    public static void StaticMethod() { }

    // ❌ NB0003: Argument arity mismatch (expected 1, got 2)
    [Benchmark]
    [Arguments(10, 20)] 
    public void Process(int size) { }

    // ✅ Fixed
    [Benchmark]
    [Arguments(10)]
    public void ProcessFixed(int size) { }
}

The range family: Sanity checking the knobs

Configuring samples and warmup counts is usually automatic, but when you pin them manually, it is easy to provide a value that is either too low to be statistically significant or high enough to hang your CI pipeline.

NB0008 and NB0009 monitor these values. If you set Samples to 1 or a WarmupSamples count that exceeds reasonable bounds, the analyzer flags it.

The contamination family: Guarding state independence

Benchmark results are only valid if each sample is independent. If a benchmark modifies an instance field that is shared across samples, the second sample is measuring a different state than the first.

NB0011 and NB0013 target these "contaminated" benchmarks. They flag the use of mutable instance fields or scoped services when the class is marked with InstanceLifetime.PerClass. The full story on managing expensive state is covered in post 19.

Finally, NB0014 flags when a benchmark body captures local state. This is an Info severity notification rather than an error. Because NBenchmark runs benchmarks in isolated worker processes, any captured state must be serialized and transferred across a process boundary. If you need complex setup that cannot be easily transferred, you should use [BenchmarkPlan].

public class StateBenchmark
{
    private int _counter = 0;

    // ❌ NB0013: Mutable field modified under PerClass lifetime
    [Benchmark]
    public void Increment() => _counter++; 
}

Managing severities in CI

Analyzers use standard Roslyn severities:

• Error: Blocks the build. These are non-negotiable honesty or discovery failures.
• Warning: Notifies you of a likely mistake.
• Info: Provides a hint about how the engine is handling your code.

If you encounter a false positive - or if you have a highly specific reason to bypass a guard - you can suppress the warning using #pragma warning disable or a .editorconfig file. In CI environments, it is recommended to treat warnings as errors to ensure no "silent lies" reach your main branch.

For more information on troubleshooting your benchmarks, see the Troubleshooting guide.

But as a project grows, benchmarks stop being scripts and start being assets. You find yourself wanting to categorize them, filter which ones run in CI, or change the sample count without recompiling the whole project.

This is where Harness mode comes in. It transforms your benchmarks from a sequence of calls into a structured project.

Setting up the Harness

To move to Harness mode, you create a dedicated console project and replace your manual suite building with a single entry point:

using NBenchmark;

// This one line handles CLI argument parsing, benchmark discovery, 
// and execution orchestration.
await BenchmarkHarness.Create(args).RunAsync();

Once this is in place, you no longer write code to run the benchmarks. Instead, you write classes to define them.

Defining Benchmarks with Attributes

In Harness mode, any public method in a public class marked with the [Benchmark] attribute is automatically discovered and measured.

using NBenchmark;

public class StringBenchmarks
{
    [Benchmark(Baseline = true, Description = "Standard concatenation")]
    public string LegacyConcat() => "Hello " + "World";

    [Benchmark(Samples = 100, WarmupSamples = 20, Description = "Using StringBuilder")]
    public string BuilderConcat() 
    {
        var sb = new System.Text.StringBuilder();
        sb.Append("Hello ");
        sb.Append("World");
        return sb.ToString();
    }
}

The [Benchmark] attribute allows you to pin specific configurations to a method:

• Baseline: Marks the method as the reference point for ratios and significance tests.
• Description: A human-readable name used in the final report.
• Samples & WarmupSamples: Overrides the engine's auto-tuning for this specific method.
• LaunchCount: Determines how many fresh worker processes are spawned to measure run-to-run variance.

The Benchmark Lifecycle

Real-world benchmarks often need state. Harness mode provides a set of attributes to manage this state without polluting the timed window:

• [GlobalSetup] / [GlobalTeardown]: Run once per class. Use these for expensive operations like opening a database connection or loading a large file into memory.
• [SampleSetup] / [SampleTeardown]: Run around every single sample. Use these to reset a collection or clear a cache so that each sample starts from a clean slate.

All lifecycle methods must be public and are executed outside the measurement window.

The Rules of Discovery

For the harness to find your benchmarks, your classes must follow a few simple rules:

1. The class must be public and non-abstract.
2. The class must have a public parameterless constructor.

If you violate these rules, the NBenchmark analyzers (like NB0001) will flag it as a build-time warning, ensuring your benchmarks don't silently disappear from your runs.

Controlling the Run via CLI

The real power of Harness mode is that you can change how your benchmarks run without touching the code. Because BenchmarkHarness.Create(args) consumes the command line, you have total control:

# Run only benchmarks related to "String"
dotnet run -- --filter String

# List all discovered benchmarks without running them
dotnet run -- --list

# Run a "smoke test" to ensure everything wires up (invokes nothing)
dotnet run -- --dry-run

# Output results to a specific directory using the Markdown reporter
dotnet run -- --reporter markdown --output ./bench-results

The Hierarchy of Truth

When a setting is defined in multiple places, NBenchmark follows a strict precedence order: Host MeasurementOptions $\rightarrow$ [Benchmark] attribute pins $\rightarrow$ CLI flags

If you set Samples = 100 on a method but pass --samples 500 in the CLI, the CLI wins. There are two special flags that override everything: --strict-isolation and --in-process. Use the latter only for fast smoke tests, as it bypasses the worker process and produces numbers that are often 20x wrong.

The dotnet benchmark Global Tool

If you want to run benchmarks without even having the project open, you can use the dotnet benchmark global tool. This tool can target an assembly directly:

dotnet benchmark --project ./src/MyBenchmarks.csproj

This allows you to integrate performance gates into your CI/CD pipeline without needing to write a custom runner.

Which mode should you choose?

Go deeper: Now that your benchmarks have a home, you need to make sure they are actually correct. In the next post, we'll look at the built-in analyzers that catch silent mistakes before they hit your report.

The reviewer that ships with the compiler $\rightarrow$

Comparing two separate Benchmark.Run calls is a mistake. Even on the same machine, the environment shifts: the CPU clocks down, a background process spikes, or the GC kicks in. If you run Implementation A, then run Implementation B five minutes later, you aren't comparing code - you're comparing two different moments in time.

NBenchmark's Suite mode solves this by putting every candidate into a single, coordinated run.

The BenchmarkSuite Builder

Suite mode uses a fluent builder to define a group of benchmarks that should be compared against one another.

using NBenchmark;
using NBenchmark.Reporters.Console;

var suite = new BenchmarkSuite()
    .Add("Legacy Approach", () => LegacyAlgorithm.Execute())
    .Add("Optimized Approach", () => OptimizedAlgorithm.Execute())
    .Add("Experimental Approach", () => ExperimentalAlgorithm.Execute())
    .WithBaseline("Legacy Approach")
    .WithReporter(new ConsoleReporter());

var results = await suite.RunAsync();

Every name in a suite must be distinct. If you add two benchmarks with the same name, NBenchmark will throw an ArgumentException. This is because the significance tests use these names as keys to pair samples and calculate ratios.

Beyond the Averages: Ratio, Sig, and Magnitude

When you run a suite, NBenchmark doesn't just give you three separate results. It generates a comparison table. The magic happens in three specific columns:

1. Ratio: This tells you exactly how much slower (or faster) a candidate is compared to the baseline. A ratio of 1.20 means the implementation is 20% slower than the baseline.
2. Sig (Significance): This is the most important column. A ✓ means the difference is statistically significant - it is highly unlikely to be the result of random noise. A ✗ means that while the numbers look different, there isn't enough evidence to prove the difference is real.
3. Magnitude: This categorizes the effect as Negligible, Small, Medium, or Large. It prevents you from spending a week optimizing a "significant" difference that only saves 2 nanoseconds.

Tuning the Suite

While NBenchmark's "Auto" mode is usually correct, you can override the engine's decisions using the builder:

• WithSamples(n) / WithWarmupSamples(n): Force a specific number of samples. Useful when you need a perfectly reproducible run for a report.
• WithOutlierMode(mode): Change how the engine handles spikes. If you are specifically studying the "tail" of your distribution, you can set this to None to keep every single sample.
• WithProgress(): Enables a live progress bar in the console, which is essential for long-running suites.

Lifecycle and Error Handling

Suites support WithSuiteSetup and WithSuiteTeardown hooks. These run once per suite, making them ideal for initializing shared resources like a database connection or a large static cache. Teardown is guaranteed to run even if the suite fails.

Crucially, if one benchmark in your suite throws an exception, it doesn't abort the entire run. NBenchmark marks that specific result as Errored, captures the ErrorMessage, and moves on to the next candidate. Your evidence survives even when your code doesn't.

What's Next?

Suite mode is powerful for comparisons, but it still requires you to write a Program.cs and manually build your suite. As your set of benchmarks grows from three to thirty, this becomes a maintenance burden.

In the next post, we'll look at Harness mode, where your benchmarks live in classes with attributes, and a CLI tool handles the execution.

Go deeper: If you're wondering exactly how NBenchmark decides if a difference is "significant," check out the documentation on significance testing.

A home for your benchmarks $\rightarrow$