Two implementations, one table

The most common question in performance engineering isn't "How fast is this?" but "Is A faster than B?"

Comparing two separate Benchmark.Run calls is a mistake. Even on the same machine, the environment shifts: the CPU clocks down, a background process spikes, or the GC kicks in. If you run Implementation A, then run Implementation B five minutes later, you aren't comparing code - you're comparing two different moments in time.

NBenchmark's Suite mode solves this by putting every candidate into a single, coordinated run.

The BenchmarkSuite Builder

Suite mode uses a fluent builder to define a group of benchmarks that should be compared against one another.

using NBenchmark;
using NBenchmark.Reporters.Console;

var suite = new BenchmarkSuite()
    .Add("Legacy Approach", () => LegacyAlgorithm.Execute())
    .Add("Optimized Approach", () => OptimizedAlgorithm.Execute())
    .Add("Experimental Approach", () => ExperimentalAlgorithm.Execute())
    .WithBaseline("Legacy Approach")
    .WithReporter(new ConsoleReporter());

var results = await suite.RunAsync();

Every name in a suite must be distinct. If you add two benchmarks with the same name, NBenchmark will throw an ArgumentException. This is because the significance tests use these names as keys to pair samples and calculate ratios.

Beyond the Averages: Ratio, Sig, and Magnitude

When you run a suite, NBenchmark doesn't just give you three separate results. It generates a comparison table. The magic happens in three specific columns:

1. Ratio: This tells you exactly how much slower (or faster) a candidate is compared to the baseline. A ratio of 1.20 means the implementation is 20% slower than the baseline.
2. Sig (Significance): This is the most important column. A ✓ means the difference is statistically significant - it is highly unlikely to be the result of random noise. A ✗ means that while the numbers look different, there isn't enough evidence to prove the difference is real.
3. Magnitude: This categorizes the effect as Negligible, Small, Medium, or Large. It prevents you from spending a week optimizing a "significant" difference that only saves 2 nanoseconds.

Tuning the Suite

While NBenchmark's "Auto" mode is usually correct, you can override the engine's decisions using the builder:

• WithSamples(n) / WithWarmupSamples(n): Force a specific number of samples. Useful when you need a perfectly reproducible run for a report.
• WithOutlierMode(mode): Change how the engine handles spikes. If you are specifically studying the "tail" of your distribution, you can set this to None to keep every single sample.
• WithProgress(): Enables a live progress bar in the console, which is essential for long-running suites.

Lifecycle and Error Handling

Suites support WithSuiteSetup and WithSuiteTeardown hooks. These run once per suite, making them ideal for initializing shared resources like a database connection or a large static cache. Teardown is guaranteed to run even if the suite fails.

Crucially, if one benchmark in your suite throws an exception, it doesn't abort the entire run. NBenchmark marks that specific result as Errored, captures the ErrorMessage, and moves on to the next candidate. Your evidence survives even when your code doesn't.

What's Next?

Suite mode is powerful for comparisons, but it still requires you to write a Program.cs and manually build your suite. As your set of benchmarks grows from three to thirty, this becomes a maintenance burden.

In the next post, we'll look at Harness mode, where your benchmarks live in classes with attributes, and a CLI tool handles the execution.

---

Go deeper: If you're wondering exactly how NBenchmark decides if a difference is "significant," check out the documentation on significance testing.

A home for your benchmarks $\rightarrow$