Up until now, we've looked at benchmarks as scripts: a few lines of code in a Program.cs that you run once and move on from. This works for one-off measurements or simple comparisons.

But as a project grows, benchmarks stop being scripts and start being assets. You find yourself wanting to categorize them, filter which ones run in CI, or change the sample count without recompiling the whole project.

This is where Harness mode comes in. It transforms your benchmarks from a sequence of calls into a structured project.

Setting up the Harness

To move to Harness mode, you create a dedicated console project and replace your manual suite building with a single entry point:

using NBenchmark;

// This one line handles CLI argument parsing, benchmark discovery, 
// and execution orchestration.
await BenchmarkHarness.Create(args).RunAsync();

Once this is in place, you no longer write code to run the benchmarks. Instead, you write classes to define them.

Defining Benchmarks with Attributes

In Harness mode, any public method in a public class marked with the [Benchmark] attribute is automatically discovered and measured.

using NBenchmark;

public class StringBenchmarks
{
    [Benchmark(Baseline = true, Description = "Standard concatenation")]
    public string LegacyConcat() => "Hello " + "World";

    [Benchmark(Samples = 100, WarmupSamples = 20, Description = "Using StringBuilder")]
    public string BuilderConcat() 
    {
        var sb = new System.Text.StringBuilder();
        sb.Append("Hello ");
        sb.Append("World");
        return sb.ToString();
    }
}

The [Benchmark] attribute allows you to pin specific configurations to a method:

Baseline: Marks the method as the reference point for ratios and significance tests.
Description: A human-readable name used in the final report.
Samples & WarmupSamples: Overrides the engine's auto-tuning for this specific method.
LaunchCount: Determines how many fresh worker processes are spawned to measure run-to-run variance.

The Benchmark Lifecycle

Real-world benchmarks often need state. Harness mode provides a set of attributes to manage this state without polluting the timed window:

[GlobalSetup] / [GlobalTeardown]: Run once per class. Use these for expensive operations like opening a database connection or loading a large file into memory.
[SampleSetup] / [SampleTeardown]: Run around every single sample. Use these to reset a collection or clear a cache so that each sample starts from a clean slate.

All lifecycle methods must be public and are executed outside the measurement window.

The Rules of Discovery

For the harness to find your benchmarks, your classes must follow a few simple rules:

The class must be public and non-abstract.
The class must have a public parameterless constructor.

If you violate these rules, the NBenchmark analyzers (like NB0001) will flag it as a build-time warning, ensuring your benchmarks don't silently disappear from your runs.

Controlling the Run via CLI

The real power of Harness mode is that you can change how your benchmarks run without touching the code. Because BenchmarkHarness.Create(args) consumes the command line, you have total control:

# Run only benchmarks related to "String"
dotnet run -- --filter String

# List all discovered benchmarks without running them
dotnet run -- --list

# Run a "smoke test" to ensure everything wires up (invokes nothing)
dotnet run -- --dry-run

# Output results to a specific directory using the Markdown reporter
dotnet run -- --reporter markdown --output ./bench-results

The Hierarchy of Truth

When a setting is defined in multiple places, NBenchmark follows a strict precedence order: Host MeasurementOptions $\rightarrow$ [Benchmark] attribute pins $\rightarrow$ CLI flags

If you set Samples = 100 on a method but pass --samples 500 in the CLI, the CLI wins. There are two special flags that override everything: --strict-isolation and --in-process. Use the latter only for fast smoke tests, as it bypasses the worker process and produces numbers that are often 20x wrong.

The `dotnet benchmark` Global Tool

If you want to run benchmarks without even having the project open, you can use the dotnet benchmark global tool. This tool can target an assembly directly:

dotnet benchmark --project ./src/MyBenchmarks.csproj

This allows you to integrate performance gates into your CI/CD pipeline without needing to write a custom runner.

Which mode should you choose?

Your Goal	Recommended Mode	Why?
"I just need one number right now"	Single	Fast, zero setup.
"I need to compare A vs B"	Suite	Coordinates samples for honest ratios.
"I have a living set of benchmarks"	Harness	Organized, filterable, and CLI-driven.
"I'm auditing a 3rd party library"	Global Tool	No need to modify the source code.

Go deeper: Now that your benchmarks have a home, you need to make sure they are actually correct. In the next post, we'll look at the built-in analyzers that catch silent mistakes before they hit your report.

The reviewer that ships with the compiler $\rightarrow$

A home for your benchmarks