When the numbers look wrong

You've done everything by the book. You used a dedicated worker process, you pinned your CPU affinity, and you ran five launches. But when the report prints, the numbers look impossible.

Maybe the median is exactly 0 ns. Maybe the error margin is so wide that the result is useless. Or maybe the numbers change every time you shuffle the order of your benchmarks.

When the output looks wrong, the engine is usually trying to tell you something. This is the field guide to the nine most common symptoms in NBenchmark and how to fix them.

Symptom 1: MarginOfError ±0

The Problem: Your result is too precise to be real. The median and mean are identical, and the error margin is zero. The Cause: The "Clock Resolution Problem." Your operation is finishing faster than the system clock can tick. If a method takes 5ns but the clock only updates every 100ns, the engine just sees a sequence of zeros. The Fix: Increase the batch size. Use --ops-per-sample <n> (or WithOpsPerSample) to run more iterations per sample. This pushes the total sample time above the clock's resolution floor.

Symptom 2: The Interval Won't Narrow

The Problem: You've increased your sample count from 100 to 10,000, but the Error margin remains stubbornly wide. The Cause: Environmental interference. Adding more samples only helps if the noise is random. If the OS is preempting your thread every few milliseconds, you aren't measuring a distribution; you're measuring the OS scheduler. The Fix: Check your host. Use --host-quality-warnings to see if the machine is overloaded. Close background apps or move the run to a dedicated performance server.

Symptom 3: The Bimodal Warning

The Problem: The report contains a warning about a "bimodal distribution" or a "split cluster." The Cause: You aren't measuring one code path; you're measuring two. This typically happens when a method has a "cold" path (e.g., first-time cache miss) and a "warm" path. The engine sees two distinct clusters of timings and warns you that a simple median is misleading. The Fix: Separate the paths. Create two different benchmarks - one for the cold start and one for the steady state - to see exactly where the divergence happens.

Symptom 4: Order-Dependent Results

The Problem: Benchmark A is fast when it runs first, but slow when it runs after Benchmark B. The Cause: State contamination. Benchmark B is likely filling a cache, allocating a massive amount of memory, or changing a static flag that affects Benchmark A. The Fix: Enforce isolation. Ensure your benchmarks are not sharing mutable state. If you are using InstanceLifetime.PerClass, implement IStateReset to scrub the state between runs.

Symptom 5: The 20x Stability Gap

The Problem: The results are perfectly stable (tiny error margin), but the number is 20 times slower than you expect. The Cause: You are measuring in-process. When you use --in-process, you bypass the worker process and the RuntimeProfile optimizations. You are measuring the code with JIT tiering, background GC, and other "production" noise that the engine usually strips away. The Fix: Remove the --in-process flag. Let the engine spawn a dedicated worker process to get a clean, steady-state number.

Symptom 6: NotTested in the Sig Column

The Problem: The Significance column for a comparison is blank or says NotTested. The Cause: Insufficient data or incompatible profiles. Either the group has fewer than two valid samples (due to extreme outlier trimming), or you are trying to compare results measured under different RuntimeProfile settings. The Fix: If it's a data issue, check for extreme noise. If it's a profile issue, ensure all benchmarks in the suite use the same RuntimeProfile.

Symptom 7: Drift Restarts in the Log

The Problem: The run logs show that the engine restarted the measurement several times due to "Clock Drift." The Cause: The host machine's clock shifted or the load changed drastically mid-run. NBenchmark uses a "canary" benchmark to monitor the host; if the canary's timing drifts beyond the tolerance, the engine knows the current environment is compromised. The Fix: Stabilize the host. Disable Turbo Boost or switch to a dedicated machine with a stable power profile. You can tune the sensitivity with --drift-tolerance.

Symptom 8: The Errored Benchmark

The Problem: A benchmark is marked as Errored with an ErrorMessage. The Cause: The body threw an exception. NBenchmark doesn't abort the whole suite when one benchmark fails; it captures the error and moves to the next. Note that an OperationCanceledException also counts as an error. The Fix: Fix the bug in the benchmark body. Check the ErrorMessage to see exactly where the failure occurred.

Symptom 9: "Fast here, Slow in CI"

The Problem: The absolute numbers are wildly different between your laptop and the CI runner, but the ratios remain the same. The Cause: Hardware variance. This is normal. A Xeon processor in a data center and an M3 chip in a laptop will never produce the same nanosecond count. The Fix: Stop looking at absolute numbers. Use a ReferenceMethod to measure the ratio. If the ratio is stable across both machines, your performance characteristic is real.

Summary Table: Symptom to Fix

Symptom	Likely Cause	The Fix
MarginOfError ±0	Clock Resolution	Raise OpsPerSample (K)
Wide, stubborn interval	Host Interference	Use dedicated hardware
Bimodal Warning	Dual code paths	Split into two benchmarks
Order dependency	State contamination	Implement IStateReset
Stable but 20x slow	In-process run	Use isolated worker process
NotTested (Sig)	Profile mismatch	Match RuntimeProfile
Drift restarts	Host clock shift	Disable Turbo Boost / Stabilize
Errored status	Thrown exception	Fix body logic
Diff absolute numbers	Hardware variance	Use ReferenceMethod ratios

---

Go deeper: You've now finished the "Running it for real" section. You know how to measure, report, and gate your performance. Now, it's time to open the hood and see how the engine actually works.

Why your numbers come from another process →