The engine, opened up
A deep dive into the NBenchmark measurement engine, from the worker protocol and clock probing to optional-stopping correction.
We've spent the last twenty-one posts learning how to use NBenchmark to get honest numbers. We've discussed isolation, sampling, and environmental noise. But for a number to be truly defensible, it cannot be a "black box." It must be based on a transparent, verifiable process.
In this final post, we're opening the hood. We'll explore the mechanics of the worker protocol and the mathematical corrections the engine applies to turn a raw stopwatch reading into a statistical fact.
The Worker Protocol: Orchestrating Isolation
As we established in post 18, the only way to ensure a clean measurement is to move the work to a separate process. But a separate process is, by definition, a separate memory space. The "Worker Protocol" is the bridge that makes this possible.
Discovery and Addressing
When you call Run(), the harness doesn't just start a process; it establishes a high-speed communication channel. The harness acts as the orchestrator, while the worker acts as the executor. The protocol handles the discovery of the worker's address and the negotiation of the measurement budget.
The Wire Format and State Transfer
To minimize the impact of the communication itself, the protocol uses a compact binary wire format. When the engine moves a prepare block or a [BenchmarkPlan] to the worker, it doesn't just send code; it transfers the captured state.
The protocol enforces a strict SampleReservoir cap. The worker collects thousands of samples, but it only sends the statistically significant subset back to the harness. This prevents the communication channel from becoming a bottleneck and ensures that the harness doesn't run out of memory when analyzing massive suites.
The Clock: Probing for Truth
Every system clock is a liar. Some have high resolution but drift over time; others are stable but "quantized," meaning they only tick in large jumps.
NBenchmark does not assume the clock is perfect. Before a single sample is taken, the engine performs clock probing. It runs a series of empty loops to determine the minimum detectable increment of the system timer.
If the probe reveals that the clock is too coarse for the operation being measured, the engine automatically switches strategies. This is where the "Ops-per-Sample" (K) calibration from post 10 comes from. The engine calculates the exact K-factor needed to push the measurement window above the clock's resolution floor, ensuring that the resulting number is a measurement, not a guess.
Jitter Calibration and the Detector Auto-Switch
Beyond resolution is the problem of jitter - the tiny, high-frequency fluctuations in timing that happen even on a perfectly stable machine.
The engine uses a jitter calibration phase to establish the "noise floor" of the current environment. It then employs a detector auto-switch. Depending on the distribution of the jitter, the engine chooses between different outlier detection algorithms.
If the jitter is Gaussian (normal), the engine uses standard deviation-based trimming. If the jitter is skewed or heavy-tailed, it switches to the IQR (Interquartile Range) fence. This ensures that the engine isn't accidentally trimming real performance data or leaving in systemic noise.
The Host Drift Canary
Even a perfectly calibrated run can be ruined by a "drift event" - a sudden change in the host's environment, such as a CPU frequency shift or a background system task.
To detect this, NBenchmark runs a drift canary. This is a hidden, ultra-stable benchmark that runs periodically alongside your actual code. The canary's timing is known and constant. If the canary's result shifts beyond the --drift-tolerance, the engine knows that the host is no longer stable.
Rather than reporting a corrupted number, the engine triggers a restart. It flushes the current samples and starts the run over. It is better to take an extra ten seconds to restart than to report a "confident" number that was skewed by a background process.
Optional Stopping and the Unbiasing Correction
One of the most subtle challenges in statistics is the "Optional Stopping" problem.
In NBenchmark, we sample adaptively: we keep taking samples until the confidence interval is tight enough to stop. However, in statistics, stopping a test precisely when it hits a target can introduce a subtle bias. You are more likely to stop a run when a "lucky" string of fast samples narrows the interval, potentially underestimating the true mean.
To solve this, NBenchmark applies an unbiasing correction. It uses a mathematical adjustment based on the total number of samples and the achieved interval width to remove this bias. This ensures that the reported mean is an unbiased estimate of the population, not just the result of a lucky stopping point.
Validation: Pinned against the Giants
Math is only useful if it's correct. To ensure that NBenchmark's statistical implementations - from the Mann-Whitney U test to the IQR fence - are accurate, they are not developed in a vacuum.
Every core numerical function in the engine is pinned against SciPy 1.17.1 and NumPy 2.4.6. We run the same raw sample sets through both NBenchmark and the industry-standard Python libraries. If the results diverge by more than a specified tolerance (typically $10^{-9}$), the build fails.
When you see a Sig checkmark or a Median in an NBenchmark report, you aren't trusting a custom implementation - you are trusting math that has been verified against the same libraries used by data scientists worldwide.
The Final Word
Benchmarking is a battle against noise. From the process boundary and the clock probe to the drift canary and the unbiasing correction, every feature in NBenchmark exists to remove one specific way that a number can lie to you.
By combining rigorous isolation with verified statistics, NBenchmark turns the "vibe" of performance into an engineering discipline. You no longer have to guess if your code is fast; you can prove it.
Go deeper: You've reached the end of the series. If you want to dive even deeper into the internals, the full technical specifications for the worker protocol and the measurement engine are available in the Deep Dives section of the docs.