Descriptive Statistics

Descriptive statistics

Given a sorted, trimmed array of n samples:

Mean

$$\overline{x}=\frac{1}{n}\sum _{i=1}^n{x}_i$$

Median

The mid-average convention: the middle value for odd n, and the mean of the two middle order statistics for even n. This matches numpy.median and the median NBenchmark uses elsewhere (per-launch aggregation, jitter calibration), so the reported Median and the P50 percentile agree. Every other percentile uses the nearest-rank method (see below); the median is the sole exception, because the mid-average removes the small systematic downward bias nearest-rank has on even n.

Percentiles

Configurable percentile values computed via the nearest-rank method: i = ceil(p × n). The median (p = 0.50) is the one exception - it mid-averages the two middles on even n (see Median above); Q1/Q3 and every other percentile stay nearest-rank. Controlled by MeasurementOptions.ReportedPercentiles (default: P50, P95, P99, P99.9, Max). Each entry is a PercentileEntry with a Percentile (0-1) and Value (nanoseconds). Access a specific percentile with result.GetPercentile(0.95).

Tail metrics are computed from the full pre-trim distribution by default. Percentiles, Min, Max, and the histogram describe the shape of the distribution, so they are computed from the raw (pre-trim) sample set - MeasurementOptions.TailMetricsBasis = Raw, the default. This keeps them honest: the IQR/MAD fence removes exactly the slow tail that P99/P99.9/Max exist to describe, so a GC pause the Realistic profile deliberately timed appears in Max rather than being trimmed out of it. Central-tendency and dispersion statistics (mean, standard deviation, CI, CV, skewness, kurtosis, MAD, median, median CI) always stay on the trimmed set, so a fenced-out spike never moves the mean or inflates the interval. Set TailMetricsBasis = Trimmed (or --tail-basis trimmed) to compute tail metrics from the inlier set instead.

Percentiles describe samples, and a sample may be a batch

When ops-per-sample calibration resolves K > 1 (the norm for sub-10 µs bodies), each sample is the mean of K back-to-back operations. Percentiles, Min, Max, and the histogram are therefore over batch means, not individual operations - a single slow op is averaged with its K-1 neighbours, so the tail percentiles understate true per-operation tail latency. This is the deliberate cost of amortising timer overhead on fast bodies. When you need genuine per-op tail latency, pin OpsPerSample = 1 (accepting that at that scale the reported values are dominated by timer resolution and read overhead - compare against a baseline measured the same way). Bodies that already span ≥ AutoTune.TargetSampleDurationNs (10 µs) keep K = 1, so their percentiles are already per-operation.

Min and Max

samples[0] and samples[n-1] of the sorted tail source (the full pre-trim set by default; see the tail-metrics note above).

Sample standard deviation (Bessel's correction)

$$s=\sqrt{\frac{1}{n-1}\sum _{i=1}^n(x_i-\overline{x}{)}^2}$$

The n-1 denominator (Bessel's correction) makes s an unbiased estimator of the population standard deviation. For n = 1, the standard deviation is reported as 0.

Standard error of the mean

$$\text{SEM}=\frac{s}{\sqrt{n}}$$

SEM measures how precisely the mean is estimated. For n = 1, SEM is 0.

Confidence interval on the mean

The margin of error is the half-width of the confidence interval:

$$\text{MoE}=t_{\alpha /2,n-1}^{\ast }\times \text{SEM}$$

where $t_{\alpha /2,n-1}^{\ast }$ is the two-tailed critical value of Student's t-distribution at the configured confidence level and n − 1 degrees of freedom.

The confidence interval is:

$$\overline{x}\pm \text{MoE}=[\overline{x}-\text{MoE},\overline{x}+\text{MoE}]$$

Why Student's t and not the normal distribution?

The normal distribution's critical value (e.g. 1.96 for 95%) assumes the population standard deviation is known. In benchmarking it is not - we estimate it from the sample. Student's t compensates by using wider critical values for small sample sizes, shrinking towards the normal as n grows.

With a typical auto-resolved sample count (tens to low hundreds), the t critical value at 95% sits around 1.97-1.98 - very close to the normal 1.960, so the practical difference is small.

Honest caveats

The CI is on the mean and relies on the Central Limit Theorem - the assumption that the sample mean is approximately normally distributed. For n ≥ 30 this is generally safe even when the underlying distribution is not normal. For very small sample counts (e.g. a parameterised benchmark with 10 iterations) the approximation is weaker, but the t-distribution's heavier tails at low degrees of freedom provide some protection.

t-critical values in practice

Confidence level	n = 10 (df=9)	n = 30 (df=29)	n = 200 (df=199)	Normal (df=∞)
90%	1.833	1.699	1.652	1.645
95%	2.262	2.045	1.972	1.960
99%	3.250	2.756	2.601	2.576

Dependency-free implementation

NBenchmark computes the t critical value without any external libraries using exact closed forms for df = 1 and df = 2, and the Cornish-Fisher expansion (Abramowitz & Stegun §26.7.5) for df ≥ 3. The normal quantile uses Acklam's rational approximation (max error < 1.15 × 10⁻⁹).

These approximations are cross-checked against SciPy on every build: the t critical value matches scipy.stats.t.ppf to machine precision for df = 1, 2 and to better than 1% for df ≥ 3 (worst case ≈ 0.79% at df = 3, 99%). See Validation & Accuracy for the full tolerance table.

Confidence interval on the median

The t-interval above is on the mean, but the median is NBenchmark's headline comparison metric (ratios and the Mann-Whitney semantics both key off it). MedianCiLower/MedianCiUpper report a distribution-free confidence interval on the median built from order statistics - no normality assumption.

For n < 50 the rank bounds are exact, from the binomial(n, ½) distribution: the interval [X(l), X(u)] (1-based order statistics) covers the median with probability 1 − 2·CDF(l−1), and l is the largest rank whose lower-tail mass does not exceed α/2. This can only be conservative (coverage ≥ the requested level). For n ≥ 50 the normal approximation to the binomial gives l = ⌊(n − z√n)/2⌋, u = ⌈1 + (n + z√n)/2⌉ with z = Φ⁻¹((1+CL)/2). Ranks are clamped into [1, n]; when even the widest interval cannot reach the requested level (tiny n, high CL) the full range is returned.

The interval is computed on the same (trimmed) set as Median. It appears in the advanced-detail stats block and is always present in JSON.

Coefficient of variation

$$\text{CV}=\frac{s}{\overline{x}}$$

A dimensionless relative measure of variability. A CV of 0.05 means the standard deviation is 5% of the mean - the benchmark is fairly stable. A CV of 0.5 or higher indicates high variability and the results should be treated with caution.

Distribution shape

Three fields describe the shape of the sample distribution, not just its central tendency or spread.

Skewness

$$g_1=\frac{n\sum (x_i-\overline{x}{)}^3}{(n-1)(n-2)s^3}$$

• Positive skew (right-tailed): a few slow outliers pull the mean above the median. Common in benchmarks where scheduler preemption or GC adds occasional spikes.
• Negative skew (left-tailed): most samples are slow and a few are fast - unusual in benchmarking, but can appear after compiler warmup where early iterations are slower.
• Near zero: roughly symmetric distribution.

Skewness is reported as 0 when n < 3.

Kurtosis (excess)

$$g_2=\frac{n(n+1)\sum (x_i-\overline{x}{)}^4}{(n-1)(n-2)(n-3)s^4}-\frac{3(n-1{)}^2}{(n-2)(n-3)}$$

This is excess kurtosis (kurtosis minus 3), so the normal distribution benchmarks at 0.

• Positive excess kurtosis (leptokurtic): heavier tails than a normal distribution. More extreme outliers than expected under normality. A benchmark with occasional GC pauses or page faults often shows this.
• Negative excess kurtosis (platykurtic): lighter tails, fewer extremes. Rare in benchmarking; can occur when samples are tightly clamped by hardware limits.
• Near zero: tail weight similar to a normal distribution.

Excess kurtosis is reported as 0 when n < 4.

Median absolute deviation (MAD, scaled)

$$\text{MAD}=\text{median}(|x_i-\text{median}(x)|)\times 1.4826$$

MAD is a robust measure of spread - it uses the median rather than the mean, so it is far less sensitive to outliers than the standard deviation. The scaling factor 1.4826 makes MAD consistent with the standard deviation $\sigma$ for normally distributed data, which means the two can be compared directly: if MAD is noticeably smaller than the standard deviation, outliers are inflating the standard deviation more than the bulk of the data warrant.

Reported as 0 when `n < 1$.

Summary of all reported fields

Core fields on BenchmarkResult

Field	Formula / method	Description
Median	Mid-average P50 (mean of the two middles on even n)	Robust central tendency.
Mean	$\overline{x}=\frac{1}{n}\sum x_i$	Arithmetic average.
Percentiles	IReadOnlyList<PercentileEntry>	Configurable percentile values. Default set includes P50 (0.50), P95 (0.95), P99 (0.99), P99.9 (0.999), Max (1.0). Controlled by MeasurementOptions.ReportedPercentiles. Access via GetPercentile(p).
Histogram	LatencyHistogram?	Latency histogram with bucket boundaries and sample counts. null when EnableHistogram is false or fewer than 2 samples.
Min	$x_1$ (sorted)	Fastest measured sample.
Max	$x_n$ (sorted)	Slowest measured sample.
Q1	Nearest-rank P25	First quartile.
Q3	Nearest-rank P75	Third quartile.
InterquartileRange	Q3 - Q1	Spread of the middle 50% of samples.
LowerFence	Detector-dependent	Lower outlier boundary; set only by fence-based detectors. IqrFence: $Q1-k\times \text{IQR}$ (default $k=1.5$). MedianAbsoluteDeviation: $m-t\times \text{scaledMAD}$ (default $t=3$). null otherwise.
UpperFence	Detector-dependent	Upper outlier boundary; set only by fence-based detectors. IqrFence: $Q3+k\times \text{IQR}$ (default $k=1.5$). MedianAbsoluteDeviation: $m+t\times \text{scaledMAD}$ (default $t=3$). null otherwise.
OutliersRemoved	Count of discarded samples	Number of samples removed by outlier trimming.
N	Post-trim length	Sample count after outlier removal.
StandardDeviation	$s=\sqrt{\frac{1}{n-1}\sum (x_i-\overline{x}{)}^2}$	Spread of measurements (Bessel).
StandardError	$s/\sqrt{n}$	Precision of the mean estimate.
MarginOfError	$t^{\ast }\times \text{SEM}$	Half-width of CI on the mean.
ConfidenceIntervalLower	$\overline{x}-\text{MoE}$	Lower CI bound.
ConfidenceIntervalUpper	$\overline{x}+\text{MoE}$	Upper CI bound.
MedianCiLower / MedianCiUpper	Order-statistic interval	Distribution-free confidence interval on the median (exact binomial for $n<50$, normal approximation above). null when undefined ($n<2$, dry-run, errored).
MedianShift	Hodges-Lehmann + Lehmann CI	Location shift vs. baseline (median of pairwise candidate − baseline differences) with a rank-based interval, in ns/op. Positive = candidate slower. null for the baseline or when significance did not run.
CoefficientOfVariation	$s/\overline{x}$	Relative variability.
Skewness	$g_1=\frac{n\sum (x_i-\overline{x}{)}^3}{(n-1)(n-2)s^3}$	Sample skewness. Zero for $n<3$.
Kurtosis	$g_2=\frac{n(n+1)\sum (x_i-\overline{x}{)}^4}{(n-1)(n-2)(n-3)s^4}-\frac{3(n-1{)}^2}{(n-2)(n-3)}$	Excess kurtosis. Zero for $n<4$.
Mad	$\text{median}(|x_i-\text{median}(x)|)\times 1.4826$	Median absolute deviation (scaled to $\sigma$). Zero for $n<1$.
PValue	Mann-Whitney U	Two-tailed pairwise p-value vs. baseline. null for the omnibus case (three or more groups - see Omnibus).
SignificanceVerdict	$p<\alpha$	Whether the pairwise difference is real (Significant, NotSignificant, or NotTested).
Omnibus	Kruskal-Wallis	The across-all-groups verdict when three or more benchmarks are compared; null otherwise. Holds TestName, Statistic, DegreesOfFreedom, GroupCount, PValue, and Verdict.
SignificanceTestName	-	Display name of the pairwise significance test used (e.g. "Mann-Whitney U").
OutlierDetector	-	Display name of the outlier detector applied (e.g. "IQR fence (1.5×)" or "MAD (3×)").
MeanAllocatedBytes	Mean of iteration deltas	Mean heap allocation per iteration.
AllocMedian	Mid-average P50 of iteration deltas	Median allocation per iteration (only when MeasureAllocations = true).
AllocP95	Nearest-rank P95 of iteration deltas	P95 allocation per iteration (only when MeasureAllocations = true).
AllocMax	Max of iteration deltas	Max allocation per iteration (only when MeasureAllocations = true).

Throughput fields

Field	Formula	Description
OperationsPerSecond	1e9 / Mean when timing is in nanoseconds	Mean operations per second. NaN for errored or dry-run results.
MedianOperationsPerSecond	1e9 / Median when timing is in nanoseconds	Median operations per second. NaN for errored or dry-run results.
NanosecondsPerOperation	Alias for Mean	Convenience alias that expresses the mean timing as nanoseconds per operation.
TotalOperations	MeasuredIterations + WarmupIterations, or AutoTuneDiagnostic.TotalBodyInvocations when auto-tuning	Total body invocations executed across warmup and measurement.

Computed properties

Property	Formula	Description
Range	Max - Min	Full spread of trimmed samples.
StandardErrorPercent	$\text{SEM}/\overline{x}\times 100$	Standard error as a percentage of the mean.
MarginPercent	$\text{MoE}/\overline{x}\times 100$	Margin of error as a percentage of the mean.
CoefficientOfVariationPercent	$\text{CV}\times 100$	Coefficient of variation as a percentage.