LLM Eval Confidence Intervals: Compare Prompts Fairly

Choose the Difference Before Calculating Its Interval

Suppose your baseline averages 60 points and your candidate averages 63.5 on a score ranging from 0 to 100, with higher scores preferred. The improvement is 3.5 score points. Those numbers describe the synthetic fixture below, not a model benchmark. The remaining question is how much the estimated improvement could vary when the evaluated cases change.

Define the target as the mean of case-level differences: for each case, subtract baseline from candidate, then average those differences. Both prompts must answer the same task under the same scoring rule. If the prompt asks for a correct migration diagnosis, a score for fluent explanation measures a different target. Freeze the rubric, model configuration, dataset, and missing-result policy before comparing prompts.

With identical case weights, the mean paired difference equals the difference between the two means. Pairing still matters for uncertainty because the scores belong to the same cases. Separately resampling baseline and candidate destroys that relationship. Better than Average explains why instance-level pairing and aggregation choices matter in NLP evaluation. Its proposed pairwise ranking target differs from the mean score improvement estimated here.

Resample the Unit That Can Vary Independently

For this fixture, one case ID is one assumed independent unit, with exactly one baseline and one candidate score. A bootstrap draw samples 20 case IDs with replacement and keeps each selected pair intact. Repeated IDs within that draw are intentional bootstrap resampling. Duplicate cases in the original dataset are a separate data-quality issue.

If five generations answer each of 20 questions, you have 100 generated answers per prompt but still only 20 question units for uncertainty across questions. One explicit target is the average per-question score after averaging its generations. Resample those question summaries together. That estimates case-sampling uncertainty for the observed summaries; it does not separately quantify all uncertainty from generating fresh answers.

If several questions share a source document or customer conversation, independence at question level may be implausible. Define the document or conversation as a cluster and resample whole clusters, preserving prompt pairs within them. Decide whether the target weights clusters equally or weights individual cases equally: those are different estimands when cluster sizes differ. The flat fixture below is not a cluster-bootstrap implementation.

The Hitchhiker’s Guide to Testing Statistical Significance in NLP discusses how evaluation measures and experimental assumptions affect test selection. Treat the sampling design as part of your analysis, rather than choosing an interval solely because a library makes it convenient.

A Local Fixture You Can Reproduce

The following complete program uses Python’s standard library. Save it as paired_bootstrap.py and run python paired_bootstrap.py. It creates synthetic scores with explicit IDs, joins by those IDs, and computes a two-sided 95% percentile bootstrap interval from 20,000 resamples. The percentile endpoints use linear interpolation between ordered bootstrap means. The fixed seed makes the saved run reproducible in the recorded runtime; preserve the runtime when comparing exact output.

This deliberately transparent implementation illustrates the percentile method. SciPy’s bootstrap reference documents percentile, basic, and BCa intervals and a paired option that reuses sampled indices across arrays. The methods need not give identical endpoints. This fixture does not call SciPy, and its percentile interval is not a claim that this method is optimal for every score distribution.

"""Synthetic paired-score fixture. Python standard library only; no model calls."""
import json
import platform
import random
import sys
from statistics import mean

SEED = 20261110
RESAMPLES = 20000
DELTAS = [-20, -10, -10, -10, 0, 0, 0, 0, 0, 0,
          0, 0, 10, 10, 10, 10, 10, 20, 20, 30]
baseline = {f"case-{i:02d}": 60 for i in range(1, 21)}
candidate = {key: baseline[key] + delta
             for key, delta in zip(baseline, DELTAS)}

def paired_differences(left, right):
    if set(left) != set(right):
        raise ValueError("Case IDs must match")
    if len(left) < 2:
        raise ValueError("At least two units required by this fixture")
    return [right[key] - left[key] for key in sorted(left)]

def percentile(sorted_values, probability):
    position = (len(sorted_values) - 1) * probability
    low = int(position)
    high = min(low + 1, len(sorted_values) - 1)
    return sorted_values[low] + (position - low) * (
        sorted_values[high] - sorted_values[low])

def interval(differences):
    rng = random.Random(SEED)
    draws = sorted(mean(rng.choices(differences, k=len(differences)))
                   for _ in range(RESAMPLES))
    return [percentile(draws, 0.025), percentile(draws, 0.975)]

differences = paired_differences(baseline, candidate)
ci = interval(differences)
assert mean(differences) == 3.5
assert ci[0] < 0 < ci[1]
assert paired_differences(baseline, dict(reversed(list(candidate.items())))) == differences
try:
    paired_differences(baseline, {"wrong-id": 70})
except ValueError:
    pass
else:
    raise AssertionError("Mismatched IDs accepted")
assert interval([0] * 20) == [0, 0]
output = {
    "fixture": "synthetic; 20 independent case units assumed; no model calls",
    "runtime": sys.version,
    "platform": platform.platform(),
    "dependencies": "Python standard library only",
    "seed": SEED,
    "resamples": RESAMPLES,
    "method": "paired percentile bootstrap; linear interpolated quantiles",
    "case_count": len(differences),
    "baseline_mean_points": mean(baseline.values()),
    "candidate_mean_points": mean(candidate.values()),
    "candidate_minus_baseline_points": mean(differences),
    "confidence_level": 0.95,
    "confidence_interval_points": ci,
    "degenerate_zero_interval_points": interval([0] * 20),
    "checks_passed": 5,
    "records": [{"case_id": key, "baseline": baseline[key], "candidate": candidate[key]}
                for key in sorted(baseline)]
}
print(json.dumps(output, indent=2))

Executed locally on 2026-11-10 with Python 3.14.2 on Windows, with no external dependencies and no model or API calls. The actual result was: 20 cases; baseline mean 60 points; candidate mean 63.5 points; difference +3.5 points; 95% percentile interval −1.5 to +8.5 points. Five fixture checks passed: expected mean, zero-crossing interval, invariance to candidate dictionary order, rejection of mismatched IDs, and the all-zero difference case.

The ID mismatch check fails closed: it does not silently compare whichever rows happen to remain. A real run also needs explicit handling for duplicate IDs, invalid scores, timeouts, and missing generations before constructing these dictionaries. Quietly discarding hard cases can change the comparison. Record exclusions and investigate whether the missingness changes the population represented.

Read the Interval Without Turning It Into a Guarantee

The synthetic result includes negative and positive differences. Under this analysis it does not resolve the direction of the mean change. It also does not establish equivalence: the interval permits a decline of 1.5 points and an improvement of 8.5 points. Failure to demonstrate an improvement is not evidence that the prompts are interchangeable.

A frequentist 95% confidence procedure concerns coverage over repeated samples under its assumptions. It does not assign a 95% probability to the fixed true difference being inside this particular realized interval. Bootstrap coverage is approximate; an interval label does not make the sampling assumptions true.

More bootstrap resamples reduce simulation variability in the computed endpoints; they do not create new independent cases. A convenience set of favorite examples remains a convenience set after resampling. The interval cannot recover missing task categories or correct a biased scoring rubric. If the target is a fixed, fully enumerated case set with fixed scores, its observed mean difference is already known exactly; the interval only has a purpose after defining what varying cases or outcomes it represents.

Small or highly discrete samples can yield unstable endpoints. If every observed difference is zero, this program returns [0, 0]. That reveals no observed variation, not guaranteed equality on unseen cases. SciPy documents that BCa can instead produce undefined endpoints for degenerate data. Inspect the records and the method’s assumptions rather than replacing an awkward interval with a preferred conclusion.

Keep Uncertainty Separate From the Release Margin

For an illustrative decision rule, suppose your team requires an improvement greater than 2 score points to justify a slower prompt. That margin is a hypothetical product judgment, not a statistical constant. The fixture’s interval crosses both zero and +2, so it does not clear that rule. An illustrative interval of +0.2 to +1.0 points would support a positive direction while still falling short of the same margin.

Equivalence needs its own predeclared acceptable range and suitable analysis. Merely including zero is insufficient. There is no universal case count: plan around the target population, clustering, score variability, and the practical difference the decision must resolve. A pilot can expose score variability and case coverage gaps, but selecting more cases only after seeing a disappointing result changes the analysis plan.

Report the signed difference, interval method and level, independent unit count, repeat policy, exclusions, and intended population together. For regression workflow and operating thresholds, use AI Evals in Production. If judgments depend on answer placement or repeated judging, address that measurement issue through the position-bias analysis before treating the scores as a stable instrument.