Skip to content

crp.eval

Auto-generated reference for the crp.eval subpackage.

eval

crp.eval

Evaluation statistics toolkit for CRP benchmarks (CRP-SPEC-026).

Small, dependency-light statistical primitives shared by CRP benchmark harnesses (SQB, governed-vs-bare):

pass_hat_k       - unbiased pass@k / pass^k estimator (Chen et al., 2021)
bootstrap_ci     - deterministic percentile bootstrap confidence interval
mcnemar_pvalue   - exact binomial McNemar p-value for paired comparisons

bootstrap_ci(values, statistic=None, n_boot=2000, alpha=0.05, seed=42)

Percentile bootstrap confidence interval for a statistic.

Resamples values with replacement n_boot times, applies statistic to each resample, and returns the (alpha/2, 1-alpha/2) percentiles of the bootstrap distribution. Deterministic for a fixed seed.

Parameters:

Name Type Description Default
values Sequence[float]

Observed sample (must be non-empty).

 required
statistic Callable[[Sequence[float]], float] | None

Statistic to bootstrap; defaults to the mean (numpy.mean when numpy is available, else statistics.mean).

 None
n_boot int

Number of bootstrap resamples.

 2000
alpha float

Significance level; the interval has coverage 1 - alpha.

 0.05
seed int

Seed for the resampling RNG.

 42

Returns:

Type Description
tuple[float, float]

(lower, upper) bounds of the confidence interval.

Raises:

Type Description
ValueError

If values is empty or arguments are out of range.

mcnemar_pvalue(b, c)

Exact binomial McNemar p-value for a paired comparison.

b counts discordant pairs where only A succeeded; c counts pairs where only B succeeded. Under the null hypothesis of no difference, the count of A-only wins is Binomial(n = b + c, p = 0.5); the exact two-sided p-value is::

p = 2 * sum_{i=0}^{min(b, c)} C(n, i) * 0.5**n   (capped at 1.0)

Parameters:

Name Type Description Default
b int

Discordant pairs won only by A (must be >= 0).

 required
c int

Discordant pairs won only by B (must be >= 0).

 required

Returns:

Type Description
float

Exact two-sided p-value in (0.0, 1.0]. b = c = 0 yields 1.0.

Raises:

Type Description
ValueError

If b or c is negative.

pass_hat_k(n, c, k)

Unbiased pass@k estimator (Chen et al., 2021 - Codex/HumanEval).

Given n samples per task of which c are correct, the unbiased estimate of pass@k (probability that at least one of k sampled completions is correct) is::

pass@k = 1 - C(n - c, k) / C(n, k)

Parameters:

Name Type Description Default
n int

Total number of samples (must be >= 1).

 required
c int

Number of correct samples (0 <= c <= n).

 required
k int

Number of samples drawn for the estimate (1 <= k <= n).

 required

Returns:

Type Description
float

The unbiased pass@k estimate in [0.0, 1.0]. c = 0 yields 0.0
float

(no correct sample can ever be drawn); c = n yields 1.0.

Raises:

Type Description
ValueError

If the arguments are out of range.

eval.stats

crp.eval.stats

Statistical primitives for CRP benchmark evaluation (CRP-SPEC-026).

Stdlib-only core; numpy is used opportunistically (it is available in the dev environment) but never required - the module imports and runs without it, in keeping with the zero-dependency core.

All estimators are deterministic: :func:bootstrap_ci seeds its resampling RNG, and :func:pass_hat_k / :func:mcnemar_pvalue are closed-form.

pass_hat_k(n, c, k)

Unbiased pass@k estimator (Chen et al., 2021 - Codex/HumanEval).

Given n samples per task of which c are correct, the unbiased estimate of pass@k (probability that at least one of k sampled completions is correct) is::

pass@k = 1 - C(n - c, k) / C(n, k)

Parameters:

Name Type Description Default
n int

Total number of samples (must be >= 1).

 required
c int

Number of correct samples (0 <= c <= n).

 required
k int

Number of samples drawn for the estimate (1 <= k <= n).

 required

Returns:

Type Description
float

The unbiased pass@k estimate in [0.0, 1.0]. c = 0 yields 0.0
float

(no correct sample can ever be drawn); c = n yields 1.0.

Raises:

Type Description
ValueError

If the arguments are out of range.

bootstrap_ci(values, statistic=None, n_boot=2000, alpha=0.05, seed=42)

Percentile bootstrap confidence interval for a statistic.

Resamples values with replacement n_boot times, applies statistic to each resample, and returns the (alpha/2, 1-alpha/2) percentiles of the bootstrap distribution. Deterministic for a fixed seed.

Parameters:

Name Type Description Default
values Sequence[float]

Observed sample (must be non-empty).

 required
statistic Callable[[Sequence[float]], float] | None

Statistic to bootstrap; defaults to the mean (numpy.mean when numpy is available, else statistics.mean).

 None
n_boot int

Number of bootstrap resamples.

 2000
alpha float

Significance level; the interval has coverage 1 - alpha.

 0.05
seed int

Seed for the resampling RNG.

 42

Returns:

Type Description
tuple[float, float]

(lower, upper) bounds of the confidence interval.

Raises:

Type Description
ValueError

If values is empty or arguments are out of range.

mcnemar_pvalue(b, c)

Exact binomial McNemar p-value for a paired comparison.

b counts discordant pairs where only A succeeded; c counts pairs where only B succeeded. Under the null hypothesis of no difference, the count of A-only wins is Binomial(n = b + c, p = 0.5); the exact two-sided p-value is::

p = 2 * sum_{i=0}^{min(b, c)} C(n, i) * 0.5**n   (capped at 1.0)

Parameters:

Name Type Description Default
b int

Discordant pairs won only by A (must be >= 0).

 required
c int

Discordant pairs won only by B (must be >= 0).

 required

Returns:

Type Description
float

Exact two-sided p-value in (0.0, 1.0]. b = c = 0 yields 1.0.

Raises:

Type Description
ValueError

If b or c is negative.