solprop.calibration.selection#

The calibration’s statistical rule: which setting replaces the table (SEP 46).

One rule throughout (SEP 46 section 6.5), the ab harness’s:

Pairs. Each candidate is timed interleaved with the built-in value, in rounds; a round times each once in a fresh order. A candidate’s score is the median of its per-round ratios to the built-in.
Noise. Each session also times the built-in against itself (A/A). The noise is the largest per-round deviation of that ratio from one, and at least NOISE_FLOOR.
Admission. A candidate replaces the built-in value only if its largest per-round ratio is below one minus the noise: it wins in every round by more than the noise. And no problem may be slower beyond the noise and TOLERANCE: its median ratio on each problem timed must stay within 1 + noise + TOLERANCE, so that a large win on some problems cannot carry a real loss on another, while a loss within the bar does not veto a broad win. Ties keep the built-in. A failed round is dropped; fewer than the level’s minimum rounds keeps the built-in.
Combined validation. The admitted settings are timed together against the table on fresh rounds; the combination is published only if no window is slower than that same bar and the geometric mean is below one. Otherwise groups are dropped in DROP_ORDER and the remainder validated again.

Every function here is pure: it reads timings and returns a verdict.

Module Attributes

`NOISE_FLOOR`	The least noise a session claims: a win must exceed 2 % in every round.
`TOLERANCE`	How much slower than its noise any one problem may run under a candidate, so that a small loss on one problem does not veto a broad win (the maintainer's choice; SEP 46 section 7 item 1 asked for 2 %).
`NOISE_LIMIT`	A/A noise above which a session is inconclusive and nothing is published.
`MAX_CANDIDATES`	The most candidates a setting is chosen from, so that the search cannot select noise from a large set.
`DROP_ORDER`	The groups dropped, in this order, when a combination fails validation: operation grain, FFT planning, then the waiting policy and the team.
`Times`	Seconds per round, `None` for a round that failed or was skipped.

Functions

`admit`(candidate, builtin, noise, minimum[, ...])	Decide one candidate against the built-in value.
`choose`(candidates, builtin, noise, minimum)	Choose among a setting's candidates: the admitted one with the lowest score.
`fallbacks`(groups)	Return the combinations to validate in turn, dropping groups in order.
`noise`(builtin, again, minimum)	Return the A/A noise of a session, or `None` with too few rounds.
`ratios`(candidate, builtin)	Return the per-round ratios of candidate to builtin, dropping failed rounds.
`validate`(windows, noise, minimum)	Validate a combination on fresh rounds of every calibration window.

Classes

`Validation`(passed, windows, geomean, reason)	Whether a combination of admitted settings may be published.
`Verdict`(admitted, score, worst, rounds, reason)	Whether one candidate replaces the built-in value.

solprop.calibration.selection.DROP_ORDER = ('grain', 'fft', 'process')#: The groups dropped, in this order, when a combination fails validation: operation grain, FFT planning, then the waiting policy and the team.

solprop.calibration.selection.MAX_CANDIDATES = 4#: The most candidates a setting is chosen from, so that the search cannot select noise from a large set.

solprop.calibration.selection.NOISE_FLOOR = 0.02#: The least noise a session claims: a win must exceed 2 % in every round.

solprop.calibration.selection.NOISE_LIMIT = 0.1#: A/A noise above which a session is inconclusive and nothing is published.

solprop.calibration.selection.TOLERANCE = 0.05#: How much slower than its noise any one problem may run under a candidate, so that a small loss on one problem does not veto a broad win (the maintainer’s choice; SEP 46 section 7 item 1 asked for 2 %).

type solprop.calibration.selection.Times = Sequence[float | None]#: Seconds per round, None for a round that failed or was skipped.

class solprop.calibration.selection.Validation(passed, windows, geomean, reason)[source]#

Bases: object

Whether a combination of admitted settings may be published.

Attributes:

passedbool: No window slower than 1 + noise + TOLERANCE, and a geometric mean below one.
windowsMapping[str, float]: Each window’s median per-round ratio to the table.
geomeanfloat or None: Their geometric mean; None when a window has too few rounds.
reasonstr: Why it passed or failed.

geomean: float | None#

passed: bool#

reason: str#

record()[source]#

Return the validation as JSON data.

Returns:

dict: The fields as JSON data.

windows: Mapping[str, float]#

class solprop.calibration.selection.Verdict(admitted, score, worst, rounds, reason, windows=<factory>)[source]#

Bases: object

Whether one candidate replaces the built-in value.

Attributes:

admittedbool: Whether it won every round by more than the noise.
scorefloat or None: The median per-round ratio to the built-in; None without enough rounds.
worstfloat or None: The largest per-round ratio.
roundsint: The complete rounds compared.
reasonstr: Why it was or was not admitted.
windowsMapping[str, float]: Each problem’s median per-round ratio, where the problems were timed apart.

admitted: bool#

reason: str#

record()[source]#

Return the verdict as JSON data.

Returns:

dict: The fields as JSON data.

rounds: int#

score: float | None#

windows: Mapping[str, float]#

worst: float | None#

solprop.calibration.selection.admit(candidate, builtin, noise, minimum, windows=None)[source]#

Decide one candidate against the built-in value.

Parameters:

candidate, builtinsequence of float or None: Seconds per round, interleaved.
noisefloat: The session’s A/A noise (noise()).
minimumint: The fewest complete rounds that decide.
windowsMapping[str, sequence of float or None] or None, optional: Each problem’s per-round ratio of the candidate to the built-in value.

Returns:

Verdict: Admitted only if every round’s ratio is below 1 - noise and no problem’s median ratio is above 1 + noise + TOLERANCE.

Examples

>>> admit([0.9, 0.92, 0.91], [1.0, 1.0, 1.0], noise=0.02, minimum=3).admitted
True
>>> admit([0.8, 0.99, 0.8], [1.0, 1.0, 1.0], noise=0.02, minimum=3).reason
'lost or tied in a round: worst ratio 0.990 against 0.980'
>>> admit(
...     [0.8] * 3, [1.0] * 3, 0.02, 3, {"small": [0.5] * 3, "large": [1.3] * 3}
... ).reason
'slower on large: median ratio 1.300 against 1.070'

solprop.calibration.selection.choose(candidates, builtin, noise, minimum, windows=None)[source]#

Choose among a setting’s candidates: the admitted one with the lowest score.

Parameters:

candidatesMapping[str, sequence of float or None]: At most MAX_CANDIDATES candidates’ seconds per round, by name.
builtinsequence of float or None: The built-in value’s, interleaved with them.
noisefloat: The session’s A/A noise.
minimumint: The fewest complete rounds that decide.
windowsMapping[str, Mapping[str, sequence]] or None, optional: Each candidate’s per-problem per-round ratios (admit()).

Returns:

tuple: The chosen name, or None to keep the built-in value; and every candidate’s verdict.

Raises:

ValueError: For more than MAX_CANDIDATES candidates.

solprop.calibration.selection.fallbacks(groups)[source]#

Return the combinations to validate in turn, dropping groups in order.

Parameters:

groupssequence of str: The groups with an admitted setting, among DROP_ORDER.

Returns:

list of tuple of str: Every admitted group, then that set less grain, less grain and FFT, and so on; empty combinations are left out.

Raises:

ValueError: For a group outside DROP_ORDER.

Examples

>>> fallbacks(["process", "grain", "fft"])
[('grain', 'fft', 'process'), ('fft', 'process'), ('process',)]
>>> fallbacks(["process"])
[('process',)]

solprop.calibration.selection.noise(builtin, again, minimum)[source]#

Return the A/A noise of a session, or None with too few rounds.

Parameters:

builtin, againsequence of float or None: The built-in setting timed twice in each round.
minimumint: The fewest complete rounds that make a session.

Returns:

float or None: The largest per-round deviation of the ratio from one, at least NOISE_FLOOR.

Examples

>>> noise([1.0, 1.0, 1.0], [1.01, 0.97, 1.0], minimum=3)
0.03
>>> noise([1.0, 1.0], [1.0, 1.0], minimum=3) is None
True

solprop.calibration.selection.ratios(candidate, builtin)[source]#

Return the per-round ratios of candidate to builtin, dropping failed rounds.

Parameters:

candidate, builtinsequence of float or None: Seconds per round, in round order.

Returns:

list of float: One ratio per round where both were timed.

Examples

>>> ratios([0.9, None, 1.8], [1.0, 1.0, 2.0])
[0.9, 0.9]

solprop.calibration.selection.validate(windows, noise, minimum)[source]#

Validate a combination on fresh rounds of every calibration window.

Parameters:

windowsMapping[str, tuple]: Each window’s seconds per round under the combination and under the table.
noisefloat: The session’s A/A noise.
minimumint: The fewest complete rounds a window needs.

Returns:

Validation: The verdict, with each window’s median ratio.

Examples

>>> validate({"a": ([0.9] * 3, [1.0] * 3)}, noise=0.02, minimum=3).passed
True
>>> validate(
...     {"a": ([0.8] * 3, [1.0] * 3), "b": ([1.08] * 3, [1.0] * 3)},
...     noise=0.02,
...     minimum=3,
... ).reason
'b is slower: 1.080 against 1.070'