What is being measured

The unit of study is a matched pair of tasks, identical in content and different only in framing. Everything else in the design exists to make the comparison within that pair trustworthy.

The paired design

Each item is presented twice: once as a problem the system is working on for itself, once as a problem posed by someone else. The formal content is held constant. The outcome measure is whether the response is correct under an answer key fixed in advance of the run.

Pairing matters because absolute performance on any such task is dominated by how hard the item happens to be. Within a pair, item difficulty cancels, and what remains is attributable to framing.

Three sources of variation, held apart

A single number from a single run tells you almost nothing, because at least three different things can move it. The design varies them independently so their contributions can be estimated separately rather than blamed on each other.

Construction

Which concrete items were generated for this run. Different draws of material produce different difficulty, and this is often mistaken for an effect of the manipulation.

Policy sampling

The model's own stochastic choice among continuations. Two runs on identical material can differ for this reason alone.

Execution

The environment the model actually ran in — the serving stack, how weights were loaded, how work was batched. This is the component most often left unmeasured, and the one most likely to be mistaken for a property of the model.

Estimating these separately requires a crossed design rather than a sequence of runs, and a grid large enough that a component can be distinguished from the noise of its own estimate. Both requirements cost machine time, and both are load-bearing: a grid too small to resolve a component will report zero and look like a clean result.

What counts as a result

A claim is eligible for publication when the decision rule was fixed before the data was seen, the effect is larger than the resolution floor of the estimate that produced it, and an independent rerun reproduces it. Exploratory analysis is reported as exploratory, and never as confirmation on the same data that suggested it.

Claims that fail a rerun are withdrawn. The documentation records the withdrawal, what was originally observed, and what is currently unknown — leaving a retracted claim in place with softer wording would be the more comfortable option and the dishonest one.

Read the methods documentation