The paired design

The problem this solves

Suppose you want to know whether a model behaves differently when a task is framed as its own problem rather than someone else’s. The obvious approach is to run a set of tasks under one framing, run another set under the other, and compare accuracy.

This does not work, and it fails for a reason that is easy to underestimate. Accuracy on any finite set of generated items is dominated by how hard those particular items turned out to be. The variation between two draws of material is routinely larger than any effect of framing. A comparison between two independently drawn sets measures mostly the difference between the sets.

The design

Each item is presented under both conditions. The formal content is held identical; only the framing differs. The comparison is made within the pair.

Item difficulty is a property of the item, so it enters both members of the pair equally and cancels in the difference. What remains is attributable to the manipulation and to the noise sources documented separately.

The answer key

Correctness is judged against a key fixed before the run. This is a constraint on what can be studied: only tasks with a defensible fixed answer are eligible. Open-ended reasoning is out of scope for this design, not because it is uninteresting but because grading it would introduce a judge whose own behaviour would then need the same scrutiny as the subject.

Ceiling effects

A condition on which the model answers everything correctly cannot show variation, and cannot be compared meaningfully with one that has room to move. Working points are therefore chosen so that both conditions sit away from the ceiling.

This turns out to be model-specific in an inconvenient way: a difficulty setting that keeps one model off the ceiling can put another one on it, and the harder condition is not necessarily the same one across models. A single shared configuration for cross-model comparison may not exist, and where it does not, cross-model comparison is not reported.