Known limitations

Scope of models

Experiments run on local hardware with open-weight models. Frontier hosted models are out of reach, both because of cost and because their serving environment cannot be held fixed or even observed. Nothing measured here should be assumed to transfer to them.

Scope of tasks

Only tasks with a fixed answer key are eligible. This excludes most of what makes reasoning interesting — open-ended argument, planning under ambiguity, anything where the quality of an answer is a matter of degree.

One machine, one stack

The execution component is measured on a single machine with a single serving stack. A component that is zero here may not be zero elsewhere, and the estimate carries no information about other hardware.

Statistical scope

The grid sizes that are affordable set a floor on what can be resolved. Effects below that floor are invisible to this design. Where a component is reported as unresolved, that is a statement about the design’s power, not about nature.

Framing is one manipulation among many

The self-directed and externally framed conditions are two specific phrasings. A difference between them is a difference between those phrasings, and attributing it to the concept of self-directedness in general requires more phrasings than have been tested.

Wording of the answer options has already been observed to interact with model identity — the same rephrasing that makes a condition harder for one model can make it easier for another. Any result resting on a single phrasing should be read as provisional.

Reproducibility across sessions

Runs taken in one continuous session and runs taken days apart are not interchangeable. Whether the instrument reproduces across sessions is treated as an open question and is measured directly rather than assumed; see withdrawn claims for what happened when it was assumed.