Separating sources of variance

Status: Method

Describes how measurements are taken. Asserts no result.

Three things move the number

A measured rate from one run can move for at least three distinct reasons, and they are not interchangeable:

• Construction — which items were generated this time.
• Policy sampling — the model’s stochastic choice among continuations.
• Execution — the state of the serving stack on this particular occasion.

Running the same configuration repeatedly does not separate these. It varies some of them and holds others fixed, in a combination that depends on details of the runtime that are not visible from the outside.

Crossed, not sequential

Construction and sampling are therefore varied independently, in a grid: every construction is run under every sampling seed. The variance of the resulting table decomposes into a component for each factor plus their interaction, and the components can be estimated rather than assumed.

The decomposition is done on the rate scale with the binomial contribution of finite items subtracted explicitly. Without that subtraction, a component absorbs ordinary counting noise and looks larger than it is.

The resolution floor

Every estimate has noise of its own. An estimated variance component that is small is not evidence that the true component is zero — it may be below what a grid of that size can resolve.

The floor is established by simulation: generate data with the component truly set to zero, run the same estimator, and take a high percentile of what comes out. That percentile is the smallest value the design can distinguish from zero. Components below it are reported as unresolved, not as absent.

Grid size follows from this rather than from convenience. A grid is chosen large enough that the components of interest sit above their own floors with adequate probability; designs that cannot are not run, because their most likely output is a clean-looking zero that means nothing.

Confirmation runs use separate data

Data used to choose the design — grid size, which cells to replicate, how the analysis will be interpreted — is not reused for confirmation. If those decisions were made by looking at a set of cells, that set has already been spent, and a confirmatory analysis over it is circular no matter how it is written up.

The cost is a repeated run. The alternative is a result that cannot be defended, which is not cheaper.

Replicates

A fraction of cells is run more than once with everything held fixed. This is what makes the execution component visible: if repeats agree exactly, repeated execution under those conditions contributes nothing, and any cell-specific noise would otherwise have been silently absorbed into the interaction term and reported as a real interaction.