Skip to main content

Experiments and comparisons

Running a model once tells you what happened once. clockwright's experiment machinery exists to turn a handful of runs into a defensible answer to "what does this model actually do, on average, and how sure are we?" — and, from there, into a defensible answer to "is this change worth making?"

Replications​

cwright run (see run) executes the model N independent times — --replications, minimum 2, since a single run cannot bound anything. Each replication draws its own random numbers (see Determinism) but is otherwise the same model, so the spread across replications is a direct measurement of how much a KPI varies for reasons that have nothing to do with your model being wrong.

Warm-up truncation​

A simulation that starts empty is not yet in steady state, and averaging from time zero mixes a transient into a number meant to describe ongoing behaviour. warmup_seconds in the model document is a hard boundary: every statistic in the engine is reset at that mark.

⚠️ MSER-5 is an additional truncation, and it covers one KPI. cwright run applies it to the pooled cycle-time series only, and then identically to every replication. Utilisation, state percentages, queue lengths, consumable levels and operator figures are truncated by warmup_seconds and by nothing else — and cwright simulate does not apply MSER-5 at all. Setting warmup_seconds to zero because "the engine chooses the real truncation" measures the transient in everything except cycle time.

⚠️ That last point is deliberate and easy to get backwards. Choosing a different truncation point per replication would make the replications no longer identically distributed, which silently invalidates the confidence interval computed from them afterwards — and the resulting interval still looks entirely reasonable, just narrower than it should be, which is the more dangerous failure. clockwright always determines one common truncation point across the whole batch and reports what it discarded (warmup_discarded in the result document) so a wrong warm-up choice is visible rather than hidden inside an average.

Confidence intervals, never a bare number​

Every KPI in a result document is an interval, not a point estimate: a point, a low and high bound, a half-width, a relative precision, and the replication count behind it (see The results document for the exact field names). A number without an attached uncertainty invites treating noise as a finding — the whole reason to run more than one replication in the first place.

Comparing two models​

cwright compare (and cwright propose, which applies a change and then calls compare on the result) runs a baseline and a candidate the same number of times, pairs them replication-for-replication — replication k of the baseline against replication k of the candidate, which share the same random draws wherever the two models are structurally the same — and reports the interval on the difference, not two separate absolute estimates. Pairing this way (common random numbers) can sharply reduce the noise in the comparison; compare reports how much it actually helped as pairing_efficiency, because CRN is capable of inflating variance in some cases and "we used CRN" is not, by itself, evidence that it worked here.

The four verdicts​

A comparison concludes with exactly one of:

VerdictMeaning
BetterThe candidate improves the KPI by more than the indifference threshold, with enough confidence to say so.
WorseSame, in the other direction.
No meaningful differenceThe difference is real or not, but it is smaller than the indifference threshold either way — not worth acting on.
Insufficient evidenceThe interval on the difference is too wide to place it on either side of the threshold.

⚠️ Insufficient evidence is not the same as no meaningful difference, and treating them as interchangeable is how a search loop convinces itself that a promising direction did nothing. When the verdict is inconclusive, clockwright also reports what would resolve it: a specific number of additional replications if that would plausibly settle it, or an honest "more replications will not help" if the indifference threshold is finer than the model's variance can ever resolve — in which case the real fix is reducing variance (longer runs, more common random numbers) or widening the threshold, not running the same experiment for longer.

⚠️ "longer runs" has a ceiling, and it is not gradual

Cycle times are held in a bounded buffer of 65,536 observations (results). Once a run passes that, one observation in every two is dropped and the retained count stops growing — so a longer horizon buys no precision at all, and can measurably make the spread worse. ⭐ Above the cap, replications are the only lever that does not change what is being measured. horizon × arrival rate against 65,536 tells you which case you are in, and needs no experiment.

The indifference threshold has no default, on purpose​

--indifference is a required flag on both compare and propose, and clockwright refuses to guess one. A difference smaller than the threshold is defined to be not worth acting on — and only someone who understands the domain the model represents can say what that is. A cycle-time improvement of two minutes might be decisive on a model whose whole cycle time is ten minutes, and meaningless on one whose cycle time is measured in days.

⚠️ The threshold must be re-derived for every model, never copied from another one — even a similar one. An indifference threshold sized for a large factory model, applied unchanged to a small test model with a much shorter cycle time, does not fail loudly: it produces confident, correctly-computed verdicts that are simply asking the wrong question. A real improvement can be reported as "no meaningful difference" because the threshold was calibrated for a model twenty times its scale, and a genuine regression can sit permanently unresolved as "insufficient evidence" because the interval straddles a threshold that was never really about this model at all. Set it as a fraction of the baseline's own scale, or reason about it fresh for the model in front of you — never inherit an absolute number from somewhere else.

Next​