cwright compare

cwright compare <baseline> <candidate> --kpi NAME
            [--replications N] [--indifference X] [--higher-is-better] [--out FILE]

Runs a baseline and a candidate model the same number of times, pairs them replication-for- replication, and returns a decision on one named KPI. See Experiments for how the pairing and the four possible verdicts work.

Flags

Flag	Default	Meaning
--kpi NAME	(required)	Which KPI to compare — e.g. cycle_seconds, completed, released, utilisation_pct.<resource>, queue.<resource>, cycle_seconds.<product>, money. ⭐ Every name cwright run reports is a name this accepts; a test asserts that on the artefact rather than on two lists.
--replications N	10	Replications for each side.
--indifference X	(required)	Differences smaller than this are not worth acting on. No default — see below.
--higher-is-better	off	By default a lower KPI value is treated as better (cycle time, cost, queueing). Pass this flag to invert that for KPIs where more is better (throughput).
--out FILE	stdout	Where to write the comparison document.

--indifference has no default, deliberately

compare needs --indifference X. There is deliberately no default: a difference smaller than it
is not worth acting on, and only the domain knows what that is.

Guessing a value here would turn a judgement about the domain into a silent default that happens to be wrong for most models. See Experiments for why a threshold copied between models — even similar-looking ones — produces confident, wrong verdicts rather than an error.

Output

A JSON document containing both full result documents (baseline, candidate), the interval on their difference, the verdict (Better, Worse, NoMeaningfulDifference, or InsufficientEvidence), a next_step describing what would resolve an inconclusive comparison, the pairing_efficiency achieved by common random numbers, and whether the two runs were actually paired (used the same replication ids). See The results document for the full field list.

When the verdict is InsufficientEvidence, a note is also printed to stderr:

note: the comparison could not conclude. run about 40 replications

or, when more replications genuinely will not help:

note: the comparison could not conclude. more replications will not help (~140000 would be
needed). Reduce variance or widen --indifference; the threshold may be finer than the model can
resolve.

Example

cwright compare baseline.yaml candidate.json \
    --kpi cycle_seconds --indifference 3600 --replications 12

Comparing money

money appears only when the model has a charges block, and it is a net cost — a negative price is revenue, so the total is costs minus earnings. ⚠️ Lower is therefore better, which is this command's default: do not pass --higher-is-better for it.

The case the charges feature exists to expose is a policy that moves the two in opposite directions — cycle time down, net cost up — because that is the one a throughput comparison cannot settle on its own.

Check first whether your model can show you that

⭐ It can only do so if the priced quantities do not all track throughput. On tests/corpus/job-board.json, revenue is 2.8× total cost, so money is very nearly −45 × completed: across 12 replications the correlation is −0.997, and the obvious cost lever — a third technician — moves money by +0.16 on a total of −14,772, reported as NoMeaningfulDifference while cycle time moves too. Two variants measured the same way both came back Better on both KPIs (a second test bay: −46.22% cycle, −0.80% money; a 20% faster bench: −6.22%, −11.05%).

⚠️ So a money verdict that always agrees with completed is telling you about your charges block, not about your policy: the capacity you are adding is not priced. Price the resource the policy actually buys — a server's state.Processing, or the operator hours behind it — and the two can then disagree.

⭐ Passing --higher-is-better with --kpi money prints a note rather than refusing: it is right only if the model prices profit instead of cost. charges explains why that cannot be checked from the document.