Validation & known limitations
A discrete-event simulator has no compiler for its semantics. A wrong queue discipline, a wrong
warm-up truncation, or a mis-modelled maintenance policy does not crash — it shifts a KPI by a few
percent and looks exactly like a result. Coverage percentage cannot catch that, so clockwright's
test suite is built the other way round: against mathematics first, then against models other
people have already published numbers for.
Forty-nine checks in ten families. Each one runs the engine and compares a quantity it
produced against a value that follows from a closed-form result, an exact algorithm (mean value
analysis, an exactly solved Markov chain) or an invariance a theorem guarantees. Every row below
is one test in the suite, and the gate refuses a row whose test is missing or switched off.
Twenty-one of them compare through one shared function, agrees, whose bound is a Student-t bound
from the replications' own spread and which fails rather than passes when the runs are too
short to resolve the error it is asked about. The others compare against a confidence interval,
a standard-error bound, a tolerance stated in the test, or an exact equality.
⚠️ What is not counted, and why. A test counts once however many formulas it checks. Not
counted: refusals, regression and byte-identity tests, comparisons between two runs of the engine,
tests of a sampler or of the statistics layer on their own, a probability that is only the input
read back, deterministic arithmetic (a duty cycle d/T, a cost summed along a fixed sequence),
flow identities of the form rate × mean duration (utilisation equals offered load, L = λ·E[T]
on a transport leg), a search that lands on the right decision rather than a measured value,
and approximations. Several of those are
good tests; none is a check against exact mathematics in the sense above.
Single queues
| Result | What it checks |
|---|
| M/M/1 | L, W and Wq on the engine core |
| M/M/1 | W = 1/(μ−λ) through the modelling API |
| M/M/1 | W = 1/(μ−λ) for a model written as a document |
| Erlang C | Wq, W and L of M/M/3 on the engine core |
| Erlang C | W and L of M/M/3 through the modelling API |
| Erlang C | A pool of operators that is the bottleneck queues as M/M/c |
| M/M/1 | W = 1/(μ−λ) for a model built in Python and run by the binary |
| Pollaczek–Khinchine | Wq of M/D/1 |
| Pollaczek–Khinchine | Wq over five service distributions of the same mean: waiting is linear in E[S²] |
| Pollaczek–Khinchine | A walk is part of the pool's service, and the wait sees its variance |
E[Wq²]_LIFO = E[Wq²]_FCFS / (1−ρ) | The discipline moves the second moment of the wait and not its mean |
Arrivals and batching
| Result | What it checks |
|---|
M^X/M/1 | Batch arrivals of a fixed size enter as distinct entities at one instant |
E_K/M/1 | A quorum batch forms on every K-th Poisson arrival |
Finite queues, impatience and blocking
| Result | What it checks |
|---|
| M/M/1/K | A bounded queue, including the uniform distribution at ρ = 1 |
| M/M/1+M | Abandonment, pinned through the abandonment probability |
| Burke and Erlang C | Transport is an infinite-server delay: cycle time and the downstream queue against Erlang C |
Saturated tandem (B+2)/(B+3) | Blocking throughput over a sweep of buffer sizes |
| Exact Markov chain | A four-station blocked line with interior stations |
Priorities and disciplines
| Result | What it checks |
|---|
| Cobham | Non-preemptive priority, with the residual summed over every class |
| Preemptive-resume priority | The top class is blind to the load of the bottom one |
| Cobham, preemptive form, Kleinrock | Four classes, every one asserted |
| Kleinrock's conservation law | Σ ρ_k Wq_k is the same under every non-preemptive work-conserving rule |
Networks
| Result | What it checks |
|---|
| Jackson / BCMP product form | A re-entrant route visiting one station twice |
| BCMP product form | A five-rung ladder of composition: classes, re-entry, stations, an infinite-server delay |
| Burke's theorem | The second station of a tandem is indifferent to the first one's service rate |
| Mean value analysis | A closed network over several population sizes |
| Multi-class mean value analysis | A closed network with several classes |
| Exact Markov chain | A composed model outside BCMP: blocking, priority and breakdowns at once |
Breakdowns, availability and repair
| Result | What it checks |
|---|
| Calendar-breakdown WIP | WIP under calendar-driven failures, and that the naive derated model is distinguishably wrong |
| Gaver | WIP under operation-dependent failures, at parameters where the calendar model would fail it |
| Alternating renewal | Availability MTTF/(MTTF+MTTR) |
| Alternating renewal | The same availability for every distribution family of the same means |
| Machine-repairman chain | Availability and crew utilisation when repairs wait for a technician |
Rework, scrap and timers
| Result | What it checks |
|---|
M/M/1 sojourn tail e^{−(μ−λ)t} | The share of lots that break a queue timer |
| Wald, Blackwell–Girshick | Rework passes are geometric, isolated with constant service |
| Blackwell–Girshick | Both variance terms live: passes draw independently |
Independent thinning 1 − 0.95³ | Three 5% scrap steps destroy 14.26% of wafers, not 5% |
Independent thinning 1 − 0.95³ | The same 14.26% whether scrap is spread over three steps or taken at one |
1 − (1 − s)^N | A lot survives if any of its wafers does, which is not the per-wafer rate |
Inventory
| Result | What it checks |
|---|
| M/M/1 shortfall | Stockout probability ρ^(r+1), swept over the reorder point |
| M/M/1 shortfall | The mean on-hand level |
| Pollaczek–Khinchine | The wait for stock doubles with the variance of the lead time |
| Palm's theorem | Orders outstanding are Poisson whatever the lead time's shape |
First passage and order books
| Result | What it checks |
|---|
M/M/1 first passage n/(μ−λ) | The first time a queue of n empties |
| Exactly solved lattice chain | Two queues racing to empty, as in a limit order book |
| M/G/∞ is Poisson | The depth is Poisson; deterministic releases keep the mean and lose the law |
Welfare and control
| Result | What it checks |
|---|
| Naor | The engine's welfare-optimal queue bound is strictly below the selfish one |
| Naor | A search over the queue bound finds the same optimum |
| Exact Markov chain | Capacity that reacts to the queue, with a call-up delay |
Asserted on every run, not counted above
| Identity | Where |
|---|
| Little's law, conservation, the state partition | Asserted inside every model's own outcome computation, on every run — not only in the test suite. |
Test beds
Beyond closed-form mathematics, clockwright is checked against models whose numbers have been
independently published:
| Test bed | What it is | Where the engine stands |
|---|
| SMT2020 | A full semiconductor-fab benchmark dataset: 1,043 tools across 105 tool groups, 10 products, routes of 242–583 re-entrant steps. | Reproduces the published reference's own tool-hours of processing per lot to within 0.1%, and the dataset's own workload arithmetic on 100 of its 106 groups (the delay station among them). Mean cycle time within 1.4% (HVLM) and 0.8% (LVHM), four replications — see below. |
| Intel Minifab (MIMAC) | The classic 5-machine, 3-product testbed from the 1994 fixed-width dataset archive. | Loaded, cross-checked across its five files (which disagree with each other about naming in several places), and validated in strict mode. |
| PHOLD | The standard discrete-event-simulation stress benchmark. | Used for throughput measurement rather than correctness — see Performance for the caveats that apply to any such number. |
Determinism, checkpointing, and the chaos test
Closed-form targets and test beds check that the engine computes the right answer. A separate set
of checks exists to verify something different: that the engine computes the same answer every
time it is asked the same question, including across a crash.
- Five determinism properties are asserted directly, not just assumed: the same seed always
reproduces the same result; replications are independent of one another; a structural change to
the model that shouldn't affect a given entity's random draws doesn't silently perturb them; a
run driven straight through and the same run driven in checkpointed chunks agree regardless of
chunk size; and declarations that shouldn't affect behaviour are actually inert.
- ⭐ The chaos test. A running replication is
SIGKILLed at six random points, resumed from
its checkpoints each time, and the final result is required to be bit-identical to a run that
was never interrupted at all — same trace hash, same event count, every KPI equal. It runs on
every change to the engine, not as a one-off demonstration. See
Checkpointing for what recover and fork mean and
Architecture for how a checkpoint is actually
taken.
SMT2020 cycle time
For a long time this was the largest honest gap in the project, and it is now closed to within
2% on both of SMT2020's configurations. Measured 2026-10-09 (#14; one warm-up year, one reported
year, the mean of four replications):
| published reference | clockwright |
|---|
| HVLM mean cycle time | 39.71 days | 40.26 (+1.4%) |
| HVLM median flow factor | 1.85 (paper) | 1.865 (+0.8%) |
| LVHM mean cycle time | 37.39 days | 37.71 (+0.8%) |
| LVHM median flow factor | 1.95 (paper) | 1.988 (+2.0%) |
Both runs release exactly the lots the order file declares (20,877 and 20,879 a year, ratio
1.0000), and our own decomposition of a lot's cycle time — at a tool, queued, waiting for a batch,
in transit — adds up to the reported cycle time within 0.2% (rung 2.5 of
scripts/smt2020-validate.py).
What closed it was not a dispatch rule. Three investigations refuted eight candidates, and
the excess turned out to be a queue level at tool groups exactly as busy as the reference's.
⭐ The cause was which free tool takes a lot. Picking the lowest-numbered one loaded identical
tools unevenly: PROC% 86 down to 70 across ten tools where the reference reads 75–76 on
every one. That unevenness had also been staggering usage-counted maintenance by accident. The
dataset declares that stagger, as a first occurrence in pieces, and it is now modelled
(first_at_units). The tool that started a lot longest ago now takes
the next one. On HVLM, sharing the work evenly without the declared stagger read 48.8 days; with
it, 40.3.
⭐ The shape that cannot be fitted reproduces too. The paper reports that on LVHM the upper
flow-factor percentiles rise again as the fab is unloaded — lots wait longer for a batch of
their own product to form — and that HVLM does not, having five times as many lots of each kind.
Over eight simulated years per point, two replications each, regular lots only
(scripts/smt2020-operating-curve.py, measured 2026-10-10 at the tool-choice change), going
from the minimum of the curve down to 160 lots a week LVHM's 75th and 95th percentiles rise by
+0.13 to +0.18, and HVLM's by +0.000 to +0.001.
⚠️ What still misses: how OFTEN a tool changes setup. The dataset's own documentation states
the rule in two sentences — a freed tool takes the waiting lot by priority, then least setup time,
then first-in-first-out; an arriving lot wakes the idle tool with the least setup time — and the
engine is tested against each sentence, each test shown to fail when its rule is broken. Under that
rule the reference still changes setup more
often than we do, on all nine groups that carry one (HVLM, four replications, changeovers per visit
to a step with a setup):
| tool groups | ours | reference | SETUP% ours / reference |
|---|
DE_BE_13, DE_BE_66 | 0.16 | 0.34–0.37 | 1.2–1.3 / 2.4–2.9 |
LithoTrack_FE_95, _115 | 0.52–0.61 | 0.94 | 3.4–7.1 / 6.2–10.8 |
Implant_90, _119 | 0.22 | 0.43–0.47 | 6.4–6.9 / 6.9–7.2 |
Implant_91, _128, _132 | 0.06 | 0.39–0.47 | 9.8–13.8 / 11.4–15.7 |
On the Implant groups the reference changes setup up to seven times as often and books a fraction
of the declared time per change, so SETUP% nearly agrees while the count does not — the
reference there does not follow its own declared durations, in either published format. On
DE_BE and LithoTrack the declared durations agree with the reference's, and the count does not.
Nothing in the published data explains it, so it is reported here rather than fitted: adjusting
setup avoidance to reproduce the reference's counts would be fitting its output, not modelling
its plant. The cycle time above is reached with the documented rule.
An earlier comparison, kept for its shape (LVHM, measured 2026-09-12 at 3741f18, before any of
this):
| published reference | clockwright then | independent open-source implementation (PySCFabSim, MIT-licensed) |
|---|
| tool-hours of processing | 322.44 | 322.12 (−0.1%) | 315.24 (−2.2%) |
| queue-days | 16.24 | 28.06 | 26.23 |
| median flow factor | 1.95 | 2.59 (+32.6%) | 2.39 (+22.3%) |
Other open items, stated plainly
- Interacting mechanisms are under-tested relative to individual ones. Every closed-form
target above exercises at most a handful of resources at once; a fab-scale model exercises
interactions between over a hundred tool groups simultaneously, and there is currently no
closed-form check that spans that scale.
- Warm-up truncation under-truncates on at least one simple model. The automatic (MSER-5)
truncation described in Experiments has a known case where it discards
less of the initial transient than it should on the very simplest models — worth watching for on
models with an unusually short natural cycle time.
What this means for you
Treat clockwright as validated for relative comparisons between structurally similar variants
of the same model — which is the property its whole design optimises for (see
Introduction) — and treat any absolute cycle-time figure on a large, highly
re-entrant, heavily dispatch-rule-driven model (a real semiconductor fab being the sharpest
current example) as carrying a known, measured overstatement relative to the published reference
case above — 11% too long on one of its configurations and 3% on the other — until this item closes. Simpler models — those closer in shape to
the closed-form targets above — do not carry this caveat; it is specific to large-scale,
dispatch-rule-dominated queueing of exactly the kind SMT2020 exercises.