Skip to main content

The results document

Every command that runs a model produces a JSON document carrying the recipe as well as the answer — a number with no engine version, model hash, and seed attached to it is not reproducible, and a verdict that cannot be reproduced is not a verdict.

The run report (run, and each side of compare)​

{
  "manifest": {
    "model_name": "inspection-booth",
    "model_hash": "a258a8d2af4acd3a",
    "engine_version": "0.0.0",
    "rustc_version": "1.97.1",
    "build_target": "x86_64-unknown-linux-gnu",
    "schema_version": 1,
    "replications": 12,
    "replication_ids": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
  },
  "kpis": {
    "cycle_seconds": {
      "point": 312.4,
      "lo": 298.1,
      "hi": 326.7,
      "half_width": 14.3,
      "relative_precision": 0.0458,
      "replications": 12
    },
    "completed": { "point": 2016.0, "lo": 1987.2, "hi": 2044.8, "half_width": 28.8, "relative_precision": 0.0143, "replications": 12 },
    "utilisation_pct.booth": { "point": 71.2, "lo": 68.9, "hi": 73.5, "half_width": 2.3, "relative_precision": 0.0323, "replications": 12 }
  },
  "warmup_discarded": [15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15],
  "trace_hashes": ["a91c...", "7bd0...", "..."],
  "events_total": 1048203,
  "wall_seconds": 0.986,
  "seconds_per_replication": 0.0822
}
  • manifest identifies exactly what produced this result. ⚠️ rustc_version and build_target are recorded because bit-identical determinism is only promised on a fixed toolchain — see Determinism — so a result carried across a toolchain change is a result whose comparability nobody has checked. replication_ids matters for comparisons: two results sharing the same ids are paired, which is what makes a later compare between them legitimate.
  • kpis is a map from KPI name to an interval — never a bare number. Every entry has a point estimate, a lo/hi bound, a half_width, a relative_precision (the half-width as a fraction of the point estimate — the usual stopping criterion for "have I run enough replications"), and the replication count behind it. A KPI whose true value is zero has infinite relative precision by construction (there is nothing to be a fraction of) — this is normal and expected for a count that never happened, not a formatting error.
  • Beyond cycle_seconds, completed, and released, the map includes one entry per resource (utilisation_pct.<id>, availability_pct.<id>, queue.<id>), per product (cycle_seconds.<product>), and per consumable (level.<id>, stockouts.<id>, stockout_wait_s.<id>) that the model declares.
  • warmup_discarded is what MSER-5 actually threw away, per replication — always one shared value across every replication (see Experiments), reported per replication anyway so a wrong warm-up choice is visible in the document rather than hidden inside an average.
  • trace_hashes — one per replication — is what makes two runs' agreement checkable rather than assumed: two runs with the same hash executed the same events, in the same order, with the same outcome.
  • seconds_per_replication is reported because optimal sample allocation across several candidate models is proportional to sigma / sqrt(cost), not to sigma alone — comparing variants that differ in how expensive they are to simulate without this figure silently costs a constant factor.

The comparison document (compare, propose)​

Wraps a full run report for baseline and one for candidate, plus:

{
  "kpi": "cycle_seconds",
  "difference": { "point": -41.2, "lo": -58.0, "hi": -24.4, "half_width": 16.8, "relative_precision": 0.4078, "replications": 12 },
  "verdict": "Better",
  "next_step": "none -- the comparison concluded",
  "indifference": 3600.0,
  "lower_is_better": true,
  "pairing_efficiency": 4.7,
  "paired": true,
  "baseline": { "...": "a run report" },
  "candidate": { "...": "a run report" }
}
  • difference is the interval on the paired difference (candidate minus baseline) — the quantity the verdict is actually computed from, not a difference of two separately-reported points.
  • verdict is one of Better, Worse, NoMeaningfulDifference, or InsufficientEvidence — see Experiments for what each one means and why there are four rather than two.
  • next_step describes what would resolve an inconclusive comparison: a specific number of further replications, or an honest statement that more replications will not help.
  • pairing_efficiency is the variance reduction actually achieved by running both arms with the same random draws (common random numbers), as a ratio of unpaired to paired variance — 1.0 means pairing achieved nothing, and values are capped at 1,000,000 to avoid reporting spurious precision when the two arms are nearly perfectly correlated.
  • paired is true only when baseline and candidate used the same replication ids; a comparison can still be valid without this, just considerably less precise.

Flow records (run --flow DIR, read by cwright trace)​

--flow DIR on run writes one zstd-compressed Parquet file per replication — one row per entity per step:

columntypemeaning
entityuint32The entity (lot) this row is about.
productuint16Which product's route it is following, as an index.
stepuint16Which step of that route, as an index. ⚠️ A rework excursion is walked as steps appended after the declared route, so an index at or past products[].steps.length is an excursion step rather than a step you can find in the document.
resourceuint16Which resource (tool group) performed it, as an index.
serveruint16Which individual server within that group — not just the group, so a per-equipment question can be answered without falling back to an aggregate.
visituint8Which visit this is, for a re-entrant route that returns to the same step more than once.
t_arriveint64 (ticks)When the entity arrived at this step's queue.
t_startint64 (ticks)When service actually began.
t_endint64 (ticks)When service finished.
setup_ticksint64Time spent on a changeover immediately before this step's service.
queue_ticksint64Time spent waiting (derived from t_arrive/t_start).
process_ticksint64Time spent actually being processed (derived from t_start/t_end, minus setup).

Times are in ticks, not seconds — divide by the model's tick_scale_per_second to convert. Rows are sorted by t_end, then entity, then step; row groups are 1,000,000 rows, which is a deliberate floor rather than a tunable knob — groups under roughly 5,000 rows make queries 5–10× slower, since a reader pays per-group metadata for almost no data in return.

Reading it​

The whole point of Parquet here is that nothing bespoke is needed to read it:

import pandas as pd
df = pd.read_parquet("flow/r0.parquet")
df["queue_seconds"] = df["queue_ticks"] / 1000  # tick_scale_per_second from the model
df.groupby("resource")["queue_seconds"].mean()
import polars as pl
pl.scan_parquet("flow/*.parquet").group_by("resource").agg(
    pl.col("queue_ticks").mean()
).collect()
-- DuckDB, across every replication at once
SELECT resource, server, avg(queue_ticks) AS mean_queue_ticks
FROM read_parquet('flow/*.parquet')
GROUP BY resource, server
ORDER BY mean_queue_ticks DESC;

A time-window query (everything between two ticks) prunes row groups efficiently, because the file is sorted on t_end; filtering by entity or server alone will scan the whole file — a known and accepted trade rather than an oversight, since time-window queries are the common case.