Definition
Statistical Validity is a quantitative validation technique that uses statistical tests to compare the output data of a simulation model against historical data from the real system.
Why It Matters
Statistical validity is the ‘credibility check’ for any simulation; it provides the mathematical proof that a model’s output matches the real-world history it represents, protecting decision-makers from acting on ‘valid-looking’ but incorrect data.
Core Concepts
- Measures of Performance (MOPs): Identifying the critical variables to compare (e.g., throughput, average wait time, resource utilization).
- Hypothesis Testing:
- Null Hypothesis (): There is no significant difference between the simulation output and the historical data.
- How to read: “H-naught.”
- Meaning: The null hypothesis—the default assumption that the model matches reality. You seek evidence strong enough to reject and conclude the simulation differs significantly from historical data.
- Null Hypothesis (): There is no significant difference between the simulation output and the historical data.
- Common Tests:
- t-test: Comparing the means of two populations.
- Chi-Square / KS: Comparing the distributions of output data.
- Confidence Intervals: Determining if the actual system mean falls within the simulation’s confidence interval.
- The 60/40 Rule: In some industrial contexts, being within 5-10% of reality is considered “statistically valid” for decision-making.