Search

When a simulation operates over an array of input settings, it is critical that its validity across all settings is considered. This paper proposes an F-statistic, based on a ratio of mean square errors, to assess model validity. System data is fit to an Ordinary Least Squares response surface and the mean square errors between both system and model data and the fitted surface are calculated. The hypothesis under test is that system and model data that possess the same distributional properties will produce similar measures of disagreement with regard to the fitted surface. If the F-statistic indicates the mean square errors are statistically different, it is determined that the system and model data do not share distributional properties and the model is assessed as invalid. A notional example demonstrates the methodology and considerations to the method's confidence with respect to the model sample size is discussed.

As we continue to model more complex systems, the validation of dynamical responses has come to the forefront of modeling and simulation. One form of dynamic response is when the output is a function of time. The proper evaluation of functional data over an array of desired input parameters is critical to achieving a robust validation assessment of a simulation model. We extend the correlation analysis (CORA) objective rating system to validate functional data across experimental regions. Functional regression analysis is used to generate surrogate estimations of the system response functions at points within the region where experimental observations are absent. These CORA scores provide a measure of disagreement at each desired parameter configuration. An overall score for model validity is achieved using a weighted linear combination of the individual CORA scores. Finally, an improved CORA size scoring metric is introduced.