PHYSICS-GUIDED ANALYTICAL MODEL VALIDATION

Abstract

This invention relates to a parameter or response assist filter that ensures that the predictions of a post-validation calibrated physics system simulator will remain within boundaries of a predetermined model validation domain. Embodiments utilize one or more filters to ensure calibrated model parameters {acute over (P)} and/or calibrated responses {tilde over (ϕ)} cause physics simulator model predictions to remain within the boundaries of the model validation domain MVD for a target application. The filters can be constructed prior to use or automatically inferred, or otherwise determined, from available measurements and other renditions of the physics system simulator during operation.

Claims

1. A method for post-validation adjustment of physics system simulators configured to predict behavior and/or state of a physical system based on an application model (M.sub.A) and multiple physical or adhoc parameters (P), denoted as model parameters, and their corresponding known parameter variations (ΔP), wherein the application model (M.sub.A) is related to one or more scaled-down experimental models (M.sub.E.sub.1, M.sub.E.sub.2, . . . ), each scaled-down experimental model (M.sub.E.sub.j) being associated with a respective set of experimental measurements (φ.sub.E.sub.j), where j=1, 2, . . . , wherein the physics system simulator is validated for a target application model (M.sub.A), as described by a model validation domain (MVD) the boundaries of which are evaluated mathematically based on deterministic or stochastic multi-variate functions of the target application model's responses, a set of scaled-down experimental-models' responses, the corresponding sets of experimental measurements (φ.sub.E.sub.1, φ.sub.E.sub.2, . . . ), derivatives thereof, and the model parameters (P), the corresponding parameter variations (ΔP), and an uncertainty estimator, the method comprising: predicting, by a first implementation (810) of a physics system simulator, first experimental responses (Φ.sub.E.sub.1, Φ.sub.E.sub.2, . . . ) of the physical system by modeling the physical system using the scaled-down experimental models (M.sub.E.sub.1, M.sub.E.sub.2, . . . ) based on the model parameters (P) and their corresponding parameter variations (ΔP); filtering, by a validation assist parameter filter (820) having an MVD boundary filter operator, the parameter variations (ΔP) corresponding to variations in responses for experimental models (ΔΦ.sub.E) that cause the predictions of a second physics system simulator, denoted by post-validation calibrated (PVC) physics system simulator, for the target application responses to fall outside of the MVD of the first physics system simulator; updating, by the first implementation (810) of the physics system simulator, the first scaled-down experimental responses (Φ.sub.E.sub.1, Φ.sub.E.sub.2, . . . ) of the physical system by modeling the physical system using the scaled-down experimental models (M.sub.E.sub.1, M.sub.E.sub.2, . . . ) based on the physical parameters (P) and their corresponding filtered parameter variations (fΔP); adjusting, with a parameter calibration module (830), the physical parameters (P) based on the updated first scaled-down experimental responses (Φ.sub.E.sub.1*, Φ.sub.E.sub.2*, . . . ), the corresponding sets of experimental measurements (φ.sub.E.sub.1, φ.sub.E.sub.2, . . . ), and the filtered parameter variations (fΔP); and predicting, by the first implementation (810) of the physics system simulator, a posteriori application response ({tilde over (Φ)}.sub.A) of the physical system by modeling the physical system using the application model (M.sub.A) based on the adjusted physical parameters .

2. The method of claim 1, wherein the filtering comprises predicting, by a second implementation (821) of the physics system simulator different from the first implementation (810) of the physics system simulator, second scaled-down experimental responses (Φ.sub.E′.sub.1, Φ.sub.E′.sub.2, . . . ) of the physical system by modeling the physical system using the same or second scaled-down experimental models (M(E′.sub.1), M(E′.sub.2), . . . ) based on the same model parameters (P) and their corresponding parameter variations (ΔP); selecting, by a parameter-feature selector (823), parameter features comprising mathematical expressions derived from the multi-variate functions used to describe the boundaries of the model validation domain (MVD); determining, by a validator (825) of the filter module, whether the first scaled-down experimental responses (Φ.sub.E.sub.1, Φ.sub.E.sub.2, . . . ) and the second scale-down experimental responses (Φ.sub.E′.sub.1, Φ.sub.E′.sub.2, . . . ) corresponding to the selected parameter features are within the boundaries of the model validation domain (MVD); and in response to the parameter features falling outside the boundaries of the MVD, removing, by a remover (827), the parameter features for which the first scaled-down experimental responses (Φ.sub.E.sub.1, Φ.sub.E.sub.2, . . . ) and the second scaled-down experimental responses (Φ.sub.E′.sub.1, Φ.sub.E′.sub.2, . . . ) are outside the boundaries of the model validation domain (MVD).

3. The method of claim 2, wherein the first implementation (810) of the physics system simulator is a high-fidelity implementation of the physics system simulator, and the second implementation (821) of the physics system simulator is a low-fidelity implementation of the physics system simulator.

4. The method of claim 2, wherein selecting the parameter features is performed using one or more of singular value decomposition, project pursuit techniques, or neural networks.

5. A method for post-validation adjustment of physics system simulators configured to predict behavior and/or state of a physical system based on an application model (M.sub.A) and multiple model parameters (P) and their corresponding known parameter variations (ΔP), wherein the application model (M.sub.A) is related to one or more scaled-down experimental models (M.sub.E.sub.1, M.sub.E.sub.2, . . . ), each scaled-down experimental model (M.sub.E.sub.j) being associated with a respective set of experimental measurements (φ.sub.E.sub.j), where j=1, 2, . . . , wherein the physics system simulator has been validated for a target application model (M.sub.A), as described by a model validation domain (MVD) the boundaries of which are described mathematically based on deterministic or stochastic multi-variate functions of experimental-model responses, the corresponding sets of experimental measurements (φ.sub.E.sub.1, φ.sub.E.sub.2, . . . ), derivatives thereof, and the parameter variations (ΔP), and an uncertainty estimator, the method comprising: predicting, by a physics system simulator (1010), the application response (Φ.sub.A) and scaled-down experimental responses (Φ.sub.E.sub.1, Φ.sub.E.sub.2, . . . ) of the physical system by modeling the physical system using the application model (M.sub.A) and the scaled-down experimental models (M.sub.E.sub.1, M.sub.E.sub.2, . . . ) based on the model parameters (P) and their corresponding parameter variations (ΔP); filtering, by a validation assist response filter (1020), the predicted scaled-down experimental responses (Φ.sub.E.sub.1, Φ.sub.E.sub.2, . . . ) to remove variations thereof that cause the predicted scaled-down experimental responses (Φ.sub.E.sub.1, φ.sub.E.sub.2, . . . ) to be outside the boundaries of the model validation domain (MVD); determining, by a response calibration module (1030), a posteriori application response ({tilde over (Φ)}.sub.A) of the physical system based on the application response (Φ.sub.A), the parameter variations (ΔP) of the model parameters (P), and the filtered scaled-down experimental responses (fΦ.sub.E.sub.1, fΦ.sub.E.sub.2, . . . ) and the corresponding sets of experimental measurements (φ.sub.E.sub.1, φ.sub.E.sub.2, . . . ).

6. The method of claim 5, wherein the filtering comprises selecting, by a response feature selector (1023), response features comprising mathematical expressions derived from the multi-variate functions used to describe the boundaries of the model validation domain (MVD); determining, by a validator (1025) of the filter module, whether the selected response features are within the boundaries of the model validation domain (MVD); and in response to the selected response features falling outside the boundaries of the MVD, removing, by a remover (1027), the response features that are outside the boundaries of the model validation domain (MVD).

7. The method of claim 6, wherein selecting the response features is performed using one or more of singular value decomposition, project pursuit techniques, or neural networks.

8. A system for supporting separate-effect experiments for nuclear-power plants, the system comprising: one or more hardware processors; an memory encoding instructions that, when performed by the hardware processors, cause the system to perform the methods of claim 1.

9. A system for supporting integral-effect experiments for nuclear-power plants, the system comprising: one or more hardware processors; and memory encoding instructions that, when performed by the hardware processors, cause the system to perform the methods of claim 1.

10. A system for supporting small-scale mock-up experiments for nuclear-power plants, the system comprising: one or more hardware processors; and memory encoding instructions that, when performed by the hardware processors, cause the system to perform the methods of claim 1.

11. A system for validating first-of-a-kind reactor designs for nuclear-power plants, the system comprising: one or more hardware processors; and memory encoding instructions that, when performed by the hardware processors, cause the system to perform the methods of claim 1.

12. A system for validating advanced-fuel designs for nuclear power-plants, the system comprising: one or more hardware processors; and memory encoding instructions that, when performed by the hardware processors, cause the system to perform the methods of claim 1.

13. A system for supporting transportation of irradiated nuclear fuel for fuel-testing facilities, the system comprising: one or more hardware processors; and memory encoding instructions that, when performed by the hardware processors, cause the system to perform the methods of claim 1.

14. A system for evaluating burn-up credit for fuel-testing facilities, the system comprising: one or more hardware processors; and memory encoding instructions that, when performed by the hardware processors, cause the system to perform the methods of claim 1.

15. A system for destructively or non-destructively assessing of irradiated nuclear fuel inventory for fuel-testing facilities, the system comprising: one or more hardware processors; and memory encoding instructions that, when performed by the hardware processors, cause the system to perform the methods of claim 1.

16. A system for detecting anomalies as part of condition monitoring of fuel-testing facilities, the system comprising: one or more hardware processors; and memory encoding instructions that, when performed by the hardware processors, cause the system to perform the methods of claim 1.

17. The method of claim 1, wherein the filtering is based upon an increase in mutual information beyond a threshold determined by comparison of scaled-down experimental responses (Φ.sub.E1, Φ.sub.E2, . . . ), (Φ.sub.E′.sub.1, Φ.sub.E′.sub.2, . . . ) from two separate physics system simulator instances.

18. The method of claim 6, wherein the filtering is based upon an increase in mutual information beyond a threshold determined by comparison of a response from a pseudo target application model simulated by the physics system simulator, and a response from a pseudo set of scaled-down experimental models simulated by the same physics system simulator.

19. A method of constructing a validation assist parameter filter comprising: comparing scaled-down experimental responses (Φ.sub.E1, Φ.sub.E2, . . . ), (Φ.sub.E′.sub.1, Φ.sub.E′.sub.2, . . . ) from two separate physics system simulator instances; obtaining an entropy-based filtration criterion using a statistical module that quantifies common information content between two sets of responses associated with the scaled-down experimental model and the target application model, as simulated by two different instances of physics system simulators; selecting a plurality of parameter features with a parameter feature selector based upon the entropy-based filtration criterion; and configuring the validation assist parameter filter to filter the selected plurality of parameter features to ensure post-validation calibration of the physics system simulator with a calibration module does not generate parameter variations that violate the MVD boundaries.

20. A method of constructing a validation assist response filter comprising: obtaining a set of simulated experimental responses (Φ.sub.E1, Φ.sub.E2, . . . ) for each of a plurality of different scaled-down experimental models of the target application model; selecting one of the plurality of different scaled-down experimental models of the target application model as a pseudo target application model; selecting a subset of the plurality of different scaled-down experimental as a pseudo set of scaled-down experimental models, wherein the pseudo set of scaled-down experimental models excludes the pseudo target application model; calculating mutual information between at least one response from the pseudo target application model simulated by the physics system simulator, and at least one response from the pseudo set of scaled-down experimental models simulated by the same physics system simulator; determining the calculated mutual information exceeds a pre-determined threshold; configuring the validation assist response filter to exclude the at least one response from the pseudo set of scaled-down experimental models.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0030] FIG. 1 illustrates a representative block diagram showing a method of operating a prior art physics system simulator to estimate, i.e., simulate, a system response.

[0031] FIG. 2 illustrates a representative block diagram of a method of operating a prior art physics system simulator to estimate a system response with a calibration module.

[0032] FIG. 3 illustrates a high level representative block diagram of a physics system simulator of multiple experiments, parameter adjustments, and a physics system simulator of a target application model utilizing the adjusted parameters.

[0033] FIG. 4 illustrates another prior art representative block diagram of a physics system simulator simulating responses of multiple experiments models and a target application model along with a response calibration module.

[0034] FIG. 5 illustrates a representative block diagram of a method of system model validation using multiple experiment models, experiment measurements, and an uncertainty estimator to predict boundaries of a model validation domain for a target application.

[0035] FIG. 6 illustrates a representative block diagram of a physics-guided analytical model validation in accordance with the present disclosure utilizing validation assist filtration.

[0036] FIG. 7 illustrates a representative block diagram of a method of validation assist filtration.

[0037] FIG. 8 illustrates a representative block diagram of a method of physics-guided analytical model validation utilizing validation assist parameter filtration.

[0038] FIG. 9 illustrates a representative block diagram of validation assist parameter filtration.

[0039] FIG. 10 illustrates a representative block diagram of a physics-guided analytical model validation utilizing validation assist response filtration.

[0040] FIG. 11 illustrates a representative block diagram of validation assist response filtration.

DESCRIPTION OF THE CURRENT EMBODIMENT

[0041] The present disclosure generally relates to a filter that ensures that the predictions of a PVC (post-validation calibrated) physics system simulator will remain within boundaries of a predetermined model validation domain (MVD). Specifically, embodiments of the present disclosure utilize one or more filters to ensure calibrated model parameters {tilde over (P)} and/or calibrated responses {tilde over (ϕ)}, i.e., aka posteriori values, cause physics simulator model predictions to remain within the boundaries of the model validation domain MVD for the target application. In some embodiments, predetermined filters can be utilized, while in other embodiments filters can be automatically inferred, or otherwise determined, from available measurements and other renditions of the physics system simulator during operation.

[0042] One way to ensure that the calibrated model remains within the boundaries of the MVD in accordance with the present disclosure is via a filter operator (f) configured to remove certain parameters' variations (ΔP) and therefore corresponding variations in responses for experimental models (ΔΦ.sub.E) and the target application model (ΔΦA) that have an undesirable impact on the model validation domain (MVD). Removing certain parameters' variations implies reducing the degrees of freedom available for parameters' variations. For example, n parameters have up to n degrees of freedom to vary. Removing a degree of freedom implies the n parameters are effectively varying along an n−1 mathematical manifold. At least one variation of the parameters or responses is referred to as a feature. A feature is a mathematical function of n variables, which captures one degree of freedom from n variables, e.g., the average of two variables is denoted as a feature, the sum of squares of two variables is denoted as a feature, etc. An undesirable impact is where the model predictions are outside the MVD for the target application. Another way to ensure that the PVC physics system simulator's predictions remain within the boundaries of the MVD in accordance with the present disclosure is via a filter operator (f) configured to remove features in the scaled-down experimental responses (ΔΦ.sub.E) that have an undesirable impact, when used by a calibration module, on the model validation domain (MVD). In some embodiments, filters can be utilized to filter certain features from parameters' variations (ΔP) and certain features from responses for the scaled-down experimental models that would have an undesirable impact, when used by a calibration module, on the model validation domain (MVD), i.e., that would result in the PVC physics system simulator's predictions to be outside the MVD for the target application.

[0043] Referring to FIG. 6, a representative block diagram of a physics-guided analytical model validation system 600 in accordance with one embodiment of the present disclosure is illustrated. In this embodiment, the MVD of the target application model, as well as the target application model, are provided as inputs to the physics-guided validation assist filtration system 600.

[0044] Before describing several exemplary embodiments of systems and methods in accordance with various aspects of the present disclosure, it should generally be understood that the systems and methods of the present disclosure can include and can be implemented on or in connection with one or more computers, microcontrollers, microprocessors, and/or other programmable electronics that are programmed to carry out the functions described herein. The systems may additionally or alternatively include other electronic components that are programmed to carry out the functions described herein, or that support the computers, microcontrollers, microprocessors, and/or other electronics. The other electronic components can include, but are not limited to, one or more field programmable gate arrays, systems on a chip, volatile or nonvolatile memory, discrete circuitry, integrated circuits, application specific integrated circuits (ASICs) and/or other hardware, software, or firmware. Such components can be physically configured in any suitable manner, such as by mounting them to one or more circuit boards, or arranging them in another manner, whether combined into a single unit or distributed across multiple units. The various system models, parameters, and other data can be stored in local or remote memory. In some embodiments, the validation system can be provided on a general purpose computer, while in other embodiments the validation system can be implemented within a dedicated hardware framework.

[0045] In general, the validation assist filtration system 600 hardware includes a physics system simulator 602, a validation assist filter 120, and a calibration module 120. The validation assist filter 120 can include a validation assist parameter filter and/or a validation assist response filter, depending on the implementation. The calibration module can be based on any number of PVC-type changes such as a parameter calibration module that adjusts model parameters, a response calibration module that adjusts predicted responses, a module that changes at least one of the inherent assumptions or numerical approximations or the numerical solver of the physics system simulator, to name a few. The validation assist filter(s) filter out the parameters and/or responses that cause the PVC physics system simulator's predictions to be outside of the MVD of the original physics system simulator for which the MVD is constructed.

[0046] The validation hardware 600 accepts a number of different inputs including a target application model M(A), a model validation domain (MVD) for that target application based on the original physics system simulator, at least one experimental model (E.sub.v), experimental measurements (φ.sub.Ev) from the at least one experimental model, a set of one or more model parameters (P), and one or more parameter variations (ΔP). Inputs are utilized by the physics system simulator to simulate the model response for each of the scaled-down experimental models and the target application model. In one aspect of the disclosure the validation assist filter 120 filters the responses from the physics system simulator 602. In another aspect of the disclosure the validation assist filter 120 filters the parameters variations ΔP. In yet another aspect, both are filtered. The filtration process will be discussed in more detail below. Suffice it to say, the calibration module 604 accepts the filtered values (or combination of filtered and unfiltered values) and performs its intended function, that is to improve predictions of the PVC system physics simulator as compared to the predictions of the original physics system simulator. Due to the use of the filtered values, the predictions of the PVC system physics simulator become consistent with the MVD.

[0047] A representation of a target application model M(A) as well as a representation of its model validation domain MVD are passed to a physics simulation system 602. In addition, a set of experimental models (M(E.sub.1), M(E.sub.2), . . . M(E.sub.v)), and various parameters (P) and their variations (ΔP) are also passed to the simulator 602. The simulator simulates the target application model and the scaled-down experimental models to predict a target application model response ΦA and a scaled-down experimental model response ΦE.sub.v for each scaled-down experimental model M(E.sub.v), which is passed to a calibration module 604.

[0048] The calibration module introduces a PVC-type change, for example, it adjusts the parameters (see FIG. 8) and/or the responses (see FIG. 10) based upon its inputs, which may include filtered parameter variations and/or filtered experimental model responses, with the filtration tailored to ensure that the calibration module output does not result in target application model responses that are outside of the MVD. That is, the filters of the present disclosure work in conjunction with the calibration module 604 are designed to ensure that the target application responses remain within the boundary of the MVD.

[0049] Filter implementation can be pre-determined using knowledge of the model validation domain (MVD) and/or other instances of the physics system simulator, and/or inferred using available measurements, as illustrated in FIG. 7, for instance.

[0050] A validation assist filter 120 can be utilized in connection with the physics-guided analytical model validation assist filtration system as illustrated in FIG. 7. The filtration process 700 includes an MVD boundary feature selector 702 that operates in conjunction with a physics system simulator 704. In general, the process includes checking whether at least one of variations in model parameters, at least one of the variations in the responses of the target application model, at least one of the variations in the responses of the scaled-down experimental models are within the boundaries of MVD 706. The boundaries of MVD can be described mathematically using deterministic or stochastic multivariate functions, generated using an uncertainty estimator 703 as part of the MVD boundary feature selector module 702, which involves the scaled-down experimental measurements, parameters variations, assumptions and approximations inherent in the construction of the physics system simulator and system-customized expert-guided scaling recipes. If one or more of the response variations is not within the MVD boundaries, the feature or features associated with the one or more response variations are excluded from the filter operator (f) whereas for response variations that stay within the MVD boundaries, the associated feature or features are not removed by the filter operator 708.

[0051] An MVD boundary feature generally refers to any mathematical expression derived from the multi-variate functions of the variables used to describe the MVD boundary, including the parameters, at least one response from the scaled-down experiments, and at least one response from the target application. Many mathematical techniques may be used to select MVD boundary features, e.g., Singular Value Decomposition, Project Pursuit Techniques, Neural Networks, or Autoencoders, to name a few. For example, a feature may describe the component of a vector of responses projected onto a basis function. The basis function may be derived from a cloud of simulation results of the scaled-down experimental and target application models. A cloud of simulation refers to multiple executions of the physics system simulator with the parameters varied within the range of their uncertainties.

[0052] The filter operator can be constructed by (i) excluding all features that are not within the boundaries of the MVD, or (ii) including only the features that are within the boundaries of the MVD.

[0053] One embodiment of a validation assist filter for a parameter calibration-based system 800 will now be described in detail in connection with FIG. 8. This embodiment of the present disclosure utilizes a parameter validation assist filter to ensure adjusted parameters P remain within boundaries of the model validation domain MVD.

[0054] As discussed below in more detail in connection with FIG. 9, a validation assist parameter filter 820 with an MVD boundary filter operator (f) is configured to remove features in parameters variations (ΔP) and features corresponding to variations in responses for experimental models (ΔΦ.sub.E) that have an impact on the model validation domain (MVD). That is, the filter is configured to identify and filter out parameters variations, described mathematically by features, that cause PVC physics system simulator's predictions for the target application responses to fall outside of the MVD. That is, the filtration can be performed by blocking certain parameters features that cause the corresponding responses to have a value outside of a particular range from being passed through the filter operator 827 or allowing only parameters features within a particular range to be passed through the filter operator. The filtration can be a complex combination conditional filtration such that certain values of certain parameters features are filtered or unfiltered dependent upon the values of other parameters' features or groups of parameters' features because different combinations of values of different parameters' features can affect whether or not the response or the response's variations fall within the MVD. The filter implementation can be predetermined based upon knowledge of the MVD as discussed in connection with FIG. 9, for example.

[0055] FIG. 9 illustrates an exemplary filtration construction process of the validation assist filter for parameters 820 from FIG. 8. In essence, the filtration construction process 900 includes predicting with another rendition, i.e., a separate instance, of the physics system simulator (821) a set of alternatively-derived experimental responses (Φ.sub.E′.sub.1, Φ.sub.E′.sub.2, . . . ) of the scaled-down experimental models (M(E′.sub.1), M(E′.sub.2), . . . ) based on the same set of parameters (P) and their corresponding parameter variations (ΔP) that are to be fed to the original physics system simulator (810). The separate instance of the physics system simulator 821 may optionally be a more accurate version of the physics system simulator 820 utilized in the validation 600 generally. If the separate instance of the physics system simulator (821) is more accurate than the original physics system simulator (810), it is typically referred to in the literature as high-fidelity or advanced simulator, implying that it produces much more accurate predictions of a system behavior. For example, it could be based on a finer mesh allowing for more detailed representation of the geometry, composition, or another aspect of the model. It could also employ more accurate mathematical equations to describe system behavior. In practice a high-fidelity or advanced simulator is not feasible to execute many times, hence it is not employed for routine engineering calculations involving the target application model.

[0056] The scaled-down experimental responses from the original physics system simulator 810 and the alternatively-derived scaled-down experimental responses from the other instance of the physics system simulator 821 are fed to a parameter-feature selector module 823, as shown in FIG. 9. The parameter-feature selector module 823 is configured to select one or more parameter features including mathematical expressions derived from the multi-variate functions of the set of responses and/or parameters used to describe the boundaries of the model validation domain (MVD), denoted earlier as MVD boundary feature.

[0057] The features causing at least one response's variations to fall within the MVD boundaries are included in the filter operator (f), while features causing at least one response's variations to fall outside the MVD boundaries are filtered out. Put another way, a validator sub-module 825 of the filter constructor 900 determines whether the scaled-down experimental responses (Φ.sub.E1, Φ.sub.E2, . . . ) and the alternatively-derived scaled-down experimental responses (Φ.sub.E′.sub.1, Φ.sub.E′.sub.2, . . . ) corresponding to the selected parameter features are within the MVD boundaries according to an entropy-based filtration criterion; and if not so, removing (827) the parameter features for which the experimental responses (Φ.sub.E.sub.1, Φ.sub.E.sub.2, . . . ) and the alternatively-derived experimental responses (Φ.sub.E′.sub.1, Φ.sub.E′.sub.2, . . . ) are outside the MVD boundaries.

[0058] Details related to the filter construction 900 generally and the parameter-feature selector module 823 will now be discussed in detail. The filter 820 is used to ensure that response variations calculated based on parameter variations remain within the MVD boundaries by filtering out parameter features which cause response variations to trigger a filtration criterion. Constructing one embodiment of the validation assist parameter filter 820 involves a second physics system simulator instance 821 (apart from the first instance of the physics system simulator 810). The parameter-feature selector module 823 compares the scaled-down experimental responses (Φ.sub.E.sub.1, Φ.sub.E.sub.2, . . . ), (Φ.sub.E′.sub.1, Φ.sub.E′.sub.2, . . . ) from the two separate physics system simulator instances 810, 821 and decides whether or not to filter a certain parameters' variation based upon whether at least one response to the associated parameters' feature is within MVD boundaries according to a filtration criterion. The filtration criterion can be quantified by an information-theoretic measure, which can be calculated or estimated using an off the shelf statistical module, such as those employing the KL-divergence theorem, f-divergence theorem, transfer entropy, mutual information neural estimators, to name a few. This filtration criterion is entropy-based and can be referred to as mutual information. Mutual information is an entropy-based measure which quantifies the common information content between two variables, e.g., at least one response from the target application and at least one response from the scaled-down experimental model, as simulated by the physics simulator.

[0059] In our embodiment, mutual information is used to set an upper limit on the maximum allowable increase in confidence that can be reached when introducing any PVC-type changes to a physics system simulator that has already been validated. Such entropy-based limit is not employed by the state-of-the-art methods to ensure that PVC-type changes would not violate the MVD boundaries. Put differently, the extant methods do not have a vetting process by which the PVC physics system simulator's predictions can be shown to lie within the MVD boundaries of the original physics system simulator.

[0060] The filtration criterion is such that any PVC-type change, for example a calibration or an adjustment of at least one parameter and/or at least one response, would not lead to an increase in the mutual information between at least one response from the target application and at least one response from the scaled-down experiments. Techniques used by the state-of-the-art calibration modules, e.g., Bayesian estimation, Generalized Least-Squares Methodology, Maximum Likelihood Estimation, to name a few, do not enforce this requirement and hence run the risk of violating the boundaries of the MVD, which can be discovered only after the target application is rendered in real life in the form of a real physical system from which real measurements can be collected. An increase in the mutual information, caused for example by an adjustment/calibration of the physics system simulator's parameters and/or responses, implies that the physics system simulator predictions for the scaled-down experiments become more informative on the target application response. Such increase in mutual information is not warranted by the MVD construction and hence should not be allowed, i.e., it is to be filtered out by the filter 820.

[0061] For the parameter-filter selector module 823, the feature selection can be conducted using any number of off the shelf techniques such as singular value decomposition, project pursuit techniques, autoencoder-based machine learning methods, or any custom expert-guided feature engineering techniques. A selected feature is removed if it triggers a filtration criterion 825 based on an entropy-based mutual information measure. The mutual information can be calculated using any standard statistical module, such as those employing KL-divergence theorem, f-divergence theorem, Fischer information, machine-learning and autoencoder-based tools, etc., between the simulated samples of at least on target application response and the simulated samples of at least one of scaled-down experimental responses. If a statistical-based method is employed to calculate the mutual information, a threshold-based criterion may be employed to determine when the mutual information has increased. This is because statistical-based methods incur some variability when the mutual information is recalculated using different sets of samples. A minimum threshold may be set by the user, above which the mutual information is considered to have experienced an increase large enough to trigger the filtration criterion.

[0062] The feature selection techniques in the parameter-feature selector module 823 can be configured to operate based on a comparison of responses from multiple physics system simulators (810 and 821), which can be especially useful where experimental measurements are not available for experimental models M(E.sub.1), M(E.sub.2) that correspond to the target application model. Put another way, the parameter-feature selector module 823 can be used when the analyst has access to a multi-fidelity physics system simulation environment. For example, one implementation 810 of the physics system simulator can correspond to a low-fidelity implementation used for routine calculations for which the MVD is constructed, and 821 is a high-fidelity implementation used to support the design and optimization of the target application model. In an alternative embodiment, both 810 and 821 can correspond to low-fidelity implementations developed by two independent entities, e.g., a regulator and a licensee trying to obtain license from the regulator, where the first implementation is employed by one entity, e.g., the regulator, to verify that the calibrated parameters are consistent with the MVD of the second implementation, e.g., by the licensee. In alternative embodiments both may be high-fidelity, where one is interested in assessing the value of new improvements to a physics system simulator.

[0063] Referring to FIG. 10 an exemplary embodiment of a validation assist response filter for a validation system 1000 is illustrated. In this embodiment, the validation assist response filter 1020 ensures adjusted, i.e., calibrated, responses {tilde over (ϕ)} remain within boundaries of the model validation domain MVD. This is achieved via a filter operator (f) configured to remove variations in simulated responses for scaled-down experimental models (Φ.sub.E1, Φ.sub.E2, . . . ) that have an impact on the MVD, as defined earlier by an entropy-based filtration criterion.

[0064] Specific filter implementations can be inferred using available measurements (φE.sub.1, ΦE.sub.2, . . . ), as show in FIG. 10, for instance. This type of filter 1020 can be configured to be used when a single physics system simulator is available for which the MVD is constructed. Where the filtration has access to experimental measurements (ΦE.sub.1, φE.sub.2, . . . ) and simulated scaled-down experimental responses (Φ.sub.E1, Φ.sub.E2, . . . ), the MVD boundaries can be used to select features by a response feature selector 1023 which are checked against a filtration criterion 1025 to determine whether they should be removed or retained in the construction of the filter operator 1027. The resulting filter 1020 produces filtered scaled-down experimental responses (fΦE.sub.1, fΦE.sub.2) which ensure that a subsequent calibration module 1030 does not lead to response variations that do not violate the MVD boundaries.

[0065] The response feature selector 1023 can be configured to operate based on an execution of multiple pseudo runs as follows. In each pseudo run, the response feature selector 1023 selects one of the scaled-down experimental models as a pseudo target application model. This selection can be done at random or in an ordered one-at-a-time fashion. To begin, select a subset of the scaled-down experiments to represent a pseudo set of scaled-down experiments. The pseudo set of scaled-down experimental models is to exclude the pseudo target application model. The filtration criterion 1025 calculates an increase in the mutual information between at least one response from the pseudo target application, as simulated by the physics system simulator, and at least one response from the pseudo scaled-down experiments, as simulated by the same physics system simulator, with the physics system simulator employing the adjusted, i.e., calibrated, parameters and/or responses. The response feature selector 1023 repeats this process by selecting another pseudo target application model and another set of pseudo scaled-down experiments. In each pseudo run, the features that increase the mutual information beyond a pre-determined threshold are configured to be filtered out 1025. When no more features are filtered out by executing a single or few additional pseudo runs by the feature selector 1023, the process may be terminated. The filter operator 1027 can be constructed by excluding all the features that increase the mutual information beyond the threshold from all the pseudo runs.

[0066] A number of exemplary use cases of systems and methods of physics-guided analytical model validation in accordance with the present disclosure will now be discussed in detail.

[0067] A number of use cases involve the validation of computer models used in support of the design of nuclear systems. Validation of models for these types of systems can be especially relevant because it can be incredibly expensive to build a physical system based upon a computer model without assurances that the fully built system will act as predicted by the target application model.

[0068] A number of the models relate to activities supporting operation and safety of the nuclear systems. Some models relate to nuclear power plants, for example some relate to: analyses for supporting small-scale mock-up experiments; analyses for supporting separate-effects experiments; analyses for supporting integral-effects experiments; validation of computer models for first-of-a-kind reactor designs; and validation of computer models for advanced fuel designs. Other models relate to fuel testing facilities, for example some relate to transportation of irradiated nuclear fuel; burnup credit evaluation; destructive and nondestructive assessment of irradiated nuclear fuel inventory; anomaly detection for condition monitoring during nuclear facility operation; and validation of codes used to model aging nuclear weapons systems.

[0069] Looking more closely at first-of-a-kind reactor design, model validation can be especially important because by its nature no experimental measurements exist. Multi-fidelity computer codes exist, including low- and high-fidelity physics models of the design of the first of a kind reactor. The high-fidelity physics models can be used to generate virtual experimental measurements, e.g., representing the measurements collected from the scaled-down experimental models M(E.sub.1), M(E.sub.2) and a low-fidelity physics model can be used to represent the target application model M(A). This allows the methodology of FIGS. 8-9 to be utilized to compare multiple different physics model simulation responses. It can also be employed to test the value of new improvements to a physics system simulator. This can be done by assessing the value of a new improvement in reshaping the MVD boundaries.

[0070] Online monitoring of a nuclear power plant or process can include modeling various nuclear processes such as: (i) material accountability and traceability in fuel reprocessing plants; (ii) fuel enrichment plants; (iii) fuel manufacturing plants; and (iv) monitoring of nuclear power plant operation.

[0071] Sensor measurements are continuously collected for forecasting and to identify anomalous behavior. For forecasting purposes: (i) the collected sensor's measurements represent the responses collected from the scaled-down experimental models (M(E.sub.1), M(E.sub.2), . . . ) and (ii) the future predictions represent the responses of a target application model M(A). Anomaly detection algorithms can be employed to flag certain measurements as anomalous. These algorithms can be validated using the disclosed methodologies (e.g., the filtration described in connection with FIGS. 10-11 where experimental measurements are available) by splitting historical data into two groups: (i) one representing the experimental conditions, corresponding to the scaled-down experimental models M(E.sub.1), M(E.sub.2), . . . and (ii) the other representing the target application model M(A).

[0072] Another use case is for validating modeling of nuclear material transportation applications. A target application model M(A) can represent the condition of the nuclear materials after discharge from the nuclear reactor. Scaled-down experimental model(s) M(E.sub.1), M(E.sub.2) can describe the irradiation conditions inside the nuclear reactor.

[0073] Another use case relates to advanced fuel designs. New fuels to be introduced in an existing nuclear reactor can be modeled and validated using the filtration validation methodologies described herein. Scaled-down experimental models M(E.sub.1), M(E.sub.2), . . . can represent the conditions of current fuel designs after irradiation during residence time in the reactor. A target application model M(A) represents the modeled behavior of the advanced fuel design.

[0074] Another use case relates to the physics system simulators used to simulate the detonation of high explosives, especially the aging stock of nuclear weapons, and how a particular weapon would perform if a detonation is initiated. Given the complexity of the physics models, researchers are forced to detonate samples of the explosives in small amounts and different geometries, and observe the explosions with specialized tools and compare the results to the simulation. These scaled-down experiments are used to refine the model via a calibration module by adjusting some of the model parameters. The state-of-the-art calibration techniques employed are empirical and cannot be proven to be consistent with the MVD established for the original weapon upon its construction.

[0075] The filtration validation methodologies of the present disclosure can be used to (i) validate the predictions of the physics system simulator, and (ii) support its continual update based on collected operational data. For validation, historical data may be split into two batches, one representing the scaled-down experimental models M(E.sub.1), M(E.sub.2), . . . and the other the target application model M(A). For continuous update, scaled-down experimental models M(E.sub.1), M(E.sub.2), . . . represent historical data, and the target application model M(A) represents future conditions.

[0076] Another aspect of the present disclosure relates to establishing prediction confidence. Confidence can be established in the prediction of a reduced order model, trained against predictions of a higher fidelity model. In many situations, it is not feasible to work with a high-fidelity physics system simulator due to its complexity and taxing computational requirements. In order to address this, the present disclosure executes the high-fidelity physics system simulator many times to train the reduced order model using existing machine learning methods.

[0077] The available training data are then split into two batches, one to represent scaled-down experimental models, e.g., as M(E.sub.1), M(E.sub.2), . . . and one target application model, e.g., M(A). The reduced order model is used to model both the experimental and application models, and the higher fidelity physics system simulator predictions are taken to represent the experimental model from the first batch, which are employed to make predictions about the application model of the second batch, thus providing a basis to validate the predictions of reduced order models.

[0078] The above description is that of current embodiments of the invention. Various alterations and changes can be made without departing from the spirit and broader aspects of the invention as defined in the appended claims, which are to be interpreted in accordance with the principles of patent law including the doctrine of equivalents. This disclosure is presented for illustrative purposes and should not be interpreted as an exhaustive description of all embodiments of the invention or to limit the scope of the claims to the specific elements illustrated or described in connection with these embodiments. For example, and without limitation, any individual element(s) of the described invention may be replaced by alternative elements that provide substantially similar functionality or otherwise provide adequate operation. This includes, for example, presently known alternative elements, such as those that might be currently known to one skilled in the art, and alternative elements that may be developed in the future, such as those that one skilled in the art might, upon development, recognize as an alternative. Further, the disclosed embodiments include a plurality of features that are described in concert and that might cooperatively provide a collection of benefits. The present invention is not limited to only those embodiments that include all of these features or that provide all of the stated benefits, except to the extent otherwise expressly set forth in the issued claims. Any reference to claim elements in the singular, for example, using the articles “a,” “an,” “the” or “said,” is not to be construed as limiting the element to the singular.