GridLAB-D™ Model Validation Process

This framework outlines a systematic approach for modeling a physical device within a software environment. Please feel free to click the boxes in the flowchart below for additional information and potential questions at each step.

Initially developed to help model microgrid-capable devices within the GridLAB-D™ software, the overall approach in this framework can be applied to many software modeling applications. Questions are categorized and decision points interjected to help insure sufficient information is available and the modeling process can continue. The framework begins with very high-level, study-oriented questions and progresses into the specific implementation of hardware and software testing. Successfully applying the framework process will result in a software-equivalent model of a physical hardware device.

Flowchart GridLAB-D flowchart GridLAB-D flowchart GridLAB-D flowchart GridLAB-D flowchart GridLAB-D flowchart GridLAB-D flowchart Quantities of Interest GridLAB-D flowchart GridLAB-D flowchart GridLAB-D flowchart GridLAB-D flowchart GridLAB-D flowchart GridLAB-D flowchart GridLAB-D flowchart GridLAB-D flowchart GridLAB-D flowchart GridLAB-D flowchart GridLAB-D flowchart GridLAB-D flowchart GridLAB-D flowchart

Project Scope

These blocks address questions and process steps related to the overall scoping and goals of the final model. They are meant to provide an initial guide to translating high-level requirements towards a specific final model.

Return to flowchart

Software

These blocks address questions related to the software model and the modeling environment. This will include questions on the actual implementation, as well as limitations imposed by the chosen environment/platform on both the model implementation and subsequent simulations.

Return to flowchart

Hardware

These blocks address questions relating to the physical device being modeled and the testing environment. This will include specifics properties of the physical device, testing capabilities of the available facility and equipment, and the process to obtain validation data for refining the software model.

Return to flowchart

Decision

These blocks represent decision points or development of evaluation criteria in the framework process. They often represent a point in the process where the effectiveness of the approach or sufficiency of information must be decided. Affirmative outcomes result in progression through the flowchart. Negative outcomes don't necessarily constitute a failure, but may require a reiteration within the current section to refine details of the approach or implementation.

Return to flowchart

Problem Statement/Goals of Study

The plan for the systems and problems that the model will be used to address after validation need to be a constant guide throughout the validation process. The purpose of this block is to create a high level mission statement for the project and clearly identify concrete goals for future use of the model.

Part of determining the study goals is examining what level the model is expected to influence the system, and clearly defining the boundaries between the model to be validated and the system that will host the model. Another question is what quantities are expected to be examined. If a specific quantity is to be evaluated, the final model must either provide that quantity or influence it in the overall structure appropriately.

Detailed questions...

Time Scale

One of the fundamental inputs for defining the initial modeling approach is the time scale of the physical phenomena that the model will address. Key parameters to specify are resolution (how small of a time step and how small of a quantization step is required to capture all desired phenomena) and duration (how long the simulation must run in order to capture all desired phenomena). This will inform both how the model is constructed and the requirements for the test system data collection equipment. Clearly specifying the timing requirements helps streamline the model, as well as avoid efforts to obtain unnecessary levels of detail.

Detailed questions...

Quantities of Interest

The other fundamental input to the modeling process is information about the quantities of interest in the model. These are typically physically observable quantities such as voltage, power, current, temperature, electromagnetic field, etc. For each relevant physical observable, the magnitude, resolution, and dynamic range must be specified; both should follow directly from the goals of the study.

The model must be able to simulate the required observables. If the observables are derived quantities, then it will be necessary to simulate the fundamental quantities with sufficient magnitude and dynamic range to compute the quantities of interest.

It is crucial that the test system data collection equipment is able to fully capture the size of the signals of interest and that it has adequate dynamic range and responsiveness to capture the full range of target physical phenomena.

Detailed questions...

Decision Point: Goals

Before moving on, carefully compare the stated goals of the project with the physical scope of model and hardware tests. Are the phenomena of interest going to be adequately captured given the time step, the time duration, and the resolution and dynamic range of the relevant physical observables in the model and in the hardware tests? If not, then either the project goals need to be reconsidered, the physical scope needs to be adjusted, or the project needs to be aborted.

It is ideal for the physical and temporal scale to be not only adequate to address the phenomena of interest, but to be parsimoniously adequate. Efforts to obtain resolution significantly better than necessary to meet project goals are not a good use of resources.

Detailed questions...

Fatal

  • Can this problem be fixed with a parameter or procedural adjustment?
  • Is the process missing any equipment or software capabilities needed?
  • Does this require starting the model validation process from the beginning?

Return to flowchart

Model Capabilities

The required capabilities of the model to be validated must be specified, closely informed by decisions made in the Project Scope section. At this stage it can be easy to be overly ambitious about the range of behavior that the model will be able to capture. The decisions made in this section should be firmly grounded in the long-term goals of what kinds of problems the model will address. Many of the decisions in this section will simply require a repetition of the characteristics decided in the Project Scope section.

At this stage, specify what the boundaries of the model are and how the model will interact with a host environment. The model may calculate many detailed internal parameters, but not all of those are physically observable or comparable to quantities that may be measured. The model should be capable of predicting behavior of signals that correspond to things that may be measured in the hardware test.

Detailed questions...

Test Capabilities

One of the largest restrictions to the modeling framework will be constraints on available validation data. Especially in cases of an operational demonstration or some laboratory testing of a physical device, there may be restrictions on the type and quality of data available or the types of testing that can be performed at that facility. These restrictions need to be factored into the details of the model structure and validation plan, mainly for the sake of feasibility.

Disturbance restrictions for the validation of the model can basically be divided into two categories: device restrictions and environment restrictions. Device restrictions are limitations on the device being modeled itself. These could be prohibited because they could cause physical damage to the device, or because they are outside the tested operating envelope. Environment disturbance restrictions are restrictions of the physical testing space, including any administrative restrictions.

Availability of suitable data acquisition channels is also a major limitation on any validation studies. Another data acquisition is the resolution and range of the testing equipment within the validation laboratory. While the temporal resolution of testing equipment may determine portions of the modeling and validation process, the second dimension of "measurement resolution" must also be considered. This consideration is mainly on the quantization resolution of any acquisition devices, and can tie closely with "output resolution" desired by the overall scope of the model. Whatever level of modeling and validation is desired, the measurement steps of the acquisition device must be capable of measuring that detail.

Detailed questions...

Acceptance Criteria

One of the most important items to define for the model creation and validation process is an outline of the performance criteria. That is, what is considered the proper accuracy and at what point will model simulation and validation results be considered equivalent? It is important to define this beforehand to make sure the model isn’t being defined in greater detail than required, or in greater detail than the validation tests are capable of providing.

As with the previous sections, this portion is heavily integrated with other test aspects. The acceptance criteria could be a direct relationship with measurement accuracy or data acquisition rates, as well as test conditions. It could also be heavily influenced by the study goals, with a 10% error being acceptable in one case and a 0.01% error being required for another. Known influences of noise (environmental, measurement, or other sources) should also factor into the acceptance criteria.

The performance criteria can general fall into a mix of two categories: scalar criteria and temporal criteria. Scalar criteria are a pure comparison to another value at a given instance in time or all time. Temporal criteria are a comparison of a response delay or specific behavior over a period of time. Sources of error and other uncertainties in the measurements must be accounted for in these comparisons.

Detailed questions...

Decision Point: Capabilities

Before moving on, carefully cross-check the required capabilities of the model and hardware tests. Compare these required capabilities back to the project goals and the physical scope that were decided upon in the first section. At this point, it should be very clearly determined exactly which properties of the model will be examined and what criteria are necessary to declare them "validated" at the end of the process.

Detailed questions...

Structure

In this section, higher-level questions about the model should be addressed. The style of model, the structure, and potentially the interactions of sub-units should be decided. Specifications for the model should be completed in this section. The block diagram of the model should be finalized, and the flow of information, as well as the inputs and outputs of each calculation, determined.

Once the overall structure of the model has been determined, the details of how to implement in code are addressed. Simple models may be a single differential equation. More complex models may be an interacting set of differential equations (care must be taken in the update order). If there are multiple support control modes, different sets of equations may apply to each case. Some models may be best implemented as a set of cascaded blocks, or as a set of sub-devices.

Detailed questions...

Parameters

The model should have an organized hierarchy of physical parameters. Some are obtained from some outside source and fixed, and some are adjusted as the model runs.

Fixed parameters are those physical or preselected parameters that the model will take as an input. These parameters are not expected to change in the model and may include limit values imposed by the hardware or test environment, or specifications associated with the particular hardware being tested. Quantities for fixed parameters must be obtained from some trusted source.

Tunable or adjustable parameters are those that vary dynamically through the course of the model runs. These may be outputs to an overall environment, such as terminal voltage, or intermediate parameters of the model. A slate of initial conditions or formulae to produce the initial conditions will be needed to populate each tunable parameter.

The uncertainty inherent in the measurement of each of these parameters must be known and properly accounted for in calculations. Careful attention should be paid to the proper use of sensible units.

An overall model parameter is the time step, which controls the evolution of the model run. The time steps may be equally spaced in some pre-determined interval, or may adjust dynamically depending on rates of change or other aspects of model behavior. Time steps should be short enough to capture all of the physical phenomena of interest, but the simulation must also be able to be carried out in a reasonable length of time on available computational resources. If required time steps are longer or shorter than the normal values for the simulation environment, alternative approaches or model-specific approaches may be for proper simulation of the model.

Detailed questions...

Simulation

Once the model has been fully developed, it should be populated with the appropriate fixed parameters relevant to the hardware testing facilities, and simulations performed that predict the results of hardware tests. It is important to keep in mind that only those predictions which correspond to measurable, testable quantities will be useful for validation. If a model capability is the use of hardware test data to fit physical properties of the system under study, then the model should be capable of taking in test data in the format that it will be recorded during testing.

Detailed questions...

System Staging

The hardware testing facility needs to be staged and prepared for tests. This will include considerations related to physical equipment, administrative procedures, and preliminary data acquisition. Anything that needs to be done before the actual hardware test procedure is executed will be performed during this time.

All necessary equipment should be gathered and tested to ensure it is functioning properly. This can include measurement equipment, general laboratory equipment, and any additional devices needed to carry out specific portions of the hardware tests (e.g., switched shunt devices). Any permits, procedures, and other administrative permissions needed at the test facility will also need to be obtained and be properly executed. The necessary equipment should be connected to the test device and system. All preliminary data acquisition, equipment settings, and field calibrations should be performed.

Detailed questions...

Test Procedure

Create a detailed plan for the tests that will be carried out during the hardware validation data tests. This will be in the form of a step-by-step recipe with specific details in many steps. The steps should including specific times of actions, durations of events and measurements, and appropriate sequencing and coordinating of assets. The test plan and procedure should be detailed enough that any appropriately technical person can execute the test procedure successfully.

Detailed questions...

Carry Out Tests

Execute the test plan, carrying out the specific steps outlined from the Test Procedure block. Obtain any measurements and data sets for use in validating the software model being developed. Note any anomalous conditions or events in the individual steps that may influence the validation process.

Detailed questions...

Apply Acceptance Criteria

Once the simulations have been run and the corresponding hardware tests are complete, compare the results. Ensure the comparison covers all quantities of interest determined earlier in the flowchart. Apply the acceptance criteria for each quantity and item of interest from the original test scope.

Return to flowchart

Decision Point: Acceptance

After applying the acceptance criteria, the success or failure of the software simulation of the hardware device is determined. If the criteria are met, then the model is validated and the process is complete. For proper validation, the properties of the model that have been explicitly compared to hardware tests meet the acceptance criteria. There may be other aspects of the model that require further validation work, but are not within the scope defined at the beginning of the process. Care should be taken to not assume that that any aspects of the model that have not been compared to hardware tests have been validated.

If the simulation results and results of the hardware tests do not match within the previously stated convergence criteria, then engineering judgment must be applied to determine whether the model might need refinements or the hardware test procedure might have been subject to experimental error. One or the other must be re-done, and then results compared again, until validation is achieved.

It is extremely advantageous to be able to iterate hardware testing and comparison to simulation results more than once. Multiple observations and executions of the hardware tests will reduce any uncertainties in the hardware results, improving the overall validation data set and providing better information for any additional software iterations required to meet the model acceptance criteria.

Detailed questions...

Apply Acceptance Criteria

Once the simulations have been run and the corresponding hardware tests are complete, compare the results. Ensure the comparison covers all quantities of interest determined earlier in the flowchart. Apply the acceptance criteria for each quantity and item of interest from the original test scope.

Finished

    Congratulations! You have successfully implemented and validated your model!

Return to flowchart

Restart or Exit

    Reaching this block represents a problem in the model validation process. Early discovery of this problem prevents excess time and effort from being invested in a model validation that may be impossible, or require a change in scope.

Return to flowchart