Calibration Wizard for catchments

The Calibration Wizard is used to calibrate the streamflow characteristics of a Source model. It can be used to calibrate rainfall runoff, constituent and link routing models and is primarily intended for use in unregulated systems. Much of the functionality relies on characteristics of catchments scenarios - specifically the presence of a sub-catchment map.

In summary, the steps involved in undertaking calibration analysis are:

• Launch Source, open your project, and then open the scenario you wish to calibrate;
• Start the wizard and work through its steps (see Calibration Wizard below);
• Click Run on the Simulation toolbar to perform a calibration run; and
• Examine the results.

Characteristics

The calibration wizard offers a lot of flexibility. It supports:

• Calibrating to single or multiple observation gauges;
• Multiple gauges in different parts of the catchment (parallel) or nested (i.e. upstream/downstream);
• Ability to calibrate modelled flow at points or links (downstream flow), and the point can be node or outlet of catchment;
• Ability to calibrate modelled constituents at points or links, and the point can be node or  (outlet of ) catchment;
• Equal weighting between gauges or different weightings (e.g. by area, length of observed record, or quality of observed record);
• Automatic calibration using a mathematical optimiser and an objective function, or manual calibration using objectives functions and visual inspection;
• Compound objective functions including the ability to manipulate weighting between objective components;
• Ability to calibrate multiple gauges for the same time period or different time periods;
• Ability to group parameters across the catchment or keep them independent, with the ability to choose this for each parameter;
• Ability to calibrate rainfall runoff models or link flow routing models or to calibrate them at the same time;
• Ability to calibrate the model with options for different Scenario Input Sets;
• Ability to calibrate any model parameters with Scenario Input Sets Calibration property;
• Ability to specify relationships between meta-parameters; and
• Ability to calibrate the model parallelly.

The Calibration Wizard is organised as a step-by-step process, which guides you through configuring the model for calibration, and a Calibration Runner, which handles the actual calibration, either in an automated fashion using an optimiser, or manually, with the ability to directly specify parameter values.

Calibration Wizard

To start the wizard, choose Flow Calibration Analysis from the Select Analysis Type popup menu on the Simulation toolbar and then click Configure, which is also on the Simulation toolbar. The wizard comprises of four steps:

• Calibration targets -define model elements ( e.g. catchment, node or link)  and associate them with the observed data loaded in the model , and select the objective function(s) that you want the optimiser to use to assess how well it fits (Figure 1);
• Period definition - define the time period(s) for your calibration. When you have more than one observation data set, you can define a distinct calibration period for each gauge. You also define a warm up period, where the overall model simulation starts at some point in time before the first calibration period commences (Figure 2, Figure 3);
• Metaparameter definition - define parameters that will be modified during the calibration to improve the model fit to the observed data. Metaparameters are distinct from the actual parameters in the model. They can correspond directly to model parameters, or multiple model parameters can be grouped together to match a single metaparameter. Usually some combination of these approaches is used to reduce the number of parameters being calibrated, while retaining distinct parameters where it is expected to improve the fit (Figure 4); and
• Select optimisation function - choose and configure the optimisation strategy, including the option of using manual calibration in lieu of optimisation (Figure 11).

Configure Calibration Points ( previous Select elements to record )

In the first wizard step, you define the point(s) or link(s) in the model where you will be calibrating to observed data (Figure 1). You can select (the outlet of) catchments, nodes or the downstream end of links, or miscellaneous if applicable. For each of these points you associate observed data and optionally provide a site weighting. You also define your calibration objectives at this step, including optionally weighting different components. You can set up the wizard to calibrate downstream from the link ,or at a node, or at the outlet from a catchment. Similarly, you can calibrate the modeled constituents.

You can select either a node or link on which to load observed data using either the Project Hierarchy or clicking the element in the geographic editor panel. The steps to load time series at a Gauge for which you have observed data are similar for a link:

• Click the gauge node in the geographic editor panel so that it becomes selected;
• Click observed flow and select Data Source radio button;
• Choose the time-series of  its associated observed data from Data Sources frame; and
• Click OK.

The same method is used to load time series for links as well.

Figure 1. Calibration Wizard (Configure Calibration Points)

At the beginning, the interface of Step 1 in Flow Calibration Wizard (i.e.  the central Windows of Figure 1) only has two components (i.e. Add New Model Element button and Input set drop down list):

Add New Model Element button is used to add a modelled element (node or link or catchment) where the observed data was loaded in Data Sources of the Source model and calibration can be performed. Clicking on this button (action 1) will add one frame of Model Element to the interface.

The user can have multiple frames of Model Element. In another word, the user can select multiple model elements (e.g. nodes or links) for the calibration at same time.

Input set drop down list allows the user to select an available Input Set for the calibration (action 2).

In each added Model Element Frame:

Model Element | Select… can open Modelled Element Selector windows by clicking on Select… (action 3). The user can navigate Modelled Element Selector windows to select the required calibration element and parameter by clicking on the triangle symbol (action 4). For example, the action 4 in Figure 1 is to select the modelled downstream flow at gauge 410024 to calibrate.

Delete button in each frame of Model Element can remove that frame from the interface. It will also remove the element, which is defined by that frame, from the calibration.

Objective Function drop down list can be used to define the calibration objective(s) (action 5 ). Your choices are:

• Minimise Absolute Bias
• NSE Daily & Flow Duration
• NSE Daily & Log Flow Duration
• NSE Daily
• NSE Monthly
• NSE Daily & Bias Penalty
• NSE Monthly & Bias Penalty
• NSE Log Daily
• NSE Log Daily & Bias Penalty
• Square-root Daily, Exceedance, and Bias (previously called Sum of Daily Flows, Daily Exceedance (Flow Duration) Curve and Bias) (SDEB)

In most cases, model calibration seeks to maximize the selected objective function. The exceptions are the Absolute Bias and SDEB, which are minimized.

The general Objectives in Source  are described in (https://wiki.ewater.org.au/display/SD520/Calibration+analysis+-+SRG). The term and algorithm used in those objective function such as NSE, bias can refer to following links:

• Nash-Sutcliffe Efficiency (NSE)
• NSE of Log Data (NSE Log)
• Relative Bias
• Bias Penalty
• Pearson's Correlation Coefficient
• NSE of Flow Duration (Flow Duration)
• NSE of Flow Duration of Log Data (Log Flow Duration)

Observed Data Source | Select… can open Observed Data Selector interface by clicking on Select… (action 6). The Observed Data Selector interface allows the user to select the observed data corresponding to the selected element and parameter. The observed data need to  be loaded in the Data Sources already.

Calibration Weighting box allows you to define an alternative weighting between the objective functions defined for the elements, where you have multiple elements such as gauges. The default is 1 for each gauge, so each gauge would contribute equally to the overall objective function. The use of Calibration Weighting is optional, and the meaning of the weights can be defined by you. You might use this in a situation where for instance, there are 20 years of data at one gauge and 10 years of data at another, then apply a weighting of 2 to the first gauge and 1 to the second gauge. Alternatively, you might place higher weighting on observation points that represent more of the catchment area.

NSE Weighting box can be used to change the weighting between the components of composite objective functions. The value should be between 0 and 1 and is applied to the "first" component of the compound objective - i.e. in NSE with log flow duration, the weighting you specify will apply to a "regular" Nash Sutcliffe efficiency on the observed and predicted data (1-Weighting) and will apply to the NSE on the logged flow duration curves.

When you have finished loading time series, click Next.

Define calibration period

In this wizard step, (Figure 2), you define the period over which the calibration should be performed for each observation site.

The following dates are required:

• The start and end dates for the overall simulation, which is constrained by the available input data (typically climate records) - with the goal of providing a warm up period for each gauge. The Simulation End Date is automatically set to the latest Calibration End Date; and

• The Start and End dates for calibration at each site, which are constrained by the available observation data for the site - allows the calibration to take place over a subset of the observed data. This might be in order to save some data for validation, or to filter out poor quality observations.

Source provides Start date and End date values which represent the lowest common denominator of the time series you loaded in the preceding step and all other input time series that have been used within the Source model (such as rainfall and evaporation data). Source also proposes a warm-up period; calculation of the objective function for the calibration commences after the simulation reaches the Warmup end date. You can adjust all dates as required and then click Next.

As an example, consider the calibration of a model with a climate record from 1 January 1950 to 31 December 1999 (as shown in Figure 3). There are two observations sites, with the following observed streamflow records:

• S1: 1 January 1960 - 31 December 1989; and
• S2: 1 January 1975 - 31 December 2010.

Figure 3. Timeline for calibration period

Consider the following valid calibration periods:

• S1 starts on 1/1/1960 and ends on 31/12/1989; S2 starts on 1/1/1975 and ends on 31/12/1999 - for a simulation start date of 1/1/1959 and end date of 31/12/1999, the calibration periods of both S1 and S2 are valid as they fall within the simulation period; and
• S1 and S2 have the same start (1/1/1975) and end (31/12/1989) dates; the simulation start and end dates are 1/7/1974 and 31/12/1989 respectively - a subset of both S1 and S2 are used with a 6-month warm-up period for both records.

The period from the start date to the warm up end date should be sufficiently long to ensure that the soil moisture and groundwater stores represented within the rainfall runoff models have a sufficiently long warm up run period. That way, the volume of water in each of the stores will not be influenced by the volume that was set within those stores at the start of the warm up period. For most catchments and models, a warm up period of between 3 and 12 months should be sufficiently long for this to occur.

Working with metaparameters

In the calibration tool, metaparameters are parameters defined specifically for the purposes of calibration, with each metaparameter mapping to one or more parameters in the underlying model. In this way, metaparameters allow you to selectively reduce the number of independent model parameters that need to be calibrated. Each metaparameter defines the allowable ranges across which each of the associated model parameters can be varied as the calibration executes. Any model parameter which is not associated with a metaparameter will not be varied during a calibration run and will remain at the same value as it was set to during the set up of the model via the Edit menu before entry into the wizard.

Metaparameters can be extremely fine-grained with a one-to-one relationship between a metaparameter and a model parameter, coarse-grained where a single metaparameter controls all the like-named model parameters in the model, or anything in between. Metaparameters are created by grouping one or more like-named model parameters and, accordingly, it can be helpful to treat metaparameter and group as synonyms.

In Source, each node, link, catchment and functional unit is a model in its own right, with its own, independent parameters. Thus, even when a catchment model is built using a rainfall runoff model with a small set of parameters, such as GR4J which has four parameters, the application of the model across multiple functional units in many catchments has a multiplicative effect on the number of model parameters.

Defining metaparameters

The Metaparameter definition interface (Figure 4) is divided into a display table of metaparameters and three functional panel frames.

• Existing Meta Parameters - displays all exiting metaparameters in the table. Metaparameters can be created from the previous version of Source,  or just created.
• Meta Parameter Creation - This panel frame can be used to create the metaparameter for different model types:

Define from catchment model: Clicking on this button will display the interface (Figure 5) to define the metaparameter for the catchment rainfall runoff models or Constituent models. The routing link associated with a catchment model can be also defined here;

Define from Links: Clicking on this button will display the interface (Figure 6) to define the metaparameter for the link routings; and

Define Custom: The user can use this button to create the metaparameters undefined by Source.

• Selected Meta Parameter(s) - This panel can manage the existing metaparameter records in the table of Existing Meta Parameters:

Edit: this will open the interface (Figure 7) for editing the selected metaparameter record in the table in Existing Meta Parameter; and

Delete: this will delete all selected metaparameter records from the table of Existing Meta Parameter.

• Others – this panel provides the function to create/edit the other items involved in the calibration such as input sets calibration property, enforced relationship:

Edit Calibration Input Set properties: This button allows to open the interface of Calibration Input Set properties to edit or enter the Input Set calibration properties; and 

Edit Relationships: This button will define the enforced relationship between the metaparameters during the calibration. Such relationship normally has an evidence or physical meaning.

Some functions will be detailed in the separated sections late in this page.

Once you have finished defining and editing metaparameters, clicking on Next to go to next step (Figure 11).

Figure 4. Calibration Wizard (Metaparameter definition)

In the calibration tool, metaparameters are parameters that are defined specification for the purposes of calibration, with each metaparameter mapping to one or more parameters in the underlying model. In this way, metaparameters allow you to selectively reduce the number of independent model parameters that need to be calibrated. Each metaparameter defines the allowable range across which each of the associated model parameters can be varied as the calibration executes. Any model parameter which is not associated with a metaparameter will not be varied during a calibration run and will remain at the same value as it was set to during the set up of the model or via the Edit menu before entry into the wizard.

Creating metaparameters for Rainfall Runoff models or Constituent models

To create a metaparameter for a rainfall runoff model/Constituent model:

Filter:

• Clicking on the button Define from catchment model (Figure 4) to display the interface (Figure 5);
• Select one from Rainfall Runoff or Constituents  in the top left corner (Figure 5) for the calibrated model (rainfall runoff model or constituent model);
• Select one or more functional units (Figure 5). You can add functional units to the selection using either the shift or control keys. You can select all functional units by clicking AllFUs; and 
• Select one or more sub-catchments. (a) If you select a node in Figure 1 ( and only one gauge node in Figure 1), only associated upstream catchments are displayed on the map in Figure 5 and you can easily to define to catchment(s)  using Select All button. (b) if you want to define a point from the whole catchment on Figure 5, you can add sub-catchments to the selection using the Control key or by dragging an extent across the map of catchments. Whenever at least one FU and one sub-catchment is selected, a list of available model parameters will be displayed in the right panel ( i.e.. Available Parameters to choose from). A model parameter is only available in the right panel if it is not already associated with a metaparameter.

Group:

• From the list of available model parameters, select one or more like-named parameters. You can add model parameters to the selection using either the Shift or Control keys;
• Click Group Selected. This creates a metaparameter and associates it with the selected model parameters. The selected model parameters are then removed from the list of available model parameters; and
• In addition to the process outlined above, you can also create metaparameters using the AutoGroup All button. Auto-grouping creates one metaparameter for each distinct type of model parameter. After clicking AutoGroup All, the list of available model parameters will always be empty – for that combination of functional units and sub-catchments.

Result:

• Clicking on AutoGroup All or Group Selected will also close the interface ( Figure 5) and created metaparameter (s) will be added to the table of Existing Meta Parameter (Figure 4).

Figure 5. Calibration Wizard (Create metaparameter for catchment model)

You continue the process of creating metaparameters until all of the model parameters you wish to have varied during the calibration run have been associated with metaparameters, or the list of available model parameters is empty. Remember that the available parameters list depends on your FU and sub-catchment selections.

Creating metaparameters for the Link routing

To create a metaparameter for the link routing:

• Clicking on the button Define from Links (Figure 4) to display the interface (Figure 6);
• Select one or more links by clicking on the check box under Show Parameter. The user can click on the check box near the title of Show Parameter to select all links. Clicking will display all available parameters, which can be calibrated, in the right panel (i.e., Parameter frame box);
• From the list of available model parameters, select one or more like-named parameters. You can add model parameters to the selecting using either the Shift or Control keys;
• Clicking on AutoGroup All or Group Selected will create a metaparameter (s) and associate it with the selected model parameters. The selected model parameters are then removed from the list of available model parameters; and
• The Interface (Figure 6) will be then closed and created metaparameter(s) such as $StorageRouting_InflowBias and $StorageRouting_RoutingConstant will be added to the table of Existing Meta Parameters (Figure 4).

Edit metaparameter 

The name of each metaparameter is constructed from the name of the model parameters with which it is associated. This may produce naming collisions where the same model parameter name is used in different types of functional unit. To avoid a naming collision, you may need to rename one or more metaparameters to make them unique. To rename a metaparameter and/or edit parameter ranges, either first select the metaparameter and then click Edit  ( Figure 4) or double-click the metaparameter  in the table of Existing Meta Parameters to open the dialog shown in Figure 7. Then:

• Edit the Name field so that the name is unique. To differentiate between parameters that have the same name, but are from different FUs, enter a suffix at the end of the parameter;
• Change Units to match the model parameter if required;
• Click Minimum and Maximum checkbox to specify the Minimum and Maximum fields - the are defined target ranges for a calibration. Default values are included when you open the dialog; and
• Click OK.

Figure 7. Calibration Wizard (Edit metaparameter)

Creating Custom metaparameters and Edit Calibration Input Set properties

Custom metaparameters and Edit Calibration Input Set properties functions often work together. Although the Edit Calibration Input Set properties function allows the user enter/edit Input Set calibration properties, it was designed for calibrating metaparameters undefined in the catchment model or link routing model such as a custom metaparameters. The users should still use Scenario Input Sets tool to configure the normal input sets. Also, metaparameter, which can be created from the catchment model or link routing tool, does not need a duplication in Calibration Input Set calibration properties.

Click on Define Custom (Figure 4) will show the interface (Figure 8a) to enter the metaparameter name such as $CustomMetaParam_2, and OK button will close the Interface and add this custom defined metaparameter to the table of Existing Meta Parameter (Figure 4) for further editing such as unit, min, max.  The user normally needs to define the unit, minimum and  maximum for Custom metaparameters ( Figure 8b) by Edit button in Figure 4.

Edit Calibration Input Set properties (Figure 4) function provides the link between (Custom) metaparameter and calibrated parameters. Clicking on Edit Calibration Input Set properties button will show the interface (Figure 9).

For example, the link is Generated link name #2 and its original Loss Gain Relationship is [[0 0][10 0][50 1][100 2][1000 3][100000 10][1000000 50]]{ML/d ML/d}. For four loss values in  [50 1][100 2][1000 3][100000 10] , the Loss Gain Relationship needs the calibration, and four custom defined parameters: $CustomMetaParam_1, $CustomMetaParam_2, $CustomMetaParam_3 and $CustomMetaParam_4  are created from above Define Custom function.

The Calibration Input Set properties should be like below (Figure  9):

Links.Generated link name #2.Loss Gain Relationship=[[0 0][10 0][50 <$CustomMetaParam_1>][100 <$CustomMetaParam_2>][1000 <$CustomMetaParam_3>][100000 <$CustomMetaParam_4>][1000000 50]]{ML/d ML/d}

Figure 8. Calibration Wizard (Define Custom metaparameter and Edit it)

Figure 9. Calibration Wizard (Edit Calibration Input Set properties)

OK button (Figure 9) can save the Calibration Input Set Properties.

Edit Relationships button (Figure 4) can define the relationship during the calibration for all metaparameters. For example, the Loss Gain Relationship curve (Figure 9) is often monotonic (single value and increased with the discharge). The relationship $CustomMetaParam_2 >CustomMetaParam_1 and $CustomMetaParam_3>CustomMetaParam_2 and $CustomMetaParam_4>CustomMetaParam_3 can meet this requirement. The definition inface looks like those in Figure 10.

Figure 10. Calibration Wizard (Edit Relationships )

You continue the process of creating metaparameters until all of the calibration model parameters have been associated with metaparameters. Click Next (Figure 4) to go to next step (Figure 11).

Configure optimisation functions

In this wizard step you choose an optimisation function (Figure 11) and then adjust its parameters. The available optimisation functions are:

• Shuffled Complex Evolution (SCE) - a global optimiser that learns the parameter set for calibration from previous runs;
• Uniform Random Sampling - another global optimiser that randomly selects a parameter set from the available metaparameter list and runs the model. It then picks the best as the calibrated set. One disadvantage is that it does not learn as the process progresses;
• Rosenbrock - a local optimiser;
• Manual optimisation - allows you to manually edit metaparameter values; and
• SCE then Rosebrock – first use SCE approach then use Rosenbrock approach to continue the calibration for SCE’s results.

If you enable the Use values of metaparameters from your most recent calibration? checkbox, this will use the current metaparmaters as the starting point for the optimisation algorithm during the next round of calibration.

A typical example would be at the completion of metaparameter optimisation using SCE, the "best set" of metaparameters is used as the initial parameter set or "seed" for fine-tuning the metaparameter range using the Rosenbrock optimiser. All optimisation functions allow you to set a limit on the number of iterations. Shuffled Complex Evolution offers additional parameters but these should be left at their default values.

If you enable the Manually set seed of automatic optimisation algorithm checkbox, this will use the entered value (such as 1) as the seed for the calibration, otherwise a random seed is used. This is useful for testing and debugging. 

If you enable the Run in parallel checkbox, this will run the calibration parallelly and more details are given in Parallel calibration section below.

If Create running configuration and functions for use from command line /service checkbox is ticked, this will create a new running configuration in the scenario and create functions in the project file and this data is helpful for command line or sever run.

If you enable the Manually set SCE parameters checkbox, this will enable the SCE parameters in SCE Advanced Parameter frame for the manual adjustment.

Shuffled Complex Evolution

The Shuffled Complex Evolution-University of Arizona (SCE-UA, or often just SCE) is an efficient global optimiser that is commonly used for many models, including rainfall runoff models.

The SCE algorithm is designed to optimise the metaparameters that are assigned for the models within Source. It has its own parameters that control the optimisation process. The algorithm involves selecting a number of sets of metaparameter values at random. The model is run and the objective function is calculated for each of these sets of metaparameter values. The sets of metaparameter values are formed into a number of groups, or "complexes". The user can control:

• P - the number of complexes to be run; and
• M - the number of sets of metaparameters in each complex.

During the first step of optimisation, the optimiser will select, at random, P multiplied by M metaparameter sets and evaluate the objective function for each of those metaparameter sets. 

Following this, the complexes are shuffled with metaparameter sets randomly redistributed into new complexes. At this point, there is a process of competitive complex evolution. In this step, new metaparameter sets are derived from the metaparameter sets within each complex. The evolution is competitive because although there is a random selection, the metaparameter sets that return better values of the objective function receive higher weighting in the selection process and are therefore more likely to be selected. The new evolved metaparameter sets can (at random) replace the metaparameter sets that were within each of the original complexes.

The number of shuffles (i.e. the number of times that the SCE process is completed) is set by the user.

There are three parameters that control the complex competitive evolution process:

• Q – the number of metaparameter sets that are selected at random from within each complex to form the parents for the evolution process;
• α – the number of times that mutation of parameter sets should be attempted; and
• β – the number of new offspring that should be generated within each complex.

The parameters have default values as defined below, but you can change these values when you enable the Edit advanced SCE parameters (for advanced users) checkbox:

• α = 1;
• β = 2N + 1;
• P = 5
• M = 2N + 1;
• Q = N + 1

where N = number of metaparameters selected by the user for optimisation.

There is no specific guidance on P, the number of complexes, other than that it should be greater than 1. In practice, a value between 4 and 8 would be appropriate, and Source uses a default value of 5. For example, if you have 10 metaparameters in your model and you select M = 21 (in accordance with the above guidance) metaparameter sets in each complex, and select 5 complexes (P = 5); then the first 105 iterations of the model are all randomly sampled from within the feasible parameter space. Only at this point, would shuffling occur and new parameter sets be generated via the competitive complex evolution process.

The number of shuffles, along with the other parameters, controls the maximum number of iterations that the model will be run for.

Manual optimisation

Manual optimisation allows you to manually edit metaparameter values, and then run the model. Choose Tools » Calibration » Result to open the Simulation Runner dialog. Click on the Create new meta parameter set button and enter the desired metaparameter values in the appropriate column in the Meta parameter set editor table. Click on Simulate with current meta parameter set to start the calibration.

Calibration Runner

Click Finish to complete the wizard. If you need to make changes in any of the preceding steps, start the wizard again as described under Calibration Wizard, click Next until you reach the appropriate screen, edit as needed, then click Next until you reach the end of the wizard, and click Finish. An alternative method is to select a specific step of the calibration wizard fromTools » Calibration.

Once a scenario has been configured for a calibration run using the wizard, you can start the calibration simulation run. Click Run on the Simulation toolbar. The Simulation Runner dialog (left of Figure 12) opens and the calibration run begins.

A calibration run is a series of iterations. By default, the iterations will continue until you click Stop, which provides an option of stopping the calibration, then running the simulation with the best result, or ending it altogether.

The Simulation Runner dialog shows two traces:

• The measure returned by the objective function from each iteration, with the leading edge of the trace being the most recent iteration; and
• The best measure obtained thus far. The value of the best measure is also shown in the legend.

Measure are expressed in the units returned by the objective function. For example, the Nash-Sutcliffe Coefficient of Efficiency (R2) can range from -∞ to +1.0. The term "best" also needs to be interpreted by reference to the objective function.

You can specify both the minimum and maximum values to display for the objective function. Enable Set cap on run graph in the Simulation Runner dialog and enter the desired Min and Max values. Compare the run graph for one that does not have any limits (left of Figure 12) to one that has the minimum and maximum values specified (right of Figure 12). Note that the Min and Max values correspond to the y-axis on the graph.

Inspecting calibration results

When the calibration run completes, Source automatically selects the metaparameter values associated with the "best" iteration, applies those to the model, and executes a normal run using the same date-range as was defined at Define calibration period. This includes the warm-up period.

The results of that run appear on the Simulation Runner dialog (Figure 13). Each result is an overlay of as-modelled vs as-observed. Using this dialog, you can also inspect the metaparameter values for each iteration. The information in this window is organised in rows and columns. Each row represents one iteration. The first column contains the measure returned by the objective function for each iteration. The remaining columns contain the metaparameter values applied to the model parameters for that iteration. 

Once a simulation run has been completed, the metaparameter values from the best run are listed in the Meta parameter set editor table (below the graph). You can simulate the model with this parameter set by clicking Simulate with current meta parameter set or choose a different set (click Create new meta parameter set to enter metaparameter values for manual optimisation).

The Meta parameter sets table shows how calibration was configured, and checks the degree of consistency in the calibration runs. The rows of the results table are sorted by objective function. You can click column titles to change the sort order. For example, clicking the left-most column title once sorts the table into ascending order of objective function measure. That is, the first row of the table will reflect the "worst" iteration. Clicking the same column title a second time will sort the table into descending order of objective function measure, so the first row will then contain the "best" iteration.

You can also export the results to Microsoft Excel. Additionally, you can select one row at a time and press Ctrl+C to copy to the clipboard. Note that you cannot select individual cells. You can select also metaparameters values and copy to the clipboard.

To check how well the calibration performed, check the simulation results for other nodes or links by clicking on their respective tabs on the top left of the Simulation Runner dialog (left side of Figure 8). Click on the desired tab, then choose the node from the drop-down menu, and click View. This shows the model and gauge flow in Results Manager. To undertake further investigations for selected periods using logarithmic curves or the Statistics tab, for example, refer to Results Manager.

In addition to calibrating individual values, you can also calibrate a function using the /wiki/spaces/SD520/pages/55214470.

Parallel calibration