Calibration analysis - SRG

Overview

Description and rationale

Source includes a number of optimisation techniques and statistical measures for automated model calibration and to assist modellers with the evaluation of the quality of calibration.  These are mainly intended for application when calibrating catchment rainfall-runoff models in Source, but are also applicable when calibrating river system models (e.g. see Lerat et al, 2013).  Optimisation techniques available are:

• Shuffled complex evolution
• Genetic algorithms
• Uniform random sampling
• Rosenbrock method

Modellers have the option of selecting one optimisation technique, multiple optimisation techniques (in parallel), or combinations two optimisation techniques (in series).

Automated calibration requires the use of an objective function to direct the optimisation process. The Source calibration tool implements single objective function optimisation which reduces the comparison between the observed and modelled data during the calibration period to a single number to be optimised (Multiple objective optimisation is also available: see Multi-objective optimisation/trade-off analysis - Insight - SRG for information).  The following nine forms of objective function are available in Source:

1. Match to Nash Sutcliffe Coefficient of Efficiency (NSE) of Daily Flows
2. Minimise Absolute Bias between Observed and Modelled Flows (calculated using daily flows)
3. Match to NSE of Daily Flows but Penalise Biased Solutions
4. Match to NSE of Monthly Flows
5. Match to NSE of Monthly Flows but Penalise Biased Solutions
6. Combined Match to NSE and Match to Flow Duration Curve (Daily)
7. Combined Match to NSE and Match to Logarithm of Flow Duration Curve (Daily)
8. Combined Match to NSE of Logarithms of Daily Flows with Bias Penalty
9. Combined Bias, Daily Flows and Daily Exceedance (Flow Duration) Curve (SDEB)

Further information on the first seven of these objective functions is available in Vaze et al (2011), Section 6.  Guidance on model calibration is available in many publications, including various eWater Best Modelling Practice Guidelines (Black et al, 2011; Vaze et al, 2011; Black and Podger, 2012; and Lerat 2012).

Scale

Typically the optimisation techniques and statistical measures are used to compare observed and estimated data at a point, such as streamflow data at a gauging station.  Both the optimisation techniques and statistical measures can be applied on a daily or monthly basis.

Scientific Provenance

The statistical measures used in Source are well established.  They are described in statistics textbooks, hydrology textbooks and papers such as Aitken (1973) and Nash and Sutcliffe (1970).

Overview information on the four optimisation techniques in Source is available in Vaze et al (2011).  Further information is in textbooks and papers, particularly for the genetic algorithm and uniform random sampling[DB1] .  Publications on the shuffled complex evolution method include papers by Duan et al (1992) and Sorooshian et al (1993).  Publications on the Rosenbrock method include the paper by Rosenbrock (1960).

Version

Source version 3.8.8.

Dependencies

Requires observed data suitable for comparison of results from model calibration runs.

Availability

Provided with Source.

Structure & processes

As the optimisation techniques and statistical measures of calibration performance used in Source are well established, they are not re-described here.  However, as the objective functions used in the optimisation techniques have been customised for Source, further information on these follows and as many of them rely on the Nash Sutcliffe Coefficient of Efficiency (NSE), its formulation is restated below.

The traditional formula for NSE is:

where:

Qobs_i    is the observed flow on day i,

Qsim_i    is the modelled flow on day i,

N           is the number of days

Alternatively,

This formulation obviates the necessity to calculate the average of the observed flows before evaluating the denominator in the traditional version.

The choice of any particular objective function will depend on the intended application. Each of the pre-defined objective functions are formulated to put emphasis (reproduce as closely as possible) on different flow characteristics (Vaze et al, 2011).

1. Match to Nash Sutcliffe Coefficient of Efficiency (NSE) of Daily Flows

Application of this objective function involves maximising the NSE (i.e. getting it as close to 1.0 as possible).  The calculation of the NSE is in accordance with Nash and Sutcliffe (1970) and uses observed and modelled daily flow data for all days within the calibration period for which observed daily flow data, including zero flow values (i.e. cease to flow), is available.

The NSE tends to produce solutions that match high and moderate flows very well but often will produce poor fits to low flows. It will also tend to favour solutions that provide a good match to the timing and shape of runoff events (Vaze et al, 2011).

2. Minimise Absolute Bias between Observed and Modelled Flows

This objective function will produce a match on the overall volume of flow generated but often will produce a poor fit to the timing of flows (Vaze et al, 2011). It has the following form:

The evaluation of this objective function uses observed and modelled daily flow data for all days within the calibration period for which observed daily flow data, including zero flow values, is available.

3. Match to NSE of Daily Flows but Penalise Biased Solutions

This objective function is a weighted combination of the daily NSE and a logarithmic function of bias based on Viney et al (2009), and the aim is to find its maximum value. 

where:

B is the bias; and

The evaluation of this objective function uses observed and modelled daily flow data for all days within the calibration period for which observed daily flow data, including zero flow values, is available.

This formulation makes sure that the models are calibrated predominantly to optimise NSE while ensuring a low bias in the total streamflow. It avoids solutions that produce biased estimates of overall runoff, which can produce marginal improvements in low flow performance over the NSE objective function. However, NSE-Bias will still be strongly influenced by moderate and high flows and by the timing of runoff events, which can still often result in poor fits to low flows (Vaze et al, 2011).

4. Match to NSE of Monthly Flows

This objective function works in the same way as for the case “Match to NSE of Daily Flows” except that monthly flows are used to evaluate the NSE instead of daily flows. The NSE calculation ignores observed and modelled data for all months where there are one or more days of missing data in the observed flow series.

The NSE of monthly flows and NSE-Bias of monthly flows (below) can be useful for initial calibration because they tend to find solutions that will match the overall movement of water through the conceptual stores in the rainfall-runoff model, without being influenced by the timing of individual runoff events (Vaze et al, 2011).