Overview
The SubSource plugin is developed to run a second Source model at different points during the main model run. This plugin is typically used to forecast water allocations or integrate different time steps in the operation model. The sub-model is run using the Source Service.
The basic concept, as shown in Figure 1, can be explain as follows:
The main model (blue) steps through the simulation period, using some information from the sub-model (yellow) to calculate water allocations. At each call from the main model, sub-model (yellow) simulates the scenario such as a worst dry case until the end of the water year (or a defined end time by the user) to estimate harvestable inflows and losses for the main model.
Figure 1 The basic concept of SubSource
Although the plugin was originally developed for a specific project, it has since been extended to be a flexible generic tool and may be used to address different needs in more complex water system models. The main model and sub-model scenarios can be defined in the same Source project file or separate Source project files. It is also possible to use the same scenario for both the main model and sub-model using different running configurations, input sets, and recorder sets. The developed interface allows the user to choose different parameter values for their requirements. The models can use catchment mode or network mode. Users can design their own data mappings between the main model and sub-model.
| Table of Contents | ||
|---|---|---|
|
Basic procedure for
use of the SubSource plugin
The basic procedure for the use of the SubSource plugin is described below:
- Step 1 Create the Source model using SubSource function
Load the SubSource plugin from CommunityPlugins\SourcePlugin.SubSource.dll in the Plugin Manager.
- Create a new project (or alternatively open an existing project to add the plugin to)
- Step 2 Create models
- Step 3 Prepare the mapping between the main model and sub-model.
Make the Main Model (scenario) active
Select Sub-model analysis from the Analysis Type dropdown list (see Figure 2(3))
Click Configure (see Figure 2(4)) on the Simulation toolbar to display the Sub-Model Analysis Configuration dialog (Figure 3)
Set up Sub-Model Analysis configuration and mapping.
- Step 4 Save the Source project file(s) after the Sub-model analysis has been set up or modified.
- Although the latest version of the SubSource plugin can automatically save the project to a temporary location prior to run, it is recommended to save the project to make sure changes to the the sub-model are applied.
- Step 5 Run
- the main model using the defined Sub-model analysis configuration.
Figure 2 SubSource plugin components
Sub-Model Analysis configuration
The Sub-Model Analysis running configuration is based on the Single Analysis running configuration with one additional section, Sub-Model (see Figure 3).
The Sub-Model section allows the configuration of:
- the mapping between the main model and sub-model; and
- the parameters for the sub-model run;
Figure 3 Sub-Model Analysis Configuration dialog
Main model to sub-model mapping
Select the mapping to use for passing values between the main and sub-models. Click Edit to open the Sub-model Mapping Manager to modify or create new mappings.
Sub-model run parameters
The options in the lower section of the Configuration tab (Figure 3) are for controlling the sub-model run. In this section, "Main Model" is used to denote values / settings taken from the current state of the main model when the sub-model run is triggered, while "Sub-Model" denotes the original values / settings defined in the sub-model running configuration. Using "Main Model" will update the running configuration of the sub-model prior to starting the sub-model run.
Time Step: this parameter provides three options (i.e. From Main Model, From Sub-Model and Custom Time Step) to set the simulation time step for the sub-model run:
From Main Model: the time step used in the sub-model run will be the same as the one defined in the Main Model section of the Configuration tab.
From Sub-Model: the time step used in the sub-model run will not be modified, using the the original running configuration setting.
Custom Time Step: this option will display an additional dropdown list for the user to select the required time step. Users need to consider the time step used in the main model and data sources in the sub-model for a reasonable matched time step.
Start Time: this parameter provides four options, called From Main Model, From Sub-Model, Custom Start Time and Main Model Trigger Time, to set up the simulation start time for sub-model run. The first three options are self-explanatory, and their mechanisms are same as those in Time Step. Specifying the Main Model Trigger Time option, the default option, will use the time when the main model triggered the sub-model run as the simulation start time of the sub-model run. For example, if the main model triggered a sub-model run on the 1st September 2000, the start time of the sub-model run will be modified to simulate the water process from 1st September 2000.
Run Period: this parameter provides four options (i.e. From Sub-Model, End of Water Year, Number of Time Steps and Custom End Time) to set up the simulation period for the sub-model run:
From Sub-Model: the sub-model run period will be adopted from the sub-model's running configuration. The run end time could be different to the one defined in the sub-model's running configuration if the selected
Start Time is not also set to From Sub-Model.
End of Water Year: the run period will be from the simulation start time to the end of the water year. This option will display an additional entry field (at the bottom of the tab) to define the water year.
Number of Time Steps: the sub-model run period will be a fixed number of time steps from the simulation start time. This option will display an additional entry field for the number of timesteps to run.
Custom End Time: allows the user to specify the end time to be used for the sub-model run. This option will display an additional entry field to enter a date as the simulation end time.
Run Frequency: this parameter provides five options (i.e. Start of Week, Start of Month, Start of Water Year, Number of Time Steps and Custom Trigger) to specify when the sub-model will be triggered during the main model run:
Start of Week: the sub-model run will be triggered at the beginning of each week.
Start of Month: the sub-model run will be triggered at the beginning of each month.
Start of Water Year: the sub-model run will be triggered at the beginning of each water year. This option will display an additional entry field (at the bottom of the tab) to define the water year.
Number of Time Steps: the sub-model run will be triggered after a recurring number of timesteps. This option will display an additional entry field for the number of timesteps.
Custom Trigger: the sub-model run will be triggered when the specified Boolean function evaluates to True. The function can
use system conditions (such as the water level at
a reservoir
reaching a certain level
or the flow rate at a node exceeding a threshold) to determine whether to trigger the sub-model run. The function time of evaluation must be prior to when the sub-model run will be triggered, such as End of Timestep. This option will display an additional entry field for selecting the function.
Water Year: this parameter is linked to options of End of Water Year and Start of Water Year. It is used to specify the start day and month of the water year.
The user needs to consider the above parameters together to define their values in a systematic approach.
The models also need a proper design and debugging, especially for data transfer between main model and sub model.
The latest version of Source can read the meta data from the loaded project even if that project was created by a previous version of Source with a different user interface for SubSource function.
Sub-model Mapping Manager
The Sub-model Mapping Manager dialog (Figure 5) is used for defining mappings between main and sub-models. A Source scenario may have multiple mappings defined and each mapping may be used by multiple Sub-Model Analysis running configurations within the scenario. Mappings are defined in the main model scenario. No mappings settings areSub-model Mapping Manager
The Sub-model Mapping Manager dialog (Figure 5) is used for defining mappings between main and sub-models. A Source scenario may have multiple mappings defined and each mapping may be used by multiple Sub-Model Analysis running configurations within the scenario. Mappings are defined in the main model scenario. No mappings settings are required in the sub-model scenario (other than the required functions and recorders).
The Sub-model Mapping Manager dialog (Figure 5) may be accessed either by clicking the Edit button next to the Main model to sub-model mapping setting in the Sub-Model Analysis Configuration dialog (Figure 3), or from the Source Tools menu, Tools » Sub-model Mapping Manager (Figure 4).
Figure 4 Access Sub-model Mapping Manager from the Tools menu
Figure 5 Sub-model Mapping Manager dialog
Sub-model
mapping management
The left
pane of the dialog (Figure 5
Right click on any empty place in the left box of Figure 5 will display the Add Mapping button shown as Figure 6. Click on the Add Mapping will display all parameters, which are associated with the definition of a Sub-model Mapping, in the right box with empty value.
Figure 6 Add Mapping
Right click on any name in the left box of Figure 5 will display the Delete button and click on this Delete will delete this name (, or a definition of Sub-model Mapping,) from the modelling.
Click on any name in the left box of Figure 5 will display all parameters values associated with the definition of this selected Sub-model Mapping in the right box. The user can display or edit this sub model mapping definition.
The details of all parameters associated with a definition of one Sub-model Mapping are given below:
) lists the sub-model mappings defined in the scenario, allowing for adding, deleting and editing of mappings:
- To add a new mapping, either click the Add Mapping toolbar button or right click an empty location in the list and click Add Mapping from the context menu. A new mapping will be created and selected, displaying parameters in the right pane.
- To delete a mapping, select the mapping from the list and either click the Delete toolbar button or right click the mapping entry and click Delete from the context menu. The mapping will be deleted from the scenario. Any running configurations which referenced the mapping will need to be edited to specify a different mapping.
- To display and edit a mapping click / select the mapping from the list to display the parameters in the right pane.
Sub-model mapping parameters
Sub-model mapping parameters are displayed in the right pane (Figure 5) when a mapping is selected. The details of all parameters defined in a sub-model mapping are as follows:
Mapping details
Name: the unique name of the current
sub-model
mapping.
Use Current Project:
this option controls whether to use the current Source project file
or a different project file for selecting the sub-model scenario. Ticking this option will load all scenarios from the current project into the Scenario dropdown list
. Unticking this option will enable the Project Path field for specifying a different Source project file.
Project Path: this parameter allows the user to choose a Source project file (other than the current project file)
for selecting the sub-model scenario. It is disabled when Use Current Project is
checked.
Loading…: this progress bar will display the progress information loading all scenarios in the project file to Scenario dropdown list ( talk below).
Sub-
model Running Configuration
Scenario:
select the scenario from the dropdown list to use as the sub-model. All scenarios in the
specified Source project file are available for choice.
The sub-model scenario may be either another scenario in the current project, a scenario in another Source project file, or the same scenario as the main model in the current project.
Override Run Configuration: this
option controls whether to use the default running configuration defined in the selected scenario or to specify another running configuration from the Configuration drop down list below. When ticked, the Configuration dropdown list will be
enabled.
The Running Configuration names are defined by Running Configuration Management from Scenario Options menu.
Configuration dropdown list: It provides the available options of Running Configuration names (such as single analysis, linked analysis, custom 1) for Sub-Model. The model will run Sub-Model using selected Running Configuration name from this parameter and ignore default one in the sub model / original scenario. It is only available when the Override Run Configuration tick boxConfiguration: select the running configuration from the dropdown list to use for the sub-model run. Only Single analysis running configurations are supported for sub-model runs. The listed running configurations are those defined within the Scenario Options » Running Configuration Management settings for the selected sub-model Scenario. The parameter is only enabled if Override Run Configuration option is ticked.
Scenario Input Set:
Displays the Scenario Input Set for which will be used for the sub-model run. Currently cannot be modified.
Variable Mapping From Main Model to Sub-
Model
In this list the user can define data transferred from the main model to
the sub-model at the time the sub-model run is triggered:
Main Model Value:
specifies the source of the value in the main model
; a specific value, a data
source timeseries, or a function.
Sub-Model Function:
Note that:
The time for the main model is the end of last time step while the time trigged to run the Sub-Model is the start of the time step. For example, the main model (daily time step) trigged Sub-Model (Monthly time step) to run from 1 September 2000. The time for the main model is the end time step of 31 August 2000, and the condition (data) at this time step will be used to setup the initial condition of Sub-Model. The time for Sub-Model is the start of 1 September 2000. This time will be used to receive the initial data from the main model. These concepts are practically for Time of Evaluation.
For the function in this frame, the user can add as many as they need.
Each function only allows one value to be used for the data transfer. If the time series is used, the last value will be taken.
All initial conditions of the water system in Sub-Model need the setting up such as storages' initial volume, water account balance in RAS. These initial values often are passed though the functions from the man model to Sub-Model.
“Run Initialisation" and "Start Of Run” of Time of Evaluation are specifically developed for this plugin:
"Run Initialisation" functions are functions which are used when initialising the run and network elements. Functions which are set to this Time of Evaluation cannot use Model Variables
"Start Of Run" will be evaluated after initialisation (reset), and can be used to prime functions and Model Variables with calculated initial values
From Variable Mapping From Sub-Model to Main Model frame, the user ca use it to set up data transfer from Sub-Model model to the main:
Sub-Model Recorder: it is a recorder in Sub-Model to save desired running results as a data source and this source value(s) will be signed to an objective function in the main model when Sub-Model completed a running process.
Main Model Function: it can be used to specify a function (existing in the main model), which will adopt the value(s) from Sub-Model Recorder when Sub-Model completed a run process. Again, the user needs to select a relevant function to transfer the data.
Note that:
The user only needs to save the recorders, which are necessary for data transfer, in Sub-Model. This can reduce the model running time and the computer memory.
The principle of above notes for Variable Mapping From Main Model to Sub-Model is also applied for this frame.
Select a mapping
Th selected mapping in Figure 5 by the user will be displayed on the dropdown list of Mapping such as “my Sub Model Mapping” in Figure 3, and this mapping will be used during the model running. The user also can directly select the existed /pre-defined mapping items from the dropdown list of Mapping.
Note that if the main model and Sub-model are from different project files, the user must remember to save the Source project file before you run the mode because the setting up is functionalized only after the project is saved.
tocspecifies the function in the sub-model scenario which the main model value will be set to. All functions in the sub-model scenario are listed in the drop down list which also supports filtering and auto-complete if the path to the function is typed. The text of the function will be overwritten with the value from the main model. Use of the function in the sub-model will need to be setup as normal for functions.
To add a new mapping entry, double click the empty row at the bottom of the list. To delete a mapping entry, click the entry in the list to select it and press the Delete key on your keyboard.
Note that:
- Each function only allows a single value to be passed from the main model to sub-model. Arrays, multi-value variables or modelled variables using date ranges are not supported. If a time series is used, the value for the current main model time step is used.
Generally, all initial conditions of the water system in the sub-model need to be set from main model values, such as storage initial volumes, water account balances in RAS. These initial values should be defined using functions in the sub-model and corresponding mapping entries added in the mapping defined in the main model.
Run Initialisation and Start Of Run function Time of Evaluations were specifically added to Source for use by SubSource and should the the Time of Evaluation used for functions within this list:
Run Initialisation functions are evaluated and used when initialising the run and network elements.
Start Of Run functions are evaluated and used after initialisation (reset), and can be used to prime functions and modelled variables with calculated initial values
Variable Mapping From Sub-Model to Main Model
In this list the can define data transferred from the sub-model to the main model after the completion of a sub-model run:
Sub-Model Recorder: specifies a recorder in the sub-model scenario. The text box supports auto-complete for searching for a recorder.
Main Model Function: specifies the function in the main model scenario which the value from the sub-model recorder will be set to. All functions in the main model scenario are listed in the drop down list which also supports filtering and auto-complete if the path to the function is typed. The text of the function will be overwritten with the value from the sub-model recorder. Use of the function in the main model will need to be setup as normal for functions. These function will appear to be constant functions to Source so it is crucial that they have Force Evaluate set to true for their values to be updated and used by the main model.
To add a new mapping entry, double click the empty row at the bottom of the list. To delete a mapping entry, click the entry in the list to select it and press the Delete key on your keyboard.
Note that:
The sub-model only needs to have the recorders for the mappings enabled. All other recorders should be disabled in order to reduce the sub-model running time and memory use. It is recommended to use recorder sets for this purpose.
- Only the last recorded value of the sub-model recorder is assigned to the main model function. Arrays, multi-value variables or modelled variables using date ranges are not supported. If multiple values are required then functions should be defined in the sub-model which use date ranges and modelled variables to provide values from previous timesteps and these functions recorded.
Running the analysis and general guidelines
The sub-model analysis run is run from the main model in the same way as other analysis types in Source. The sub-model run will be run when triggered from the main model using the Source Service in a separate process.
Points to consider when designing the main and sub-models:
- When triggered, the sub-model run will run between the End of Timestep and Start of Timestep simulation phases. Variable mappings from the main model to the sub-model will thus use values from the end of the time step before any new values are applied (functions evaluated at Start of Timestep and timeseries values played onto a model). Similarly, variable mappings from sub-model to main model will have values applied to the main model before Start of Timestep so that any functions evaluated at Start of Timestep in the main model can use the values from the sub-model run.
- For example, the main model (daily time step) triggered the sub-model (monthly time step) to run from 1 September 2000. The time for the main model is the end time step of 31 August 2000, and the condition (data) at the end of this time step will be used to setup the initial condition of the sub-model. The time for the sub-model is the start of 1 September 2000.
- See simulation phases and function times of evaluation for further details of timings.
- Sub-model run results are not recorded in the main model results. It can therefore be difficult to debug what is happening in the sub-model runs. To help with debugging a few approaches may be used:
- Auto export results scenario option in the sub-model scenario may be enabled to export run results to a results file and later loaded and viewed in Results Manager. Note that both the Append Run Number and Append Date and Time options should be enabled to prevent subsequent run results overwriting previous run files which may have the same name.
- Auto export simulation log scenario option in the sub-model may similarly be used to export results summaries and log messages from the sub-model runs. It may be useful to use this together with function logging.
- Record functions and data sources used in the main model to sub-model mappings.
- Set up and record additional functions in the sub-model and define functions in the main model to receive these values which and subsequently recorded. Set Force Evaluate to true to make sure the functions are updated correctly.
- It is recommended to create a Single analysis configuration in the sub-model scenario for exclusive use by the sub-model run. A recorder set should be created for this configuration and the Current recorder tree recorder set disabled. This will help prevent runtime and configuration errors due to changes to recorders for doing other types of analysis runs in the scenario.
- When using the same project and scenario as both main and sub-models use of separate running configurations and recorder sets is crucial. It is also important to understand that the functions used in the sub-model run will not contain values from the sub-model if they are recorded during the main model run. The models should be treated as separate models as much as possible to avoid confusion.
- If using the same project file for main and sub-model scenarios, the project file will be saved to a temporary location prior to run so as to get the latest changes made to the project. The project is not required to be saved prior to changing mapping settings as meta-data will be loaded from the current project rather than using the Source Service.
- If using separate project files for the main and sub-model scenarios, the sub-model project file must be saved prior to run if it has being modified in a separate instance of Source. If it is not saved the previous version of project file will be used for the sub-model runs. Similarly, the sub-model project file will need to be saved after any recorder or function name changes in order for these changes to be used in the main model mapping configuration.
- The latest version of Source and the SubSource plugin can upgrade projects created with previous version of the SubSource plugin which previously didn't have the Sub-model Mapping manager.
- If using a new version of Source and/or the SubSource plugin it is recommended to upgrade and save the project file(s) prior to any runs. If it is not upgraded prior to run then the project file(s) will be upgraded each time the Source Service loads the sub-model project file which could take a long time for some projects.
- Although the SubSource plugin is not specifically used by the sub-model project, it is recommended to add SubSource (and all other plugins used in the main model project) to the sub-model project. All plugins present in Source when running the main model will be loaded by the Source Service which will result in a database upgrade being performed on load if the plugins aren't already added to the project.
- Running a sub-model analysis requires two Source processes to be running: the main model project either loaded in the Source UI or through the command line runner; and, the sub-model project loaded through the Source Service. The machine used to run the analysis will thus need to be capable of running thew two processes simultaneously, with enough CPU cores and RAM. Multiple sub-model run results may be held in memory for each single main model run requiring the use of system memory. Using run results streamed to file may help reduce system memory use but will increase physical drive use.