This section describes how to load time series data into Source and how to manually enter data in to tables. Additional sub-pages include:
| Child pages (Children Display) |
|---|
Scenario Input Sets allow you to easily keep model structure, while switching between groups of input data and parameters.
| Anchor | ||||
|---|---|---|---|---|
|
There are two types of time series data inputs:
- File – a text-based time series data file in a /wiki/spaces/SD45/pages/54025926. Refer supported file format. Refer to Loading a file data source.
- Scenario – the results of a run from a different scenario can be used as an input for the current scenario. Refer to Loading a scenario data source.
When time series data is loaded in to Source, it is referred to as a data source and is available throughout Source. To either load new data sources, or access previously loaded data sources, you can use:
- the Data Sources Explorer
- the Data Source option in a component's feature editor (Figure 1)
- the /wiki/spaces/SD45/pages/53969981Feature Table
- the /wiki/spaces/SD45SD516/pages/5397010354986558
- the /wiki/spaces/SD45/pages/53970373Command Line Runner Options
- the output of the /wiki/spaces/SD45/pages/53969813Climate data import tool
| Info | ||
|---|---|---|
| ||
Note: Data sources are scenario-specific. If you want to use the same data source in two scenarios, you must load them individually for each scenario. |
Figure 1. Loading a data source from within a component's feature editor
| Anchor | ||||
|---|---|---|---|---|
|
Source allows you to load and manage time series data at a central location using the Data Sources Explorer (Figure 2). Note that all data sources for the active scenario appear in the Explorer regardless of whether they were loaded using the Explorer, or one of the other methods listed above (eg. by using a component's feature editor, Figure 1).
Figure 2 shows features of the Data Sources Explorer containing all data sources available in the scenario, both from time series data files and another scenario. Additionally, the nodes that some of the data sources are associated with are also shown i.
Figure 2.e. the data source usage.
Figure 2. Data Sources Explorer
The contextual menu options allow you to:
- Add Source to add a new file or scenario data source;
- Add Folder to create a new folder;
- Delete the selected file/folder;
- Edit the selected data source. This opens the Load Data Source dialog (shown in Figures
- 5 and
- 6 late) where you can change various features of the data source;
- Export data to a .csv file;
- Rename the input data file/folder; and
- View Data in the charting tool;
Figure 3. Data Sources toolbar
The Data Sources toolbar (Figure 3) has the following functions:
- Filter Data in the data source manager tree. The filter box at the top of the data source manager tree allows filtering of data sources to those with names that match, or have children with names that match the entered text. Filtering also works in a component's feature editor.
Figure 3. Data Sources toolbar
The Data Sources toolbar (Figure 3) has the following functions:
- New Data Source creates a new data source, using either the output of a scenario (Scenario Data Source...) or a time-series file (File Data Source...);
- New Folder creates a new folder, allowing organisation of data sources;
- Expand All expands the file hierarchy;
- Collapse All collapses the file hierarchy;
- Data Sources Overview opens the Data Sources Overview window, displaying metadata for all file and scenario data sources in the active scenario (Figure 3). This is particularly useful for checking the start and end dates of your data. For example, in Figure 3, WheatDemand_csv has an end date a decade before all other data sources;. The user can choose an overview focused on the timeseries or usage (Figure 4a,b). Export button can open Save As dialog to output the metadata overview records to a csv file. The Drag a column header here to group by the column allows to view the metadata records by the group based on the dragged column header. Refresh button can load the latest metadata information to the overview interface.
If selected an overview based on the timeseries (Figure 4a), the metadata will display the overview records based on the time series, which normally appeared as a column-based data in the original source file or scenario output data:
Group Name: The label/name of a file or a scenario data source displayed in Data Sources
Folder Path: The logic folder path where Group Name is located at in Data Sources
Match method: The used method (By Name or By Position) to match columns when reloading the data file
Scenario input sets: The scenario input set name(s) associated with this time series data
Column and Name: The column order number (start from 0) for the time series during the loading or reloading and time series name related the column name. The Date column in the original data file does not count
Start and End. The start and end date of the data content in the time series
Aggregation Method: Which method was setup for the aggregation when the model uses data at a smaller timestep to the time step the model.
Units: The used unit for the time series data
Null Count: How many null ( missing) values in the loaded time series data
Usage: How many times of the time series used in the modelling such as by a network element, a functions, or models.
If selected an overview based on the usages (Figure 4b), the metadata will be displayed based on the usage:
Data Group Name, Folder Path, Column Name, Start, End, Unit (in Figure 4b): Same as those in overview based on the timeseries
Property Name: Name of the property used in the modelling such as rainfall, PET, Constituent Load, Recorded Gauging Station Flow
Feature Type: Network element feature type such as Storage, Gauge, Water User
Network Element Name: Name of Network element feature.
- Reload all time series with 'Reload on Run' reloads all time series when Reload on Run is . This can also be selected from within the Load Data Source dialog (Figures 4 and 5); and
- Sets if a failed 'Reload on Run will stop the current run works in conjunction with Reload on Run to determine if source data will cause a scenario run failure. It is shown toggled on in Figures 2 and 3.
- dialog (Figures 4 and 5); and
Figure 4 a) and b). Data Sources Overview
| Anchor | ||||
|---|---|---|---|---|
|
To load a time series data file, either:
- Click the New Data Source button on the Data Sources toolbar and choose File Data Source... from the drop down menu; or
- Right-click on a Folder and choose Add Source File from the contextual menu.
The Load File Data Source dialog opens (Figure 5). Then, follow these steps:
. Then, follow these steps:
- Click the Load data file button and navigate to and select your time series data file.
The path to the data file is displayed. If you enable the Relative Path check box, the path displayed is the location of the time-series file relative to the project location. Note that the project must have been saved prior to this.Anchor Relative path Relative path - Each column of data in the time series data file becomes a row in the Data table. All time series data in the file is imported by default. You can select what columns to import by toggling the appropriate check box in the Import column.
- If there are empty column headings in the file, there will be an error and you will have to name these columns at this point.
- Choose whether to Match data items By Name or By Position. By default this is set to By Name. This determines how Source will match columns when reloading the data file if the the Reload on Run option is selected or find the equivalent series between each file within the data source if e.g. used for different input sets.
- Choose the scenario input set(s) associated with this data source by clicking on the arrow for the Default Input Set tab and toggling the appropriate check boxes. Refer to Assigning data sources to scenario input sets for more detail.
- Click on Reload on Run to toggle whether Source reloads the data into the internal store when the Scenario is next run (for more information, see below).
- Click on a Default Units cell and choose the appropriate units from the drop down menu. To filter the list of units, select the Default Units input box a begin typing the name of the unit that you require. Right-click on a Default Units cell to bring up a contextual menu that allows you to Apply Unit to all data.
- Select an an Aggregation Method can be set to allow you to use data at a smaller timestep to the time step the model will run at. E.g. you can load daily data and then run the model at a monthly time step if you have selected an Aggregation Method.
Table 1. Aggregation Methods
| Name | Description |
|---|---|
| First Value | Takes the first value. e.g. Going from Daily to monthly, the first value for each month will be used. |
| Max | Uses the Maximum value from the data |
| Mean | Uses the Mean of the data |
| Median | Uses the Median of the data |
| Min | Uses the Minimum value from the data |
| Not Set | Using this option won't allow you to run at a larger time step when the Data Source is being used by the model |
| Sum | Uses the sum of the data. e.g. Going from Daily to monthly, the value used for the month will be the total of all the days added together. |
| Info | ||
|---|---|---|
| ||
Note: It is imperative that you specify compatible units for a time series after loading. If you do not, the time series will either not display in the Data Sources option of the component's feature editor or a run will provide incorrect results. The time series must be visible in the chart view to ensure that the loading process is complete. |
Once a file data source has been loaded, it will be listed in the Data Source Explorer. Its default name is that of the associated file, with "." replaced with "_". Grouped underneath each data source are the time series columns you have selected to import from that file, with the names taken from the column headings in the file (if there are no column headings, Source will name them Column01, Column02, …unnamed columns in the file, you will have to name these columns in the Load File Data Source dialog). The data source and time series can be renamed by right clicking on the item and selecting Rename from the contextual menu. Like functions and variables, each data source must have a unique full name. Two data sources can have a non-unique name if they are in different folders because the full path to the data source is still unique (ege.g. Flows_csv in Figure 2).
| Anchor | ||||
|---|---|---|---|---|
|
| Anchor | ||||
|---|---|---|---|---|
|
| Anchor | ||||
|---|---|---|---|---|
|
| Info | ||||
|---|---|---|---|---|
| ||||
|
You can only load the result(s) of scenario run(s). Therefore, ensure that the donor scenario has been run before you load the scenario data source into the acceptor scenario.
To load the results of a scenario, either:
- Click the New Data Source button on the Data Sources toolbar and choose Scenario Data Source... from the drop down menu; or
- Right-click on a Folder and choose Add Source Scenario from the contextual menu.
This opens the Load Scenario Data Source dialog (Figure 6). Then:
- Click on the ellipsis (…) button on the first column of the Data table, which opens the Results Results dialog (Figure 7).
- This dialog lists the most recent results for each scenario that you have run in your project. Choose the result that you wish to use as input data to your current scenario. In Figure 7, this data is Upstream Scenario Confluence 1's Downstream Flow;
- Click OK to close the Results Results dialog.
- If required, enable the Reload on Run checkbox (see below).
- The units are set automatically from the result chosen.
- Just as with time series, you can select the scenario input set(s) associated with this data source.
- To load another scenario data source, repeat this process.
Once a scenario data source has been loaded, it will be listed in the Data Source Explorer. Its default name is New Group1. Grouped underneath the scenario data source are the results you selected to import from the donor scenario(s). The data source and time series can be renamed by right-clicking on the item and selecting Rename Rename from the contextual menu.
Once you assign a time series to a node or link, the associated component is shown underneath that time series (Figure 2). A time series can be disconnected from a component.
Figure 6. Load Scenario Data Source
Figure 7. New Data Source, Results
Anchor EditingDataSources EditingDataSources
Editing data sources
| EditingDataSources | |
| EditingDataSources |
You can edit a data source after it has been imported to modify the options you selected during loading, eg. the associated scenario input set(s). In the Data Sources Explorer, right-click on the data source you wish to change, and choose choose Edit from the contextual menu, and the appropriate Load Data Source dialog (Figures 5, 6) will re-open.
| Anchor | ||||
|---|---|---|---|---|
|
In many cases, there is a Reload on Run checkbox associated with the controls for loading data into Source. Regardless of whether the data source is a file or another scenario, when you first load a data source, it is loaded into an internal store.
When Reload on Run is enabled, and the Data Source is used in the model, Source will reload its internal store from the original source file each time you click the Run button. Therefore, for every run, your data sources must be accessible, that is:
- For File Data Sources, the source file cannot be open in another program; and the file must be located at the file path specified when you loaded your data. If Relative Path is also enabled, the data sources needs to be accessible at the same file path relative to your project.For Scenario Data Sources, the donor scenario must be run before running the acceptor scenario.
| icon | false |
|---|
- program; and the file must be located at the file path specified when you loaded your data. If Relative Path is also enabled, the data sources needs to be accessible at the same file path relative to your project.
- For Scenario Data Sources, the donor scenario must be run before running the acceptor scenario.
The severity of notification levels on Reload on Run can be customized by accessing Assurance Rule via Edit>>Scenario Options>>Data Sources .
| Anchor | ||||
|---|---|---|---|---|
|
Scenario input sets allow you to easily switch between groups of input data and parameters that represent different conditions, such as normal, wet and dry (see /wiki/spaces/SD45/pages/53970066Scenario Input Sets for more information). Data sources are assigned to specific scenario input set(s) using the Load Data Source dialogs (Figures 5, 6).
Each scenario has a Default Input Set created automatically when you create the scenario, and by default all loaded data sources will be assigned to this root-level scenario input set. To create another root-level scenario input set, see/wiki/spaces/SD45/pages/53970066 Adding Scenario Input Sets. It is also possible to create a/wiki/spaces/SD45/pages/53970066 child scenario input set, which inherits all parameters and data sources specified in its parent. However, data sources can only be assigned to root-level scenario input sets.
Once configured, a file data source may group several time series data files together; up to one file per scenario input set. Assign a file data source to root-level scenario input set(s) as follows:
- Open the Load file data source dialog (Figure 5) either by creating a new data source or editing an existing one (see Loading a file data source for more information).
- The first time series data file you load is associated with all scenario input sets that exist when it is loaded.
- For each additional time series data file you want to add to the data source:
- Create a new tab by clicking the Add New Tab button. By default new tabs are not associated with any scenario input sets.
Load the time series data file (see Loading a file data source for more information)
Info icon false Note: Each time series data file must contain the same number of time series columns as the other files in that data source, in the same order.
A button on the "Default Units" column will allow you to copy units from the first Input Set Configuration:
- Choose the scenario input set(s) to associate with each time series data file by selecting the tab's drop-down menu and toggling the check box next to the appropriate scenario input set(s). The associated scenario input sets are listed in the tab's heading. For example, in Figure 5, Flows.csv is associated with Default Input Set and Dry Weather. Source will warn you if you assign more than one time series data file to the same scenario input set.
- The check box next to the tab name determines which time series data file Source will use to name the time series columns; only one of the tabs can be toggled on. If none of the check boxes are selected, the column names from the file in the first tab will be used.
The default name of the data source will be the file name in the first tab. Both the data source and the time series columns can be renamed in the Data Source Explorer (see above).
Assign a scenario data source to scenario input set(s) as explained for a file data sources, but use a new tab for each input scenario.
Data from Climate data import tool
Once data is generated using the /wiki/spaces/SD45/pages/53969813Climate data import tool, it can be used as a data source in the Data Sources Explorer, as shown in Figure 8.
Figure 8. Data Source Explorer, Climate Data Import
| Anchor | ||||
|---|---|---|---|---|
|
Some Source windows allow you to manually enter data into tables and view a graphical representation alongside. This holds for most tables across Source.
Follow the steps shown in Figure 9 to enter data into the table, one row at a time.
Additional shortcuts for entering data include:
- Press the tab button to move across the columns, then down to the next row. This button maintains the 'Edit status' of a cell;
- Pressing F2 is synonymous to using the Enter key - they both enter or exit the Edit status;
- Press the Delete key to remove a row; and
- As you enter data points in the table, the graph on the right will get populated.
An example of a completed table is shown in Figure 10.
Figure 9. Steps to manually enter data tables
Figure 10. Manual data entry in tables (completed)
