Page Comparison

There are two types of time series data inputs:

• File – a text-based time series data file in a supported file format. Refer to Loading a file data source.
• Scenario – the results of a run from a different scenario can be used as an input for the current scenario. Refer to Loading a scenario data source.

When time series data is loaded in to Source, it is referred to as a data source and is available throughout Source. To either load new data sources, or access previously loaded data sources, you can use:

• the Data Sources Explorer
• the Data Source option in a component's feature editor (Figure 1)
• the Feature Table
• the /wiki/spaces/SD516/pages/54986558
• the Command Line Runner Options
• the output of the Climate data import tool

Figure 1. Loading a data source from within a component's feature editor

Anchor
DataSourcesExplorer DataSourcesExplorer

Data Sources Explorer

Source allows you to load and manage time series data at a central location using the Data Sources Explorer (Figure 2). Note that all data sources for the active scenario appear in the Explorer regardless of whether they were loaded using the Explorer, or one of the other methods listed above (eg. by using a component's feature editor, Figure 1).

Figure 2 shows features of the Data Sources Explorer containing all data sources available in the scenario, both from time series data files and another scenario. Additionally, the nodes that some of the data sources are associated with are also shown i.e. the data source usage.

The Data Sources toolbar (Figure 3) has the following functions:

• New Data Source creates a new data source, using either the output of a scenario (Scenario Data Source...) or a time-series file (File Data Source...);
• New Folder creates a new folder, allowing organisation of data sources;
• Expand All expands the file hierarchy;
• Collapse All collapses the file hierarchy;
• Data Sources Overview opens the Data Sources Overview window, displaying metadata for all file and scenario data sources in the active scenario. The user can choose an overview focused on the timeseries or usage (Figure 4a,b). Export button can open Save As dialog to output the metadata overview records to a csv file. The Drag a column header here to group by the column allows to view the metadata records by the group based  on the dragged column header. Refresh button can load the latest metadata information to the overview interface.

If selected an overview based on the timeseries (Figure 4a), the metadata will display the overview records based on the time series, which normally appeared as a column-based data in the original source file or scenario output data:

Group Name: The label/name of a file or a scenario data source displayed in Data Sources

Folder Path: The logic folder path where Group Name is located at in Data Sources

Match method: The used method (By Name or By Position) to match columns when reloading the data file

Scenario input sets: The scenario input set name(s) associated with this time series data

Column and Name: The column order number (start from 0) for the time series during the loading or reloading and time series name related the column name. The Date column in the original data file does not count

Start and End. The start and end date of the data content in the time series

Aggregation Method: Which method was setup for the aggregation when the model uses data at a smaller timestep to the time step the model. 

Units: The used unit for the time series data

Null Count: How many null ( missing) values in the loaded time series data

Usage: How many times of the time series used in the modelling such as by a network element, a functions, or models.

If selected an overview based on the usages (Figure 4b), the metadata will be displayed based on the usage:

Data Group Name, Folder Path, Column Name, Start, End, Unit (in Figure 4b):  Same as those in overview based on the timeseries

Property Name: Name of the property used in the modelling such as rainfall, PET, Constituent Load, Recorded Gauging Station Flow       

Feature Type: Network element feature type such as Storage, Gauge, Water User

Network Element Name: Name of Network element feature.

• Reload all time series with 'Reload on Run' reloads all time series when Reload on Run. This can also be selected from within the Load Data Source dialog (Figures 4 and 5); and

Figure 4 a) and b). Data Sources Overview

Anchor
LoadingDataFile LoadingDataFile

Loading a file data source

To load a time series data file, either:

• Click the New Data Source button on the Data Sources toolbar and choose File Data Source... from the drop down menu; or
• Right-click on a Folder and choose Add Source » File from the contextual menu.

The Load File Data Source dialog opens (Figure 5). Then, follow these steps:

• Click the Load data file button and navigate to and select your time series data file. 

• Anchor
  Relative path Relative path

  The path to the data file is displayed. If you enable the Relative Path check box, the path displayed is the location of the time-series file relative to the project location. Note that the project must have been saved prior to this. 

• Each column of data in the time series data file becomes a row in the Data table. All time series data in the file is imported by default. You can select what columns to import by toggling the appropriate check box in the Import column.
• If there are empty column headings in the file, there will be an error and you will have to name these columns at this point.
• Choose whether to Match data items By Name or By Position. By default this is set to By Name. This determines how Source will match columns when reloading the data file if the the Reload on Run option is selected or find the equivalent series between each file within the data source if e.g. used for different input sets.
• Choose the scenario input set(s) associated with this data source by clicking on the arrow for the Default Input Set tab and toggling the appropriate check boxes. Refer to Assigning data sources to scenario input sets for more detail. 
• Click on Reload on Run to toggle whether Source reloads the data into the internal store when the Scenario is next run (for more information, see below).
• Click on a Default Units cell and choose the appropriate units from the drop down menu. To filter the list of units, select the Default Units input box a begin typing the name of the unit that you require. Right-click on a Default Units cell to bring up a contextual menu that allows you to Apply Unit to all data.
• Select an Aggregation Method can be set to allow you to use data at a smaller timestep to the time step the model will run at.  E.g. you can load daily data and then run the model at a monthly time step if you have selected an Aggregation Method.

Table 1. Aggregation Methods