The changes proposed to res.csv are to do with how we use the metadata in the file.
The metadata fields are: Field, Units, ScenarioName, ScenarioInputSetName, Name, Site, ElementName, WaterFeatureType, ElementType, Structure, Custom
Note: Units has moved to directly after "Field" for ease of finding it.
Above the data section (See image below), each column header will be a "key". The key is aiming to uniquely identify each column but also retain some information of the column metadata to make it more user friendly. The key is made up of two parts, the "Field" metadata (unique within the res.csv file) and a descriptive part (not necessarily unique within the res.csv file). The metadata should be referenced via the field when distinguishing the differences between each column. The key will always start with the "Field" metadata to make it unique, but to descriptive part depends on where it is exported from:
From Results Manager or any place in which has all the adequate metadata it will be: Field > Site > Structure
From Data Sources or any place that DOESN'T have all the metadata it will be: Field > Name
When exporting
From Results Manager or any place in which has all the adequate metadata it will populate the metadata fields such as scenario, run name, input sets, site, structure, etc but NOT the "Name" field
From Data Sources or any place that DOESN'T have all the metadata we will populate the name field, and only some of the other metadata fields will be used like units and input sets (Although input sets will have a different meaning) all unused fields will be empty. This is because these metadata fields will be stripped when reading them in to certain places like Data Sources.
Reading into Results Manager
When Data was exported from Results Manager a it should be able to use the metadata to recreate the key in such a way it will replicate a tree structure that it had before it was exported
When Data was exported from a place that didn't insert all the metadata (such as Data Sources) it will appear to flatten the data.
Reading into Data Sources
We think it will be best to strip most of the metadata for a few reasons:
The user can rename the Data Source item/column and it would be difficult to maintain that
As the metadata doesn't necessarily represent what it did when it was exported:
Scenario name: if imported into Data Sources in a different scenario you might expect it to now have that new Scenario name for this data when you export it
Run name: the data should no longer be connected with a run
Input set: input sets in Data Sources refers to which input sets this data source is assigned to rather than the input set that was used in the run when the data was created
etc...
Backwards Compatibility for the original format
We will bring back the ability to READ the old undocumented format for the following reasons:
- It shouldn't be too hard to distinguish the old version by a lack of version field
- Results Manager Persisted Run Data was stored in res.csv
- People have data stored in the old format they wanted to be able to use in the new version rather than causing it to crash.