The changes proposed to res.csv are to do with how we use the metadata in the file.
The metadata fields are: Field, Units, ScenarioName, ScenarioInputSetName, Name, Site, ElementName, WaterFeatureType, ElementType, Structure, Custom
Note: Units In File version 1: When saving, units has moved to the second column, directly after "Field" for ease of finding it. . In order to do this, the metadata fields can now be read in any order and it depends on the metadata headings to map the metadata to the correct field. See attached example.res.csv.
When exporting
Populates as many metadata fields as possible, including name.
When exporting from Data Sources, we no longer append the input set name (Corresponding with the Data Source Group) to the Name metadata.
Above the data
...
(Arrow pointing to the row described, ), each
...
header will be a "key". The key is aiming to uniquely identify each column but also retain some information of the column metadata to make it more user friendly. The key is made up of two parts, the "Field" metadata (unique within the res.csv file) and a descriptive part (not necessarily unique within the res.csv file). The metadata should be referenced via the field when distinguishing the differences between each column. The key will always start with the "Field" metadata to make it unique, but to descriptive part depends on where it is exported from:
From Results Manager or any place in which has
...
the "Site" and "Structure" metadata it will be: Field > Site > Structure
From Data Sources or any place
...
that DOESN'T
...
have all the
...
metadata it will be: Field > Name
When
...
From Results Manager or any place in which has all the adequate metadata it will populate the metadata fields such as scenario, run name, input sets, site, structure, etc but NOT the "Name" field
From Data Sources or any place that DOESN'T have all the metadata we will populate the name field, and only some of the other metadata fields will be used like units and input sets (Although input sets will have a different meaning) all unused fields will be empty. This is because these metadata fields will be stripped when reading them in to certain places like Data Sources.
...
importing
The metadata will be loaded into the domain object, but will for the most part be hidden from the UI. We have chosen not to strip any of the metadata in attempt to be able to round trip as close to the original file and the UI will determine how the data is displayed.
Importing into Results Manager
When Data was exported from a Results Manager a run it should be able to use contain all the metadata to recreate the key in such a way , it will be able to replicate a the tree structure that it had before it was exported.
When Data was exported from a place that didnthe res.csv doesn't insert contain all the required metadata (such as Data Sources, importing a csv than exporting as a res.csv) it will appear to flatten the data.
Reading into Data Sources
We think it will be best to strip most of the metadata for a few reasons:
The user can rename the Data Source item/column and it would be difficult to maintain that
As the metadata doesn't necessarily represent what it did when it was exported:
Scenario name: if imported into Data Sources in a different scenario you might expect it to now have that new Scenario name for this data when you export it
Run name: the data should no longer be connected with a run
Input set: input sets in Data Sources refers to which input sets this data source is assigned to rather than the input set that was used in the run when the data was created
...
The name chosen for display will be "Site > Structure" if the metadata exists, otherwise the "Name" Field will be used. However, as names need to be unique, we will append the Field metadata to this name for any non unique columns
Backwards Compatibility for the original format
We will bring back the ability to READ the old undocumented format for the following reasons:
- It shouldn't be too hard to distinguish the old version by a lack of version field
- Results Manager Persisted Run Data was stored in res.csv
- People have data stored in the old format they wanted to be able to use in the new version rather than causing it to crash.