Versions Compared

Old Version 2

changes.mady.by.user Trudy Green

Saved on

compared with

New Version 3

changes.mady.by.user Trudy Green

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

eWater is pleased to announce the latest Production Release of Source version 4.11.   

...

Version 5.0 includes three significant features:

  • Urban Developer... 


Date Range changes

We have improved Date Ranges and fixed up some issues with their implementation. This will change results for a number of users, so we have implemented a Scenario Option to turn on the new Date Ranges algorithm. 

Existing projects won't be using the new method and will have to manually swap over. This gives users a chance to test the new implementation before it becomes the only option.

Image Modified

In a later Beta release, we will implement a utility to convert across while keeping results consistent as much as possible along with user interface improvements to help users understand Date Ranges better.

Results Streaming

...

Results can now be streamed to files on disk rather than stored in memory. This is to prevent memory issues with large runs.  Results streaming is off by default, however, it can be turned on in Project Options:

Image Modified

The format used (called Source Time Series Database .sdb) is now also available as a general timeseries format.  

Source.DataUtility.exe - eWater Source Time Series Data Utility                                                                                                                                                                                              
Usage:                                                                                                                                                                                                                                                       
        -?, --help[optional]... Show usage help                                                                                                                                                                                                              
                                                                                                                                                                                                                                                               
        -i, --input[optional]... Input file to read timeseries from. All standard TIME timeseries formats are supported.                                                                                                                                     
Example: -i "Run1.sdb" or -i "Run1.res.csv"                                                                                                                                                                                                                  
                                                                                                                                                                                                                                                               
        -o, --output[optional]... Output file to save timeseries to. The extension selects the output format based on the standard TIME I/O formats. A directory of the name specified will be created if the format only supports a single timeseries.      
Example: -o "output.res.csv" for Source Results CSV or -o "output.sdb" for SourceDB etc.                                                                                                                                                                     
                                                                                                                                                                                                                                                               
        -k, --key[optional]... Key of a timeseries to output. This can appear multiple times. If no keys are specified then all timeseries will be exported. Use the '--list' option to list timeseries in the input file. The DisplayName is used as the key.
Example: -k "someResult" -k "someOtherResult"                                                                                                                                                                                                                
                                                                                                                                                                                                                                                               
        -q, --query[optional]... SourceDB filter query. Use the '--list' option to list timeseries in the input file together with possible MetadataKeys to filter on.                                                                                       
Example: -q "WaterFeatureType=Storage" -q "ElementName=Storage Volume"                                                                                                                                                                                       
                                                                                                                                                                                                                                                               
        --startTime, --st[optional]... Start time for extracting partial results. If not specified the start of the input timeseries is used.                                                                                                                
Example: --st 1/1/2011                                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                               
        --endTime, --et[optional]... End time for extracting partial results. If not specified the end of the input timeseries is used.                                                                                                                      
Example: --et 31/12/2011                                                                                                                                                                                                                                     
                                                                                                                                                                                                                                                               
        -l, --list[optional]... List the timeseries and metadata contained in the input timeseries file. Standard output may be redirected to a '.json' file.                                                                                                
                                                                                                                                                                                                                                                               
        -a, --addOrReplace[optional]... Add input timeseries to an existing output SourceDb (.sdb) file. Existing timeseries with matching metadata will be replaced. Only supported for SourceDb output files.                                              
Example: --addOrReplace                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                               
        -f, --force[optional]... Don't prompt for confirmation when overwriting an existing file.

New Version Number system

We have changed the version number system to help users understand when they need a new version of Source to open projects.  The previous database version was a letter that wasn't in the version number.

The previous system was:

Order

Name


Order

Name


1Major Version number
2Prod/Beta identifierAll Prod numbers are even. All Beta identifiers are +1 the most recent Prod identifier
3Regular Functionality ReleaseIncremented when a new Beta is released every 1 to 2 months. Rarely updated for Prod releases.
Not included in version numberDatabase Upgrade letterNeeded when changes are made that require DB upgrades between official releases.
This is used to identify what the current database structure of the project will be. Resets to 'a' when the 'Regular Functionality Release' changes.
4Build numberEssentially a unique identifier for Source builds

New system:

Order

Name


1Major Version number
2Prod/Beta identifier/Regular Functionality ReleaseReset when the Major Version number increments. Increments with every official Prod and Beta release. We may increase by more than +1 for production releases.
3Database Upgrade NumberNeeded when changes are made that require DB upgrades between official releases. This is used to identify what the current structure of the database's project will be.
4Build number

Essentially a unique identifier for Source builds

This is why this release is 4.13.1beta rather than 4.12.4beta, as it would have been in the previous system.

...

For example, under the previous system the last beta release would have been 4.12.4beta, but under the new system it was 4.13.1beta

River Operations 

The Operations tabular editor is now independent of the recording selection.  You no longer get all recorders in the tabular editor. Recorders are now selected using the new column editor:

Image Modified

The tree on the left has also had major changes

...

Results Streaming

...

.

Resource Assessment Allocation table interface improvements 

Could this be under minor?

You can now freeze columns and drag columns to reorder the columns in the Allocation table within resource assessment:

Image Added

Results Streaming

Results can now be streamed to files on disk rather than stored in memory. This is to prevent memory issues with large runs.  Results streaming

...

is off

...

by default, however, it can be turned

...

on

...

in Project Options:

...

Image Removed

The format used (called Source Time Series Database .sdb) is now also available as a general timeseries format.  

Please test it out and let us know if you have any issues.

Custom Chart Improvements

We have included a search bar and more menu items under right-click.

Image Removed

Resource Assessment Allocation table interface improvements

Image Removed

You can now freeze columns and drag columns to reorder the columns in the Allocation table within resource assessment:

Image Removed

Results Streaming

Results can now be streamed to files on disk rather than stored in memory. This is to prevent memory issues with large runs.  Results streaming is off by default, however, it can be turned on in Project Options:

Image Removed

Performance improvements for the command line and Service interface

We have done some work to improve the performance of the command line, less memory (RAM) is now used during a run.  Run times have been reduced in the command line.

The command line now has additional options for results saving:

--resultsOutputMode

There are three modes currently available:

  1. NoOutput - Does not save results anywhere, hence does not fetch results either. Can be used where the result data files are ignored. E.g. in tests, or where a plugin gives a different pathway to results. This improves performance when using automatic exporting tools such as Scenario Options: Simulation results or Simulation Log which automatically exports results after each run to a specific location. 
  2. ClientNamedFormat- Results fetched to client, saved to output file from there, in a format based on output file extension. The data is transferred back to the client and saved (current behaviour).
  3. ServerNamedFormat - Results saved to output file directly from server, in a format based on output file extension. Does not fetch results to client. This "server side" save is more efficient. Default for InProcess mode, since "server side" is local here. Can be used in Client mode, assuming the end user has access to the path provided to the server e.g. a network drive.

Custom charts

Custom charts are no longer saved by default! 

All automatically generated charts are created under Temporary and only saved when moved into Saved. Users can also create folders and subfolders for organising custom charts:

Image Removed

We will also want to include a toolbar with a filter/search for custom charts:  Image RemovedRM-19119 - Toolbar for custom charts including Filter / Search READY TO DOCUMENT  will be available in a beta soon.

Performance improvements for the command line

We have done some work to improve the performance of the command line, less memory (RAM) is now used during a run and to speed up run time in the command line.

The command line now has additional options for results saving:

--resultsOutputMode

There are three modes available:

...

Image Added

Performance improvements for the command line and Service interface

We have done some work to improve the performance of the command line, less memory (RAM) is now used during a run.  Run times have been reduced in the command line.

The command line now has additional options for results saving:

--resultsOutputMode

There are three modes currently available:

  1. NoOutput - Does not save results anywhere, hence does not fetch results either. Can be used where the result data files are ignored. E.g. in tests, or where a plugin gives a different pathway to results. This improves performance when using automatic exporting tools such as Scenario Options: Simulation results or Simulation Log which automatically exports results after each run to a specific location. 
  2. ClientNamedFormat- Results fetched to client, saved to output file from there, in a format based on output file extension. The data is transferred back to the client and saved (current behaviour).
  3. ServerNamedFormat - Results saved to output file directly from server, in a format based on output file extension. Does not fetch results to client. This "server side" save is more efficient. Default for InProcess mode, since "server side" is local here. Can be used in Client mode, assuming the end user has access to the path provided to the server e.g. a network drive.

A new Water User and Constraint Phase

There has been a change to allow modellers finer control over function execution during the ordering process.

Phase logic has changed from:

10. Ordering phase:
  1. Evaluate functions and modelled variables - where Time of Evaluation equals Order Phase and where the function is not used at a network element.
  2. Process Water Users: 
    1. Water User order calculations can be run in Parallel at this point in the ordering phase with the option enabled: "Process Water User Orders in Parallel": Scenario Options \ Performance
    2. For each Water User:
      1. Evaluate functions and modelled variables - where Time of Evaluation equals Order Phase, and where the function is used at this Water User.
      2. Calculate and distribute the demands of each Water user.
  3. Constraint Phase:
    1. For each Network Element:
      1. Calculate minimum and maximum constraints at each element and pass the constraints downstream.
  4. Ordering Phase:
    1. For each Network Element:
      1. Evaluate functions and modelled variables - where Time of Evaluation equals Order Phase, and where the function is used at this network element.
      2. Calculate regulated orders and pass upstream.
      3. Calculate off allocation orders and pass upstream.

To:

10. Water User and Constraint Phase

  1. Evaluate functions and modelled variables - where Time of Evaluation equals Water User & Constraint Phase and where the function is not used at a network element.
  2. Process Water Users: 
    1. Water User order calculations can be run in Parallel at this point in the ordering phase with the option enabled: "Process Water User Orders in Parallel": Scenario Options \ Performance
    2. For each Water User:
      1. Evaluate functions and modelled variables - where Time of Evaluation equals Water User and Constraints Phase, and where the function is used at this Water User.
      2. Calculate and distribute the demands of each Water user.
  3. Constraint Phase:
    1. For each Network Element:
      1. Evaluate functions and modelled variables - where Time of Evaluation equals Order Phase, and where the function is used at this network element.
      2. Calculate minimum and maximum constraints at each element and pass the constraints downstream.

11. Ordering phase:

  1. Evaluate functions and modelled variables - where Time of Evaluation equals Order Phase and where the function is not used at a network element or is used at a Water User.
  2. Ordering Phase:
    1. For each Network Element:
      1. Evaluate functions and modelled variables - where Time of Evaluation equals Order Phase, and where the function is used at this network element.
      2. Calculate regulated orders and pass upstream.
      3. Calculate off allocation orders and pass upstream.

The important thing to note is that the timing has changed.

Example: A function assigned to a Water User, that says it evaluates in the Ordering Phase, will now happen later than it previously did. To evaluate it at the same time as it did in v4.11, the time of evaluation needs to be moved to the Water User & Constraint Phase.

Migration Code

So to minimise the effect in splitting this phase in two, we've written some migration code.

The logic is:

  • IF a function or modelled variable has a time of evaluation of 'Ordering Phase' (in v4.10.1)
  • AND it is used on a model that is either:
    • a Non-Network Element (e.g. Resource Assessment, Env Flow Manager)
    • a Water User
  • THEN move its time of evaluation to 'Water User & Constraint Phase'

also

  • IF a function or modelled variable has a time of evaluation of 'Ordering Phase' (in v4.10.1)
  • AND it is not used anywhere
  • AND it has Force Evaluate set to On
  • THEN move its time of evaluation to 'Water User & Constraint Phase'

otherwise, we leave it as 'Ordering Phase'

Limitations

We know our migration codes is not perfect:

  • We don't cater for Functions activated via input sets. 
  • Functions not used on a model, but only used by other Functions will not be moved

Recommended steps

We recommend you review any functions that are evaluated in the ordering phase and ensure they are configured correctly using: Edit \ Feature Table.. \ Functions.


Minor changes

  • We have introduced the concept of Locking to the Function Modelled Variables, selected variables can now be locked, making it less likely for mistakes to occur. 

  • Ordering for the Irrigator demand model now includes an option to make the Flow phase and Order phase identical.

  • Data Sources can now find data based on the column name rather than the index. Being able to match by name is important when using Reload on Run. Match data items by Name is now the default option.
    It will validate and fail to run if you are matching based on name and reload a file with duplicate names.

  • The Operations Tabular Editor now allows you to copy-paste into multiple cells and view charts directly in the Results Manager. 

  • We have made changes to significantly improve performance for editing and saving in some large models, for one project this reduced the time to save from 12 minutes to 45 seconds.

  • Constituent Trend and % removal model. Links and Storages decay models have been extended to include a decay to a background concentration rather than to 0 and a % removal that can occur after the decay.

  • Annual Accounting recorder changes. Some annual accounting recorders at RAS system level were giving incorrect values. The recorded values of "Total account spill reduction", "Total usage water year" and "Total Account Adjustments" at RAS system level values will change. See RM-18149 for details.
  • The order crawler recorder is a new recorder to show all the nodes and links in a network and what their orders and requirements are on a particular date.  Available under: Miscellaneous \ Ordering \ Order Crawler. The Order Crawler is a fully interactive zoomable schematic. Note that for large networks, it does record a lot of data and is primarily for problem-solving and not intended to always be turned on.
  • Ownership adjustment for storages and weirs - 

...

  • weirs - End of day volumes in Storages and Weirs

...

  • can now be adjusted.
  • Custom charts now include a search bar and more menu items under right-click. Customs charts are also no longer saved by default. All automatically generated charts are created under Temporary and only saved when moved into Saved. Users can also create folders and subfolders for organising custom charts.

Urban Developer Recorder improvements

...