Versions Compared

Old Version 9

changes.mady.by.user teamcity

Saved on

compared with

New Version Current

changes.mady.by.user SysAdmin eWater

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

About the command line runner

The riversystem.commandline.exe provides a mechanism to run existing projects without the Source interface. It is possible to modify or change the behaviour of the project from the command line, by changing the values of global expressions. Results can be returned for recorded items or global expressions within the project.

It can run in three modes:

...

 riversystem.commandline.exe is included with Source.

The command line is used where it is impractical to run Source from the main user interface. For example, the command line can be used for batch runs, where Source is run many times with a range of different parameter values. The command line can also be used in optimisation scenarios and when integrating Source with other models or business systems.

The command line can be used in either a standalone (in-process) mode or in a client/server configuration. When run in client/server mode, the Source project is loaded and run at the server side, with parameters provided by and results through a separate client. In this way, the server can be used for many runs of the same project, without needing to reload the project file each time.

Configuring a scenario from the command line

Any valid Source project can be used with the command line, but to modify behaviour of the model, you must configure global expressions, which can be modified from the command line and alter the model’s behaviour in some way.

Figure 1 illustrates the process of manipulating the operating target for a storage from the command line, by manipulating a global expression, $x. This must first be specified in terms of an expression, which must reference $x. Global expressions can be used to manipulate the model in this way, either before the start of the simulation, or before each time-step.

Figure 1. Manipulating global expressions

Image Added

The Metaparameter explorer (Metaparameter Explorer) can be used to attach global expressions directly to model variables for situations where the Function Editor has not been enabled by default.

Working with the command line

This section describes how to load, run, retrieve results, and modifying parameters using the command line.

 

Info
iconfalse

Note: All characters in the command line are case-sensitive.

Loading and Running

You can run a project in the following modes:

  • Standalone mode - the project loads and the simulation runs with a single line command:

    Code Block
    RiverSystem.CommandLine -p C:\Temp\ExampleProject.rsproj

    Client/server mode - one command line runs projects. It does anything except for loading a project until a second command line is run in client mode. In this way, multiple runs can be completed with the server loading the project only once. It also enables the server to run on a different machine to the client. The examples provided assume a client/server mode, illustrating the calls made by the client. They could also work in standalone mode. Refer to Advanced configuration options for two options available in this mode.

Retrieving results

By default, all recorded items are returned, as configured from within the Source application (Running scenarios). Alternatively, you can specify specific items to record by using a command line argument:

Code Block
RiverSystem.CommandLine -m Client -r "Inflow 1/Inflow Volume"

The command after -r refers to the exact name of the recordable item as it appears in Source, ie Node Name/Variable. In this case, the variable, Inflow Volume, of the inflow node will be recorded.

In client/server mode, results are always computed on the server, but returned through the client. By default, results are displayed on the console (in both standalone and client modes), but it is possible to save the results directly to a file using the following command (file name is resultsfilename.csv):

Code Block
RiverSystem.CommandLine -m Client -o resultsfilename.csv

Modifying parameters

Several global expressions can be set using the -v argument:

  • Set the values: 

    Code Block
    RiverSystem.CommandLine -m Client -v "$metaParam=2.5"
  • Set multiple expressions:
Code Block
RiverSystem.CommandLine -m Client -v "$metaParam=2.5" -v "$anotherMetaParam=3.0".

Examples

The following set of examples indicate the type of commands used

...

to run projects in the two modes.

Inprocess

The next example runs the project from start to finish and puts all recorded values in the file ‘test.csv’. 

Code Block
>RiverSystem.CommandLine.exe -p "C:\ProjectDir\test.rsproj" -o test.csv

...


Loading 100%

...


Meta-Parameters:

...


+

...

 $inflow
-r "Forest\Catchment for node #2\Quick Flow\Flow"
+ Straight-Through Routing\Default Link #1\Upstream Flow Volume

...


+ Straight-Through Routing\Default Link #1\Downstream Flow Volume

...


+ Straight-Through Routing\Default Link #1\Mass Balance

...


+ Inflow\Inflow 1\Upstream Flow Volume

...


+ Inflow\Inflow 1\Downstream Flow Volume

...


+ Inflow\Inflow 1\Inflow Volume

...


+ Inflow\Inflow 1\Mass Balance

...


+ Gauge\Gauge 2\Upstream Flow Volume

...


+ Gauge\Gauge 2\Mass Balance

...


+ Global Expressions\$inflow\Value

...


Running 100%

Server

This command starts a basic server on the same machine as the client:

Code Block
>RiverSystem.CommandLine.exe --mode server


The service is ready at net.tcp://localhost:8523/eWater/Services/RiverSystemService
Press Enter to stop the service.

In this command, a basic server runs on the same machine as the client. This server will look for project files passed from the client in the directory "C:\ProjectDir":

Code Block
>RiverSystem.CommandLine.exe --mode server --d "C:\ProjectDir"


The service is ready at net.tcp://localhost:8523/eWater/Services/RiverSystemService
Press Enter to stop the service.

The following command will start up a server, load a project from the directory C:\Users\man25n\Documents\eWater Source Projects using the address specified (net.tcp://192.168.1.5:8080/myWaterService/):

Code Block
>RiverSystem.CommandLine.exe --mode server -d "C:\Users\man25n\Documents\eWater Source Projects" -a "net.tcp://192.168.1.5:8080/myWaterService/"


Press Enter to stop the service.

The address consists of the communication protocol (net.tcp) and the name of the end-point. For a machine that runs both the client and server, the address of the end-point must match.

Client

...

The following command

...

assumes that the server is already running from the previous example:

Code Block
> RiverSystem.CommandLine.exe --mode

...

 client --d "C:\ProjectDir"


The service is ready at net.tcp://localhost:8523/eWater/Services/RiverSystemService

...

Option descriptions

Notice in the examples that the command line contains various options, which result in different outputs. The function of these options is provided in Table 110. Note that these are optional extras.

Client/Server configuration

A command line can run in a server/client mode, meaning that there would be one command line that would run projects. It would not do anything except for loading a project until a second command line is run in client mode. In this way, multiple runs can be completed with the server loading the project only once. It also enables the server to run on a different machine to the client.

...

.

Additional examples of running Source using a DOS batch file, a Python script or an R script can be found here.

Advanced configuration options

In the client/server mode, there are two command line windows and riversystem.commandline.exe is run twice:

  • XML file: riversystem.commandline.exe.config - This file contains a section called <system.serviceModel> which can configure hundreds of options for the services and client including transport types, maximum data size, security etc. These are the default options used when running in client or server mode. The location of the server defined in this file, by default, is "localhost" which means the server and client are configured to run on the same machine

...

  • .
    Option A - A project is loaded when the server starts (first command line) and is run in response to a client call (second command line):
Code Block
RiverSystem.CommandLine -p C:\Temp\ExampleProject.rsproj -m Server
RiverSystem.CommandLine -m Client
  • For testing purposes and ad-hoc configuration, two main command line options are also provided to configure the location/type of the server and message time out. When run in this mode the <system.serviceModel> section of the riversystem.commandline.exe.config file may need to be deleted as creating two servers of the same type is not supported. This means that the net.tcp server on localhost defined in the exe.config file will stop you creating an adhoc server of the same type.

Using riversystem.commandline in Source

This section provides a few examples of some common functions that can be carried out in Source using the command line. Refer to Table 111.

PEST calibration using the command line

PEST (Parameter ESTimation and uncertainty analysis) allows analysis of complex environmental models, and can be used in Source for calibration of catchments. You can use the command line to do calibration and uncertainty in a catchments model. The steps to do the calibration are as follows:

  • Starting with a Source Catchment project file, use the new calibration tool to set up ‘global expressions’ for the parameters of the rainfall runoff models. When defining the metaparameters in the Calibration Wizard (Figure 148), the list of Existing Metaparameters will be used in the parameter file for invoking riversystem.commandline;
  • Turn on any recorders for the hydrological quantities that will be used in the objective function. In step 4 of the Calibration Wizard, choose Manual optimisation and enable Create global expressions for use from an external tool;
  • Save the project;
  • Set up the Riversystem.Commandline input file;
  • Setup the PEST input and template files;
  • Prepare a batch file to invoke the commandline and pre-processors; and
  • Run PEST.

Once the Calibration wizard has been set up, start the command line server (riversystem.commandline -p projectname.rsproj -m server) and then PEST (pest pestcontrolfile.pst).

...

  • Option B - In the example below, the server starts without loading any project (first command line). The client then specifies the project, which gets loaded onto the server and runs immediately (second command line):
Code Block
RiverSystem.CommandLine -m Server
RiverSystem.CommandLine -m Client -p C:\Temp\ExampleProject.rsproj

Option descriptions

Notice in the examples that the command line contains various options, which result in different outputs. The function of these options is provided in Table 1. Note that these are optional extras.

 
Anchor
CommandLineOptions
CommandLineOptions
Table 1. Command Line Runner (options)

 

Display option commands

Function

Example

-p, --project

Path to project, or project name on server

...

Printing results

a[0,] # header

a[,1] # date column

...

followed by these optional parameters: scenario within that project, start and end times, name of data input set

-p "projectName.rsproj" or -p "c:\somePath\projectName.rsproj;someScenarioName;1/1/2011;31/12/2011;dry"

-m, --mode

Application mode Client/Server/InProcess.

-m Client or -m Server

-d, --directory

Directory in which to look for projects.

-d "c:\ProjectFiles"

-o, --output

File to save output into. The extension selects the output format based on the standard TIME I/O formats. A directory of the name specifed will be created if the format only supports a single time series.

-o "output.res.csv" for CSV or -o "output.nc" for NetCDF etc.

-r, --results

Which results to return. This can appear multiple times.

Info
iconfalse
Note: By default, all recorded results are saved into the specified output file. When this options is used, it only allows you to reduce the number of results saved out. Requested results must be set to record in the project file. To specify the output file name, save the result to a file with a .res.csv extension. Use the column name for the time series.

-r "someResult" -r "someOtherResult"

-v, --value

Set value of meta-parameter or internal setting.

-v "$metaparam=25", or -v "$anotherMetaParam=2.3"

-s, --step

Number of time-steps to run.

--step 1

-t, --reset

Reset the loaded project.

 

-e, --timeout

Operation timeout for WCF service in seconds.

-e 2000

-b, -baseSIUnits

Ignore units set within project and output base SI units

 

-a, --address

Address for server to create, or client to connect.

-a "net.pipe://localhost/PipeReverse" or "net.tcp://localhost:8523/eWater/Services/RiverSystemService"

Using command line runner in Source

...

Function

...

Command

...

Define the location of the executable

...

SourceExe = shQuote("C:\\Program Files\\eWater\\Source 2.19.1.27\\RiverSystem.CommandLine.exe")

...

Define the project that you want to load

...

ProjectFile = shQuote("D:\\Working\\FloodPlain.rsproj")

...

Start the server in background with project

...

system (paste(SourceExe,"-p",ProjectFile,"-m Server"), wait=FALSE)

...

Load a client and run the server (results to screen)

...

system (paste(SourceExe,"-m Client"))

...

Load a client and send the results to a temporary directory

...

DefaultOutputFile = paste(tempdir(), "SourceOutput.csv", sep="\\")

system (paste(SourceExe,"-m Client","-o",shQuote(DefaultOutputFile)))

...

Reading an output csv back into R

...

results=read.csv (DefaultOutputFile, nrows=365*120)

PEST calibration using the command line

PEST (Parameter ESTimation and uncertainty analysis) allows analysis of complex environmental models, and can be used in Source for calibration of catchments. You can use the command line to do calibration and uncertainty in a catchments model. The steps to do the calibration are as follows:

  • Starting with a Source Catchment project file, use the new calibration tool to set up ‘global expressions’ for the parameters of the rainfall runoff models. When defining the metaparameters in the Calibration Wizard, the list of Existing Metaparameters will be used in the parameter file for invoking riversystem.commandline;
  • Turn on any recorders for the hydrological quantities that will be used in the objective function. In step 4 of the Calibration Wizard, choose Manual optimisation and enable Create global expressions for use from an external tool;
  • Save the project;
  • Set up the Riversystem.Commandline input file;
  • Setup the PEST input and template files;
  • Prepare a batch file to invoke the commandline and pre-processors; and
  • Run PEST.

Once the Calibration wizard has been set up, start the command line server (riversystem.commandline -p projectname.rsproj -m server) and then PEST (pest pestcontrolfile.pst).

Batch running using the command line

Insert excerpt
SD50:Batch running using the command line
SD50:Batch running using the command line
nopaneltrue

Please see /wiki/spaces/SD50/pages/50137610 in the Source community for details and an example.