Exploring solutions

Fetch information from solution

library(simplaceUtil)

We can read a solution xml file via getSolutionFromFile and get its content as an xml_document object. From the xml object we can get the components, user variables and links between components as dataframes.

solfile <- system.file("solution","Complete.sol.xml",
                       package="simplaceUtil")

sol <- getSolutionFromFile(solfile)
components <- getComponents(sol)
variables <- getUserVariables(sol)
links <- getLinks(sol, components)

As a shorthand we can use getElementsFromSolutionFile, which returns the xml object as well as the dataframes in a list.

el <- getElementsFromSolutionFile(solfile)
sol <- el$solution
components <- el$components
variables <- el$variables
links <- el$links

The components dataframe contains not only sim components, but also interfaces, resources and outputs.

components[c(1:3,30:31,42:44),c("id","type","subtype","nr")]

##                   id         type   subtype nr
## 1            CURRENT          var       var  1
## 2          variables          var       var  2
## 3            compare    interface       CSV  3
## 30              crop     resource  resource 30
## 31           ndemand     resource  resource 31
## 42    init_transform     resource transform 42
## 43 DefaultManagement simcomponent       mgm 43
## 44    EvapTranDemand simcomponent    normal 44

The links dataframe contains the links between resources, simcomponents, outputs, but also dependencies via key variables or via variables used in rules. Relationships between resources and transformers are included too.

links[c(1:2,23,300,313,322,325),]

##                  from         name                to     rel
## 1             CURRENT          DOY DefaultManagement   value
## 2           phenology emergencedoy DefaultManagement   value
## 23  DefaultManagement    DoHarvest   LintulPhenology   value
## 300       weatherfile            -           weather storage
## 313              soil            -    soil_transform    data
## 322         variables    vSoiltype              soil     key
## 325 DefaultManagement        DoSow              crop    rule

User variables show the values as defined in the solution. Notice that user variables can be changed when running a solution in project mode.

variables[c(1:5,18:20),]

##                      id      value unit datatype kind
## 1          vCLIMATEZONE      00002 <NA>     CHAR  var
## 2             vSoiltype sandy loam <NA>     CHAR  var
## 3            vTempLimit         35 <NA>   DOUBLE  var
## 4  vTempSlopeAboveLimit          7 <NA>   DOUBLE  var
## 5              vBaseLUE        3.0 <NA>   DOUBLE  var
## 18        vCO2StartYear       2000 <NA>      INT  dyn
## 19            startdate 01.01.1991 <NA>     DATE  dyn
## 20              enddate 31.12.2004 <NA>     DATE  dyn

Filenames for interfaces may use placeholder variables like _WORKDIR_ or user defined variables (e.g. vFilename)

head(components[,c("id","ref")])

##              id
## 1       CURRENT
## 2     variables
## 3       compare
## 4   weatherfile
## 5 phenologyfile
## 6      cropfile
##                                                                 ref
## 1                                                            SYSTEM
## 2                                                              User
## 3        \\${_WORKDIR_}/gk/data/lintul/compare/compare_Complete.csv
## 4                                                    \\${vFileName}
## 5     \\${_WORKDIR_}/gk/data/lintul/LintulPhenologyObservations.csv
## 6 \\${_WORKDIR_}/gk/data/lintul/CropPropertiesLintulWaterstress.xml

With replaceVariablesWithValues we can replace the placeholder with the values from the user value dataframe. We can supply additional values for system variables like _WORKDIR_.

components$ref <- replaceVariablesWithValues(
  components$ref,
  variables, 
  additional=c("_WORKDIR_"="~"))
head(components[,c("id","ref")])

##              id                                                  ref
## 1       CURRENT                                               SYSTEM
## 2     variables                                                 User
## 3       compare        ~/gk/data/lintul/compare/compare_Complete.csv
## 4   weatherfile                    ~/gk/data/lintul/00002Weather.txt
## 5 phenologyfile     ~/gk/data/lintul/LintulPhenologyObservations.csv
## 6      cropfile ~/gk/data/lintul/CropPropertiesLintulWaterstress.xml

Working with component and links dataframe

Subsetting the components dataframe to get only normal components

ncomp <- components[components$type=="simcomponent" & components$subtype=="normal",]
ncomp$id

##  [1] "EvapTranDemand"                "Co2InfluenceOnLUE"            
##  [3] "Co2InfluenceOnTranspiration"   "LintulPhenology"              
##  [5] "VernalisationAndPhotoresponse" "SlimRoots"                    
##  [7] "SlimWater"                     "SlimNitrogen"                 
##  [9] "LintulWaterStress"             "LintulPartitioning"           
## [11] "LintulBiomass"                 "NitrogenDemand"               
## [13] "SoilCN"                        "SnowCoverCalculator"          
## [15] "STMPsimCalculator"

Getting only links between normal components

nlinks <- links[links$from %in% ncomp$id & links$to %in% ncomp$id,]
head(nlinks)

##                             from                name
## 12                 LintulBiomass                sLAI
## 16                     SlimWater ActualTranspiration
## 21 VernalisationAndPhotoresponse               RTSUM
## 31               LintulPhenology           sDevStage
## 44             STMPsimCalculator       SoilTempArray
## 45                     SlimWater              DRYFAC
##                               to   rel
## 12                EvapTranDemand value
## 16   Co2InfluenceOnTranspiration value
## 21               LintulPhenology value
## 31 VernalisationAndPhotoresponse value
## 44                     SlimRoots value
## 45                     SlimRoots value

Counting the links between components

nlinks <- aggregate(name ~ from + to , data=nlinks, length)
nlinks <- nlinks[order(-nlinks$name),]
head(nlinks,10)

##                  from                          to name
## 23          SlimWater                SlimNitrogen    9
## 14      LintulBiomass              NitrogenDemand    5
## 3  LintulPartitioning               LintulBiomass    4
## 13          SlimWater           LintulWaterStress    2
## 22          SlimRoots                SlimNitrogen    2
## 30          SlimRoots                   SlimWater    2
## 31      LintulBiomass         SnowCoverCalculator    2
## 34      LintulBiomass                      SoilCN    2
## 1           SlimWater Co2InfluenceOnTranspiration    1
## 2       LintulBiomass              EvapTranDemand    1

Get Metadata about solution

getSolutionInfoAsDataframe gives information about the solutions content (simcomponents) as well as the solution itself (filename, modification date and possibly simulation experiment information derived from folder structure).

wd <- system.file(package="simplaceUtil")
solfiles <- list.files(wd,pattern=".sol.xml",recursive=TRUE,full.names = TRUE)

df <- do.call(rbind,
        lapply(solfiles,getSolutionInfoAsDataframe,workdir=wd))
df$file <- basename(df$file)
df[df$type=="simcomponent",c("id","file")]

##                               id             file
## 43             DefaultManagement Complete.sol.xml
## 44                EvapTranDemand Complete.sol.xml
## 45             Co2InfluenceOnLUE Complete.sol.xml
## 46   Co2InfluenceOnTranspiration Complete.sol.xml
## 47               LintulPhenology Complete.sol.xml
## 48 VernalisationAndPhotoresponse Complete.sol.xml
## 49                     SlimRoots Complete.sol.xml
## 50                     SlimWater Complete.sol.xml
## 51                  SlimNitrogen Complete.sol.xml
## 52             LintulWaterStress Complete.sol.xml
## 53            LintulPartitioning Complete.sol.xml
## 54                 LintulBiomass Complete.sol.xml
## 55                NitrogenDemand Complete.sol.xml
## 56                        SoilCN Complete.sol.xml
## 57           SnowCoverCalculator Complete.sol.xml
## 58             STMPsimCalculator Complete.sol.xml
## 78             DefaultManagement    Yield.sol.xml
## 79                   LintulPheno    Yield.sol.xml
## 80 VernalisationAndPhotoresponse    Yield.sol.xml
## 81            LintulPartitioning    Yield.sol.xml
## 82                 LintulBiomass    Yield.sol.xml

Visualising solution structure

A graph showing components and links can be created from a solution file:

graph <- solutionToGraph(solfile)

DiagrammeR::render_graph(graph)

We can restrict the graph to specific component types and the neighborhood of selected components:

subgraph <- getNeighborhood(graph, c("SlimWater","SlimRoots"),
                            distance=1, type="simcomponent")

DiagrammeR::render_graph(subgraph)

Modifying solutions

Sensitivity analysis and calibration

A solution can be modified on the fly to meet requirements for sensitivity analysis or calibration by

removing non-memory outputs for performance reasons
adding memory outputs with target variables
adding user variables for parameters that will be varied
replacing variables from resource by user variables

solfile <- system.file("solution","Yield.sol.xml",
                       package="simplaceUtil")
sol <- getSolutionFromFile(solfile)
newsol <- sol |>
  removeNonMemoryOutputs() |>
  addMemoryOutput("Yields",frequence = "YEARLY") |>
  addOutputVariable("Yields","Year","CURRENT.YEAR","INT") |>
  addOutputVariable("Yields", "ClimateZone","vClimateZone","CHAR") |>
  addOutputVariable("Yields", "Yield","LintulBiomass.sWSO","DOUBLE","MAX") |>
  addUserVariable("vLUE",3.0,"DOUBLE") |>
  addUserVariable("vSLA",0.02,"DOUBLE") |>
  addUserVariable("vRGRL",0.009,"DOUBLE") |>
  replaceVariable("crop.LUE","vLUE") |>
  replaceVariable("crop.SLA","vSLA") |>
  replaceVariable("crop.RGRL","vRGRL")

Let’s output the beginning of the solution as text:

soltext <- getTextFromSolution(newsol)

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE solution PUBLIC "-//SIMPLACE/DTD SOL 1.0//EN" "http://simplace.net/dtd/SimSolution_4_1.dtd">
<solution version="4.1" check="LAZY">
  <description title="Yield" author="XXX_XXX" qualitylevel="I">WIKI_START
=== Purpose ===
Example solution for simplaceUtil package.
WIKI_END 
 * removed non-memory outputs for interfaces yieldout, sumyieldout 
 * added memory output  Yields 
 * added output variable Year  to output  Yields  
 * added output variable ClimateZone  to output  Yields  
 * added output variable Yield  to output  Yields  
 * added user variable vLUE with value 3 
 * added user variable vSLA with value 0.02 
 * added user variable vRGRL with value 0.009 
 * replaced crop.LUE with vLUE 
 * replaced crop.SLA with vSLA 
 * replaced crop.RGRL with vRGRL</description>
  <variables>
    <var id="vTempLimit" datatype="DOUBLE">35</var>
    <var id="vTempSlopeAboveLimit" datatype="DOUBLE">7</var>
    <var id="vCropName" datatype="CHAR">winter wheat</var>
    <var id="vBaseLUE" datatype="DOUBLE">3.0</var>
    <var id="vBaseTempBeforeAnt" datatype="DOUBLE">3.0</var>
    <var id="vBaseTempAfterAnt" datatype="DOUBLE">3.0</var>
    <var id="startdate" datatype="DATE">01.01.1991</var>
    <var id="enddate" datatype="DATE">31.12.2004</var>
    <var id="vClimateZone" datatype="CHAR">C02</var>
    <var id="vLUE" datatype="DOUBLE">3</var>
    <var id="vSLA" datatype="DOUBLE">0.02</var>
    <var id="vRGRL" datatype="DOUBLE">0.009</var>
  </variables>
  <interfaces defaul

Notice that the modifications are logged in the description section of the new solution.

The solution can be saved as xml file:

solution_modified <- "simulation/solution/YieldModified.sol.xml"  
writeSolutionToFile(newsol, solution_modified)

Changing order of SimComponents

As the order of SimComponents can have a considerable impact, one can generate random orders of components and run the reordered solutions to analyse the impact.

solfile <- system.file("solution","Yield.sol.xml",
                       package="simplaceUtil")
sol <- getSolutionFromFile(solfile)

cmp <- getComponents(sol)
cmp[cmp$type=="simcomponent",c("id","subtype","nr")]

##                               id subtype nr
## 14             DefaultManagement     mgm 14
## 15                   LintulPheno  normal 15
## 16 VernalisationAndPhotoresponse  normal 16
## 17            LintulPartitioning  normal 17
## 18                 LintulBiomass  normal 18

We want to swap the components 2 to 5

toswap <- 2:5
swapped <- sample(toswap, length(toswap))
swapped

## [1] 5 3 4 2

newsol <- swapComponents(sol,swapped)
cmp <- getComponents(newsol)
cmp[cmp$type=="simcomponent",c("id","subtype","nr")]

##                               id subtype nr
## 14             DefaultManagement     mgm 14
## 15                 LintulBiomass  normal 15
## 16 VernalisationAndPhotoresponse  normal 16
## 17            LintulPartitioning  normal 17
## 18                   LintulPheno  normal 18

The order of the SimComponent is now different.

Adding timing SimComponent

There is a special SimComponent that records the processing time of SimComponents.

To add timing to a solution, one has to

add an interface for the timing output
add the timing SimComponent and configure it to measure the time of all or only of distinct components
add the output for the timing information

This can be done for every solution automatically via addTimingSimComponent.

solfile <- system.file("solution","Yield.sol.xml",
                       package="simplaceUtil")
sol <- getSolutionFromFile(solfile)

newsol <- addTimingSimComponent(sol, componentlist=c("LintulBiomass","LintulPheno"))
cmp <- getComponents(newsol)
cmp[cmp$type=="simcomponent" | grepl("timing",cmp$id),c("id","type","subtype","nr")]

##                               id         type subtype nr
## 8     automatic_timing_interface    interface  MEMORY  8
## 15             DefaultManagement simcomponent     mgm 15
## 16                   LintulPheno simcomponent  normal 16
## 17 VernalisationAndPhotoresponse simcomponent  normal 17
## 18            LintulPartitioning simcomponent  normal 18
## 19                 LintulBiomass simcomponent  normal 19
## 20               AutomaticTiming simcomponent  normal 20
## 23       automatic_timing_output       output  output 23

The new solution has now an additional SimComponent AutomaticTiming as well as an additional output together with it’s interface.

Standard transformation and visualisation of outputs

Transform output data

With parseDate the standard Date column of Simplace output files is converted to Date.

outfile <- system.file("output","water.csv",
                       package="simplaceUtil")
water <- read.csv(outfile, header=TRUE, sep=";")
water <- parseDate(water)

By default the simplace outputs are in wide format, e.g. each layer is in an own column. We can transform the dataframe from wide format to long format, by putting values from all layers into one column and distinguish them by an extra column layer.

water[1:4,c(2,9:11,49)]

##   CURRENT.DATE MobileWater_1 MobileWater_2 MobileWater_3 RetainedWater_1
## 1   1991-07-18             0     0.5214681      1.197492        3.402230
## 2   1991-07-19             0     0.0000000      0.000000        2.045055
## 3   1991-07-20             0     0.0000000      0.000000        1.584985
## 4   1991-07-21             0     0.0000000      0.000000        2.906192

water_long <- transformLayeredData(water)
water_long[c(1:3,41:43,81:83),c(2,9:11)]

## # A tibble: 9 × 4
##   CURRENT.DATE layer MobileWater RetainedWater
##   <date>       <int>       <dbl>         <dbl>
## 1 1991-07-18       1       0              3.40
## 2 1991-07-18       2       0.521          5.4 
## 3 1991-07-18       3       1.20           5.4 
## 4 1991-07-19       1       0              2.05
## 5 1991-07-19       2       0              4.27
## 6 1991-07-19       3       0              5.31
## 7 1991-07-20       1       0              1.58
## 8 1991-07-20       2       0              4.40
## 9 1991-07-20       3       0              5.04

Visualise output variables

Multiple scalar values can be plotted in a common diagram.

plotScalarOutput(water,"CURRENT.DATE",c("Evaporation","Transpiration"))

Default plot for layered values.

plotLayeredOutput(water,"RetainedWater")

Simple GUI as Shiny App

The package includes a Shiny App that runs locally within a webbrowser.

runSimplaceGuiApp()

We can interactively explore solutions by filtering and displaying solution graphs.

We can also use a local Simplace installation to run the simulation defined by the solution and plot some of the output variables.

Assistance for creating new Solutions (simulation configuration)

Creating XML stubs for resources

We can create the interface (location and format) and resource (structure) description for a solution as XML stubs from given input data.

Data can be given in CSV or XML format. The structure of the data has to be compatible with Simplace.

stubs <- createResourceStubsFromCsv(
  filename = system.file("input","weather.csv", package="simplaceUtil"),
  id = "weather",
  sep =",",
  keyvals = c("CURRENT.DATE" = "Date")
)
cat(stubs$interface)

## <interface id="weather_file" type="CSV">
##   <poolsize>100</poolsize>
##   <divider>,</divider
##   <filename>/tmp/RtmpIwIYma/Rinst109d23572467/simplaceUtil/input/weather.csv</filename>
## </interface>

cat(stubs$resource)

## <resource id="weather" interface="weather_file" frequence="DAILY">
##   <header>
##     <res id="DOY" datatype="INT" unit=""/>
##     <res id="Date" key="CURRENT.DATE" datatype="CHAR" unit=""/>
##     <res id="Rain" datatype="DOUBLE" unit=""/>
##     <res id="Radiation" datatype="DOUBLE" unit=""/>
##     <res id="Tmean" datatype="DOUBLE" unit=""/>
##     <res id="RelHum" datatype="DOUBLE" unit=""/>
##   </header>
## </resource>

The function tries to determine the datatype from the data. We have to check and possibly correct the inferred datatype (e.g. replacing ‘CHAR’ by ‘DATE’ or ‘INT’ by ‘DOUBLE’ or ‘BOOLEAN’).

Assistance for developing new SimComponents

Creating code stubs for SimVariables

When creating a new SimComponent, we can put all SimVariables (constants, inputs, states, rates, outputs) in a CSV table.

varfile <- system.file("variables","translocation.csv",package="simplaceUtil")
vartable <- read.csv(varfile, header=TRUE, sep=";")
vartable[c(4,8:9),]

##   contenttype                 id                              description
## 4    constant FruitingEfficiency Number of grains per storage organs mass
## 8       state                WEA                           Weight of ears
## 9        rate               RWEA                      Change rate of ears
##   datatype        unit min max default
## 4   Double        g^-1   0  NA      .1
## 8   Double      g m^-2   0  NA       0
## 9   Double g m^-2 d^-1  NA  NA       0

With the function createCodeStubsForSimVariables we can generate so called boilerplate code for Java as well as XML stubs for solutions.

stubs <- createCodeStubsForSimVariables(varfile, "BMTransloc", sep=";")
names(stubs)

## [1] "JavaFields"            "JavaVariables"         "JavaInit"             
## [4] "ParametersXML"         "ResourceParametersXML" "ResourceInputsXML"    
## [7] "ComponentInputsXML"    "OutputsXML"

When we specify an output folder, the stubs are written to files.

stubs <- createCodeStubsForSimVariables(varfile, "BMTransloc", sep=";",
                                        outfolder="translocation_stubs")

Example code stubs

Defining fields in Java (stubs$JavaFields):

private FWSimVariable<Double> cFruitingEfficiency;
private FWSimVariable<Double> rRWEA;
private FWSimVariable<Double> sWEA;

Creating the SimVariables (stubs$JavaVariables):

addVariable(FWSimVariable.createSimVariable("cFruitingEfficiency", "Number of grains per storage organs mass", DATA_TYPE.DOUBLE, CONTENT_TYPE.constant, "g^-1",  0d, null, .1d, this));
addVariable(FWSimVariable.createSimVariable("rRWEA", "Change rate of ears", DATA_TYPE.DOUBLE, CONTENT_TYPE.rate, "g m^-2 d^-1", null, null, 0d, this));
addVariable(FWSimVariable.createSimVariable("sWEA", "Weight of ears", DATA_TYPE.DOUBLE, CONTENT_TYPE.state, "g m^-2",  0d, null, 0d, this));

Default initialisation of states and rates (sic!) (stubs$JavaInit):

rRWEA.setDefaultValue();
sWEA.setDefaultValue();

XML stub for linking inputs (stubs$ComponentInputsXML):

<input id="cTranslocationFraction" source="bmtransloc_parameters.TranslocationFraction" />
<input id="cFruitingEfficiency" source="bmtransloc_parameters.FruitingEfficiency" />
<input id="iEarPartitioningReductionFactor" source="bmtransloc_inputs.EarPartitioningReductionFactor" />

XML stubs for parameter file (stubs$ParametersXML) and coresponding resource section in solution (stubs$ResourceParametersXML):

<parameter id="EarsPartitioningTableDVS" datatype="DOUBLEARRAY" unit="" description="DVS for fraction of above-ground dry matter to ears">
      <value>0</value>
      <value>1</value>
      <value>2</value>
  </parameter>
<parameter id="EarsPartitioningTableFraction" datatype="DOUBLEARRAY" unit="" description="Fraction of above-ground dry matter to ears as function of DVS">
      <value>.0</value>
      <value>.1</value>
      <value>.0</value>
  </parameter>
<parameter id="TranslocationFraction" datatype="DOUBLE" unit="" description="Fraction of stems and leaves biomass  that is reserved for translocation at anthesis">0.2</parameter>
<parameter id="FruitingEfficiency" datatype="DOUBLE" unit="g^-1" description="Number of grains per storage organs mass">.1</parameter>

<res id="EarsPartitioningTableDVS" datatype="DOUBLEARRAY" unit="" description="DVS for fraction of above-ground dry matter to ears" />
<res id="EarsPartitioningTableFraction" datatype="DOUBLEARRAY" unit="" description="Fraction of above-ground dry matter to ears as function of DVS" />
<res id="TranslocationFraction" datatype="DOUBLE" unit="" description="Fraction of stems and leaves biomass  that is reserved for translocation at anthesis" />
<res id="FruitingEfficiency" datatype="DOUBLE" unit="g^-1" description="Number of grains per storage organs mass" />

Accessing SimComponent documentation

With fetchSimComponentlistFromWebsite() one can get the java class names of the SimComponents from Simplace’s website.

With the class name one can then fetch the SimVariables as a dataframe

fetchSimVariablesFromWebsite('net.simplace.sim.components.evapotran.fao56.ReferenceETHargreaves')[,1:8]

##   contenttype                              id
## 1    constant                cConvertLeByTemp
## 2       input                 iSolarRadiation
## 3       input                           iTMax
## 4       input                           iTMin
## 5         out ReferenceCropEvapotranspiration
##                                                                                                           description
## 1 Use latent heat (Le) of vaporisation as a function of temperature to convert radiation from MJ/(m^2 day) to mm/day.
## 2                                                                                                     solar radiation
## 3                                                                                           maximum daily temperature
## 4                                                                                           minimum daily temperature
## 5                                                                                  reference evapotranspiration (ET0)
##   datatype      unit min max default
## 1  BOOLEAN                     false
## 2   DOUBLE MJ/(m2 d)             0.0
## 3   DOUBLE        °C             0.0
## 4   DOUBLE        °C             0.0
## 5   DOUBLE      mm/d             0.0