Fetch information from solution

library(simplaceUtil)

We can read a solution xml file via getSolutionFromFile and get its content as an xml_document object. From the xml object we can get the components, user variables and links between components as dataframes.

solfile <- system.file("solution","Complete.sol.xml",
                       package="simplaceUtil")

sol <- getSolutionFromFile(solfile)
components <- getComponents(sol)
variables <- getUserVariables(sol)
links <- getLinks(sol, components)

As a shorthand we can use getElementsFromSolutionFile, which returns the xml object as well as the dataframes in a list.

el <- getElementsFromSolutionFile(solfile)
sol <- el$solution
components <- el$components
variables <- el$variables
links <- el$links

The components dataframe contains not only sim components, but also interfaces, resources and outputs.

components[c(1:3,30:31,42:44),c("id","type","subtype","nr")]

##                   id         type   subtype nr
## 1            CURRENT          var       var  1
## 2          variables          var       var  2
## 3            compare    interface       CSV  3
## 30              crop     resource  resource 30
## 31           ndemand     resource  resource 31
## 42    init_transform     resource transform 42
## 43 DefaultManagement simcomponent       mgm 43
## 44    EvapTranDemand simcomponent    normal 44

The links dataframe contains the links between resources, simcomponents, outputs, but also dependencies via key variables or via variables used in rules. Relationships between resources and transformers are included too.

links[c(1:2,23,300,313,322,325),]

##                  from         name                to     rel
## 1             CURRENT          DOY DefaultManagement   value
## 2           phenology emergencedoy DefaultManagement   value
## 23  DefaultManagement    DoHarvest   LintulPhenology   value
## 300          cropfile            -              crop storage
## 313           CURRENT         DATE     referencedata     key
## 322 DefaultManagement        DoSow           ndemand    rule
## 325         variables    vSoiltype          nitrogen     key

User variables show the values as defined in the solution. Notice that user variables can be changed when running a solution in project mode.

variables[c(1:5,18:20),]

##                      id      value unit rule datatype kind
## 1          vCLIMATEZONE      00002 <NA> <NA>     CHAR  var
## 2             vSoiltype sandy loam <NA> <NA>     CHAR  var
## 3            vTempLimit         35 <NA> <NA>   DOUBLE  var
## 4  vTempSlopeAboveLimit          7 <NA> <NA>   DOUBLE  var
## 5              vBaseLUE        3.0 <NA> <NA>   DOUBLE  var
## 18        vCO2StartYear       2000 <NA> <NA>      INT  dyn
## 19            startdate 01.01.1991 <NA> <NA>     DATE  dyn
## 20              enddate 31.12.2004 <NA> <NA>     DATE  dyn
##                     description
## 1                          <NA>
## 2                          <NA>
## 3                          <NA>
## 4                          <NA>
## 5                          <NA>
## 18 Base year of CO2 development
## 19     Start date of simulation
## 20       End date of simulation

Filenames for interfaces may use placeholder variables like _WORKDIR_ or user defined variables (e.g. vFilename)

head(components[,c("id","ref")])

##              id
## 1       CURRENT
## 2     variables
## 3       compare
## 4   weatherfile
## 5 phenologyfile
## 6      cropfile
##                                                                 ref
## 1                                                            SYSTEM
## 2                                                              User
## 3        \\${_WORKDIR_}/gk/data/lintul/compare/compare_Complete.csv
## 4                                                    \\${vFileName}
## 5     \\${_WORKDIR_}/gk/data/lintul/LintulPhenologyObservations.csv
## 6 \\${_WORKDIR_}/gk/data/lintul/CropPropertiesLintulWaterstress.xml

With replaceVariablesWithValues we can replace the placeholder with the values from the user value dataframe. We can supply additional values for system variables like _WORKDIR_.

components$ref <- replaceVariablesWithValues(
  components$ref,
  variables, 
  additional=c("_WORKDIR_"="~"))
head(components[,c("id","ref")])

##              id                                                  ref
## 1       CURRENT                                               SYSTEM
## 2     variables                                                 User
## 3       compare        ~/gk/data/lintul/compare/compare_Complete.csv
## 4   weatherfile                    ~/gk/data/lintul/00002Weather.txt
## 5 phenologyfile     ~/gk/data/lintul/LintulPhenologyObservations.csv
## 6      cropfile ~/gk/data/lintul/CropPropertiesLintulWaterstress.xml

Working with component and links dataframe

Subsetting the components dataframe to get only normal components

ncomp <- components[components$type=="simcomponent" & components$subtype=="normal",]
ncomp$id

##  [1] "EvapTranDemand"                "Co2InfluenceOnLUE"            
##  [3] "Co2InfluenceOnTranspiration"   "LintulPhenology"              
##  [5] "VernalisationAndPhotoresponse" "SlimRoots"                    
##  [7] "SlimWater"                     "SlimNitrogen"                 
##  [9] "LintulWaterStress"             "LintulPartitioning"           
## [11] "LintulBiomass"                 "NitrogenDemand"               
## [13] "SoilCN"                        "SnowCoverCalculator"          
## [15] "STMPsimCalculator"

Getting only links between normal components

nlinks <- links[links$from %in% ncomp$id & links$to %in% ncomp$id,]
head(nlinks)

##                             from                name
## 12                 LintulBiomass                sLAI
## 16                     SlimWater ActualTranspiration
## 21 VernalisationAndPhotoresponse               RTSUM
## 31               LintulPhenology           sDevStage
## 44             STMPsimCalculator       SoilTempArray
## 45                     SlimWater              DRYFAC
##                               to   rel
## 12                EvapTranDemand value
## 16   Co2InfluenceOnTranspiration value
## 21               LintulPhenology value
## 31 VernalisationAndPhotoresponse value
## 44                     SlimRoots value
## 45                     SlimRoots value

Counting the links between components

nlinks <- aggregate(name ~ from + to , data=nlinks, length)
nlinks <- nlinks[order(-nlinks$name),]
head(nlinks,10)

##                  from                          to name
## 20          SlimWater                SlimNitrogen    9
## 14      LintulBiomass              NitrogenDemand    5
## 3  LintulPartitioning               LintulBiomass    4
## 13          SlimWater           LintulWaterStress    2
## 19          SlimRoots                SlimNitrogen    2
## 28          SlimRoots                   SlimWater    2
## 29      LintulBiomass         SnowCoverCalculator    2
## 32      LintulBiomass                      SoilCN    2
## 1           SlimWater Co2InfluenceOnTranspiration    1
## 2       LintulBiomass              EvapTranDemand    1

Get Metadata about solution

getSolutionInfoAsDataframe gives information about the solutions content (simcomponents) as well as the solution itself (filename, modification date and possibly simulation experiment information derived from folder structure).

wd <- system.file(package="simplaceUtil")
solfiles <- list.files(wd,pattern=".sol.xml",recursive=TRUE,full.names = TRUE)

df <- do.call(rbind,
        lapply(solfiles,getSolutionInfoAsDataframe,workdir=wd))
df$file <- basename(df$file)
df[df$type=="simcomponent",c("id","file")] 

##                               id             file
## 43             DefaultManagement Complete.sol.xml
## 44                EvapTranDemand Complete.sol.xml
## 45             Co2InfluenceOnLUE Complete.sol.xml
## 46   Co2InfluenceOnTranspiration Complete.sol.xml
## 47               LintulPhenology Complete.sol.xml
## 48 VernalisationAndPhotoresponse Complete.sol.xml
## 49                     SlimRoots Complete.sol.xml
## 50                     SlimWater Complete.sol.xml
## 51                  SlimNitrogen Complete.sol.xml
## 52             LintulWaterStress Complete.sol.xml
## 53            LintulPartitioning Complete.sol.xml
## 54                 LintulBiomass Complete.sol.xml
## 55                NitrogenDemand Complete.sol.xml
## 56                        SoilCN Complete.sol.xml
## 57           SnowCoverCalculator Complete.sol.xml
## 58             STMPsimCalculator Complete.sol.xml
## 78             DefaultManagement    Yield.sol.xml
## 79                   LintulPheno    Yield.sol.xml
## 80 VernalisationAndPhotoresponse    Yield.sol.xml
## 81            LintulPartitioning    Yield.sol.xml
## 82                 LintulBiomass    Yield.sol.xml

Sensitivity analysis and calibration

A solution can be modified on the fly to meet requirements for sensitivity analysis or calibration by

• removing non-memory outputs for performance reasons
• adding memory outputs with target variables
• adding user variables for parameters that will be varied
• replacing variables from resource by user variables

solfile <- system.file("solution","Yield.sol.xml",
                       package="simplaceUtil")
sol <- getSolutionFromFile(solfile)
newsol <- sol |>
  removeNonMemoryOutputs() |>
  addMemoryOutput("Yields",frequence = "YEARLY") |>
  addOutputVariable("Yields","Year","CURRENT.YEAR","INT") |>
  addOutputVariable("Yields", "ClimateZone","vClimateZone","CHAR") |>
  addOutputVariable("Yields", "Yield","LintulBiomass.sWSO","DOUBLE","MAX") |>
  addUserVariable("vLUE",3.0,"DOUBLE") |>
  addUserVariable("vSLA",0.02,"DOUBLE") |>
  addUserVariable("vRGRL",0.009,"DOUBLE") |>
  replaceVariable("crop.LUE","vLUE") |>
  replaceVariable("crop.SLA","vSLA") |>
  replaceVariable("crop.RGRL","vRGRL")

Let’s output the beginning of the solution as text:

soltext <- getTextFromSolution(newsol)

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE solution PUBLIC "-//SIMPLACE/DTD SOL 1.0//EN" "http://simplace.net/dtd/SimSolution_5_2.dtd">
<solution version="5.2" check="LAZY">
  <description title="Yield" author="XXX_XXX" qualitylevel="I">WIKI_START
=== Purpose ===
Example solution for simplaceUtil package.
WIKI_END 
 * removed non-memory outputs for interfaces yieldout, sumyieldout 
 * added memory output  Yields 
 * added output variable Year  to output  Yields  
 * added output variable ClimateZone  to output  Yields  
 * added output variable Yield  to output  Yields  
 * added user variable vLUE with value 3 
 * added user variable vSLA with value 0.02 
 * added user variable vRGRL with value 0.009 
 * replaced crop.LUE with vLUE 
 * replaced crop.SLA with vSLA 
 * replaced crop.RGRL with vRGRL</description>
  <variables>
    <var id="vTempLimit" datatype="DOUBLE">35</var>
    <var id="vTempSlopeAboveLimit" datatype="DOUBLE">7</var>
    <var id="vCropName" datatype="CHAR">winter wheat</var>
    <var id="vBaseLUE" datatype="DOUBLE">3.0</var>
    <var id="vBaseTempBeforeAnt" datatype="DOUBLE">3.0</var>
    <var id="vBaseTempAfterAnt" datatype="DOUBLE">3.0</var>
    <var id="startdate" datatype="DATE">01.01.1991</var>
    <var id="enddate" datatype="DATE">31.12.2004</var>
    <var id="vClimateZone" datatype="CHAR">C02</var>
    <var id="vLUE" datatype="DOUBLE">3</var>
    <var id="vSLA" datatype="DOUBLE">0.02</var>
    <var id="vRGRL" datatype="DOUBLE">0.009</var>
  </variables>
  <interfaces>
    <

Notice that the modifications are logged in the description section of the new solution.

The solution can be saved as xml file:

solution_modified <- "simulation/solution/YieldModified.sol.xml"  
writeSolutionToFile(newsol, solution_modified) 

Changing order of SimComponents

As the order of SimComponents can have a considerable impact, one can generate random orders of components and run the reordered solutions to analyse the impact.

solfile <- system.file("solution","Yield.sol.xml",
                       package="simplaceUtil")
sol <- getSolutionFromFile(solfile)

cmp <- getComponents(sol)
cmp[cmp$type=="simcomponent",c("id","subtype","nr")]

##                               id subtype nr
## 14             DefaultManagement     mgm 14
## 15                   LintulPheno  normal 15
## 16 VernalisationAndPhotoresponse  normal 16
## 17            LintulPartitioning  normal 17
## 18                 LintulBiomass  normal 18

We want to swap the components 2 to 5

toswap <- 2:5
swapped <- sample(toswap, length(toswap))
swapped

## [1] 5 3 4 2

newsol <- swapComponents(sol,swapped)
cmp <- getComponents(newsol)
cmp[cmp$type=="simcomponent",c("id","subtype","nr")]

##                               id subtype nr
## 14             DefaultManagement     mgm 14
## 15                 LintulBiomass  normal 15
## 16 VernalisationAndPhotoresponse  normal 16
## 17            LintulPartitioning  normal 17
## 18                   LintulPheno  normal 18

The order of the SimComponent is now different.

Adding timing SimComponent

There is a special SimComponent that records the processing time of SimComponents.

To add timing to a solution, one has to

• add an interface for the timing output
• add the timing SimComponent and configure it to measure the time of all or only of distinct components
• add the output for the timing information

This can be done for every solution automatically via addTimingSimComponent.

solfile <- system.file("solution","Yield.sol.xml",
                       package="simplaceUtil")
sol <- getSolutionFromFile(solfile)

newsol <- addTimingSimComponent(sol, componentlist=c("LintulBiomass","LintulPheno"))
cmp <- getComponents(newsol)
cmp[cmp$type=="simcomponent" | grepl("timing",cmp$id),c("id","type","subtype","nr")]

##                               id         type subtype nr
## 8     automatic_timing_interface    interface  MEMORY  8
## 15             DefaultManagement simcomponent     mgm 15
## 16                   LintulPheno simcomponent  normal 16
## 17 VernalisationAndPhotoresponse simcomponent  normal 17
## 18            LintulPartitioning simcomponent  normal 18
## 19                 LintulBiomass simcomponent  normal 19
## 20               AutomaticTiming simcomponent  normal 20
## 23       automatic_timing_output       output  output 23

The new solution has now an additional SimComponent AutomaticTiming as well as an additional output together with it’s interface.

Transform output data

With parseDate the standard Date column of Simplace output files is converted to Date.

outfile <- system.file("output","water.csv",
                       package="simplaceUtil")
water <- read.csv(outfile, header=TRUE, sep=";")
water <- parseDate(water)

By default the simplace outputs are in wide format, e.g. each layer is in an own column. We can transform the dataframe from wide format to long format, by putting values from all layers into one column and distinguish them by an extra column layer.

water[1:4,c(2,9:11,49)]

##   CURRENT.DATE MobileWater_1 MobileWater_2 MobileWater_3 RetainedWater_1
## 1   1991-07-18             0     0.5214681      1.197492        3.402230
## 2   1991-07-19             0     0.0000000      0.000000        2.045055
## 3   1991-07-20             0     0.0000000      0.000000        1.584985
## 4   1991-07-21             0     0.0000000      0.000000        2.906192

water_long <- transformLayeredData(water)
water_long[c(1:3,41:43,81:83),c(2,9:11)]

## # A tibble: 9 × 4
##   CURRENT.DATE layer MobileWater RetainedWater
##   <date>       <int>       <dbl>         <dbl>
## 1 1991-07-18       1       0              3.40
## 2 1991-07-18       2       0.521          5.4 
## 3 1991-07-18       3       1.20           5.4 
## 4 1991-07-19       1       0              2.05
## 5 1991-07-19       2       0              4.27
## 6 1991-07-19       3       0              5.31
## 7 1991-07-20       1       0              1.58
## 8 1991-07-20       2       0              4.40
## 9 1991-07-20       3       0              5.04

Visualise output variables

Multiple scalar values can be plotted in a common diagram.

plotScalarOutput(water,"CURRENT.DATE",c("Evaporation","Transpiration"))

Default plot for layered values.

plotLayeredOutput(water,"RetainedWater")

Default App

The package includes a Shiny App that runs locally within a webbrowser.

runSimplaceGuiApp()

We can interactively explore solutions by filtering and displaying solution graphs.

We can also use a local Simplace installation to run the simulation defined by the solution and plot some of the output variables.

Custom App

For running a specific solution / project by changing parameters interactively and get instantly custom plots, the user can create a custom shiny app by specifying

• location of Simplace and it’s work-/outputdir
• a solution file and optionally a project file
• a list of plot functions
• a list of data transformation functions
• a list of parameters (with default value, min, max or options)

instdir <- simplace::findFirstSimplaceInstallation()
simplacedirs <- list(
  instdir = instdir,
  workdir = paste0(instdir,"/simplace_run/simulation/"),
  outdir = paste0(instdir,"/simplace_run/output/")
)

solution <- paste0(simplacedirs$workdir,"/gk/solution/calibration/Yield.sol.xml")

paramlist <- list(
 vLUE = list(
   label = "Light Use efficiency",
   value = 3.0,
   min = 1.0,
   max = 6.5
 )
)

plotlist <- list(
  "Default" = simplaceUtil::plotScalarOutput,
  "Boxplots" = function(data,x,y,sim,from,to) {
    if(length(y)==1 && is.numeric(data[[y]])) {
      boxplot(formula(paste(y,"~simulationid")),data)
    }
  }
)

datalist <- list(
  "Aggregated Yield"=function(data){
    dplyr::group_by(data, simulationid) |>
    dplyr::summarise(minY=min(Yield),maxY=max(Yield))
    },
  "Raw Data"=identity
)

simplaceUtil::runSimplaceCustomGuiApp(
  simplacedirs, sol,
  plotlist = plotlist,
  datalist = datalist,
  paramlist = paramlist
)

Creating XML stubs for resources

We can create the interface (location and format) and resource (structure) description for a solution as XML stubs from given input data.

Data can be given in CSV or XML format. The structure of the data has to be compatible with Simplace.

stubs <- createResourceStubsFromCsv(
  filename = system.file("input","weather.csv", package="simplaceUtil"),
  id = "weather",
  sep =",",
  keyvals = c("CURRENT.DATE" = "Date")
)
cat(stubs$interface)

## <interface id="weather_file" type="CSV">
##   <poolsize>100</poolsize>
##   <divider>,</divider
##   <filename>/tmp/RtmppuHWYj/Rinstfa521602fda/simplaceUtil/input/weather.csv</filename>
## </interface>

cat(stubs$resource)

## <resource id="weather" interface="weather_file" frequence="DAILY">
##   <header>
##     <res id="DOY" datatype="INT" unit=""/>
##     <res id="Date" key="CURRENT.DATE" datatype="CHAR" unit=""/>
##     <res id="Rain" datatype="DOUBLE" unit=""/>
##     <res id="Radiation" datatype="DOUBLE" unit=""/>
##     <res id="Tmean" datatype="DOUBLE" unit=""/>
##     <res id="RelHum" datatype="DOUBLE" unit=""/>
##   </header>
## </resource>

The function tries to determine the datatype from the data. We have to check and possibly correct the inferred datatype (e.g. replacing ‘CHAR’ by ‘DATE’ or ‘INT’ by ‘DOUBLE’ or ‘BOOLEAN’).

Creating code stubs for SimVariables

When creating a new SimComponent, we can put all SimVariables (constants, inputs, states, rates, outputs) in a CSV table.

varfile <- system.file("variables","translocation.csv",package="simplaceUtil")
vartable <- read.csv(varfile, header=TRUE, sep=";")
vartable[c(4,8:9),]

##   contenttype                 id                              description
## 4    constant FruitingEfficiency Number of grains per storage organs mass
## 8       state                WEA                           Weight of ears
## 9        rate               RWEA                      Change rate of ears
##   datatype        unit min max default
## 4   Double        g^-1   0  NA      .1
## 8   Double      g m^-2   0  NA       0
## 9   Double g m^-2 d^-1  NA  NA       0

With the function createCodeStubsForSimVariables we can generate so called boilerplate code for Java as well as XML stubs for solutions.

stubs <- createCodeStubsForSimVariables(varfile, "BMTransloc", sep=";")
names(stubs)

## [1] "JavaFields"            "JavaVariables"         "JavaInit"             
## [4] "ParametersXML"         "ResourceParametersXML" "ResourceInputsXML"    
## [7] "ComponentInputsXML"    "OutputsXML"

When we specify an output folder, the stubs are written to files.

stubs <- createCodeStubsForSimVariables(varfile, "BMTransloc", sep=";",
                                        outfolder="translocation_stubs")

Example code stubs

Defining fields in Java (stubs$JavaFields):

private FWSimVariable<Double> cFruitingEfficiency;
private FWSimVariable<Double> rRWEA;
private FWSimVariable<Double> sWEA;

Creating the SimVariables (stubs$JavaVariables):

addVariable(FWSimVariable.createSimVariable("cFruitingEfficiency", "Number of grains per storage organs mass", DATA_TYPE.DOUBLE, CONTENT_TYPE.constant, "g^-1",  0d, null, .1d, this));
addVariable(FWSimVariable.createSimVariable("rRWEA", "Change rate of ears", DATA_TYPE.DOUBLE, CONTENT_TYPE.rate, "g m^-2 d^-1", null, null, 0d, this));
addVariable(FWSimVariable.createSimVariable("sWEA", "Weight of ears", DATA_TYPE.DOUBLE, CONTENT_TYPE.state, "g m^-2",  0d, null, 0d, this));

Default initialisation of states and rates (sic!) (stubs$JavaInit):

rRWEA.setDefaultValue();
sWEA.setDefaultValue();

XML stub for linking inputs (stubs$ComponentInputsXML):

<input id="cTranslocationFraction" source="bmtransloc_parameters.TranslocationFraction" />
<input id="cFruitingEfficiency" source="bmtransloc_parameters.FruitingEfficiency" />
<input id="iEarPartitioningReductionFactor" source="bmtransloc_inputs.EarPartitioningReductionFactor" />

XML stubs for parameter file (stubs$ParametersXML) and coresponding resource section in solution (stubs$ResourceParametersXML):

<parameter id="EarsPartitioningTableDVS" datatype="DOUBLEARRAY" unit="" description="DVS for fraction of above-ground dry matter to ears">
      <value>0</value>
      <value>1</value>
      <value>2</value>
  </parameter>
<parameter id="EarsPartitioningTableFraction" datatype="DOUBLEARRAY" unit="" description="Fraction of above-ground dry matter to ears as function of DVS">
      <value>.0</value>
      <value>.1</value>
      <value>.0</value>
  </parameter>
<parameter id="TranslocationFraction" datatype="DOUBLE" unit="" description="Fraction of stems and leaves biomass  that is reserved for translocation at anthesis">0.2</parameter>
<parameter id="FruitingEfficiency" datatype="DOUBLE" unit="g^-1" description="Number of grains per storage organs mass">.1</parameter>

<res id="EarsPartitioningTableDVS" datatype="DOUBLEARRAY" unit="" description="DVS for fraction of above-ground dry matter to ears" />
<res id="EarsPartitioningTableFraction" datatype="DOUBLEARRAY" unit="" description="Fraction of above-ground dry matter to ears as function of DVS" />
<res id="TranslocationFraction" datatype="DOUBLE" unit="" description="Fraction of stems and leaves biomass  that is reserved for translocation at anthesis" />
<res id="FruitingEfficiency" datatype="DOUBLE" unit="g^-1" description="Number of grains per storage organs mass" />