How Modelers Model: the Overlooked Social and Human Dimensions in Model Intercomparison Studies

There is a growing realization that the complexity of model ensemble studies depends not only on the models used but also on the experience and approach used by modelers to calibrate and validate results, which remain a source of uncertainty. Here, we applied a multi-criteria decision-making method to investigate the rationale applied by modelers in a model ensemble study where 12 process-based different biogeochemical model types were compared across five successive calibration stages. The modelers shared a common level of agreement about the importance of the variables used to initialize their models for calibration. However, we found inconsistency among modelers when judging the importance of input variables across different calibration stages. The level of subjective weighting attributed by modelers to calibration data decreased sequentially as the extent and number of variables provided increased. In this context, the perceived importance attributed to variables such as the fertilization rate, irrigation regime, soil texture, pH, and initial levels of soil organic carbon and nitrogen stocks was statistically different when classified according to model types. The importance attributed to input variables such as experimental duration, gross primary production, and net ecosystem exchange varied significantly according to the length of the modeler's experience. We argue that the gradual access to input data across the five calibration stages negatively influenced the consistency of the interpretations made by the modelers, with cognitive bias in "trial-and-error" calibration routines. Our study highlights that overlooking human and social attributes is critical in the outcomes of modeling and model intercomparison studies. While complexity of the processes captured in the model algorithms and parameterization is important, we contend that (1) the modeler's assumptions on the extent to which parameters should be altered and (2) modeler perceptions of the importance of model parameters are just as critical in obtaining a quality model calibration as numerical or analytical details.

Ensemble modelling, uncertainty and robust predictions of organic carbon in long-term bare-fallow soils

Simulation models represent soil organic carbon (SOC) dynamics in global carbon (C) cycle scenarios to support climate-change studies. It is imperative to increase confidence in long-term predictions of SOC dynamics by reducing the uncertainty in model estimates. We evaluated SOC simulated from an ensemble of 26 process-based C models by comparing simulations to experimental data from seven long-term bare-fallow (vegetation-free) plots at six sites: Denmark (two sites), France, Russia, Sweden and the United Kingdom. The decay of SOC in these plots has been monitored for decades since the last inputs of plant material, providing the opportunity to test decomposition without the continuous input of new organic material. The models were run independently over multi-year simulation periods (from 28 to 80 years) in a blind test with no calibration (Bln) and with the following three calibration scenarios, each providing different levels of information and/or allowing different levels of model fitting: (a) calibrating decomposition parameters separately at each experimental site (Spe); (b) using a generic, knowledge-based, parameterization applicable in the Central European region (Gen); and (c) using a combination of both (a) and (b) strategies (Mix). We addressed uncertainties from different modelling approaches with or without spin-up initialization of SOC. Changes in the multi-model median (MMM) of SOC were used as descriptors of the ensemble performance. On average across sites, Gen proved adequate in describing changes in SOC, with MMM equal to average SOC (and standard deviation) of 39.2 (±15.5) Mg C/ha compared to the observed mean of 36.0 (±19.7) Mg C/ha (last observed year), indicating sufficiently reliable SOC estimates. Moving to Mix (37.5 ± 16.7 Mg C/ha) and Spe (36.8 ± 19.8 Mg C/ha) provided only marginal gains in accuracy, but modellers would need to apply more knowledge and a greater calibration effort than in Gen, thereby limiting the wider applicability of models.

The use of biogeochemical models to evaluate mitigation of greenhouse gas emissions from managed grasslands

Simulation models quantify the impacts on carbon (C) and nitrogen (N) cycling in grassland systems caused by changes in management practices. To support agricultural policies, it is however important to contrast the responses of alternative models, which can differ greatly in their treatment of key processes and in their response to management. We applied eight biogeochemical models at five grassland sites (in France, New Zealand, Switzerland, United Kingdom and United States) to compare the sensitivity of modelled C and N fluxes to changes in the density of grazing animals (from 100% to 50% of the original livestock densities), also in combination with decreasing N fertilization levels (reduced to zero from the initial levels). Simulated multi-model median values indicated that input reduction would lead to an increase in the C sink strength (negative net ecosystem C exchange) in intensive grazing systems: -64 ± 74 g C m^-2 yr^-1 (animal density reduction) and -81 ± 74 g C m^-2 yr^-1 (N and animal density reduction), against the baseline of -30.5 ± 69.5 g C m^-2 yr^-1 (LSU [livestock units] ≥ 0.76 ha^-1 yr^-1). Simulations also indicated a strong effect of N fertilizer reduction on N fluxes, e.g. N₂O-N emissions decreased from 0.34 ± 0.22 (baseline) to 0.1 ± 0.05 g N m^-2 yr^-1 (no N fertilization). Simulated decline in grazing intensity had only limited impact on the N balance. The simulated pattern of enteric methane emissions was dominated by high model-to-model variability. The reduction in simulated offtake (animal intake + cut biomass) led to a doubling in net primary production per animal (increased by 11.6 ± 8.1 t C LSU^-1 yr^-1 across sites). The highest N₂O-N intensities (N₂O-N/offtake) were simulated at mown and extensively grazed arid sites. We show the possibility of using grassland models to determine sound mitigation practices while quantifying the uncertainties associated with the simulated outputs.

Assessing uncertainties in crop and pasture ensemble model simulations of productivity and N2 O emissions

Simulation models are extensively used to predict agricultural productivity and greenhouse gas emissions. However, the uncertainties of (reduced) model ensemble simulations have not been assessed systematically for variables affecting food security and climate change mitigation, within multi-species agricultural contexts. We report an international model comparison and benchmarking exercise, showing the potential of multi-model ensembles to predict productivity and nitrous oxide (N₂ O) emissions for wheat, maize, rice and temperate grasslands. Using a multi-stage modelling protocol, from blind simulations (stage 1) to partial (stages 2-4) and full calibration (stage 5), 24 process-based biogeochemical models were assessed individually or as an ensemble against long-term experimental data from four temperate grassland and five arable crop rotation sites spanning four continents. Comparisons were performed by reference to the experimental uncertainties of observed yields and N₂ O emissions. Results showed that across sites and crop/grassland types, 23%-40% of the uncalibrated individual models were within two standard deviations (SD) of observed yields, while 42 (rice) to 96% (grasslands) of the models were within 1 SD of observed N₂ O emissions. At stage 1, ensembles formed by the three lowest prediction model errors predicted both yields and N₂ O emissions within experimental uncertainties for 44% and 33% of the crop and grassland growth cycles, respectively. Partial model calibration (stages 2-4) markedly reduced prediction errors of the full model ensemble E-median for crop grain yields (from 36% at stage 1 down to 4% on average) and grassland productivity (from 44% to 27%) and to a lesser and more variable extent for N₂ O emissions. Yield-scaled N₂ O emissions (N₂ O emissions divided by crop yields) were ranked accurately by three-model ensembles across crop species and field sites. The potential of using process-based model ensembles to predict jointly productivity and N₂ O emissions at field scale is discussed.

Review and analysis of strengths and weaknesses of agro-ecosystem models for simulating C and N fluxes

Biogeochemical simulation models are important tools for describing and quantifying the contribution of agricultural systems to C sequestration and GHG source/sink status. The abundance of simulation tools developed over recent decades, however, creates a difficulty because predictions from different models show large variability. Discrepancies between the conclusions of different modelling studies are often ascribed to differences in the physical and biogeochemical processes incorporated in equations of C and N cycles and their interactions. Here we review the literature to determine the state-of-the-art in modelling agricultural (crop and grassland) systems. In order to carry out this study, we selected the range of biogeochemical models used by the CN-MIP consortium of FACCE-JPI (http://www.faccejpi.com): APSIM, CERES-EGC, DayCent, DNDC, DSSAT, EPIC, PaSim, RothC and STICS. In our analysis, these models were assessed for the quality and comprehensiveness of underlying processes related to pedo-climatic conditions and management practices, but also with respect to time and space of application, and for their accuracy in multiple contexts. Overall, it emerged that there is a possible impact of ill-defined pedo-climatic conditions in the unsatisfactory performance of the models (46.2%), followed by limitations in the algorithms simulating the effects of management practices (33.1%). The multiplicity of scales in both time and space is a fundamental feature, which explains the remaining weaknesses (i.e. 20.7%). Innovative aspects have been identified for future development of C and N models. They include the explicit representation of soil microbial biomass to drive soil organic matter turnover, the effect of N shortage on SOM decomposition, the improvements related to the production and consumption of gases and an adequate simulations of gas transport in soil. On these bases, the assessment of trends and gaps in the modelling approaches currently employed to represent biogeochemical cycles in crop and grassland systems appears an essential step for future research.