Testing prediction accuracy in short-term ecological studies

Near-term phytoplankton forecasts reveal the effects of model time step and forecast horizon on predictability

As climate and land use increase the variability of many ecosystems, forecasts of ecological variables are needed to inform management and use of ecosystem services. In particular, forecasts of phytoplankton would be especially useful for drinking water management, as phytoplankton populations are exhibiting greater fluctuations due to human activities. While phytoplankton forecasts are increasing in number, many questions remain regarding the optimal model time step (the temporal frequency of the forecast model output), time horizon (the length of time into the future a prediction is made) for maximizing forecast performance, as well as what factors contribute to uncertainty in forecasts and their scalability among sites. To answer these questions, we developed near-term, iterative forecasts of phytoplankton 1-14 days into the future using forecast models with three different time steps (daily, weekly, fortnightly), that included a full uncertainty partitioning analysis at two drinking water reservoirs. We found that forecast accuracy varies with model time step and forecast horizon, and that forecast models can outperform null estimates under most conditions. Weekly and fortnightly forecasts consistently outperformed daily forecasts at 7-day and 14-day horizons, a trend that increased up to the 14-day forecast horizon. Importantly, our work suggests that forecast accuracy can be increased by matching the forecast model time step to the forecast horizon for which predictions are needed. We found that model process uncertainty was the primary source of uncertainty in our phytoplankton forecasts over the forecast period, but parameter uncertainty increased during phytoplankton blooms and when scaling the forecast model to a new site. Overall, our scalability analysis shows promising results that simple models can be transferred to produce forecasts at additional sites. Altogether, our study advances our understanding of how forecast model time step and forecast horizon influence the forecastability of phytoplankton dynamics in aquatic systems and adds to the growing body of work regarding the predictability of ecological systems broadly.

PERFICT: A Re-imagined foundation for predictive ecology

Making predictions from ecological models-and comparing them to data-offers a coherent approach to evaluate model quality, regardless of model complexity or modelling paradigm. To date, our ability to use predictions for developing, validating, updating, integrating and applying models across scientific disciplines while influencing management decisions, policies, and the public has been hampered by disparate perspectives on prediction and inadequately integrated approaches. We present an updated foundation for Predictive Ecology based on seven principles applied to ecological modelling: make frequent Predictions, Evaluate models, make models Reusable, Freely accessible and Interoperable, built within Continuous workflows that are routinely Tested (PERFICT). We outline some benefits of working with these principles: accelerating science; linking with data science; and improving science-policy integration.

Ecological forecasts reveal limitations of common model selection methods: predicting changes in beaver colony densities

Over the past two decades, there have been numerous calls to make ecology a more predictive science through direct empirical assessments of ecological models and predictions. While the widespread use of model selection using information criteria has pushed ecology toward placing a higher emphasis on prediction, few attempts have been made to validate the ability of information criteria to correctly identify the most parsimonious model with the greatest predictive accuracy. Here, we used an ecological forecasting framework to test the ability of information criteria to accurately predict the relative contribution of density dependence and density-independent factors (forage availability, harvest, weather, wolf [Canis lupus] density) on inter-annual fluctuations in beaver (Castor canadensis) colony densities. We modeled changes in colony densities using a discrete-time Gompertz model, and assessed the performance of four models using information criteria values: density-independent models with (1) and without (2) environmental covariates; and density-dependent models with (3) and without (4) environmental covariates. We then evaluated the forecasting accuracy of each model by withholding the final one-third of observations from each population and compared observed vs. predicted densities. Information criteria and our forecasting accuracy metrics both provided strong evidence of compensatory density dependence in the annual dynamics of beaver colony densities. However, despite strong within-sample performance by the most complex model (density-dependent with covariates) as determined using information criteria, hindcasts of colony densities revealed that the much simpler density-dependent model without covariates performed nearly as well predicting out-of-sample colony densities. The hindcast results indicated that the complex model over-fit our data, suggesting that parameters identified by information criteria as important predictor variables are only marginally valuable for predicting landscape-scale beaver colony dynamics. Our study demonstrates the importance of evaluating ecological models and predictions with long-term data and revealed how a known limitation of information criteria (over-fitting of complex models) can affect our interpretation of ecological dynamics. While incorporating knowledge of the factors that influence animal population dynamics can improve population forecasts, we suggest that comparing forecast performance metrics can likewise improve our knowledge of the factors driving population dynamics.