Adding decision models to systematic reviews: informing a framework for deciding when and how to do so

The influence of loss to follow-up in autism screening research: Taking stock and moving forward

BACKGROUND

How best to improve the early detection of autism spectrum disorder (ASD) is the subject of significant controversy. Some argue that universal ASD screeners are highly accurate, whereas others argue that evidence for this claim is insufficient. Relatedly, there is no clear consensus as to the optimal role of screening for making referral decisions for evaluation and treatment. Published screening research can meaningfully inform these questions-but only through careful consideration of children who do not complete diagnostic follow-up.

METHODS

We developed two simulation models that re-analyze the results of a large-scale validation study of the M-CHAT-R/F by Robins et al. (2014, Pediatrics, 133, 37). Model #1 re-analyzes screener accuracy across six scenarios, each reflecting different assumptions regarding loss to follow-up. Model #2 builds on this by closely examining differential attrition at each point of the multi-step detection process.

RESULTS

Estimates of sensitivity ranged from 40% to 94% across scenarios, demonstrating that estimates of accuracy depend on assumptions regarding the diagnostic status of children who were lost to follow-up. Across a range of plausible assumptions, data also suggest that children with undiagnosed ASD may be more likely to complete follow-up than children without ASD, highlighting the role of clinicians and caregivers in the detection process.

CONCLUSIONS

Using simulation modeling as a quantitative method to examine potential bias in screening studies, analyses suggest that ASD screening tools may be less accurate than is often reported. Models also demonstrate the critical importance of every step in a detection process-including steps that determine whether children should complete an additional evaluation. We conclude that parent and clinician decision-making regarding follow-up may contribute more to detection than is widely assumed.

Partial herd hoof trimming results in a higher economic net benefit than whole herd hoof trimming in dairy herds

Sole ulcers, a common cause of lameness is the costliest non-infectious foot lesion in dairy cows and one of the most prevalent non-infectious foot lesions in freestall housing systems. Costs associated with sole ulcers are treatment costs, plus increased labor and decreased productivity and fertility. Routine hoof trimming is part of a strategy to manage sole ulcers. However, hoof trimming strategies differ among farms. The two most frequently applied strategies are: 1) partial herd hoof trimming with a 2-month interval between trims; and 2) whole herd hoof trimming at 6-month intervals. A Markov model was developed to investigate whether every 2 months partial herd hoof trimming or whole herd hoof trimming every 6 months was the most cost-effective strategy to avoid costs associated with sole ulcers. In this model, the net benefits for a 100-cow herd and the average productive life span of a dairy cow in intensive dairy systems of 3 years were evaluated. Partial herd hoof trimming was the most cost-effective strategy 100% of the time compared to whole herd hoof trimming, with a difference in 3-year net benefits of US$4,337 (95% CI: US$2,713-US$5,830). Based on sensitivity analyses, variables that were the sources of the biggest uncertainty in the model were herd size, the probability of being trimmed in a partial herd trim, and the prevalence of sole ulcers. To further investigate the impacts of herd size and of probability of being trimmed, various scenario analyses were conducted. With increasing herd size, the difference in net benefits in favor of partial herd hoof trimming increased even more. Scenario analyses about the probability of getting trimmed all indicated that targeted intervention increased the difference in net benefits in favor of partial herd hoof trimming. However, if the selection of cows to be trimmed in a partial herd trim was random, the whole herd hoof trimming strategy became cost-effective. Therefore, targeted selection and early intervention are necessary to decrease costs associated with sole ulcers.

Sonoelastography for Testicular Tumor Identification: A Systematic Review and Meta-Analysis of Diagnostic Test Accuracy

The objective of this review was to summarize the applications of sonoelastography in testicular tumor identification and inquire about their test performances. Two authors independently searched English journal articles and full conference papers from CINAHL, Embase, IEEE Xplore®, PubMed, Scopus, and Web of Science from inception and organized them into a PIRO (patient, index test, reference test, outcome) framework. Eleven studies (n = 11) were eligible for data synthesis, nine of which (n = 9) utilized strain elastography and two (n = 2) employed shear-wave elastography. Meta-analyses were performed on the distinction between neoplasm (tumor) and non-neoplasm (non-tumor) from four study arms and between malignancy and benignity from seven study arms. The pooled sensitivity of classifying malignancy and benignity was 86.0% (95%CI, 79.7% to 90.6%). There was substantial heterogeneity in the classification of neoplasm and non-neoplasm and in the specificity of classifying malignancy and benignity, which could not be addressed by the subgroup analysis of sonoelastography techniques. Heterogeneity might be associated with the high risk of bias and applicability concern, including a wide spectrum of testicular pathologies and verification bias in the reference tests. Key technical obstacles in the index test were manual compression in strain elastography, qualitative observation of non-standardized color codes, and locating the Regions of Interest (ROI), in addition to decisions in feature extractions. Future research may focus on multiparametric sonoelastography using deep learning models and ensemble learning. A decision model on the benefits-risks of surgical exploration (reference test) could also be developed to direct the test-and-treat strategy for testicular tumors.

Collaborative Modeling: Experience of the U.S. Preventive Services Task Force

Models can be valuable tools to address uncertainty, trade-offs, and preferences when trying to understand the effects of interventions. Availability of results from two or more independently developed models that examine the same question (comparative modeling) allows systematic exploration of differences between models and the effect of these differences on model findings. Guideline groups sometimes commission comparative modeling to support their recommendation process. In this commissioned collaborative modeling, modelers work with the people who are developing a recommendation or policy not only to define the questions to be addressed but ideally, work side-by-side with each other and with systematic reviewers to standardize selected inputs and incorporate selected common assumptions. This paper describes the use of commissioned collaborative modeling by the U.S. Preventive Services Task Force (USPSTF), highlighting the general challenges and opportunities encountered and specific challenges for some topics. It delineates other approaches to use modeling to support evidence-based recommendations and the many strengths of collaborative modeling compared with other approaches. Unlike systematic reviews prepared for the USPSTF, the commissioned collaborative modeling reports used by the USPSTF in making recommendations about screening have not been required to follow a common format, sometimes making it challenging to understand key model features. This paper presents a checklist developed to critically appraise commissioned collaborative modeling reports about cancer screening topics prepared for the USPSTF.

The role of mathematical modelling in the development of recommendations in the 2013 WHO consolidated antiretroviral therapy guidelines

Despite the exponential growth in the literature on modelling and simulation studies of impact and cost-effectiveness in different aspects of healthcare, there is no clear consensus on the appropriate role of modelling in the development of recommendations in clinical guidelines. This is compounded both by the lack of a standardised approach to assess the quality of modelling, and lack of clarity on its positioning within the GRADE (Grading of Recommendations, Assessment, Development, and Evaluation) method for decision-making in the development of WHO guidelines, that considers both evidence from systematic reviews of randomized clinical trials (RTCs) or observational studies, together with stakeholder values and preferences, resource use, and feasibility issues. In the development of the 2013 WHO Consolidated Guidelines on the use of Antiretroviral drugs for treating and preventing HIV infection, a series of modelling projects were undertaken to inform the recommendations on eligibility criteria for ART initiation, and approaches to monitoring for treatment response. We report our experiences, challenges encountered, and several key considerations to guide the future use of modelling in the development of WHO guidelines. These are: (1) Transparency in the conduct and reporting of model inputs and results; (2) The need for agreed standards for critical appraisal and use of modelling data in healthcare policy making; (3) recognition that modelling of cost-effectiveness is only one component of decision-making in development of WHO recommendations and in priority-setting; (4) The need for closer interaction and an ongoing dialogue between modellers and model end-users or decision-makers; (5) the important role of WHO in convening and facilitating comparative assessment of multiple models; and (6) The need to optimize research and data collection to inform modelling studies.