Decision-Making about Healthcare Related Tests and Diagnostic Strategies: User Testing of GRADE Evidence Tables

Co-creation of a step-by-step guide for specifying the test-management pathway to formulate focused guideline questions about healthcare related tests

BACKGROUND

Guideline development on testing is known to be difficult for guideline developers. It requires consideration of various aspects, such as accuracy, purpose of testing, and consequences on management and people-important outcomes. This can be outlined in a test-management pathway. We aimed to create and user-test a step-by-step guide for guideline developers for designing a test-management pathway.

METHODS

Developmental design with a co-creative strategy. We created a draft step-by-step guide, that was user tested in a workshop with 19 experts, and by interviewing 7 guideline panel members.

RESULTS

Our proposed guide consists of five blocks of signalling questions: patients/population, index test(s), current practice/comparison/control, people-important outcomes, and the link between testing and outcome(s). The user testing led to refinement of the signalling questions, the use of inclusive terminology, and addition of a test-management pathway figure with detailed explanation.

CONCLUSIONS

The step-by-step guide for formulating focused guideline questions regarding healthcare related testing can help in identifying relevant characteristics of the population, tests, and outcomes and to create a test management pathway. This should facilitate the formulation of evidence-based guideline recommendations about healthcare related testing.

Required knowledge for guideline panel members to develop healthcare related testing recommendations: a developmental study

OBJECTIVES

To define the minimum knowledge required for guideline panel members (healthcare professionals and consumers) involved in developing recommendations about healthcare related testing.

STUDY DESIGN AND SETTING

A developmental study with a multistaged approach. We derived a first set of knowledge components from literature and subsequently performed semistructured interviews with 9 experts. We refined the set of knowledge components and checked it with the interviewees for final approval.

RESULTS

Understanding the test-management pathway, for example, how test results should be used in context of decisions about interventions, is the key knowledge component. The final list includes 26 items on the following topics: health question, test-management pathway, target population, test, test result, interpretation of test results and subsequent management, and impact on people important outcomes. For each item, the required level of knowledge is defined.

CONCLUSION

We developed a list of knowledge components required for guideline panels to formulate recommendations on healthcare related testing. The list could be used to design specific training programs for guideline panel members when developing recommendations about tests and testing strategies in healthcare.

Efficacy of the hybrid closedloop insulin delivery system in children and adolescents with type 1 diabetes: a meta-analysis with trial sequential analysis

The aim of this study was to assess the efficacy and safety of hybrid closed-loop (HCL) systems for insulin delivery in children and adolescents with type 1 diabetes (T1D). We searched Embase, PubMed, and Cochrane Library for randomized controlled trials (RCTs) published until March 2023 comparing the HCL therapy with control therapies for children and adolescents with T1D. We computed weighted mean differences (WMDs) for continuous outcomes and risk ratios (RRs) with 95% confidence intervals (CIs) for binary endpoints. Four RCTs and 501 patients were included, of whom 323 were randomized to HCL therapy. Compared with control therapies, HCL significantly improved the period during which glucose level was 70-180 mg/dL (WMD 10.89%, 95% CI 8.22-13.56%) and the number of participants with glycated hemoglobin (HbA1c) level < 7% (RR 2.61, 95% CI 1.29-5.28). Also, HCL significantly reduced the time during which glucoselevel was > 180 mg/dL (WMD-10.46%, 95% CI-13.99 to-6.93%) and the mean levels of glucose (WMD-16.67 mg/dL, 95% CI-22.25 to-11.09 mg/dL) and HbA1c (WMD-0.50%, 95% CI-0.68 to-0.31). There were no significant differences between therapies regarding time during which glucose level was < 70 mg/dL or <54 mg/dL or number of episodes of ketoacidosis, hyperglycemia, and hypoglycemia. In this meta-analysis, HCL compared with control therapies was associated with improved time in range and HbA1c control in children and adolescents with T1D and a similar profile of side effects. These findings support the efficacy of HCL in the treatment of T1D in this population.

A Metasynthesis and Meta-analysis of the Impact and Diagnostic Safety of COVID-19 Symptom Agnostic Rapid Testing in Low- and Middle-Income Countries: Protocol for a Systematic Review

BACKGROUND

Amid all public health measures to contain COVID-19, the most challenging has been how to break the transmission chain. This has been even more challenging in low- and middle-income countries (LMICs). A public health emergency warrants a public health perspective, which comes down to prevention. Rapid mass testing has been advocated throughout the pandemic as a way to promptly deal with asymptomatic infections, but its usefulness in LMICs is yet to be fully understood.

OBJECTIVE

The study objectives of this paper are to (1) investigate the impact of the different rapid mass testing options for SARS-CoV-2 that have been delivered at point of care in LMICs and (2) evaluate the diagnostic safety (accuracy) of rapid mass testing for SARS-CoV-2 in LMICs.

METHODS

This review will systematically search records in PubMed, EBSCOhost, Cochrane library, Global Index Medicus COVID-19 Register, and Scopus. Records will be managed using Mendeley reference manager and SWIFT-Review. Risk of bias for randomized controlled trials will be assessed using the RoB 2 assessment tool, while nonrandomized interventions will be assessed using the tool developed by the Evidence Project. A narrative approach will be used to synthesize data under the first objective, and either a meta-analysis or synthesis without meta-analysis for the second objective. Tables, figures, and textual descriptions will be used to present findings. The overall body of evidence for the first objective will be assessed using the Grading of Recommendations Assessment, Development, and Evaluation-Confidence in the Evidence from Reviews of Qualitative research (GRADE-CERQual) approach, and for the second objective using GRADE.

RESULTS

The screening of records has been finalized. We hope to finalize the synthesis by the end of February 2023 and to prepare the manuscript for publication by April 2023. The study will be reported in accordance with standard guidelines for the reporting of systematic reviews. Review results will be disseminated through conferences and their peer-reviewed publication in a relevant journal.

CONCLUSIONS

This review highlights the role of a preventive approach in infection control using rapid mass testing. It also flags the overriding need to involve users and providers in the evaluation of such tests in the settings for which they are intended. This will be the first review to the best of our knowledge to generate both qualitative and quantitative evidence regarding rapid mass testing specific to LMICs.

TRIAL REGISTRATION

PROSPERO CRD42022283776; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=283776.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/41132.

The effectiveness and acceptability of evidence synthesis summary formats for clinical guideline development groups: a mixed-methods systematic review

INTRODUCTION

Clinical guideline development often involves a rigorous synthesis of evidence involving multidisciplinary stakeholders with different priorities and knowledge of evidence synthesis; this makes communicating findings complex. Summary formats are typically used to communicate the results of evidence syntheses; however, there is little consensus on which formats are most effective and acceptable for different stakeholders.

METHODS

This mixed-methods systematic review (MMSR) aimed to evaluate the effectiveness and acceptability (e.g. preferences and attitudes and preferences towards) of evidence synthesis summary formats for GDG members. We followed the PRISMA 2020 guideline and Joanna Briggs Institute Manual for Evidence Synthesis for MMSRs. We searched six databases (inception to April 20, 2021) for randomised controlled trials (RCTs), RCTs with a qualitative component, and qualitative studies. Screening, data extraction, and quality appraisal were performed in duplicate. Qualitative findings were synthesised using meta-aggregation, and quantitative findings are described narratively.

RESULTS

We identified 17,240 citations and screened 54 full-text articles, resulting in 22 eligible articles (20 unique studies): 4 articles reported the results of 5 RCTs, one of which also had a qualitative component. The other 18 articles discussed the results of 16 qualitative studies. Therefore, we had 5 trials and 17 qualitative studies to extract data from. Studies were geographically heterogeneous and included a variety of stakeholders and summary formats. All 5 RCTs assessed knowledge or understanding with 3 reporting improvement with newer formats. The qualitative analysis identified 6 categories of recommendations: 'presenting information', 'tailoring information' for end users, 'trust in producers and summary', 'knowledge required' to understand findings, 'quality of evidence', and properly 'contextualising information'. Across these categories, the synthesis resulted in 126 recommendations for practice. Nine recommendations were supported by both quantitative and qualitative evidence and 116 by only qualitative. A majority focused on how to present information (n = 64) and tailor content for different end users (n = 24).

CONCLUSIONS

This MMSR provides guidance on how to improve evidence summary structure and layout. This can be used by synthesis producers to better communicate to GDGs. Study findings will inform the co-creation of evidence summary format prototypes based on GDG member's needs. Trial registration The protocol for this project was previously published, and the project was preregistered on Open Science Framework (Clyne and Sharp, Evidence synthesis and translation of findings for national clinical guideline development: addressing the needs and preferences of guideline development groups, 2021; Sharp and Clyne, Evidence synthesis summary formats for decision-makers and Clinical Guideline Development Groups: A mixed-methods systematic review protocol, 2021).

GRADE summary of findings tables Enhanced Understanding of Values and Preferences Evidence

OBJECTIVE

We conceptualize patient values and preferences as the relative importance of health outcomes (RIO) which are often obtained through utility elicitation research. A transparent and structured approach to present synthesized RIO evidence and the certainty of this evidence is needed. This study aims to adapt the summary of findings (SoF) table to describe the RIO.

STUDY DESIGN AND SETTING

We performed three interactive workshops with a protype version of the SoF table for RIO evidence adapted from the SoF table for intervention effects. We then tested the new format through semi-structured interviews with professionals who interpret RIO evidence (e.g., systematic review authors and guideline developers).

RESULTS

We adapted the SoF table for the presentation of RIO evidence. This SoF table may be easy to use but bears one risk: some participants misunderstood the utility information and the variability around the RIO. We added a visual analogue scale to clarify the concept of utilities.

CONCLUSION

Through a multi-stage process including brainstorming sessions and interviews, we adapted the SoF table to present RIO evidence. This table may enhance understanding of evidence synthesis of values and preferences, facilitating the incorporation of this type of evidence in decision making.

Oral preanesthetic medication in children - comparison between midazolam alone and in combination with ketamine: a systematic review and meta-analysis

BACKGROUND

Up to 60% of pediatric surgical patients develop high levels of preoperative anxiety. This study compared the effects of oral combinations of midazolam and ketamine with oral midazolam alone for pediatric preanesthetic medication.

METHODS

The study protocol was registered in PROSPERO as CRD42020172920. A systematic literature search was conducted using Medline, Cochrane, EMBASE, CENTRAL, and Web of Science for randomized controlled trials comparing oral combinations of midazolam and ketamine with midazolam alone as preanesthetic medication in elective surgical pediatric patients. Meta-analyses included the following outcomes: anxiety and sedation levels, child's behavior during separation from parents, face mask acceptance, and venipuncture. The quality of evidence was assessed using GRADE criteria.

RESULTS

Twenty studies were included. The following effects (RR (95% CI)) were observed for combinations of ketamine and midazolam relative midazolam alone: anxiolysis (1.2 (0.94-1.52); p = 0.15; I2 = 80%; GRADE = very low); satisfactory sedation (1.2 ( 1.10-1.31); p < 0.001; I2 = 71%; GRADE = very low); behavior during parental separation (1.2 (1.06-1.36); p = 0.003; I2 = 88%; GRADE = very low); facial mask acceptance (1.13 (1.04-1.24); p = 0.007; I2 = 49%; GRADE = very low); behavior during venipuncture (1.32 (1.11-1.57); p = 0.002; I2 = 66%; GRADE = very low).

CONCLUSIONS

While similar probabilities of obtaining anxiolysis were found, adequate sedation, calm behavior during child's separation from parents, low levels of fear during face mask adaptation, and cooperative behavior during peripheral venous cannulation were more likely with midazolam-ketamine combinations.

Diagnostic accuracy of tests for type 2 diabetes and prediabetes: A systematic review and meta-analysis

AIM

This systematic review aimed to ascertain the diagnostic accuracy (sensitivity and specificity) of screening tests for early detection of type 2 diabetes and prediabetes in previously undiagnosed adults.

METHODS

This systematic review included published studies that included one or more index tests (random and fasting tests, HbA1c) for glucose detection, with 75-gram Oral Glucose Tolerance Test (or 2-hour post load glucose) as a reference standard (PROSPERO ID CRD42018102477). Seven databases were searched electronically (from their inception up to March 9, 2020) accompanied with bibliographic and website searches. Records were manually screened and full text were selected based on inclusion and exclusion criteria. Subsequently, data extraction was done using standardized form and quality assessment of studies using QUADAS-2 tool. Meta-analysis was done using bivariate model using Stata 14.0. Optimal cut offs in terms of sensitivity and specificity for the tests were analysed using R software.

RESULTS

Of 7,151 records assessed by title and abstract, a total of 37 peer reviewed articles were included in this systematic review. The pooled sensitivity, specificity, positive (LR+) and negative likelihood ratio (LR-) for diagnosing diabetes with HbA1c (6.5%; venous sample; n = 17 studies) were 50% (95% CI: 42-59%), 97.3% (95% CI: 95.3-98.4), 18.32 (95% CI: 11.06-30.53) and 0.51 (95% CI: 0.43-0.60), respectively. However, the optimal cut-off for diagnosing diabetes in previously undiagnosed adults with HbA1c was estimated as 6.03% with pooled sensitivity of 73.9% (95% CI: 68-79.1%) and specificity of 87.2% (95% CI: 82-91%). The optimal cut-off for Fasting Plasma Glucose (FPG) was estimated as 104 milligram/dL (mg/dL) with a sensitivity of 82.3% (95% CI: 74.6-88.1%) and specificity of 89.4% (95% CI: 85.2-92.5%).

CONCLUSION

Our findings suggest that at present recommended threshold of 6.5%, HbA1c is more specific and less sensitive in diagnosing the newly detected diabetes in undiagnosed population from community settings. Lowering of thresholds for HbA1c and FPG to 6.03% and 104 mg/dL for early detection in previously undiagnosed persons for screening purposes may be considered.

Data sources and methods used to determine pretest probabilities in a cohort of Cochrane diagnostic test accuracy reviews

Background

A pretest probability must be selected to calculate data to help clinicians, guideline boards and policy makers interpret diagnostic accuracy parameters. When multiple analyses for the same target condition are compared, identical pretest probabilities might be selected to facilitate the comparison. Some pretest probabilities may lead to exaggerations of the patient harms or benefits, and guidance on how and why to select a specific pretest probability is minimally described. Therefore, the aim of this study was to assess the data sources and methods used in Cochrane diagnostic test accuracy (DTA) reviews for determining pretest probabilities to facilitate the interpretation of DTA parameters. A secondary aim was to assess the use of identical pretest probabilities to compare multiple meta-analyses within the same target condition.

Methods

Cochrane DTA reviews presenting at least one meta-analytic estimate of the sensitivity and/or specificity as a primary analysis published between 2008 and January 2018 were included. Study selection and data extraction were performed by one author and checked by other authors. Observed data sources (e.g. studies in the review, or external sources) and methods to select pretest probabilities (e.g. median) were categorized.

Results

Fifty-nine DTA reviews were included, comprising of 308 meta-analyses. A pretest probability was used in 148 analyses. Authors used included studies in the DTA review, external sources, and author consensus as data sources for the pretest probability. Measures of central tendency with or without a measure of dispersion were used to determine the pretest probabilities, with the median most commonly used. Thirty-two target conditions had at least one identical pretest probability for all of the meta-analyses within their target condition. About half of the used identical pretest probabilities were inside the prevalence ranges from all analyses within a target condition.

Conclusions

Multiple sources and methods were used to determine (identical) pretest probabilities in Cochrane DTA reviews. Indirectness and severity of downstream consequences may influence the acceptability of the certainty in calculated data with pretest probabilities. Consider: whether to present normalized frequencies, the influence of pretest probabilities on normalized frequencies, and whether to use identical pretest probabilities for meta-analyses in a target condition.

Identifying children exposed to maltreatment: a systematic review update

Background

Child maltreatment affects a significant number of children globally. Strategies have been developed to identify children suspected of having been exposed to maltreatment with the aim of reducing further maltreatment and impairment. This systematic review evaluates the accuracy of strategies for identifying children exposed to maltreatment.

Methods

We conducted a systematic search of seven databases: Medline, Embase, PsycINFO, Cumulative Index to Nursing and Allied Health Literature, Cochrane Libraries, Sociological Abstracts and the Education Resources Information Center. We included studies published from 1961 to July 2, 2019 estimating the accuracy of instruments for identifying potential maltreatment of children, including neglect, physical abuse, emotional abuse, and sexual abuse. We extracted data about accuracy and narratively synthesised the evidence. For five studies—where the population and setting matched known prevalence estimates in an emergency department setting—we calculated false positives and negatives. We assessed risk of bias using QUADAS-2.

Results

We included 32 articles (representing 31 studies) that evaluated various identification strategies, including three screening tools (SPUTOVAMO checklist, Escape instrument, and a 6-item screening questionnaire for child sex trafficking). No studies evaluated the effects of identification strategies on important outcomes for children. All studies were rated as having serious risk of bias (often because of verification bias). The findings suggest that use of the SPUTOVAMO and Escape screening tools at the population level (per 100,000) would result in hundreds of children being missed and thousands of children being over identified.

Conclusions

There is low to very low certainty evidence that the use of screening tools may result in high numbers of children being falsely suspected or missed. These harms may outweigh the potential benefits of using such tools in practice (PROSPERO 2016:CRD42016039659).

GRADE guidelines: 21 part 2. Test accuracy: inconsistency, imprecision, publication bias, and other domains for rating the certainty of evidence and presenting it in evidence profiles and summary of findings tables

OBJECTIVES

This article provides updated GRADE guidance about how authors of systematic reviews and health technology assessments and guideline developers can rate the certainty of evidence (also known as quality of the evidence or confidence in the estimates) of a body of evidence addressing test accuracy (TA) on the domains imprecision, inconsistency, publication bias, and other domains. It also provides guidance for how to present synthesized information in evidence profiles and summary of findings tables.

STUDY DESIGN AND SETTING

We present guidance for rating certainty in TA in clinical and public health and review the presentation of results of a body of evidence regarding tests.

RESULTS

Supplemented by practical examples, we describe how raters of the evidence can apply the GRADE domains inconsistency, imprecision, and publication bias to a body of evidence of TA studies.

CONCLUSION

Using GRADE in Cochrane and other reviews as well as World Health Organization and other guidelines helped refining the GRADE approach for rating the certainty of a body of evidence from TA studies. Although several of the GRADE domains (e.g., imprecision and magnitude of the association) require further methodological research to help operationalize them, judgments need to be made on the basis of what is known so far.

GRADE guidelines: 22. The GRADE approach for tests and strategies-from test accuracy to patient-important outcomes and recommendations

OBJECTIVES

This article describes the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group's framework of moving from test accuracy to patient or population-important outcomes. We focus on the common scenario when studies directly evaluating the effect of diagnostic and other tests or strategies on health outcomes are not available or are not providing the best available evidence.

STUDY DESIGN AND SETTING

Using practical examples, we explored how guideline developers and other decision makers can use information from test accuracy to develop a recommendation by linking evidence that addresses downstream consequences. Guideline panels should develop an analytic framework that summarizes the actions that follow from applying a test and the consequences.

RESULTS

We describe GRADE's current thinking about the overall certainty of the evidence (also known as quality of the evidence or confidence in the estimates) arising from consideration of the often complex pathways that involve multiple tests and management options. Each link in the evidence can-and often does-lower the overall certainty of the evidence required to formulate recommendations and make decisions about tests. The frequency with which an outcome occurs and its importance will influence whether or not a particular step in the linked evidence is critical to decision-making.

CONCLUSIONS

Overall certainty may be expressed by the weakest critical step in the linked evidence. The linked approach to addressing optimal testing will often require the use of decision analytic approaches. We present an example that involves decision modeling in a GRADE Evidence to Decision framework for cervical cancer screening. However, because resources and time of guideline developers may be limited, we describe alternative, pragmatic strategies for developing recommendations addressing test use.

Interactive Visual Displays for Interpreting the Results of Clinical Trials: Formative Evaluation With Case Vignettes

BACKGROUND

At the point of care, evidence from randomized controlled trials (RCTs) is underutilized in helping clinicians meet their information needs.

OBJECTIVE

To design interactive visual displays to help clinicians interpret and compare the results of relevant RCTs for the management of a specific patient, and to conduct a formative evaluation with physicians comparing interactive visual versus narrative displays.

METHODS

We followed a user-centered and iterative design process succeeded by development of information display prototypes as a Web-based application. We then used a within-subjects design with 20 participants (8 attendings and 12 residents) to evaluate the usability and problem-solving impact of the information displays. We compared subjects' perceptions of the interactive visual displays versus narrative abstracts.

RESULTS

The resulting interactive visual displays present RCT results side-by-side according to the Population, Intervention, Comparison, and Outcome (PICO) framework. Study participants completed 19 usability tasks in 3 to 11 seconds with a success rate of 78% to 100%. Participants favored the interactive visual displays over narrative abstracts according to perceived efficiency, effectiveness, effort, user experience and preference (all P values <.001).

CONCLUSIONS

When interpreting and applying RCT findings to case vignettes, physicians preferred interactive graphical and PICO-framework-based information displays that enable direct comparison of the results from multiple RCTs compared to the traditional narrative and study-centered format. Future studies should investigate the use of interactive visual displays to support clinical decision making in care settings and their effect on clinician and patient outcomes.

[GRADE Guidelines: 16. GRADE evidence to decision frameworks for tests in clinical practice and public health]

OBJECTIVE

To describe the Grading of Recommendations Assessment, Development and Evaluation (GRADE) interactive Evidence to Decision (EtD) frameworks for tests and test strategies for clinical, public health or coverage decisions.

STUDY DESIGN AND SETTING

As part of the GRADE Working Group's DECIDE project we conducted workshops, user testing with systematic review authors, guideline developers and other decision makers, and piloted versions of the EtD framework.

RESULTS

EtD frameworks for tests share the structure, explicitness, and transparency of other EtD frameworks. They require specifying the purpose of the test, linked or related management and the key outcomes of concern for different test results and subsequent management. The EtD criteria address test accuracy and assessments of the certainty of the additional evidence necessary for decision-making. When there is no direct evidence of test effects on patient important outcomes, formal or informal modeling is needed to estimate effects. We describe the EtD criteria based on examples developed with GRADEpro (www.gradepro.org), GRADE's software that also provides interactive Summary of Findings Tables.

CONCLUSION

EtD frameworks for developing recommendations and making decisions about tests lay out the sequential steps in reviewing and assessing the different types of evidence that need to be linked.

Defining the clinical pathway in cochrane diagnostic test accuracy reviews

BACKGROUND

The value of a medical test depends on the context in which it might be used. Ideally, questions, results and conclusions of a diagnostic test accuracy (DTA) systematic review should be presented in light of this context. There is increasing acceptance of the value for knowing the impact a test can have on downstream consequences such as costs, implications for further testing and treatment options however there is currently no explicit guidance on how to address this. Authors of a Cochrane diagnostic review have recently been asked to include the clinical pathway in which a test maybe used. We aimed to evaluate how authors were developing their clinical pathways in the light of this.

METHODS

We searched the Cochrane Database of Systematic Reviews for all published DTA reviews. We included only those reviews that included a clinical pathway. We developed a checklist, based on the guidance in the Cochrane Handbook for DTA review authors. To this, we added a number of additional descriptors. We checked if the included pathways fulfilled these descriptors as defined by our checklist.

RESULTS

We found 47 reviews, of which 33 (73 %) contained aspects pertaining to a clinical pathway. The 33 reviews addressed the clinical pathway differently, both in content and format. Of these, 21 provided a textual description and 12 include visual and textual descriptions. There was considerable variation in how comprehensively review authors adhered to our checklist. Eighteen reviews (51 %) linked the index test results to downstream clinical management actions and patient consequences, but only eight went on to differentially report on the consequences for false negative results and nine on the consequences for false positive results.

CONCLUSION

There is substantial variation in the clinical pathway descriptions in Cochrane systematic reviews of test accuracy. Most reviews do not link misclassifications (i.e. false negatives and false positive) to downstream patient consequences. Review authors could benefit from more explicit guidance on how to create such pathways, which in turn can help guide them in their evidence selection and appraisal of the evidence in the context of downstream consequences of testing.