Decision making about healthcare-related tests and diagnostic test strategies. Paper 2: a review of methodological and practical challenges

GRADE guidance 37: rating imprecision in a body of evidence on test accuracy

OBJECTIVES

To provide guidance on rating imprecision in a body of evidence assessing the accuracy of a single test. This guide will clarify when Grading of Recommendations Assessment, Development and Evaluation (GRADE) users should consider rating down the certainty of evidence by one or more levels for imprecision in test accuracy.

STUDY DESIGN AND SETTING

A project group within the GRADE working group conducted iterative discussions and presentations at GRADE working group meetings to produce this guidance.

RESULTS

Before rating the certainty of evidence, GRADE users should define the target of their certainty rating. GRADE recommends setting judgment thresholds defining what they consider a very accurate, accurate, inaccurate, and very inaccurate test. These thresholds should be set after considering consequences of testing and effects on people-important outcomes. GRADE's primary criterion for judging imprecision in test accuracy evidence is considering confidence intervals (i.e., CI approach) of absolute test accuracy results (true and false, positive, and negative results in a cohort of people). Based on the CI approach, when a CI appreciably crosses the predefined judgment threshold(s), one should consider rating down certainty of evidence by one or more levels, depending on the number of thresholds crossed. When the CI does not cross judgment threshold(s), GRADE suggests considering the sample size for an adequately powered test accuracy review (optimal or review information size [optimal information size (OIS)/review information size (RIS)]) in rating imprecision. If the combined sample size of the included studies in the review is smaller than the required OIS/RIS, one should consider rating down by one or more levels for imprecision.

CONCLUSION

This paper extends previous GRADE guidance for rating imprecision in single test accuracy systematic reviews and guidelines, with a focus on the circumstances in which one should consider rating down one or more levels for imprecision.

Case studies to explore the optimal use of randomized and nonrandomized studies in evidence syntheses that use GRADE

OBJECTIVES

Randomized controlled trials (RCTs) are the preferred source of evidence for the relative effect of healthcare interventions summarized in knowledge syntheses. Nonrandomized studies of interventions (NRSI) may provide replacement, sequential, or complementary evidence to RCTs. The Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach can provide different options for properly using RCTs and NRSI integrated in health syntheses. In this article, we discuss different implications on the certainty of evidence when authors consider the use of NRSI and RCTs in systematic reviews using GRADE. Although this is a GRADE-related article, it is not an official GRADE guidance or concept article.

STUDY DESIGN AND SETTING

We present case studies used during GRADE working group meetings for discussion of the effects of using NRSI and RCTs on GRADE domains and on the certainty of evidence. Several concepts were discussed through iterative feedback with experts in GRADE methods and Cochrane authors. We compared suggested solutions for possible scenarios that can be met in evidence syntheses informing decisions and future guidance.

RESULTS

Different scenarios for the use of RCTs and NRSI in evidence syntheses are presented, focusing on how different GRADE ratings between RCTs and NRSI affect the overall assessment of the evidence and possible health recommendations.

CONCLUSIONS

Considering differences and similarities grounded in the GRADE approach between NRSI and RCTs may help complement one another and maximize the value of knowledge syntheses and health recommendations.

The Value of a Standardized Knee Functional Assessment in Predicting the Outcomes of Total Knee Arthroplasty

Questionnaires and physical tests are tools to determine the ability of an individual to perform tasks of the daily living. In our institution, a standardized knee performance evaluation including patient-reported outcome measures (PROMs) and physical performance tests has been applied to all patients undergoing total knee arthroplasty (TKA). Our goal was to identify which preoperative tools influence the outcomes of a TKA and if physical performance tests can be of value if used along with PROMs in predicting functional outcomes. Classification and regression tree was used to analyze which preoperative factors influence function after TKA. Western Ontario and McMaster Universities Arthritis Index (WOMAC) function (WOMAC-F), 6-minute walk test (6MWT), and timed up and go (TUG) test at the 12th postoperative month were the dependent variables. Age, body mass index, preoperative WOMAC function and pain score, muscle strength, 6MWT, and TUG test score were used as preoperative predictors of dependent variables. TUG ≤19.3 seconds and age <62 years were preoperative predictors of better scores in the WOMAC-F (5.5). Strength of extensor muscles of the nonoperated knee ≥99.43 N·m/kg and 6MWT> 328 m were preoperative predictors of a better postoperative 6MWT (499 m). TUG <12.3 seconds and 6MWT ≥421 m were preoperative predictors of better postoperative TUG (7.3 seconds). Preoperative performance in physical tests had an influence on postoperative outcome scores than PROMs after TKA. Less age, good muscular strength, greater capacity of walking, and smaller TUG times were associated with better outcomes.

The predictive potential of different molecular markers linked to amikacin susceptibility phenotypes in Pseudomonas aeruginosa

Informed antibiotic prescription offers a practical solution to antibiotic resistance problem. With the increasing affordability of different sequencing technologies, molecular-based resistance prediction would direct proper antibiotic selection and preserve available agents. Amikacin is a broad-spectrum aminoglycoside exhibiting higher clinical efficacy and less resistance rates in Ps. aeruginosa due to its structural nature and its ability to achieve higher serum concentrations at lower therapeutic doses. This study examines the predictive potential of molecular markers underlying amikacin susceptibility phenotypes in order to provide improved diagnostic panels. Using a predictive model, genes and variants underlying amikacin resistance have been statistically and functionally explored in a large comprehensive and diverse set of Ps. aeruginosa completely sequenced genomes. Different genes and variants have been examined for their predictive potential and functional correlation to amikacin susceptibility phenotypes. Three predictive sets of molecular markers have been identified and can be used in a complementary manner, offering promising molecular diagnostics. armR, nalC, nalD, mexR, mexZ, ampR, rmtD, nalDSer32Asn, fusA1Y552C, fusA1D588G, arnAA170T, and arnDG206C have been identified as the best amikacin resistance predictors in Ps. aeruginosa while faoAT385A, nuoGA890T, nuoGA574T, lptAT55A, lptAR62S, pstBR87C, gidBE126G, gidBQ28K, amgSE108Q, and rplYQ41L have been identified as the best amikacin susceptibility predictors. Combining different measures of predictive performance together with further functional analysis can help design new and more informative molecular diagnostic panels. This would greatly inform and direct point of care diagnosis and prescription, which would consequently preserve amikacin functionality and usefulness.

Development and evaluation of safety and effectiveness of novel cancer screening tests for routine clinical use with applications to multicancer detection technologies

Multicancer screening is a promising approach to improving the detection of preclinical disease, but current technologies have limited ability to identify precursor or early stage lesions, and approaches for developing the evidentiary chain are unclear. Frameworks to enable development and evaluation from discovery through evidence of clinical effectiveness are discussed.

2019 sickle cell disease guidelines by the American Society of Hematology: methodology, challenges, and innovations

The American Society of Hematology (ASH) convened 5 guideline panels to develop clinical practice recommendations addressing 5 management areas of highest importance to individuals living with sickle cell disease: pain, cerebrovascular complications, pulmonary and kidney complications, transfusion, and hematopoietic stem cell transplant. Panels were multidisciplinary and consisted of patient representatives, content experts, and methodologists. The Mayo Clinic Evidence-Based Practice Center conducted systematic reviews based on a priori selected questions. In this exposition, we describe the process used by ASH, including the GRADE approach (Grades of Recommendations, Assessment, Development and Evaluation) for rating certainty of the evidence and the GRADE Evidence to Decision Framework. We also describe several unique challenges faced by the guideline panels and the specific innovations and solutions used to address them, including a curriculum to train patients to engage in guideline development, dealing with the opioid crisis, and working with indirect and noncomparative evidence.

Current methods for development of rapid reviews about diagnostic tests: an international survey

BACKGROUND

Rapid reviews (RRs) have emerged as an efficient alternative to time-consuming systematic reviews-they can help meet the demand for accelerated evidence synthesis to inform decision-making in healthcare. The synthesis of diagnostic evidence has important methodological challenges. Here, we performed an international survey to identify the current practice of producing RRs for diagnostic tests.

METHODS

We developed and administered an online survey inviting institutions that perform RRs of diagnostic tests from all over the world.

RESULTS

All participants (N = 25) reported the implementation of one or more methods to define the scope of the RR; however, only one strategy (defining a structured question) was used by ≥90% of participants. All participants used at least one methodological shortcut including the use of a previous review as a starting point (92%) and the use of limits on the search (96%). Parallelization and automation of review tasks were not extensively used (48 and 20%, respectively).

CONCLUSION

Our survey indicates a greater use of shortcuts and limits for conducting diagnostic test RRs versus the results of a recent scoping review analyzing published RRs. Several shortcuts are used without knowing how their implementation affects the results of the evidence synthesis in the setting of diagnostic test reviews. Thus, a structured evaluation of the challenges and implications of the adoption of these RR methods is warranted.

Driving the Musculoskeletal Diagnosis Train on the High-Value Track

Clinicians who provide high-value musculoskeletal care offer evidence-based management that can improve pain, function, and quality of life. However, the current approach to improving outcomes emphasizes treatment strategies at the expense of accurate diagnosis. Guidelines rarely provide quality information on differential diagnosis and prognosis. The disconnect is worrying, because accurate diagnosis can lead to more appropriate care and improved patient outcomes. In this Viewpoint, we reflect on how the clinician can refine musculoskeletal diagnoses to provide high-value care. We (1) argue that the link between musculoskeletal diagnosis and patient outcomes must be strengthened, (2) introduce a diagnostic framework to help clinicians go beyond "special tests," and (3) present new methods for researchers to move forward from diagnostic accuracy studies. J Orthop Sports Phys Ther 2020;50(3):118-120. doi:10.2519/jospt.2020.0603.

GRADE guidelines: 21 part 1. Study design, risk of bias, and indirectness in rating the certainty across a body of evidence for test accuracy

OBJECTIVES

This article provides updated GRADE guidance about how authors of systematic reviews and health technology assessments and guideline developers can assess the results and the certainty of evidence (also known as quality of the evidence or confidence in the estimates) of a body of evidence addressing test accuracy (TA).

STUDY DESIGN AND SETTING

We present an overview of the GRADE approach and guidance for rating certainty in TA in clinical and public health and review the presentation of results of a body of evidence regarding tests. Part 1 of the two parts in this 21st guidance article about how to apply GRADE focuses on understanding study design issues in test accuracy, provide an overview of the domains, and describe risk of bias and indirectness specifically.

RESULTS

Supplemented by practical examples, we describe how raters of the evidence using GRADE can evaluate study designs focusing on tests and how they apply the GRADE domains risk of bias and indirectness to a body of evidence of TA studies.

CONCLUSION

Rating the certainty of a body of evidence using GRADE in Cochrane and other reviews and World Health Organization and other guidelines dealing with in TA studies helped refining our approach. The resulting guidance will help applying GRADE successfully for questions and recommendations focusing on tests.

A systematic review evaluating screening instruments for gambling disorder finds lack of adequate evidence

OBJECTIVES

To date, no research has systematically evaluated screening instruments for gambling disorder to assess their accuracy and the quality of the research. This systematic review evaluated screening instruments for gambling disorder to inform decision makers about choices for population-level screening.

STUDY DESIGN AND SETTING

On May 22, 2017 and January 4, 2019, we searched PubMed, PsycInfo, EMBASE, and Cochrane for studies that evaluated screening instruments for gambling disorder. Studies were included if (1) the screening instrument was in English, (2) the screening instrument was compared to a reference standard semistructured interview based on Diagnostic and Statistical Manual of Mental Disorders or International Classification of Diseases diagnoses of gambling disorder, and (3) data were reported on psychometric properties of the instrument.

RESULTS

We identified 31 different screening instruments from 60 studies. Only three instruments from three separate studies were eligible for inclusion in the systematic review.

CONCLUSION

Few screening instruments for gambling disorder have been validated with sufficient methodological quality to be recommended for use across a large health system.

American Society of Hematology 2019 guidelines for sickle cell disease: cardiopulmonary and kidney disease

BACKGROUND

Prevention and management of end-organ disease represent major challenges facing providers of children and adults with sickle cell disease (SCD). Uncertainty and variability in the screening, diagnosis, and management of cardiopulmonary and renal complications in SCD lead to varying outcomes for affected individuals.

OBJECTIVE

These evidence-based guidelines of the American Society of Hematology (ASH) are intended to support patients, clinicians, and other health care professionals in their decisions about screening, diagnosis, and management of cardiopulmonary and renal complications of SCD.

METHODS

ASH formed a multidisciplinary guideline panel that included 2 patient representatives and was balanced to minimize potential bias from conflicts of interest. The Mayo Evidence-Based Practice Research Program supported the guideline development process, including performing systematic evidence reviews up to September 2017. The panel prioritized clinical questions and outcomes according to their importance for clinicians and patients. The panel used the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach, including GRADE evidence-to-decision frameworks, to assess evidence and make recommendations, which were subject to public comment.

RESULTS

The panel agreed on 10 recommendations for screening, diagnosis, and management of cardiopulmonary and renal complications of SCD. Recommendations related to anticoagulation duration for adults with SCD and venous thromboembolism were also developed.

CONCLUSIONS

Most recommendations were conditional due to a paucity of direct, high-quality evidence for outcomes of interest. Future research was identified, including the need for prospective studies to better understand the natural history of cardiopulmonary and renal disease, their relationship to patient-important outcomes, and optimal management.

An algorithm for the classification of study designs to assess diagnostic, prognostic and predictive test accuracy in systematic reviews

Results of medical tests are the main source to inform clinical decision making. The main information to assess the usefulness of medical tests for correct discrimination of patients are accuracy measures. For the estimation of test accuracy measures, many different study designs can be used. The study design is related to the clinical question to be answered (diagnosis, prognosis, prediction), determines the accuracy measures that can be calculated and it might have an influence on risk of bias. Therefore, a clear and consistent distinction of the different study designs in systematic reviews on test accuracy studies is very important. In this paper, we propose an algorithm for the classification of study designs of test accuracy, that compare the results of an index test (the test to be evaluated) with the results of a reference test (the test whose results are considered as correct/the gold standard) studies in systematic reviews.

Evaluating diagnostic strategies for early detection of cancer: the CanTest framework

BACKGROUND

Novel diagnostic triage and testing strategies to support early detection of cancer could improve clinical outcomes. Most apparently promising diagnostic tests ultimately fail because of inadequate performance in real-world, low prevalence populations such as primary care or general community populations. They should therefore be systematically evaluated before implementation to determine whether they lead to earlier detection, are cost-effective, and improve patient safety and quality of care, while minimising over-investigation and over-diagnosis.

METHODS

We performed a systematic scoping review of frameworks for the evaluation of tests and diagnostic approaches.

RESULTS

We identified 16 frameworks: none addressed the entire continuum from test development to impact on diagnosis and patient outcomes in the intended population, nor the way in which tests may be used for triage purposes as part of a wider diagnostic strategy. Informed by these findings, we developed a new framework, the 'CanTest Framework', which proposes five iterative research phases forming a clear translational pathway from new test development to health system implementation and evaluation.

CONCLUSION

This framework is suitable for testing in low prevalence populations, where tests are often applied for triage testing and incorporated into a wider diagnostic strategy. It has relevance for a wide range of stakeholders including patients, policymakers, purchasers, healthcare providers and industry.

Prediction Models for Suicide Attempts and Deaths: A Systematic Review and Simulation

IMPORTANCE

Suicide prediction models have the potential to improve the identification of patients at heightened suicide risk by using predictive algorithms on large-scale data sources. Suicide prediction models are being developed for use across enterprise-level health care systems including the US Department of Defense, US Department of Veterans Affairs, and Kaiser Permanente.

OBJECTIVES

To evaluate the diagnostic accuracy of suicide prediction models in predicting suicide and suicide attempts and to simulate the effects of implementing suicide prediction models using population-level estimates of suicide rates.

EVIDENCE REVIEW

A systematic literature search was conducted in MEDLINE, PsycINFO, Embase, and the Cochrane Library to identify research evaluating the predictive accuracy of suicide prediction models in identifying patients at high risk for a suicide attempt or death by suicide. Each database was searched from inception to August 21, 2018. The search strategy included search terms for suicidal behavior, risk prediction, and predictive modeling. Reference lists of included studies were also screened. Two reviewers independently screened and evaluated eligible studies.

FINDINGS

From a total of 7306 abstracts reviewed, 17 cohort studies met the inclusion criteria, representing 64 unique prediction models across 5 countries with more than 14 million participants. The research quality of the included studies was generally high. Global classification accuracy was good (≥0.80 in most models), while the predictive validity associated with a positive result for suicide mortality was extremely low (≤0.01 in most models). Simulations of the results suggest very low positive predictive values across a variety of population assessment characteristics.

CONCLUSIONS AND RELEVANCE

To date, suicide prediction models produce accurate overall classification models, but their accuracy of predicting a future event is near 0. Several critical concerns remain unaddressed, precluding their readiness for clinical applications across health systems.

Diagnostic Biomarkers: Are We Moving from Discovery to Clinical Application?

BACKGROUND

Despite considerable research investment, moving from biomarker discovery to clinical application has presented unique challenges. We aimed to evaluate progress toward clinical application of a sample of molecular- and “omics”-based diagnostic tests over a 10-year period.

METHODS

We used Scopus to locate studies, published before the December 31, 2016, citing 107 original-research articles published in 2006 that assessed the diagnostic value of a molecular- or “omics”-based test. We identified diagnostic studies of the same test and disease and determined whether the article represented progress in the validation of the molecular test. We classified the types of progress: (a) clinical validation (measuring diagnostic accuracy in a series of patients similar to the population in which the test will be used in practice), (b) technical improvement, (c) extended diagnostic application (modification of the diagnostic question attended initially by the test), (d) economic evaluation, or (e) clinical use or implementation.

RESULTS

In the 10-year period analyzed, 4257 articles cited the 107 diagnostic studies; 118 (2.8%) were diagnostic studies of the same test, and of these papers, 25 (21.2%) did not constitute progress toward validation of the test for use in clinical practice (potential research waste). Of the 107 molecular- or “omics”-based tests described in 2006, only 28 (26.2%) appeared to have made progress toward clinical application. Only 4 (9.1%) of 44 proteomics-based tests had made progress toward clinical application.

CONCLUSIONS

Articles evaluating molecular- or “omics”-based diagnostic tests are numerous in biomedical journals. Few tests have made progress toward clinical application in the 10 years following their discovery.

Using systematic reviews in guideline development: the GRADE approach

Systematic reviews are essential to produce trustworthy guidelines. To assess the certainty of a body of evidence included in a systematic review the Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group has developed an approach that is currently used by over 100 organisations, including the World Health Organization and the Cochrane Collaboration. GRADE provides operational definitions and instructions to rate the certainty of the evidence for each outcome in a review as high, moderate, low, or very low for the effects of interventions, prognostic estimates, values and preferences, test accuracy and resource utilization. The assessment includes assessing risk of bias, imprecision, inconsistency, indirectness, and publication bias, the magnitude of effects, dose-response relations and the impact of residual confounding and bias. Summary statistical information and assessments of certainty are presented in GRADE evidence summary tables, which can be produced using GRADE's official GRADEpro software tool (www.gradepro.org/). The evidence summary tables feed into the GRADE Evidence to Decision frameworks which guideline panels can use to produce recommendations.

Serum IL-6 in combination with synovial IL-6/CRP shows excellent diagnostic power to detect hip and knee prosthetic joint infection

BACKGROUND

The diagnosis of prosthetic joint infection (PJI) is still a challenge in some patients after total joint replacement. Interleukin-6 (IL-6) strongly participates in the arrangement of the host-bacteria response. Therefore, increased levels of IL-6 should accompany every PJI.

PURPOSE

The aim of the study was to show diagnostic characteristics of serum IL-6 for the diagnosis of prosthetic joint infection (PJI). We also compared the diagnostic values of serum IL-6 with synovial IL-6 (sIL-6) and synovial C-reactive protein (sCRP).

STUDY DESIGN

We performed a prospective study of 240 patients in whom serum IL-6 was determined before total hip (n = 124) or knee (n = 116) reoperations. The PJI diagnosis was based on the MSIS (Musculoskeletal Infection Society) criteria (2011). Receiver operating characteristic plots were constructed for IL-6, sIL-6, and sCRP.

RESULTS

PJI was diagnosed in 93 patients, and aseptic revision was diagnosed in 147 patients. The AUC (area under curve) for IL-6 was 0.938 (95% CI; 0.904-0.971). The optimal IL-6 cut-off value for PJI was 12.55 ng/L. Positive and negative likelihood ratios for IL-6 were 8.24 (95% CI; 4.79-14.17) and 0.15 (95% CI; 0.09-0.26), respectively. The optimal sIL-6 and sCRP cut-off values were 20,988 ng/L and 8.80 mg/L, respectively. Positive and negative likelihood ratios for sIL-6 were 40.000 (95% CI; 5.7-280.5) and 0.170 (95% CI; 0.07-0.417), respectively. Negative likelihood ratio for sCRP was 0.083 (95% CI; 0.022-0.314).

CONCLUSIONS

The present study identified the cut-off values for serum/synovial IL-6 and synovial CRP for diagnostics of PJI at the site of THA and TKA and separately for each site. The diagnostic odds ratio for serum/synovial IL-6 and synovial CRP is very good. Simultaneous positivity of serum IL-6 either with synovial IL-6 or synovial CRP almost excludes false negative detection of PJI at the site of interest.

[GRADE Guidelines: 16. GRADE evidence to decision frameworks for tests in clinical practice and public health]

OBJECTIVE

To describe the Grading of Recommendations Assessment, Development and Evaluation (GRADE) interactive Evidence to Decision (EtD) frameworks for tests and test strategies for clinical, public health or coverage decisions.

STUDY DESIGN AND SETTING

As part of the GRADE Working Group's DECIDE project we conducted workshops, user testing with systematic review authors, guideline developers and other decision makers, and piloted versions of the EtD framework.

RESULTS

EtD frameworks for tests share the structure, explicitness, and transparency of other EtD frameworks. They require specifying the purpose of the test, linked or related management and the key outcomes of concern for different test results and subsequent management. The EtD criteria address test accuracy and assessments of the certainty of the additional evidence necessary for decision-making. When there is no direct evidence of test effects on patient important outcomes, formal or informal modeling is needed to estimate effects. We describe the EtD criteria based on examples developed with GRADEpro (www.gradepro.org), GRADE's software that also provides interactive Summary of Findings Tables.

CONCLUSION

EtD frameworks for developing recommendations and making decisions about tests lay out the sequential steps in reviewing and assessing the different types of evidence that need to be linked.

Decision making about healthcare-related tests and diagnostic test strategies. Paper 5: a qualitative study with experts suggests that test accuracy data alone is rarely sufficient for decision making

OBJECTIVES

The objective of the study was to identify the critical factors that determine recommendations and other decisions about healthcare-related tests and diagnostic strategies (HCTDS).

METHODS

We used a qualitative descriptive approach and conducted semi-structured in-depth interviews with 24 international experts (informants) in evidence and decisions about HCTDS.

RESULTS

Although test accuracy (TA) was the factor most commonly considered by organizations when developing recommendations about HCTDS, informants agreed that TA is necessary but rarely, if ever, sufficient and may be misleading when solely considered. The informants identified factors that are important for developing recommendations about HCTDS. Informants largely agreed that laying out the potential care pathways based on the test result is an essential early step but is rarely done in developing recommendations about HCTDS. Most informants also agreed that decision analysis could be useful for organizing the clinical, cost, and preference data relevant to the use of tests in the absence of direct evidence. However, they noted that using models is limited by the lack of resources and expertise required.

CONCLUSION

Developing guidelines about HCTDS requires consideration of factors beyond TA, but implementing this may be challenging. Further development and testing of "frameworks" that can guide this process is a priority for decision makers.

Decision making about healthcare-related tests and diagnostic test strategies. Paper 3: a systematic review shows limitations in most tools designed to assess quality and develop recommendations

OBJECTIVES

The objective of this study was to identify and describe critical appraisal tools designed for assessing the quality of evidence (QoE) and/or strength of recommendations (SoRs) related to health care-related tests and diagnostic strategies (HCTDSs).

STUDY DESIGN AND SETTING

We conducted a systematic review to identify tools applied in guidelines, methodological articles, and systematic reviews to assess HCTDS.

RESULTS

We screened 5,534 titles and abstracts, 1,004 full-text articles, and abstracted data from 330 references. We identified 29 tools and 14 modifications of existing tools for assessing QoE and SoR. Twenty-three out of 29 tools acknowledge the importance of assessing the QoE and SoR separately, but in 8, the SoR is based solely on QoE. When making decisions about the use of tests, patient values and preferences and impact on resource utilization were considered in 6 and 8 tools, respectively. There is also confusion about the terminology that describes the various factors that influence the QoE and SoR.

CONCLUSION

Although at least one approach includes all relevant criteria for assessing QoE and determining SoR, more detailed guidance about how to operationalize these assessments and make related judgments will be beneficial. There is a need for a better description of the framework for using evidence to make decisions and develop recommendations about HCTDS.