EXPLICIT: a feasibility study of remote expert elicitation in health technology assessment

Software with artificial intelligence-derived algorithms for analysing CT brain scans in people with a suspected acute stroke: a systematic review and cost-effectiveness analysis

Background

Artificial intelligence-derived software technologies have been developed that are intended to facilitate the review of computed tomography brain scans in patients with suspected stroke.

Objectives

To evaluate the clinical and cost-effectiveness of using artificial intelligence-derived software to support review of computed tomography brain scans in acute stroke in the National Health Service setting.

Methods

Twenty-five databases were searched to July 2021. The review process included measures to minimise error and bias. Results were summarised by research question, artificial intelligence-derived software technology and study type. The health economic analysis focused on the addition of artificial intelligence-derived software-assisted review of computed tomography angiography brain scans for guiding mechanical thrombectomy treatment decisions for people with an ischaemic stroke. The de novo model (developed in R Shiny, R Foundation for Statistical Computing, Vienna, Austria) consisted of a decision tree (short-term) and a state transition model (long-term) to calculate the mean expected costs and quality-adjusted life-years for people with ischaemic stroke and suspected large-vessel occlusion comparing artificial intelligence-derived software-assisted review to usual care.

Results

A total of 22 studies (30 publications) were included in the review; 18/22 studies concerned artificial intelligence-derived software for the interpretation of computed tomography angiography to detect large-vessel occlusion. No study evaluated an artificial intelligence-derived software technology used as specified in the inclusion criteria for this assessment. For artificial intelligence-derived software technology alone, sensitivity and specificity estimates for proximal anterior circulation large-vessel occlusion were 95.4% (95% confidence interval 92.7% to 97.1%) and 79.4% (95% confidence interval 75.8% to 82.6%) for Rapid (iSchemaView, Menlo Park, CA, USA) computed tomography angiography, 91.2% (95% confidence interval 77.0% to 97.0%) and 85.0 (95% confidence interval 64.0% to 94.8%) for Viz LVO (Viz.ai, Inc., San Fransisco, VA, USA) large-vessel occlusion, 83.8% (95% confidence interval 77.3% to 88.7%) and 95.7% (95% confidence interval 91.0% to 98.0%) for Brainomix (Brainomix Ltd, Oxford, UK) e-computed tomography angiography and 98.1% (95% confidence interval 94.5% to 99.3%) and 98.2% (95% confidence interval 95.5% to 99.3%) for Avicenna CINA (Avicenna AI, La Ciotat, France) large-vessel occlusion, based on one study each. These studies were not considered appropriate to inform cost-effectiveness modelling but formed the basis by which the accuracy of artificial intelligence plus human reader could be elicited by expert opinion. Probabilistic analyses based on the expert elicitation to inform the sensitivity of the diagnostic pathway indicated that the addition of artificial intelligence to detect large-vessel occlusion is potentially more effective (quality-adjusted life-year gain of 0.003), more costly (increased costs of £8.61) and cost-effective for willingness-to-pay thresholds of £3380 per quality-adjusted life-year and higher.

Limitations and conclusions

The available evidence is not suitable to determine the clinical effectiveness of using artificial intelligence-derived software to support the review of computed tomography brain scans in acute stroke. The economic analyses did not provide evidence to prefer the artificial intelligence-derived software strategy over current clinical practice. However, results indicated that if the addition of artificial intelligence-derived software-assisted review for guiding mechanical thrombectomy treatment decisions increased the sensitivity of the diagnostic pathway (i.e. reduced the proportion of undetected large-vessel occlusions), this may be considered cost-effective.

Future work

Large, preferably multicentre, studies are needed (for all artificial intelligence-derived software technologies) that evaluate these technologies as they would be implemented in clinical practice.

Study registration

This study is registered as PROSPERO CRD42021269609.

Funding

This award was funded by the National Institute for Health and Care Research (NIHR) Evidence Synthesis programme (NIHR award ref: NIHR133836) and is published in full in Health Technology Assessment; Vol. 28, No. 11. See the NIHR Funding and Awards website for further award information.

Direct healthcare costs of non-metastatic castration-resistant prostate cancer in Italy

OBJECTIVES

The management of non-metastatic castration-resistant prostate cancer (nmCRPC) is rapidly evolving; however, little is known about the direct healthcare costs of nmCRPC. We aimed to estimate the cost-of-illness (COI) of nmCRPC from the Italian National Health Service perspective.

METHODS

Structured, individual qualitative interviews were carried out with clinical experts to identify what healthcare resources are consumed in clinical practice. To collect quantitative estimates of healthcare resource consumption, a structured expert elicitation was performed with clinical experts using a modified version of a previously validated interactive Excel-based tool, EXPLICIT (EXPert eLICItation Tool). For each parameter, experts were asked to provide the lowest, highest, and most likely value. Deterministic and probabilistic sensitivity analyses (PSA) were carried out to test the robustness of the results.

RESULTS

Ten clinical experts were interviewed, and six of them participated in the expert elicitation exercise. According to the most likely estimate, the yearly cost per nmCRPC patient is €4,710 (range, €2,243 to €8,243). Diagnostic imaging (i.e., number/type of PET scans performed) had the highest impact on cost. The PSA showed a 50 percent chance for the yearly cost per nmCRPC patient to be within €5,048 using a triangular distribution for parameters, and similar results were found using a beta-PERT distribution.

CONCLUSIONS

This study estimated the direct healthcare costs of nmCRPC in Italy based on a mixed-methods approach. Delaying metastases may be a reasonable goal also from an economic standpoint. These findings can inform decision-making about treatments at the juncture between non-metastatic and metastatic prostate cancer disease.

Crowdsourced Perceptions of Human Behavior to Improve Computational Forecasts of US National Incident Cases of COVID-19: Survey Study

BACKGROUND

Past research has shown that various signals associated with human behavior (eg, social media engagement) can benefit computational forecasts of COVID-19. One behavior that has been shown to reduce the spread of infectious agents is compliance with nonpharmaceutical interventions (NPIs). However, the extent to which the public adheres to NPIs is difficult to measure and consequently difficult to incorporate into computational forecasts of infectious diseases. Soliciting judgments from many individuals (ie, crowdsourcing) can lead to surprisingly accurate estimates of both current and future targets of interest. Therefore, asking a crowd to estimate community-level compliance with NPIs may prove to be an accurate and predictive signal of an infectious disease such as COVID-19.

OBJECTIVE

We aimed to show that crowdsourced perceptions of compliance with NPIs can be a fast and reliable signal that can predict the spread of an infectious agent. We showed this by measuring the correlation between crowdsourced perceptions of NPIs and US incident cases of COVID-19 1-4 weeks ahead, and evaluating whether incorporating crowdsourced perceptions improves the predictive performance of a computational forecast of incident cases.

METHODS

For 36 weeks from September 2020 to April 2021, we asked 2 crowds 21 questions about their perceptions of community adherence to NPIs and public health guidelines, and collected 10,120 responses. Self-reported state residency was compared to estimates from the US census to determine the representativeness of the crowds. Crowdsourced NPI signals were mapped to 21 mean perceived adherence (MEPA) signals and analyzed descriptively to investigate features, such as how MEPA signals changed over time and whether MEPA time series could be clustered into groups based on response patterns. We investigated whether MEPA signals were associated with incident cases of COVID-19 1-4 weeks ahead by (1) estimating correlations between MEPA and incident cases, and (2) including MEPA into computational forecasts.

RESULTS

The crowds were mostly geographically representative of the US population with slight overrepresentation in the Northeast. MEPA signals tended to converge toward moderate levels of compliance throughout the survey period, and an unsupervised analysis revealed signals clustered into 4 groups roughly based on the type of question being asked. Several MEPA signals linearly correlated with incident cases of COVID-19 1-4 weeks ahead at the US national level. Including questions related to social distancing, testing, and limiting large gatherings increased out-of-sample predictive performance for probabilistic forecasts of incident cases of COVID-19 1-3 weeks ahead when compared to a model that was trained on only past incident cases.

CONCLUSIONS

Crowdsourced perceptions of nonpharmaceutical adherence may be an important signal to improve forecasts of the trajectory of an infectious agent and increase public health situational awareness.

Methods for Extrapolating Survival Analyses for the Economic Evaluation of Advanced Therapy Medicinal Products

There are two significant challenges for analysts conducting economic evaluations of advanced therapy medicinal products (ATMPs): (i) estimating long-term treatment effects in the absence of mature clinical data, and (ii) capturing potentially complex hazard functions. This review identifies and critiques a variety of methods that can be used to overcome these challenges. The narrative review is informed by a rapid literature review of methods used for the extrapolation of survival analyses in the economic evaluation of ATMPs. There are several methods that are more suitable than traditional parametric survival modelling approaches for capturing complex hazard functions, including, cure-mixture models and restricted cubic spline models. In the absence of mature clinical data, analysts may augment clinical trial data with data from other sources to aid extrapolation, however, the relative merits of employing methods for including data from different sources is not well understood. Given the high and potentially irrecoverable costs of making incorrect decisions concerning the reimbursement or commissioning of ATMPs, it is important that economic evaluations are correctly specified, and that both parameter and structural uncertainty associated with survival extrapolations are considered. Value of information analyses allow for this uncertainty to be expressed explicitly, and in monetary terms.

Remote, real-time expert elicitation to determine the prior probability distribution for Bayesian sample size determination in international randomised controlled trials: Bronchiolitis in Infants Placebo Versus Epinephrine and Dexamethasone (BIPED) study

Background

Bayesian methods are increasing in popularity in clinical research. The design of Bayesian clinical trials requires a prior distribution, which can be elicited from experts. In diseases with international differences in management, the elicitation exercise should recruit internationally, making a face-to-face elicitation session expensive and more logistically challenging. Thus, we used a remote, real-time elicitation exercise to construct prior distributions. These elicited distributions were then used to determine the sample size of the Bronchiolitis in Infants with Placebo Versus Epinephrine and Dexamethasone (BIPED) study, an international randomised controlled trial in the Pediatric Emergency Research Network (PERN). The BIPED study aims to determine whether the combination of epinephrine and dexamethasone, compared to placebo, is effective in reducing hospital admission for infants presenting with bronchiolitis to the emergency department.

Methods

We developed a Web-based tool to support the elicitation of the probability of hospitalisation for infants with bronchiolitis. Experts participated in online workshops to specify their individual prior distributions, which were aggregated using the equal-weighted linear pooling method. Experts were then invited to provide their comments on the aggregated distribution. The average length criterion determined the BIPED sample size.

Results

Fifteen paediatric emergency medicine clinicians from Canada, the USA, Australia and New Zealand participated in three workshops to provide their elicited prior distributions. The mean elicited probability of admission for infants with bronchiolitis was slightly lower for those receiving epinephrine and dexamethasone compared to supportive care in the aggregate distribution. There were substantial differences in the individual beliefs but limited differences between North America and Australasia. From this aggregate distribution, a sample size of 410 patients per arm results in an average 95% credible interval length of less than 9% and a relative predictive power of 90%.

Conclusion

Remote, real-time expert elicitation is a feasible, useful and practical tool to determine a prior distribution for international randomised controlled trials. Bayesian methods can then determine the trial sample size using these elicited prior distributions. The ease and low cost of remote expert elicitation mean that this approach is suitable for future international randomised controlled trials.

Trial registration

ClinicalTrials.govNCT03567473

Supplementary Information

The online version contains supplementary material available at 10.1186/s13063-022-06240-w.

Uncertainty Quantification with Experts: Present Status and Research Needs

Expert elicitation is deployed when data are absent or uninformative and critical decisions must be made. In designing an expert elicitation, most practitioners seek to achieve best practice while balancing practical constraints. The choices made influence the required time and effort investment, the quality of the elicited data, experts' engagement, the defensibility of results, and the acceptability of resulting decisions. This piece outlines some of the common choices practitioners encounter when designing and conducting an elicitation. We discuss the evidence supporting these decisions and identify research gaps. This will hopefully allow practitioners to better navigate the literature, and will inspire the expert judgment research community to conduct well powered, replicable experiments that properly address the research gaps identified.

The Use of Expert Elicitation among Computational Modeling Studies in Health Research: A Systematic Review

BACKGROUND

Expert elicitation (EE) has been used across disciplines to estimate input parameters for computational modeling research when information is sparse or conflictual.

OBJECTIVES

We conducted a systematic review to compare EE methods used to generate model input parameters in health research.

DATA SOURCES

PubMed and Web of Science.

STUDY ELIGIBILITY

Modeling studies that reported the use of EE as the source for model input probabilities were included if they were published in English before June 2021 and reported health outcomes.

DATA ABSTRACTION AND SYNTHESIS

Studies were classified as "formal" EE methods if they explicitly reported details of their elicitation process. Those that stated use of expert opinion but provided limited information were classified as "indeterminate" methods. In both groups, we abstracted citation details, study design, modeling methodology, a description of elicited parameters, and elicitation methods. Comparisons were made between elicitation methods.

STUDY APPRAISAL

Studies that conducted a formal EE were appraised on the reporting quality of the EE. Quality appraisal was not conducted for studies of indeterminate methods.

RESULTS

The search identified 1520 articles, of which 152 were included. Of the included studies, 40 were classified as formal EE and 112 as indeterminate methods. Most studies were cost-effectiveness analyses (77.6%). Forty-seven indeterminate method studies provided no information on methods for generating estimates. Among formal EEs, the average reporting quality score was 9 out of 16.

LIMITATIONS

Elicitations on nonhealth topics and those reported in the gray literature were not included.

CONCLUSIONS

We found poor reporting of EE methods used in modeling studies, making it difficult to discern meaningful differences in approaches. Improved quality standards for EEs would improve the validity and replicability of computational models.

HIGHLIGHTS

We find extensive use of expert elicitation for the development of model input parameters, but most studies do not provide adequate details of their elicitation methods.Lack of reporting hinders greater discussion of the merits and challenges of using expert elicitation for model input parameter development.There is a need to establish expert elicitation best practices and reporting guidelines.

Exploring the Feasibility of Comprehensive Uncertainty Assessment in Health Economic Modeling: A Case Study

OBJECTIVES

Decision makers adopt health technologies based on health economic models that are subject to uncertainty. In an ideal world, these models parameterize all uncertainties and reflect them in the cost-effectiveness probability and risk associated with the adoption. In practice, uncertainty assessment is often incomplete, potentially leading to suboptimal reimbursement recommendations and risk management. This study examines the feasibility of comprehensive uncertainty assessment in health economic models.

METHODS

A state transition model on peripheral arterial disease treatment was used as a case study. Uncertainties were identified and added to the probabilistic sensitivity analysis if possible. Parameter distributions were obtained by expert elicitation, and structural uncertainties were either parameterized or explored in scenario analyses, which were model averaged.

RESULTS

A truly comprehensive uncertainty assessment, parameterizing all uncertainty, could not be achieved. Expert elicitation informed 8 effectiveness, utility, and cost parameters. Uncertainties were parameterized or explored in scenario analyses and with model averaging. Barriers included time and resource constraints, also of clinical experts, and lacking guidance regarding some aspects of expert elicitation, evidence aggregation, and handling of structural uncertainty. The team's multidisciplinary expertise and existing literature and tools were facilitators.

CONCLUSIONS

While comprehensive uncertainty assessment may not be attainable, improvements in uncertainty assessment in general are no doubt desirable. This requires the development of detailed guidance and hands-on tutorials for methods of uncertainty assessment, in particular aspects of expert elicitation, evidence aggregation, and handling of structural uncertainty. The issue of benefits of uncertainty assessment versus time and resources needed remains unclear.

Guidance for the Harmonisation and Improvement of Economic Evaluations of Personalised Medicine

OBJECTIVE

The objective of this study was to develop guidance contributing to improved consistency and quality in economic evaluations of personalised medicine (PM), given current ambiguity about how to measure the value of PM as well as considerable variation in the methodology and reporting in economic evaluations of PM.

METHODS

A targeted literature review of methodological papers was performed for an overview of modelling challenges in PM. Expert interviews were held to discuss best modelling practice. A systematic literature review of economic evaluations of PM was conducted to gain insight into current modelling practice. The findings were synthesised and used to develop a set of draft recommendations. The draft recommendations were discussed at a stakeholder workshop and subsequently finalised.

RESULTS

Twenty-two methodological papers were identified. Some argued that the challenges in modelling PM can be addressed within existing methodological frameworks, others disagreed. Eighteen experts were interviewed. They believed large uncertainty to be a key concern. Out of 195 economic evaluations of PM identified, 56% addressed none of the identified modelling challenges. A set of 23 recommendations was developed. Eight recommendations focus on the modelling of test-treatment pathways. The use of non-randomised controlled trial data is discouraged but several recommendations are provided in case randomised controlled trial data are unavailable. The parameterisation of structural uncertainty is recommended. Other recommendations consider perspective and discounting; premature survival data; additional value elements; patient and clinician compliance; and managed entry agreements.

CONCLUSIONS

This study provides a comprehensive list of recommendations to modellers of PM and to evaluators and reviewers of PM models.

Validation of 4D Components for Measuring Quality of the Public Health Data Collection Process: Elicitation Study

Background

Identification of the essential components of the quality of the data collection process is the starting point for designing effective data quality management strategies for public health information systems. An inductive analysis of the global literature on the quality of the public health data collection process has led to the formation of a preliminary 4D component framework, that is, data collection management, data collection personnel, data collection system, and data collection environment. It is necessary to empirically validate the framework for its use in future research and practice.

Objective

This study aims to obtain empirical evidence to confirm the components of the framework and, if needed, to further develop this framework.

Methods

Expert elicitation was used to evaluate the preliminary framework in the context of the Chinese National HIV/AIDS Comprehensive Response Information Management System. The research processes included the development of an interview guide and data collection form, data collection, and analysis. A total of 3 public health administrators, 15 public health workers, and 10 health care practitioners participated in the elicitation session. A framework qualitative data analysis approach and a quantitative comparative analysis were followed to elicit themes from the interview transcripts and to map them to the elements of the preliminary 4D framework.

Results

A total of 302 codes were extracted from interview transcripts. After iterative and recursive comparison, classification, and mapping, 46 new indicators emerged; 24.8% (37/149) of the original indicators were deleted because of a lack of evidence support and another 28.2% (42/149) were merged. The validated 4D component framework consists of 116 indicators (82 facilitators and 34 barriers). The first component, data collection management, includes data collection protocols and quality assurance. It was measured by 41 indicators, decreased from the original 49% (73/149) to 35.3% (41/116). The second component, data collection environment, was measured by 37 indicators, increased from the original 13.4% (20/149) to 31.9% (37/116). It comprised leadership, training, funding, organizational policy, high-level management support, and collaboration among parallel organizations. The third component, data collection personnel, includes the perception of data collection, skills and competence, communication, and staffing patterns. There was no change in the proportion for data collection personnel (19.5% vs 19.0%), although the number of its indicators was reduced from 29 to 22. The fourth component, the data collection system, was measured using 16 indicators, with a slight decrease in percentage points from 18.1% (27/149) to 13.8% (16/116). It comprised functions, system integration, technical support, and data collection devices.

Conclusions

This expert elicitation study validated and improved the 4D framework. The framework can be useful in developing a questionnaire survey instrument for measuring the quality of the public health data collection process after validation of psychometric properties and item reduction.

Cost-effectiveness of Digital Breast Tomosynthesis in Population-based Breast Cancer Screening: A Probabilistic Sensitivity Analysis

BackgroundDigital breast tomosynthesis (DBT) is a promising screening test, but its outcomes and cost-effectiveness remain uncertain.PurposeTo determine if biennial DBT is cost-effective in a screening setting, when compared with digital mammography (DM) in the Netherlands, and to quantify the uncertainty.Materials and MethodsIn this study, performed from March 2018 to February 2019, the MIcrosimulation SCreening ANalysis model was used to conduct a probabilistic sensitivity analysis (PSA), consisting of 10 000 model runs with 1 000 000 women simulated per run. The Bayesian Cost-Effectiveness Analysis package and the Sheffield Accelerated Value of Information tool were used to process PSA outcomes. Two simulated cohorts born in 1970 were invited to undergo biennial screening between ages 50 and 74 years-one cohort was assigned to DM screening, and one was assigned to DBT screening. DM input parameters were based on data from the Dutch breast cancer screening program. DBT parameters were based on literature and expert opinion. Willingness-to-pay thresholds of €20 000 ($22 000) and €35 000 ($38 500) per life-year gained (LYG) were considered. Effects and costs were discounted at 3.5% per year.ResultsDBT resulted in a gain of 13 additional life-years per 1000 women invited to screening (7% increase, 13 of 193), followed over lifetime, compared with DM and led to 2% (four of 159) fewer false-positive results. DBT screening led to incremental discounted lifetime effects of 5.09 LYGs (95% confidence interval: -0.80, 9.70) and an increase in lifetime costs of €137 555 ($151 311) per 1000 women (95% confidence interval: €31 093 [$34 202], €263 537 [$289 891]) compared with DM, resulting in a mean incremental cost-effectiveness ratio of €27 023 ($29 725) per LYG. The probability of DBT being more cost-effective was 0.36 at €20 000 and 0.66 at €35 000 per LYG.ConclusionSwitching from digital mammography to biennial digital breast tomosynthesis is not cost-effective at a willingness-to-pay threshold of €20 000 per life-year gained, but digital breast tomosynthesis has a higher probability of being more cost-effective than digital mammography at a threshold of €35 000 per life-year gained.© RSNA, 2020Online supplemental material is available for this article.See also the editorial by Slanetz in this issue.

Cost-effectiveness analysis of newborn pulse oximetry screening to detect critical congenital heart disease in Colombia

BACKGROUND

In many countries, economic assessments of the routine use of pulse oximetry in the detection of Critical Congenital Heart Disease (CCHD) at birth has not yet been carried out. CCHDs necessarily require medical intervention within the first months of life. This assessment is a priority in low and medium resource countries. The purpose of this study was to assess the cost-effectiveness (CE) relation of pulse oximetry in the detection of cases of CCHD in Colombia.

METHODS

A full economic assessment of the cost-effectiveness type was conducted from the perspective of society. A decision tree was constructed to establish a comparison between newborn physical examination plus pulse oximetry, versus physical examination alone, in the diagnosis of CCHDs. The sensitivity and specificity of pulse oximetry were estimated from a systematic review of the literature; to assess resource use, micro-costing analyses and surveys were conducted. The time horizon of the economic evaluation was the first week after birth and until the first year of life. The incremental cost-effectiveness ratio (ICER) was determined and, to control for uncertainty, deterministic and probabilistic sensitivity analysis were made, including the adoption of different scenarios of budgetary impact. All costs are expressed in US dollars from 2017, using the average exchange rate for 2017 [$2,951.15 COP for 1 dollar].

RESULTS

The costs of pulse oximetry screening plus physical examination were $102; $7 higher than physical examination alone. The effectiveness of pulse oximetry plus the physical examination was 0.93; that is, 0.07 more than the physical examination on its own. The ICER was $100 for pulse oximetry screening; that is, if one wishes to increase 1% the probability of a correct CCHD diagnosis, this amount would have to be invested. A willingness to pay of $26.292 USD (direct medical cost) per probability of a correct CCHD diagnosis was assumed.

CONCLUSIONS

At current rates and from the perspective of society, newborn pulse oximetry screening at 24 h in addition to physical examination, and considering a time horizon of 1 week, is a cost-effective strategy in the early diagnosis of CCHDs in Colombia.Trial registration "retrospectively registered".

Use of Expert Judgement Across NICE Guidance-Making Programmes: A Review of Current Processes and Suitability of Existing Tools to Support the Use of Expert Elicitation

OBJECTIVES

This study aimed to review current use of experts within National Institute for Health and Care Excellence (NICE) guidance-making programmes, identify improvements in use of expert judgement, and to assess tools and protocols to support the elicitation of information from experts for use by NICE.

METHODS

The study comprised a review of NICE process guides; semi-structured interviews with individuals representing each NICE guidance-making programme and a comparison of the suitability of currently available tools and protocols for expert elicitation to the requirements of NICE programmes.

RESULTS

Information elicited from experts and the way in which it is used varies across NICE guidance-making programmes. Experts' involvement can be as intensive as being a member of a committee and thus having direct influence on recommendations or limited one-off consultations on specific parameters. We identified 16 tools for expert elicitation that were potentially relevant. None fully met the requirements of NICE, although an existing tool could be potentially adapted. Ongoing research to develop a reference protocol for expert elicitation in healthcare decision making may be of use in future.

CONCLUSIONS

NICE uses expert judgement across all its guidance-making programmes, but its uses vary considerably. There is no currently available tool for expert elicitation suitable for use by NICE. However, adaptation of an existing tool or ongoing research in the area could address this in the future.