Conducting online expert panels: a feasibility and experimental replicability study

Assessing the feasibility and likelihood of policy options to lower specialty drug costs

Specialty drugs are high-cost medications often used to treat complex chronic conditions. Even with insurance coverage, patients may face very high out-of-pocket costs, which in turn may restrict access. While the Inflation Reduction Act of 2022 included policies designed to reduce specialty drug costs, relatively few policies have been enacted during the past decade. In 2022-2023, we conducted a scoping literature review to identify a range of policy options and selected a set of 9 that have been regularly discussed or recently considered to present to an expert stakeholder panel to seek consensus on (1) the feasibility of implementing each policy and (2) its likely impact on drug costs. Experts rated only 1 policy highly on both feasibility and impact: grouping originator biologics and biosimilars under the same Medicare Part B reimbursement code. They rated 3 policies focused on setting payment limits as likely to have positive (downward) impact on costs but of uncertain feasibility. They considered 4 policies as uncertain on both criteria. Experts rated capping monthly out-of-pocket costs as feasible but unlikely to reduce specialty drug costs. Based on these results, we offer 4 recommendations to policymakers considering ways to reduce specialty drug costs.

Quality Indicators for Transition from Pediatric to Adult Care for Youth With Chronic Conditions: Proposal for an Online Modified Delphi Study

BACKGROUND

The transition from pediatric to adult care poses a significant health system-level challenge impeding the delivery of quality health services for youth with chronic health conditions. In Canada and globally, the transition to adult care is regarded as a top priority in adolescent health in need of readily applicable, adaptable, and relevant national metrics to evaluate and benchmark transition success across disease populations and clinical care settings. Unfortunately, existing literature fails to account for the lack of engagement from youth and caregivers in developing indicators, and its applicability across chronic conditions, primary care involvement, and health equity considerations.

OBJECTIVE

Our proposed study aims to establish a consensus-driven set of quality indicators for the transition to adult care that are universally applicable across physical, developmental, and mental health conditions, clinical care settings, and health jurisdictions.

METHODS

Using an integrated knowledge translation (iKT) approach, a panel comprising youth, caregivers, interdisciplinary health care providers, and health system leaders will be established to collaborate with our research team to ensure that the study methodology, materials, and knowledge dissemination are suitable and reflect the perspectives of youth and their families. We will then conduct an iterative 3-round Online Modified Delphi (OMD) study (n=160) to (1) compare and contrast the perspectives of youth, caregivers, health care providers, and health system leaders on quality indicators for transition; and (2) prioritize a key set of quality indicators for transition applicable across disease populations that are the most important, useful, and feasible in the Canadian context. Using the RAND/UCLA Appropriateness Method (RAM) multistage analytic approach, data from each panel and stakeholder group will be examined separately and compared to establish a key set of indicators endorsed by both panels.

RESULTS

The study is funded by the Canadian Institutes of Health Research and Physicians Services Incorporated.

CONCLUSIONS

This study will produce quality indicators to evaluate and inform action equitably to improve transition from pediatric to adult care for youth and their families in Canada.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/60860.

Using the Delphi Process to Prioritize an Agenda for Care Transition Research for Patients With Substance Use Disorders

Medical hospitalizations are increasingly recognized as important opportunities to engage individuals with substance use disorders (SUD) and offer treatment. While a growing number of hospitals have instituted interventions to support the provision of SUD care during medical admissions, post-hospitalization transitions of care remain a challenge for patients and clinicians and an understudied area of SUD care. Evidence is lacking on the most effective and feasible models of care to improve post-hospitalization care transitions for people with SUD. In the absence of strong empirical evidence to guide practice and policy, consensus-based research methods such as the Delphi process can play an important role in efficiently prioritizing existing models of care for future study and implementation. We conducted a Delphi study that convened a group of 25 national interdisciplinary experts with direct clinical experience facilitating post-hospitalization care transitions for people with SUD. Our panelists rated 10 existing care transition models according to anticipated effectiveness and facility of implementation based on the GRADE Evidence to Decision framework. Qualitative data on each care model were also gathered through comments and an online moderated discussion board. Our results help establish a hierarchy of SUD care transition models to inform future study and program development.

Hospital Quality Indicators for Opioid-Exposed Infants: Results From an Expert Consensus Panel

OBJECTIVE

To elicit expert consensus on quality indicators for the hospital-based care of opioid-exposed infants.

METHODS

We used the ExpertLens online platform to conduct a 3-round modified Delphi panel. Expert panelists included health care providers, parents in recovery, quality experts, and public health experts. We identified 49 candidate quality indicators from a literature review and environmental scan. A total of 32 experts rated the importance and feasibility of the indicators using a 9-point Likert scale (Round 1), reviewed and discussed the initial ratings (round 2), and revised their original ratings (Round 3). Numeric scores corresponded with descriptive ratings of "low" (1-3), "uncertain" (4-6), or "high" (7-9). We measured consensus using the RAND/UCLA Appropriateness Method.

RESULTS

Candidate quality indicators assessed structures, processes, and outcomes in multiple domains of clinical care. After the final round, 36 indicators were rated "high" on importance and feasibility. Experts had strong consensus on the importance of quality indicators to assess universal screening of pregnant people for substance use disorder, hospital staff training, standardized assessment for neonatal opioid withdrawal syndrome, nonpharmacologic interventions, and transitions of care. For indicators focused on processes and outcomes, experts saw feasibility as dependent on the information routinely documented in electronic medical records or billing records. To present a more complete picture of hospital quality, experts suggested development of composite measures that summarize quality across multiple indicators.

CONCLUSIONS

A panel of experts reached consensus on a range of quality indicators for hospital-based care of opioid-exposed infants, with potential for use in national benchmarking, intervention studies, or hospital performance measurement.

ROMPER: The RAND/USC OPTIC Method for Policy Expert Ratings

We developed an expert panel approach for identifying expert views on the effectiveness and implementability of population-level policy interventions. ROMPER-the RAND/USC OPTIC Method for Policy Expert Ratings-involves an online, three-round, modified-Delphi process:•Experts rate and comment on policies according to domains of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Evidence-to-Decision framework.•To identify consensus on policy effectiveness and implementability, expert ratings are analyzed using the Inter-Percentile Range Adjusted for Symmetry (IPRAS) technique from the RAND/UCLA Appropriateness Method and visualized using a forest plot. To explain consensus, expert comments are analyzed using reflexive thematic analysis and reported following the Standards for Reporting Qualitative Research.•To provide actionable information for decisionmakers, each policy is summarized in a "Policy Profile" adapted from GRADEPro Evidence-to-Decision tables.We validated ROMPER in two studies that successfully recruited the targeted sample size, retained experts through all three rounds, and examined consensus on which policies are (not) effective and implementable. ROMPER protocols, materials, data, and code are openly available on the Open Science Framework with Creative Commons licensing for replication and reuse. ROMPER provides a validated, replicable, open access approach for eliciting expert views on both policy effectiveness and implementability-and for summarizing (lack of) consensus specifically for policymakers.

Expert Views on State Policies to Improve Engagement and Retention in Treatment for Opioid Use Disorder: A Qualitative Analysis of an Online Modified Delphi Process

OBJECTIVES

The aim of this study was to examine expert views on the effectiveness and implementability of state policies to improve engagement and retention in treatment for opioid use disorder (OUD).

METHODS

We conducted a 3-round modified Delphi process using the online ExpertLens platform. Participants included 66 experts on OUD treatment policies. Experts commented on 14 hypothetical state policies targeting treatment engagement and quality of care. Using the GRADE Evidence-to-Decision framework, we conducted reflexive thematic analysis to develop patterns of meaning from the dataset.

RESULTS

Only policies for providing continued access to evidence-based treatment for highly at-risk populations, settings, and periods were seen as effective in meaningfully reducing population-level opioid-related overdose mortality. Experts commented that, although the general public increasingly supports policies expanding medications for OUD and evidence-based care, ongoing stigma about OUD encourages public acceptance of punitive and paternalistic policies. Experts viewed all policies as at least moderately feasible given the current infrastructure and resources, with affordability reliant on long-term cost savings from reduced opioid-related harms. Equitability depended on whether experts perceived a policy as inherently equitable in its design as well as concerns about the potential for inequitable implementation due to structural oppression and interpersonal biases in criminal-legal, healthcare, and other systems.

CONCLUSIONS

Experts believe that supportive (rather than punitive) policies improve engagement and retention in OUD treatment. States could prioritize implementing supportive policies that are patient-centered and take a harm-reduction approach to enhance medications for OUD access and utilization. States could consider deimplementing punitive policies that are coercive, take an abstinence-only approach, and use punitive and restrictive measures.

Categorization and recommendations for outcome measures for lower limb absence by an expert panel

BACKGROUND

Understanding the psychometric strengths and limitations of outcome measures for use with people with lower limb absence (LLA) is important for selecting measures suited to evaluating patient outcomes, answering clinical and research questions, and informing health care policy. The aim of this project was to review the current psychometric evidence on outcome measures in people with LLA to determine which measures should be included in a stakeholder consensus process.

METHODS

An expert panel was assembled, and a 3-stage review process was used to categorize outcome measures identified in a systematic literature review into 3 distinct categories (recommended for measures with better than adequate psychometric properties; recommended with qualification; and unable to recommend). Panelists were asked to individually categorize measures based on results of a systematic review of identified measures' psychometric properties. Each measure's final categorization was based on ≥70% agreement by all panelists.

RESULTS

No outcome measure attained the ≥70% consensus threshold needed to achieve a rating of "recommend." Hence, panelists suggested combining "recommend" and "recommend with qualifications" into a single category of "recommend with qualifications." Using this approach, consensus was reached for 59 of 60 measures. Consensus could not be reached on 1 outcome measure (socket comfort score). Thirty-six outcome measures were categorized as "unable to recommend" based on available evidence; however, 23 (12 patient-reported measures and 11 performance-based measures) demonstrated adequate psychometric properties in LLA samples and were thus rated as "recommend with qualification" by the expert panel. The panel of experts were able to recommend 23 measures for inclusion in the subsequent stakeholder review. A key strength of this process was bringing together international researchers with extensive experience in developing and/or using LLA outcome measures who could assist in identifying psychometrically sound measures to include in a subsequent stakeholder consensus process.

CONCLUSION

The above categorizations represent the current state of psychometric evidence on outcome measures for people with LLA and hence may change over time as additional research becomes available. The results will be used to achieve wider consensus from clinicians, health policymakers, health clinic managers, researchers, and end users (i.e., individuals with LLA) on outcome measures for the International Society of Prosthetics and Orthotics lower limb Consensus Outcome Measures for Prosthetic and Amputation ServiceS.

Study protocol for development of an options assessment toolkit (OAT) for national malaria programs in Asia Pacific to determine best combinations of vivax radical cure for their given contexts

INTRODUCTION

Recent advances in G6PD deficiency screening and treatment are rapidly changing the landscape of radical cure of vivax malaria available for National Malaria Programs (NMPs). While NMPs await the WHO's global policy guidance on these advances, they will also need to consider different contextual factors related to the vivax burden, health system capacity, and resources available to support changes to their policies and practices. Therefore, we aim to develop an Options Assessment Toolkit (OAT) that enables NMPs to systematically determine optimal radical cure options for their given environments and potentially reduce decision-making delays. This protocol outlines the OAT development process.

METHODS

Utilizing participatory research methods, the OAT will be developed in four phases where the NMPs and experts will have active roles in designing the research process and the toolkit. In the first phase, an essential list of epidemiological, health system, and political & economic factors will be identified. In the second phase, 2-3 NMPs will be consulted to determine the relative priority and measurability of these factors. These factors and their threshold criteria will be validated with experts using a modified e-Delphi approach. In addition, 4-5 scenarios representing country contexts in the Asia Pacific region will be developed to obtain the expert-recommended radical cure options for each scenario. In the third phase, additional components of OAT, such as policy evaluation criteria, latest information on new radical cure options, and others, will be finalized. The OAT will be pilot-tested with other Asia Pacific NMPs in the final phase.

ETHICS AND DISSEMINATION

Human Research Ethics Committee approval has been received from the Northern Territory, Department of Health, and Menzies School of Health Research (HREC Reference Number: 2022-4245). The OAT will be made available for the NMPs, introduced at the APMEN Vivax Working Group annual meeting, and reported in international journals.

A practical approach to establishing a critical care outreach service: An expert panel research design

BACKGROUND

For over two decades, nurse-led critical care outreach services have improved the recognition, response, and management of deteriorating patients in general hospital wards, yet variation in terms, design, implementation, and evaluation of such services continue. For those establishing a critical care outreach service, these factors make the literature difficult to interpret and translate to the real-world setting.

AIM

The aim of this study was to provide a practical approach to establishing a critical care outreach service in the hospital setting.

METHOD

An international expert panel of clinicians, managers, and academics with experience in implementing, developing, operationalising, educating, and evaluating critical care outreach services collaborated to synthesise evidence, experience, and clinical judgment to develop a practical approach for those establishing a critical care outreach service. A rapid review of the literature identified publications relevant to the study. A modified Delphi technique was used to achieve expert panel consensus particularly in areas where insufficient published literature or ambiguities existed.

FINDINGS

There were 502 publications sourced from the rapid review, of which 104 were relevant and reviewed. Using the modified Delphi technique, the expert panel identified five key components needed to establish a critical care outreach service: (i) approaches to service delivery, (ii) education and training, (iii) organisational engagement, (iv) clinical governance, and (v) monitoring and evaluation.

CONCLUSION

An expert panel research design successfully synthesised evidence, experience, and clinical judgement to provide a practical approach for those establishing a critical care outreach service. This method of research will likely be valuable in other areas of practice where terms are used interchangeably, and the literature is diverse and lacking a single approach to practice.

Barriers to the Engagement of Complementary and Integrative Health Providers in Public Health Responses to COVID-19: Recommendations From a Multidisciplinary Expert Panel

OBJECTIVE

The purpose of this project was to explore barriers to the involvement of complementary and integrative health (CIH) providers in the public health response to COVID-19 and potential solutions for future involvement in public health crises.

METHODS

An expert panel of 10 people, which included doctors of chiropractic, naturopathic doctors, public health practitioners, and researchers from the United States, was convened for a day-long online panel discussion. Facilitators asked panelists how CIH practitioners could contribute and be mobilized. We summarized themes and recommendations from the discussion.

RESULTS

Despite their skills and resources, few CIH providers participated in public health efforts like testing and contact tracing during the COVID-19 pandemic. Panelists described that CIH professionals may not have participated in those efforts due to the CIH providers possibly not having sufficient public health training and limited contact with public health professionals, as well as policy and financial challenges during the pandemic. Panelists proposed solutions to these barriers, including more public health training, stronger formal relationships between CIH and public health organizations, and improved financial support for both CIH care and public health efforts.

CONCLUSION

Through an expert panel discussion, we identified barriers that hindered the involvement of CIH providers in the public health response to the COVID-19 pandemic. During future pandemics in the United States, public health planners should recognize CIH providers as part of the existing labor resource, with clinical expertise and community-level connections that can be called upon in a crisis. During future events, CIH professional leaders should be more proactive in seeking out a supportive role and sharing their knowledge, skills, and expertise.

Expert Panel Consensus on State-Level Policies to Improve Engagement and Retention in Treatment for Opioid Use Disorder

Importance

In the US, recent legislation and regulations have been considered, proposed, and implemented to improve the quality of treatment for opioid use disorder (OUD). However, insufficient empirical evidence exists to identify which policies are feasible to implement and successfully improve patient and population-level outcomes.

Objective

To examine expert consensus on the effectiveness and the ability to implement state-level OUD treatment policies.

Evidence Review

This qualitative study used the ExpertLens online platform to conduct a 3-round modified Delphi process to convene 66 stakeholders (health care clinicians, social service practitioners, addiction researchers, health policy decision-makers, policy advocates, and persons with lived experience). Stakeholders participated in 1 of 2 expert panels on 14 hypothetical state-level policies targeting treatment engagement and linkage, evidence-based and integrated care, treatment flexibility, and monitoring or support services. Participants rated policies in round 1, discussed results in round 2, and provided final ratings in round 3. Participants used 4 criteria associated with either the effectiveness or implementability to rate and discuss each policy. The effectiveness panel (n = 29) considered policy effects on treatment engagement, treatment retention, OUD remission, and opioid overdose mortality. The implementation panel (n = 34) considered the acceptability, feasibility, affordability, and equitability of each policy. We measured consensus using the interpercentile range adjusted for symmetry analysis technique from the RAND/UCLA appropriateness method.

Findings

Both panels reached consensus on all items. Experts viewed 2 policies (facilitated access to medications for OUD and automatic Medicaid enrollment for citizens returning from correctional settings) as highly implementable and highly effective in improving patient and population-level outcomes. Participants rated hub-and-spoke-type policies and provision of financial incentives to emergency departments for treatment linkage as effective; however, they also rated these policies as facing implementation barriers associated with feasibility and affordability. Coercive policies and policies levying additional requirements on individuals with OUD receiving treatment (eg, drug toxicology testing, counseling requirements) were viewed as low-value policies (ie, decreasing treatment engagement and retention, increasing overdose mortality, and increasing health inequities).

Conclusions and Relevance

The findings of this study may provide urgently needed consensus on policies for states to consider either adopting or deimplementing in their efforts to address the opioid overdose crisis.

Planning and Reporting Effective Web-Based RAND/UCLA Appropriateness Method Panels: Literature Review and Preliminary Recommendations

BACKGROUND

The RAND/UCLA Appropriateness Method (RAM), a variant of the Delphi Method, was developed to synthesize existing evidence and elicit the clinical judgement of medical experts on the appropriate treatment of specific clinical presentations. Technological advances now allow researchers to conduct expert panels on the internet, offering a cost-effective and convenient alternative to the traditional RAM. For example, the Department of Veterans Affairs recently used a web-based RAM to validate clinical recommendations for de-intensifying routine primary care services. A substantial literature describes and tests various aspects of the traditional RAM in health research; yet we know comparatively less about how researchers implement web-based expert panels.

OBJECTIVE

The objectives of this study are twofold: (1) to understand how the web-based RAM process is currently used and reported in health research and (2) to provide preliminary reporting guidance for researchers to improve the transparency and reproducibility of reporting practices.

METHODS

The PubMed database was searched to identify studies published between 2009 and 2019 that used a web-based RAM to measure the appropriateness of medical care. Methodological data from each article were abstracted. The following categories were assessed: composition and characteristics of the web-based expert panels, characteristics of panel procedures, results, and panel satisfaction and engagement.

RESULTS

Of the 12 studies meeting the eligibility criteria and reviewed, only 42% (5/12) implemented the full RAM process with the remaining studies opting for a partial approach. Among those studies reporting, the median number of participants at first rating was 42. While 92% (11/12) of studies involved clinicians, 50% (6/12) involved multiple stakeholder types. Our review revealed that the studies failed to report on critical aspects of the RAM process. For example, no studies reported response rates with the denominator of previous rounds, 42% (5/12) did not provide panelists with feedback between rating periods, 50% (6/12) either did not have or did not report on the panel discussion period, and 25% (3/12) did not report on quality measures to assess aspects of the panel process (eg, satisfaction with the process).

CONCLUSIONS

Conducting web-based RAM panels will continue to be an appealing option for researchers seeking a safe, efficient, and democratic process of expert agreement. Our literature review uncovered inconsistent reporting frameworks and insufficient detail to evaluate study outcomes. We provide preliminary recommendations for reporting that are both timely and important for producing replicable, high-quality findings. The need for reporting standards is especially critical given that more people may prefer to participate in web-based rather than in-person panels due to the ongoing COVID-19 pandemic.

Consensus-Based Guidance on Opioid Management in Individuals With Advanced Cancer-Related Pain and Opioid Misuse or Use Disorder

Importance

Opioid misuse and opioid use disorder (OUD) are important comorbidities in people with advanced cancer and cancer-related pain, but there is a lack of consensus on treatment.

Objective

To develop consensus among palliative care and addiction specialists on the appropriateness of various opioid management strategies in individuals with advanced cancer-related pain and opioid misuse or OUD.

Design, Setting, and Participants

For this qualitative study, using ExpertLens, an online platform and methodology for conducting modified Delphi panels, between August and October 2020, we conducted 2 modified Delphi panels to understand the perspectives of palliative and addiction clinicians on 3 common clinical scenarios varying by prognosis (weeks to months vs months to years). Of the 129 invited palliative or addiction medicine specialists, 120 participated in at least 1 round. A total of 84 participated in all 3 rounds.

Main Outcomes and Measures

Consensus was investigated for 3 clinical scenarios: (1) a patient with a history of an untreated opioid use disorder, (2) a patient taking more opioid than prescribed, and (3) a patient using nonprescribed benzodiazepines.

Results

Participants were mostly women (47 [62%]), White (94 (78 [65%]), and held MD/DO degrees (115 [96%]). For a patient with untreated OUD, regardless of prognosis, it was deemed appropriate to begin treatment with buprenorphine/naloxone and inappropriate to refer to a methadone clinic. Beginning split-dose methadone was deemed appropriate for patients with shorter prognoses and of uncertain appropriateness for those with longer prognoses. Beginning a full opioid agonist was deemed of uncertain appropriateness for those with a short prognosis and inappropriate for those with a longer prognosis. Regardless of prognosis, for a patient with no medical history of OUD taking more opioids than prescribed, it was deemed appropriate to increase monitoring, inappropriate to taper opioids, and of uncertain appropriateness to increase the patient's opioids or transition to buprenorphine/naloxone. For a patient with a urine drug test positive for non-prescribed benzodiazepines, regardless of prognosis, it was deemed appropriate to increase monitoring, inappropriate to taper opioids and prescribe buprenorphine/naloxone.

Conclusions and Relevance

The findings of this qualitative study provide urgently needed consensus-based guidance for clinicians and highlight critical research and policy gaps.

Expert views on state-level naloxone access laws: a qualitative analysis of an online modified-Delphi process

Background

Expanding availability to naloxone is a core harm reduction strategy in efforts to address the opioid epidemic. In the US, state-level legislation is a prominent mechanism to expand naloxone availability through various venues, such as community pharmacies. This qualitative study aimed to identify and summarize the views of experts on state-level naloxone access laws.

Methods

We conducted a three-round modified-Delphi process using the online ExpertLens platform. Participants included 46 key stakeholders representing various groups (advocates, healthcare providers, human/social service practitioners, policymakers, and researchers) with expertise naloxone access laws. Participants commented on the effectiveness and implementability of 15 state-level naloxone access laws (NALs). We thematically analyzed participant comments to summarize views on NALs overall and specific types of NAL.

Results

Participants commented that the effectiveness of NALs in reducing opioid-related mortality depends on their ability to make sustained, significant impacts on population-level naloxone availability. Participants generally believed that increased naloxone availability does not have appreciable negative impacts on the prevalence of opioid misuse, opioid use disorder (OUD), and non-fatal opioid overdoses. Implementation barriers include stigma among the general public, affordability of naloxone, and reliance on an inequitable healthcare system.

Conclusions

Experts believe NALs that significantly increase naloxone access are associated with less overdose mortality without risking substantial unintended public health outcomes. To maximize impacts, high-value NALs should explicitly counter existing healthcare system inequities, address stigmatization of opioid use and naloxone, maintain reasonable prices for purchasing naloxone, and target settings beyond community pharmacies to distribute naloxone.

Arguments for and Against a New Diagnostic Entity for Patients With Chronic Pain on Long-Term Opioid Therapy for Whom Harms Outweigh Benefits

The goal of this study was to understand perspectives on whether a new diagnostic entity, distinct from Diagnostic and Statistical Manual - 5 (DSM-5) opioid use disorder (OUD), is needed for patients with chronic pain on long-term opioid therapy (LTOT) for whom the harms of continued opioid therapy outweigh the benefits. Data were collected as part of a larger Delphi study. We used rapid and thematic qualitative methods to analyze data from 51 panelists with expertise in internal medicine, psychiatry, psychology, and related fields. Three-quarters of panelists supported a new diagnostic entity; common themes included recognizing distinct experiences of patients prescribed LTOT, addressing problems with DSM-5 OUD criteria, facilitating research and improved treatment, and reducing stigma. Thirteen panelists opposed the creation of a new diagnostic entity; common themes included similarities in biological underpinnings of patients prescribed LTOT and diagnosed with OUD, belief that the continuum of OUD captured patients' experiences, finding better ways to address problems with DSM-5 OUD criteria, and concerns about stigma. While this expert panel disagreed about the need for a new diagnostic entity, there was an overall acknowledgement that the current implementation of DSM-5's OUD diagnosis is not meeting the needs of LTOT providers or patients. PERSPECTIVE: The DSM-5's OUD diagnosis may not adequately meet the needs of patients on LTOT for whom the harms of continued opioid therapy outweigh the benefits. Experts do not agree on how to address this problem; more work is needed to determine if a new diagnostic entity would be beneficial.

The impact of panel composition and topic on stakeholder perspectives: Generating hypotheses from online maternal and child health modified-Delphi panels

Introduction

Multistakeholder engagement is crucial for conducting health services research. Delphi‐based methodologies combining iterative rounds of questions with feedback on and discussion of group results are a well‐documented approach to multistakeholder engagement. This study develops hypotheses about the impact of panel composition and topic on the propensity and meaningfulness of response changes in multistakeholder modified‐Delphi panels.

Methods

We conducted three online modified‐Delphi (OMD) multistakeholder panels using the same protocol. We assigned 60 maternal and child health professionals to a homogeneous (professionals only) panel, 60 pregnant or postpartum women (patients) to a homogeneous panel, and 30 professionals and 30 patients to a mixed panel. In Round 1, participants rated the seriousness of 11 maternal and child health outcomes using a 0–100 scale and explained their ratings. In Round 2, participants saw their own and their panel's Round 1 results and discussed them using asynchronous, anonymous discussion boards moderated by the study investigators. In Round 3, participants revised their original ratings. Our outcome measures included binary indicators of response changes to ratings of the low, medium and high severity maternal and child health outcomes and their meaningfulness, measured by a change of 10 or more points.

Results

Participants changed 818 of 1491 (55%) of responses; the majority of response changes were meaningful. Patterns of response changes were different for patients and professionals and for different levels of outcome seriousness. Using study results and the literature, we developed three hypotheses. First, OMD participants, regardless of their stakeholder group, are more likely to change their responses on preference‐sensitive topics where there is a range of viable alternatives or perspectives. Second, patients are more likely to change their responses and to do so meaningfully in mixed panels, whereas professionals are more likely to do so in homogeneous panels. Third, the association between panel composition and response change varies according to the topic (e.g., the level of outcome seriousness).

Conclusions

Results of our work not only helped generate empirically derived hypotheses to be tested in future research but also offer practical recommendations for designing multistakeholder OMD panels.

Patient or Public Contribution

Pregnant or postpartum women were involved in this study.

Expert Panel Consensus on Management of Advanced Cancer-Related Pain in Individuals With Opioid Use Disorder

IMPORTANCE

Opioid use disorder (OUD) is an important comorbidity in individuals with advanced cancer, in whom pain is common. Full-agonist opioid medications are the cornerstone of cancer pain management, but the existing literature does not address how to manage cancer pain in patients with OUD.

OBJECTIVE

To conduct an expert panel to develop consensus on the appropriateness of management of cancer pain in individuals with co-occurring advanced cancer and OUD.

EVIDENCE REVIEW

A 3-round modified Delphi process was completed from August to October 2020 with 2 cases: patient with advanced cancer, pain, and OUD treated with buprenorphine-naloxone or methadone. Participants rated management strategies in round 1, discussed results in round 2, and provided final responses in round 3. ExpertLens, an online approach to conducting modified Delphi panels, was used. Participants were experts in palliative care, addiction, or both, recruited by email from palliative care and addiction-focused professional groups, lists from prior studies, and snowball sampling. Data analysis was performed from November 2020 to July 2021.

FINDINGS

Of 120 experts (median age, 40-49 years), most were White (78 participants [94%]), female (74 participants [62%]), and held MD or DO degrees (115 participants [96%]); 84 (70%) participated in all rounds. For a patient with OUD taking buprenorphine-naloxone, it was deemed appropriate to continue buprenorphine-naloxone with thrice-daily dosing. Continuing buprenorphine-naloxone and adding a full-agonist opioid was deemed to be appropriate for patients with a prognosis of weeks to months and of uncertain appropriateness for patients with a prognosis of months to years. For a patient with OUD taking methadone dispensed at a methadone clinic, it was deemed appropriate to take over prescribing and dose twice or thrice daily. Continuing methadone daily while adding another full-agonist opioid was deemed appropriate for patients with a prognosis of weeks to months and of uncertain appropriateness for those with a prognosis of months to years.

CONCLUSIONS AND RELEVANCE

The findings of this qualitative study provide urgently needed, consensus-based guidance for clinicians and highlight critical research and policy gaps needed to facilitate implementation.

Effectiveness and implementability of state-level naloxone access policies: Expert consensus from an online modified-Delphi process

BACKGROUND

Naloxone distribution, a key global strategy to prevent fatal opioid overdose, has been a recent target of legislation in the U.S., but there is insufficient empirical evidence from causal inference methods to identify which components of these policies successfully reduce opioid-related harms. This study aimed to examine expert consensus on the effectiveness and implementability of various state-level naloxone policies.

METHODS

We used the online ExpertLens platform to conduct a three-round modified-Delphi process with a purposive sample of 46 key stakeholders (advocates, healthcare providers, human/social service practitioners, policymakers, and researchers) with naloxone policy expertise. The Effectiveness Panel (n = 24) rated average effects of 15 types of policies on naloxone pharmacy distribution, opioid use disorder (OUD) prevalence, nonfatal opioid-related overdoses, and opioid-related overdose mortality. The Implementation Panel (n = 22) rated the same policies on acceptability, feasibility, affordability, and equitability. We compared ratings across policies using medians and inter-percentile ranges, with consensus measured using the RAND/UCLA Appropriateness Method Inter-Percentile Range Adjusted for Symmetry technique.

RESULTS

Experts reached consensus on all items. Except for liability protections and required provision of education or training, experts perceived all policies to generate moderate-to-large increases in naloxone pharmacy distribution. However, only three policies were expected to yield substantive decreases on fatal overdose: statewide standing/protocol order, over-the-counter supply, and statewide "free naloxone." Of these, experts rated only statewide standing/protocol orders as highly affordable and equitable, and unlikely to generate meaningful population-level effects on OUD or nonfatal opioid-related overdose. Across all policies, experts rated naloxone prescribing mandates relatively lower in acceptability, feasibility, affordability, and equitability.

CONCLUSION

Experts believe statewide standing/protocol orders are an effective, implementable, and equitable policy for addressing opioid-related overdose mortality. While experts believe many other broad policies are effective in reducing opioid-related harms, they also believe these policies face implementation challenges related to cost and reaching vulnerable populations.

Informing the Development of a Standardized Clinical Definition of Neonatal Abstinence Syndrome: Protocol for a Modified-Delphi Expert Panel

Background

Neonatal abstinence syndrome (NAS) is a postnatal withdrawal syndrome that most commonly results from prenatal opioid exposure. Every 15 minutes, an infant is born in the United States with signs of NAS. The field lacks a standardized clinical definition of NAS, complicating discussions on programmatic and policy development to support opioid-exposed mothers and infants.

Objective

The goal of this paper is to describe a protocol for a systematic expert panel process to inform the development of a clinical definition of NAS.

Methods

We will conduct two three-round online modified-Delphi panels using the ExpertLens system and will follow the recommendations for Conducting and REporting of DElphi Studies (CREDES). One panel will focus on developing key components of a clinical definition of NAS, and the second panel will focus on neonatal opioid withdrawal syndrome (NOWS), which is a term that has come into use to differentiate opioid-exposed infants from infants exposed to other substances in utero. However, there is lack of agreement on the precise clinical definition of NOWS and how it is distinct from or overlaps with NAS. Each panel will complete two rating rounds and a discussion round using a similar protocol. We will analyze all rating data descriptively and determine the presence of agreement within and between the two panels. We will also perform thematic analysis of the qualitative comments to contextualize the panel findings.

Results

The panels were convened between October 29 and December 17, 2020. Their results were disseminated and discussed at a national conference on NAS that took place on March 17-18, 2021.

Conclusions

A standardized clinical definition of NAS will help to better characterize NAS incidence and to design effective clinical, public health, and policy interventions to support opioid-exposed mother-infant dyads.

International Registered Report Identifier (IRRID)

DERR1-10.2196/25387

Development of prehospital care quality indicators for the Australian setting: a modified RAND/UCLA appropriateness method

BACKGROUND

Globally, the measurement of quality is an important process that supports the provision of high-quality and safe healthcare services. The requirement for valid quality measurement to gauge improvements and monitor performance is echoed in the Australian prehospital care setting. The aim of this study was to use an evidence-informed expert consensus process to identify valid quality indicators (QIs) for Australian prehospital care provided by ambulance services.

METHODS

A modified RAND/UCLA appropriateness method was conducted with a panel of Australian prehospital care experts from February to May 2019. The proposed QIs stemmed from a scoping review and were systematically prepared within a clinical and non-clinical classification system, and a structure/process/outcome and access/safety/effectiveness taxonomy. Rapid reviews were performed for each QI to produce evidence summaries for consideration by the panellists. QIs were deemed valid if the median score by the panel was 7-9 without disagreement.

RESULTS

Of 117 QIs, the expert panel rated 84 (72%) as valid. This included 26 organisational/system QIs across 7 subdomains and 58 clinical QIs within 10 subdomains.Most QIs were process indicators (n=62; 74%) while QIs describing structural elements and desired outcomes were less common (n=13; 15% and n=9; 11%, respectively). Non-exclusively, 18 (21%) QIs addressed access to healthcare, 21 (25%) described safety aspects and 64 (76%) specified elements contributing to effective services and care. QIs on general time intervals, such as response time, were not considered valid by the panel.

CONCLUSION

This study demonstrates that with consideration of best available evidence a substantial proportion of QIs scoped and synthesised from the international literature are valid for use in the Australian prehospital care context.

Defining Access Management in Health Care Delivery Organizations

Managing patient access to care in health care delivery organizations is instrumental in shaping patient experiences. We convened an inclusive stakeholder panel, informed by evidence, to understand the dimensions and establish definitions of access and access management. The literature varies in access definitions, but the temporal measure "time to third next available appointment" was consistently used as an indicator of access. Panel deliberations highlighted the importance of patient-centeredness and resulted in comprehensive definitions for access management, optimal access management, and optimal access. Health care organizations and researchers can use the developed definitions and concepts as starting points for initiatives to improve access management.

Prioritizing Needs and Outcomes for Adolescent Substance Use Treatment Planning: An Online Modified-Delphi Process

Key stakeholders can have differing views about which information is essential to inform placement decisions for all patients. This study examined consensus across stakeholder groups on the most important individual needs and treatment outcomes for informing decisions specifically about the level of care for an adolescent in substance use treatment.

Assessing the process and outcome of the development of practice guidelines and recommendations: PANELVIEW instrument development

BACKGROUND

Guideline recommendations may be affected by flaws in the process, inappropriate panel member selection or conduct, conflicts of interest and other factors. To our knowledge, no validated tool exists to evaluate guideline development from the perspective of those directly involved in the process. Our objective was to develop and validate a universal tool, the PANELVIEW instrument, to assess guideline processes, methods and outcomes from the perspective of the participating guideline panellists and group members.

METHODS

We performed a systematic literature search and surveys of guideline groups (identified through contacting international organizations and convenience sampling of working panels) to inform item generation. Subsequent groups of guideline methodologists and panellists reviewed items for face validity and missing items. We used surveys, interviews and expert review for item reduction and phrasing. For reliability assessment and feedback, we tested the PANELVIEW tool in 8 international guideline groups.

RESULTS

We surveyed 62 members from 13 guideline panels, contacted 19 organizations and reviewed 20 source documents to generate items. Fifty-three additional key informants provided feedback about phrasing of the items and response options. We reduced the number of items from 95 to 34 across domains that included administration, training, conflict of interest, group dynamics, chairing, evidence synthesis, formulating recommendations and publication. The tool takes about 10 minutes to complete and showed acceptable measurement properties.

INTERPRETATION

The PANELVIEW instrument fills a gap by enabling guideline organizations to involve clinicians, patients and other participants in evaluating their guideline processes. The tool can inform quality improvement of existing or new guideline programs, focusing on insight into and transparency of the guideline development process, methods and outcomes.

Consensus-based approach to managing opioids, including opioid misuse and opioid use disorder, in patients with serious illness: protocol for a modified Delphi process

INTRODUCTION

Management of opioid misuse and opioid use disorder (OUD) among individuals with serious illness is an important yet understudied issue. Palliative care clinicians caring for individuals with serious illness, many of whom may live for months or years, describe a complex tension between weighing the benefits of opioids, which are considered a cornerstone of pain management in serious illness, and serious opioid-related harms like opioid misuse and OUD. And yet, little literature exists to inform the management of opioid misuse and OUDs among individuals with serious illness. Our objective is to provide evidence-based management guidance to clinicians caring for individuals with serious illness who develop opioid misuse or OUD.

METHODS AND ANALYSIS

We chose a modified Delphi approach, which is appropriate when empirical evidence is lacking and expert input must be used to shape clinical guidance. We sought to recruit 60 clinicians with expertise in palliative care, addiction or both to participate in this study. We created seven patient cases that capture important management challenges in individuals with serious illness prescribed opioid therapy. We used ExpertLens, an online platform for conducting modified Delphi panels. Participants completed three rounds of data collection. In round 1, they rated and commented on the appropriateness of management choices for cases. In round 2, participants reviewed and discussed their own and other participants' round 1 numerical responses and comments. In round 3 (currently ongoing), participants again reviewed rounds 1 and 2, and are allowed to change their final numerical responses. We used ExpertLens to automatically identify whether there is consensus, or disagreement, among responses in panels. Only round 3 responses will be used to assess final consensus and disagreement.

ETHICS AND DISSEMINATION

This project received ethical approval from the University of Pittsburgh's Institutional Review Board (study 19110301) and the RAND Institutional Research Board (study 2020-0142). Guidance from this work will be disseminated through national stakeholder networks to gain buy-in and endorsement. This study will also form the basis of an implementation toolkit for clinicians caring for individuals with serious illness who are at risk of opioid misuse or OUD.

Quality measures for end-of-life care for children with cancer: A modified Delphi approach

BACKGROUND

The quality of adult end-of-life (EOL) cancer care has benefited from quality measures, but corresponding pediatric measures are lacking. Therefore, the authors used a validated expert panel method to recommend EOL quality measures for pediatric oncology.

METHODS

The authors used the modified Delphi method to assess potential quality measures. Panelists were selected on the basis of professional organization nominations and expert qualifications. Pediatric and adult oncology, pediatric palliative care, social work, nursing, and hospice were represented. The authors provided the panel with a literature review on 20 proposed measures derived from adult measures and bereaved family interviews. The panel first scored the importance of each measure on a 9-point scale and then discussed the measures via a conference call. The panel then rescored the measures. According to a priori standards, measures with median scores ≥ 7 with at least 7 of 9 experts ranking it as ≥4 were endorsed.

RESULTS

The 16 endorsed measures included measures related to avoidance of medically intense care (eg, intensive care unit death and intubation in the last 14 days of life), death location (eg, death in the preferred location), hospital policies/programs (eg, the removal of visitor restrictions at EOL and the presence of a bereavement program), and supportive care services (eg, pediatric palliative care involvement and sibling needs assessment). Unendorsed measures included avoidance of chemotherapy at EOL and home death.

CONCLUSIONS

Expert panel-endorsed quality measures have been developed for EOL care in pediatric oncology. The measures need validation with bereaved families and further refinement before they are ready for real-world application as a tool for standardizing EOL care in pediatric oncology.

LAY SUMMARY

Quality measures for end-of-life care for children with cancer lag behind adult quality measures. Therefore, the authors have conducted an expert panel to develop an endorsed list of quality measures for end-of-life care for children with cancer. The 16 endorsed measures include measures related to avoidance of medically intense care (eg, intensive care unit death and intubation in the last 14 days of life), location of death (eg, death in the preferred location), hospital policies/programs (eg, the removal of visitor restrictions at the end of life and the presence of a bereavement program), and supportive care services (eg, pediatric palliative care involvement and sibling needs assessment).

Why and how to engage expert stakeholders in ontology development: insights from social and behavioural sciences

BACKGROUND

Incorporating the feedback of expert stakeholders in ontology development is important to ensure content is appropriate, comprehensive, meets community needs and is interoperable with other ontologies and classification systems. However, domain experts are often not formally engaged in ontology development, and there is little available guidance on how this involvement should best be conducted and managed. Social and behavioural science studies often involve expert feedback in the development of tools and classification systems but have had little engagement with ontology development. This paper aims to (i) demonstrate how expert feedback can enhance ontology development, and (ii) provide practical recommendations on how to conduct expert feedback in ontology development using methodologies from the social and behavioural sciences.

MAIN BODY

Considerations for selecting methods for engaging stakeholders are presented. Mailing lists and issue trackers as existing methods used frequently in ontology development are discussed. Advisory boards and working groups, feedback tasks, consensus exercises, discussions and workshops are presented as potential methods from social and behavioural sciences to incorporate in ontology development.

CONCLUSIONS

A variety of methods from the social and behavioural sciences exist to enable feedback from expert stakeholders in ontology development. Engaging domain experts in ontology development enables depth and clarity in ontology development, whilst also establishing advocates for an ontology upon its completion.

Priority-Setting to Address the Geriatric Pharmacoparadox for Pain Management: A Nursing Home Stakeholder Delphi Study

BACKGROUND

Evidence to guide clinical decision making for pain management in nursing home residents is scant.

OBJECTIVE

Our objective was to explore the extent of consensus among expert stakeholders regarding what analgesic issues should be prioritized for comparative-effectiveness studies of beneficial and adverse effects of analgesic regimens in nursing home residents.

METHODS

Two stakeholder panels (nurses only and a mix of clinicians/researchers) were engaged (n = 83). During a three-round online modified Delphi process, participants rated and commented on the need for new evidence on nonopioid analgesic regimens and opioid regimens, short-term adverse effects, long-term adverse effects, comorbid conditions, and other factors in the nursing home setting (9-point scale; 1 = not essential to 9 = very essential to obtain new evidence). The quantitative data were analyzed to determine the existence of consensus using an approach from the RAND/UCLA Appropriateness Method User's Manual. The qualitative data, consisting of participant explanations of their numeric ratings, were thematically analyzed by an experienced qualitative researcher.

RESULTS

For nursing home residents, evidence generation was deemed essential for opioids, gabapentin (alone or with serotonin norepinephrine reuptake inhibitors [SNRIs]), and nonsteroid anti-inflammatory drugs with SNRIs. Experts prioritized the following outcomes as essential: long-term adverse effects, including delirium, cognitive decline, and decline in activities of daily living (ADLs). Kidney disease and depression were deemed essential conditions to consider in studies of pain medications. Coprescribing analgesic regimens with benzodiazepines, sedating medications, serotonergic medications, and non-SNRI antidepressants were considered essential areas of study. Experts noted that additional study was essential in residents with moderate/severe cognitive impairment and limitations in ADLs.

CONCLUSIONS

Stakeholder priorities for more evidence reflect concerns related to treating medically complex residents with complex drug regimens and included long-term adverse effects, coprescribing, and sedating medications. Carefully conducted observational studies are needed to address the vast evidence gap for nursing home residents.

Nature and Predictors of Response Changes in Modified-Delphi Panels

OBJECTIVES

To describe the extent and nature of changes in participants' responses after exposure to group feedback and discussion in modified-Delphi panels and to identify factors affecting those changes.

METHODS

We analyzed data from 2 online modified-Delphi panels, each consisting of 2 rating rounds and an online discussion round. We included responses from 55 participants who answered 38 questions in both rating rounds. Because not all participants answered each question twice, our sample consisted of 1846 cases (response changes). We used mixed-effect logistic and multinomial logistic regression to identify factors predicting response changes and their direction relative to group median-our consensus measure.

RESULTS

Participants changed, on average, 49% of their responses. A response was changed in 47% of the 1846 cases: 28% of responses were changed toward consensus and 19% away from it. Although some measures of subjective participation experiences had a marginally significant impact on the propensity and direction of response changes, several objective measures of discussion engagement were statistically significant predictors of both the presence and direction of response changes.

CONCLUSION

Our results illustrate the nature of response changes and highlight the importance of exposing participants to alternative perspectives and encouraging them to explain their perspectives.

Standardizing care to manage bleeding disorders in adolescents with heavy menses-A joint project from the ISTH pediatric/neonatal and women's health SSCs

BACKGROUND

Bleeding disorders (BD) are under-recognized in adolescents with heavy menstrual bleeding (HMB).

OBJECTIVES

The lack of clinical guidelines and variable symptomatic management of HMB created the imperative to standardize HMB care to identify and manage BD in adolescents.

METHODS

We convened an international working group (WG), utilized the results of a literature review to define knowledge gaps in HMB care, and used the collective clinical experience of the WG to develop care considerations for adolescents with BD and HMB. We then solicited input on the appropriateness of HMB care considerations from expert stakeholders representing hematology, adolescent medicine, and obstetrics-gynecology. We conducted an expert panel online, using the ExpertLens platform. During a three-round online modified-Delphi process, the expert panel rated the appropriateness of 21 care considerations using a 9-point scale to designate care as appropriate (7-9), uncertain (4-6), or inappropriate (1-3) covering screening for BD, the laboratory work-up, and management of adolescents with BD that present with HMB. We used the RAND/UCLA appropriateness method to determine the existence of consensus among the interdisciplinary panel of experts.

RESULTS

Thirty-nine experts participated in the panel. The experts rated fifteen HMB care considerations as appropriate, six as uncertain, and none as inappropriate.

CONCLUSIONS

The HMB care statements represent the first set of HMB care considerations in adolescents with BD, developed with broad expert input on appropriateness. Although likely to be of interest to a range of clinicians who routinely manage adolescents with HMB, additional research is required in many key areas.

Development of a Patient-centered Quality Measurement Framework for Measuring, Monitoring, and Optimizing Rheumatoid Arthritis Care in Canada

OBJECTIVE

The aim of this study was to develop a patient-centered quality measurement framework to address a predefined vision statement and 7 strategic objectives for rheumatoid arthritis (RA) care that was developed in prior qualitative work with arthritis stakeholders.

METHODS

One hundred forty-seven RA-related performance measures (PMs) were identified from a systematic review. A candidate list of 26 PMs meeting predefined criteria and addressing the strategic objectives previously defined was then assessed during a 3-round (R) modified Delphi. Seventeen panelists with expertise in RA, quality measurement, and/or lived experience with RA rated each PM on a 1-9 scale based on the items of importance, feasibility, and priority for inclusion in the framework during R1 and R3, with a moderated discussion in R2. PMs with median scores ≥ 7 on all 3 items without disagreement were included in the final set, which then underwent public comment.

RESULTS

Twenty-one measures were included in the final framework (15 PMs from the Delphi and 6 published system-level measures on access to care and treatment). The measures included 4 addressing early access to care and timely diagnosis, 12 evidence-based care for RA and related comorbidities, 1 addressing patient participation as an informed partner in care, and 4 on patient outcomes.

CONCLUSION

The proposed framework builds upon existing measures capturing early access to care and treatment in RA and adds important PMs to promote high-quality RA care and outcome measurement. In the next phase, the authors will test the framework in clinical practice in addition to addressing certain areas where no suitable PMs were identified.

Development and testing of Australian prehospital care quality indicators: study protocol

INTRODUCTION

Historically, ambulance services were established to provide rapid transport of patients to hospital. Contemporary prehospital care involves provision of sophisticated 'mobile healthcare' to patients across the lifespan presenting with a range of injuries or illnesses of varying acuity. Because of its young age, the paramedicine profession has until recently experienced a lack of research capacity which has led to paucity of a discipline-specific, scientific evidence-base. Therefore, the performance and quality of ambulance services has traditionally been measured using simple, evidence-poor indicators forming a deficient reflection of the true quality of care and providing little direction for quality improvement efforts. This paper reports the study protocol for the development and testing of quality indicators (QIs) for the Australian prehospital care setting.

METHODS AND ANALYSIS

This project has three phases. In the first phase, preliminary work in the form of a scoping review was conducted which provided an initial list of QIs. In the subsequent phase, these QIs will be developed by aggregating them and by performing related rapid reviews. The summarised evidence will be used to support an expert consensus process aimed at optimising the clarity and evaluating the validity of proposed QIs. Finally, in the third phase those QIs deemed valid will be tested for acceptability, feasibility and reliability using mixed research methods. Evidence-based indicators can facilitate meaningful measurement of the quality of care provided. This forms the first step to identify unwarranted variation and direction for improvement work. This project will develop and test quality indicators for the Australian prehospital care setting.

ETHICS AND DISSEMINATION

This project has been approved by the University of Adelaide Human Research Ethics Committee. Findings will be disseminated by publications in peer-reviewed journals, presentations at appropriate scientific conferences, as well as posts on social media and on the project's website.

ADAPT study: adaptation of evidence-informed complex population health interventions for implementation and/or re-evaluation in new contexts: protocol for a Delphi consensus exercise to develop guidance

INTRODUCTION

Complex population health interventions that are effective in one context may not be effective elsewhere, and may even be harmful. As such, an intervention may require adaptation to ensure it fits with a new context. To date, there is no overarching guidance to help researchers to adapt and evaluate interventions in new contexts, and no criteria to support research funders or journals assess proposed or reported adaptations or evaluation. There is limited assistance for policy-makers and practitioners to decide if evidence-informed interventions are appropriate to their context, or if adaptation and further evaluation is needed. This Delphi exercise will contribute to the development of guidance for these communities to support the adaptation, implementation and/or re-evaluation of complex population health interventions in new contexts.

METHODS

We will conduct a Delphi consensus exercise to gather expert opinion from researchers, research funders, journal editors and policy-makers. Expert opinion will be sought on: appropriate definitions and concepts, identifying key methodological considerations and establishing adaptations and processes to be undertaken during adaptation of complex population health interventions in new contexts.

ETHICS AND DISSEMINATION

Ethics approval for the Delphi exercise has been obtained from the University of Glasgow and and the RAND institutional research board. Dissemination of the results of this study will be through peer-reviewed publications, workshops at national and international conferences, and a summary of the guidance developed for key organisations and stakeholders.

An adapted Delphi approach: The use of an expert panel to operationally define non-judgment of internal experiences as it relates to mindfulness

OBJECTIVES

There are several definitions of mindfulness throughout the literature, many of which suggest an attitude of non-judgmental awareness. However, the concept of "non-judgment" itself has not previously been systematically operationally defined. Our purpose was to use an expert panel to generate an operational definition of non-judgment of internal experiences, as it relates to mindfulness, to be used to inform the development of an implicit measure of the construct.

DESIGN

We utilized an adapted Delphi survey method consisting of three survey rounds.

SETTING

We employed in-person and online survey methods.

RESULTS

We used three survey rounds with an adapted Delphi approach. Expert review panelists consisted of 18 mindfulness researchers or clinicians. Each round of survey results was assessed and discussed among the core team. A consensus was reached among the core team for an operational definition of non-judgment of internal experiences: "acknowledging our thoughts, feelings, and sensations, as they are, without applying valence (e.g., good, bad, right, wrong) to them."

CONCLUSIONS

An expert panel review process informed the generation of an operational definition of non-judgment of internal experiences. Our operational definition provides a foundation for the future development of an implicit task of non-judgment of internal experiences, with the aim of using this task to assess change in response to mindfulness-based treatments. To our knowledge, this is the first systematic definition of non-judgment of internal experiences within the mindfulness literature.

Engaging Patients and Professionals to Evaluate the Seriousness of Maternal and Child Health Outcomes: Protocol for a Modified Delphi Study

BACKGROUND

Maternal weight gain during pregnancy is one of the few potentially modifiable risk factors for many adverse maternal and child health outcomes. Defining the optimal pregnancy weight gain range is difficult because, while lower weight gain may prevent some outcomes, such as maternal and child obesity, it may increase the risk of others such as fetal growth restriction and infant death. These health outcomes vary in their seriousness to mothers and their health care providers, and these differences in seriousness should be taken into account when determining optimal weight gain ranges. However, the relative seriousness that women and their care providers place on different health outcomes is unknown.

OBJECTIVE

We will determine the seriousness of 11 maternal and child health outcomes that have been consistently associated with pregnancy weight gain. We will achieve this by engaging patients and maternal and child health professionals using an online modified Delphi panel process.

METHODS

We aim to recruit a racially/ethnically and geographically diverse group of 90 US maternal and child health professionals and 90 women who are pregnant or less than 2 years postpartum. We will conduct 3 concurrent panels using the ExpertLens system, a previously evaluated online modified Delphi system that combines 2 rounds of rating with 1 round of feedback and moderated online discussion. In Round 1, panelists are asked to rate the seriousness of each health outcome on a scale of 0-100 and to provide a rationale for their scores. In Round 2, panelists will review their responses relative to those of other panelists. They will discuss their seriousness ratings anonymously using a moderated online discussion board. In Round 3, participants will revise their Round 1 responses based on group feedback and discussion. Each round will be open for 1-2 weeks.

RESULTS

The study protocol was reviewed by our ethics boards and did not require approval as human research. A pilot study of 6 professionals and 7 patients was completed in December 2019.

CONCLUSIONS

Our numeric estimates of the seriousness of maternal and child health outcomes will enable future studies to determine pregnancy weight gain ranges that balance the risks of low and high weight gain for mothers and children.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/16478.

Stakeholders' engagement in co-producing policy-relevant knowledge to facilitate employment for persons with developmental disabilities

BACKGROUND

Persons with developmental disabilities (PWDD) face a number of individual, environmental and societal barriers when seeking employment. Integrated knowledge translation (IKT) involves ongoing and dynamic interactions between researchers and stakeholders for the purpose of engaging in mutually beneficial research to address these types of multi-faceted barriers. There is a knowledge gap in the IKT literature on effective stakeholder engagement strategies outside of the dissemination stage to inform policy. In this paper, we report on a number of engagement strategies employed over a 2-year period to engage a wide range of stakeholders in different stages of an IKT project that aimed to investigate the 'wicked' problem of employment for PWDD.

METHOD

Our engagement plan included multiple linked strategies and was designed to ensure the meaningful engagement of, and knowledge co-production with, stakeholders. We held two participatory consensus-building stakeholder policy dialogue events to co-produce knowledge utilising the nominal group technique and the modified Delphi technique. A total of 31 and 49 stakeholders engaged in the first and second events, respectively, from six key stakeholder groups. Focused engagement strategies were employed to build on the stakeholder dialogues for knowledge mobilisation and included a focus group attended only by PWDD, a stakeholder workshop attended only by policy/decision-makers, a webinar attended by human resources professionals and employers, and a current affairs panel attended by the general public.

RESULTS

Our findings suggest that the level of engagement for each stakeholder group varies depending on the goal and need of the project. Our stakeholder dialogue findings highlight the inherent challenges in co-framing and knowledge co-production through the meaningful engagement of multiple stakeholders who hold different ideas and interests. Focused outreach is needed to foster relationships and trust for meaningful engagement.

CONCLUSIONS

In addition to providing guidance on how to implement adaptable meaningful engagement strategies, these findings contribute to discussions on how IKT projects are planned and funded. More studies to explore effective mechanisms for engaging a wide range of stakeholders in IKT research are needed. More evidence of successful engagement strategies employed by researchers to achieve meaningful knowledge co-production is also key to advancing the discipline.

Eight Priorities for Improving Primary Care Access Management in Healthcare Organizations: Results of a Modified Delphi Stakeholder Panel

OBJECTIVE

To identify priorities for improving healthcare organization management of patient access to primary care based on prior evidence and a stakeholder panel.

BACKGROUND

Studies on healthcare access show its importance for ensuring population health. Few studies show how healthcare organizations can improve access.

METHODS

We conducted a modified Delphi stakeholder panel anchored by a systematic review. Panelists (N = 20) represented diverse stakeholder groups including patients, providers, policy makers, purchasers, and payers of healthcare services, predominantly from the Veterans Health Administration. A pre-panel survey addressed over 80 aspects of healthcare organization management of access, including defining access management. Panelists discussed survey-based ratings during a 2-day in-person meeting and re-voted afterward. A second panel process focused on each final priority and developed recommendations and suggestions for implementation.

RESULTS

The panel achieved consensus on definitions of optimal access and access management on eight urgent and important priorities for guiding access management improvement, and on 1-3 recommendations per priority. Each recommendation is supported by referenced, panel-approved suggestions for implementation. Priorities address two organizational structure targets (interdisciplinary primary care site leadership; clearly identified group practice management structure); four process improvements (patient telephone access management; contingency staffing; nurse management of demand through care coordination; proactive demand management by optimizing provider visit schedules), and two outcomes (quality of patients' experiences of access; provider and staff morale). Recommendations and suggestions for implementation, including literature references, are summarized in a panelist-approved, ready-to-use tool.

CONCLUSIONS

A stakeholder panel informed by a pre-panel systematic review identified eight action-oriented priorities for improving access and recommendations for implementing each priority. The resulting tool is suitable for guiding the VA and other integrated healthcare delivery organizations in assessing and initiating improvements in access management, and for supporting continued research.

Practical Considerations in Using Online Modified-Delphi Approaches to Engage Patients and Other Stakeholders in Clinical Practice Guideline Development

Patients and caregivers are increasingly recognized as key stakeholders in developing clinical practice guidelines (CPGs). Online engagement approaches offer the promise of a rigorous, scalable, and convenient engagement method. This paper illustrates how an online modified-Delphi approach could be used to engage patients, caregivers, and other stakeholder in CPG development. It provides practical guidance for conducting online modified-Delphi panels that covers (1) joint development of the engagement approach with relevant stakeholders, (2) adaptation of methods used by experts in guideline development, (3) pilot testing, (4) participant recruitment, (5) determining panel size and composition, (6) building participant capacity, (7) facilitation of two-way interaction, (8) continuous engagement and retention of participants, (9) rigorous data analysis, (10) evaluation of engagement activities, and (11) result dissemination. The paper is based on a recently completed study about engaging individuals with Duchenne muscular dystrophy (DMD) and their caregivers in determining the patient-centeredness of DMD care guidelines.

Practical factors determining adolescent substance use treatment settings: Results from four online stakeholder panels

Practical factors can significantly influence the setting, or level of care, where an adolescent receives substance use treatment. This study aimed to identify practical factors that stakeholders find most critical to consider when planning adolescent substance use treatment. We conducted online panels with four stakeholder groups: providers, policymakers, researchers, and parents. Stakeholders nominated, rated, and commented on the importance of 10 practical factors that could influence treatment setting decisions. We assessed consensus on the rated importance of practical factors using the RAND/UCLA Appropriateness Method. We thematically analyzed stakeholders comments to explain how they differentiated relative importance. 153 stakeholders (66 providers, 38 policymakers, 27 researchers, and 22 parents) identified continuity of care, coordination of care across service sectors, and quality of care as the practical factors of highest importance. Participants rated higher the practical factors they perceived to (1) trump clinical appropriateness as the reason for placing an adolescent in a given setting, (2) steer adolescents toward specific settings, or (3) steer an adolescent away from specific settings. Conversely, participants rated lower the practical factors they perceived (1) applicable to clinical intervention rather than treatment setting, (2) unrelated to initial recommendations, (3) relevant to any setting, or (4) applicable only to certain contexts and sub-populations. These findings help elucidate why stakeholders view certain practical factors as critical to consider in actual decisions about substance use treatment settings for adolescents. Future research should investigate how to incorporate these practical factors alongside clinical needs and treatment goals in placement criteria and treatment matching.

Developing an Evidence-Based Nursing Handover Standard for a Multi-Site Public Hospital in Switzerland: Protocol for a Web-Based, Modified Delphi Study

BACKGROUND

Poor communication processes create opportunities for errors when caregivers fail to transfer complete and consistent information. Inadequate or nonexistent clinical handovers or failures to transfer information, responsibility, and accountability can have dire consequences for hospitalized patients. Clinical handover is practiced every day, in a multitude of ways, in all health care settings.

OBJECTIVE

The goal of this study is to build a consensus, evidence-based nursing handover standard for inpatients during shift changes or internal transfers between hospital wards. The study will be based on papers published by Slade et al.

METHODS

This protocol describes a modified Delphi data-collection survey involving a targeted panel sample of 300 nurse experts. A multi-round survey will select an anonymous panel from a multi-site public hospital in Switzerland. Each survey stage will be described and will build on the previous one. The study will end with a focus group discussion involving a randomly selected panel to explain why items for the evidence-based clinical nursing handover standard were accepted or not accepted. An item must achieve a consensus of ≥70% for inclusion.

RESULTS

The present study's expected outcome is a consensus-built, evidence-based nursing handover standard for inpatients during shift changes or internal transfers between the wards of a multi-site public hospital in Switzerland.

CONCLUSIONS

This survey will enable us to develop an evidence-based nursing handover standard for use during shift changes and internal inpatient transfers in a multi-site public hospital in Switzerland.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/15910.

Optimising Management of Patients with Heart Failure with Preserved Ejection Fraction in Primary Care (OPTIMISE-HFpEF): rationale and protocol for a multi-method study

BACKGROUND

Heart failure with preserved ejection fraction (HFpEF) is less well understood than heart failure with reduced ejection fraction (HFrEF), with greater diagnostic difficulty and management uncertainty.

AIM

The primary aim is to develop an optimised programme that is informed by the needs and experiences of people with HFpEF and healthcare providers. This article presents the rationale and protocol for the Optimising Management of Patients with Heart Failure with Preserved Ejection Fraction in Primary Care (OPTIMISE-HFpEF) research programme.

DESIGN & SETTING

This is a multi-method programme of research conducted in the UK.

METHOD

OPTIMISE-HFpEF is a multi-site programme of research with three distinct work packages (WPs). WP1 is a systematic review of heart failure disease management programmes (HF-DMPs) tested in patients with HFpEF. WP2 has three components (a, b, c) that enable the characteristics, needs, and experiences of people with HFpEF, their carers, and healthcare providers to be understood. Qualitative enquiry (WP2a) with patients and providers will be conducted in three UK sites exploring patient and provider perspectives, with an additional qualitative component (WP2c) in one site to focus on transitions in care and carer perspectives. A longitudinal cohort study (WP2b), recruiting from four UK sites, will allow patients to be characterised and their illness trajectory observed across 1 year of follow-up. Finally, WP3 will synthesise the findings and conduct work to gain consensus on how best to identify and manage this patient group.

RESULTS

Results from the four work packages will be synthesised to produce a summary of key learning points and possible solutions (optimised programme) which will be presented to a broad spectrum of stakeholders to gain consensus on a way forward.

CONCLUSION

HFpEF is often described as the greatest unmet need in cardiology. The OPTIMISE-HFpEF programme aims to address this need in primary care, which is arguably the most appropriate setting for managing HFpEF.

Using an Online, Modified Delphi Approach to Engage Patients and Caregivers in Determining the Patient-Centeredness of Duchenne Muscular Dystrophy Care Considerations

Purpose. To determine the patient-centeredness of endocrine and bone health Duchenne muscular dystrophy (DMD) care considerations using the RAND/PPMD Patient-Centeredness Method (RPM), which is a novel, online, modified-Delphi approach to engaging patients and caregivers in clinical guideline development. Methods. We solicited input on the patient-centeredness of care considerations from 28 individuals with DMD and 94 caregivers, randomly assigned to 1 of 2 mixed panels. During a 3-round online modified-Delphi process, participants rated the importance and acceptability of 19 DMD care considerations (round 1), reviewed and discussed the initial results (round 2), and revised their original ratings (round 3). Patient-centeredness was operationalized as importance and acceptability of recommendations. We considered a care consideration to be patient-centered if both panels deemed it important and acceptable. Results. Ninety-five panelists (78%) participated in this study. Of these, 88 (93%) participated in round 1, 74 (78%) in round 2, and 56 (59%) in round 3. Panelists deemed 12 care considerations to be patient-centered: 3 weight management, 3 bone health, 4 vertical growth, and 2 puberty recommendations. Seven care considerations did not meet patient-centeredness criteria. Common reasons were lack of evidence specific to DMD and concerns about insurance coverage, access to treatment, and patient safety. Conclusions. Using the RPM, Duchenne families considered most care considerations to be patient-centered. Besides being clinically appropriate, these considerations are likely to be consistent with the preferences, needs, and values of Duchenne families. While all relevant care considerations should be discussed during patient-provider encounters, those that did not meet patient-centeredness criteria in particular should be carefully considered as part of joint decision making between Duchenne families and their providers. Study Registration: HSRProj 20163126.

Brain computerized tomography reading in suspected acute ischemic stroke patients: what are essentials for medical students?

BACKGROUND

Few systematic methods prioritize the image education in medical students (MS). We hope to develop a checklist of brain computerized tomography (CT) reading in patients with suspected acute ischemic stroke (AIS) for MS and primary care (PC) physicians.

METHODS

Our pilot group generated the items indicating specific structures or signs for the checklist of brain CT reading in suspected AIS patients for MS and PC physicians. These items were used in a modified web-based Delphi process using the online software "SurveyMonkey". In total 15 panelists including neurologists, neurosurgeons, neuroradiologists, and emergency department physicians participated in the modified Delphi process. Each panelist was encouraged to express feedback, agreement or disagreement on the inclusion of each item using a 9-point Likert scale. Items with median scores of 7-9 were included in our final checklist.

RESULTS

Fifty-two items were initially provided for the first round of the Delphi process. Of these, 35 achieved general agreement of being an essential item for the MS and PC physicians. The other 17 of the 52 items in this round and another two added items suggested by the panelists were further rated in the next round. Finally, 38 items were included in the essential checklist items of brain CT reading in suspected AIS patients for MS and PC physicians.

CONCLUSIONS

We established a reference regarding the essential items of brain CT reading in suspected AIS patients. We hope this helps to minimize malpractice and a delayed diagnosis, and to improve competency-based medical education for MS and PC physicians.

Use of an expert panel to identify domains and indicators of delirium severity

PURPOSE

Our purpose was to create a content domain framework for delirium severity to inform item development for a new instrument to measure delirium severity.

METHODS

We used an established, multi-stage instrument development process during which expert panelists discussed best approaches to measure delirium severity and identified related content domains. We conducted this work as part of the Better ASsessment of ILlness (BASIL) study, a prospective, observational study aimed at developing and testing measures of delirium severity. Our interdisciplinary expert panel consisted of twelve national delirium experts and four expert members of the core research group. Over a one-month period, experts participated in two rounds of review.

RESULTS

Experts recommended that the construct of delirium severity should reflect both the phenomena and the impact of delirium to create an accurate, patient-centered instrument useful to interdisciplinary clinicians and family caregivers. Final content domains were Cognitive, Level of consciousness, Inattention, Psychiatric-Behavioral, Emotional dysregulation, Psychomotor features, and Functional. Themes debated by experts included reconciling clinical geriatrics and psychiatric content, mapping symptoms to one specific domain, and accurate capture of unclear clinical presentations.

CONCLUSIONS

We believe this work represents the first application of instrument development science to delirium. The identified content domains are inclusive of various, wide-ranging domains of delirium severity and are reflective of a consistent framework that relates delirium severity to potential clinical outcomes. Our content domain framework provides a foundation for development of delirium severity instruments that can help improve care and quality of life for patients with delirium.

Key factors for the assessment of mobility in advanced dementia: A consensus approach

INTRODUCTION

Loss of mobility is common in advanced dementia and has important negative consequences related to fall risk, loss of independence, and lack of participation in meaningful activities. The causes of decline are multifactorial, including disease-specific changes in motor function, behavior, and cognition. To optimize clinical management of mobility, there is a need to better characterize capacity for safe and independent mobility. This study aimed to identify key factors that impact on mobility in dementia.

METHODS

Expert input was gathered using a modified Delphi consensus approach. The primary criterion for participation was specialist knowledge in mobility or dementia, either as a clinician or a researcher. Participants rated elements of mobility for importance and feasibility of assessment in advanced dementia and prioritized items for inclusion in a mobility staging tool. Descriptive statistics and qualitative content analysis were used to summarize responses.

RESULTS

Thirty-six experts completed the first survey with an 80% retention rate over three rounds. One-third of 61 items reached consensus for being both important and feasible to assess, representing five categories of elements. Items reaching agreement for a staging tool included walking, parkinsonism, gait, impulsivity, fall history, agitation, transfers, and posture control.

DISCUSSION

This study highlights the need for a multidimensional, dementia-specific approach to mobility assessment. Results have implications for development of assessment methods and management guidelines to support the clinical care of mobility impairment in people with dementia.

Community-Directed Bacterial Sexually Transmitted Infection Testing Interventions Among Men Who Have Sex With Men: Protocol for an E-Delphi Study in Toronto, Canada

Background

HIV-positive and HIV-negative (gay, bisexual, and other) men who have sex with men (MSM) have experienced a dramatic increase in bacterial sexually transmitted infections (STIs)—syphilis, gonorrhea, and chlamydia. STI testing and treatment mitigate adverse health outcomes and substantially reduce transmission; yet, testing rates remain below recommended levels. Innovation is needed to produce the required increases in testing levels, frequency, and the use of appropriate testing technologies in ways that are engaging, nonstigmatizing, and acceptable to men.

Objective

The aim of this study is to build consensus with regard to interventions with the greatest potential for improving local STI testing services for MSM communities in Toronto, Canada.

Methods

Following a literature review of evidence regarding the effectiveness of novel testing interventions, and focus groups, and surveys to describe local barriers and facilitators of testing among MSM, we will conduct a Web-based, modified Delphi study (e-Delphi). We will form expert panels of community members and STI test providers. Panelists will rate potential interventions in terms of their priority, using a 7-point Likert scale from definitely not a priority to definitely a priority. They will also rank their preferences by selecting their top 3 preferred interventions. Surveys will be distributed in 3 rounds, with feedback on the distribution of responses from preceding rounds provided in rounds 2 and 3. We will define consensus as having ≥60% (18/30) members indicate a preference within 2 adjacent response points. Qualitative data on disagreements will be obtained using open-ended text responses to explain for ratings and rankings that are different from the majority.

Results

On the basis of a literature review and identification of barriers and facilitators to STI testing among community members and test providers in Toronto, we have selected 8 potential interventions for inclusion in the e-Delphi panel surveys. These include 4 interventions that streamline STI testing for asymptomatic individuals, 2 interventions that are targeted at clients and 2 interventions that are targeted at providers.

Conclusions

Findings will provide community direction for informed decision making regarding the implementation of STI testing interventions in this setting. They will characterize the intervention climate for innovation to STI testing services, including perceived needs for changes to test delivery, relative priorities for change, and readiness for implementation. These methods may be transferable to other urban jurisdictions experiencing similar epidemics and for other contexts where stakeholder input is needed to manage sensitive areas of concern.

International Registered Report Identifier (IRRID)

PRR1-10.2196/13801

A Novel Stakeholder Engagement Approach for Patient-centered Outcomes Research

INTRODUCTION/OBJECTIVES

The engagement of patients and other stakeholders is a critical element in the design of patient-centered outcomes research studies. However, methodology for scalable engagement in research management particularly activities such as operationalization of principles and setting of priorities is not well-developed. The objective of this study is to describe a novel approach for scalable stakeholder engagement in research aligned with the Patient-Centered Outcomes Research Institute (PCORI) engagement principles, which was evaluated in a national clinical data research network.

MATERIALS AND METHODS

Patient, patient advocate, clinician, and researcher stakeholders were recruited from clinical sites, as well as social media sites related to the 3 conditions of focus, heart failure, obesity, and Kawasaki disease. The engagement strategy was designed, implemented, and mapped to the PCORI engagement principles. Evaluation included internal assessment and quantitative measures of online engagement.

RESULTS

We operationalized the PCORI principles with 12 stakeholder engagement strategies and convened stakeholder advisory boards and online research prioritization panels to determine research priorities in a rigorous, deliberative process. A total of 46 advisors (20 patients) and 339 panelists (159 patients) actively participated. There were not significant differences between patients and clinicians in level of online engagement. Nonetheless, while patients reported a slightly greater challenge with following online discussion, they overall had a more favorable opinion about use of the online format.

DISCUSSION/CONCLUSION

An efficient way to engage large numbers of representative stakeholders in research is a necessary first step to assure the public of trustworthy use of data networks for health research. This paper describes a comprehensive approach to engagement in patient-centered outcomes research management that informs ongoing development of rigorous methodologies in this area.

Participant experiences with a new online modified-Delphi approach for engaging patients and caregivers in developing clinical guidelines

BACKGROUND

Patient engagement in clinical practice guideline (CPG) development has increased significantly in recent years. However, only a few patients and caregivers join CPG development groups.

OBJECTIVE

To describe participant experiences with a novel online, scalable approach for patient and caregiver engagement in CPG development.

DESIGN

We developed and tested the RAND/PPMD Patient-Centeredness Method (RPM), a novel online modified-Delphi approach to patient engagement in CPG development that consists of an optional idea generation round and two rating rounds interspersed with an online discussion round.

SETTING AND PARTICIPANTS

Using the online ExpertLens™ system, we ran 2 concurrent panels of patients and caregivers of individuals with Duchenne Muscular Dystrophy (DMD).

MAIN OUTCOME MEASURES

We surveyed all 95 panel participants about their participation and satisfaction with the process. We also conducted telephone interviews with 25 participants.

RESULTS

Participants expressed satisfaction with various ExpertLens™ features, noting that the system fostered lively interaction among them. Panelists also appreciated participating in an educational, interactive and convenient discussion forum that allowed them to share their opinions on the importance and acceptability of different recommendations. The RPM was viewed as empowering by patients and their caregivers who felt it would be useful for CPG developers.

DISCUSSION AND CONCLUSION

The results of our study show the overall participant satisfaction with a novel, scalable, online approach to engaging patients and caregivers in CPG development, which allows them to share their perspectives and lived experiences using a rigorous, systematic and iterative way that is similar to how clinicians provide their input.

Smart Technology Impact on Neighborhood Form for a Sustainable Doha

This study focuses on the significance of neighborhoods in a city as the smallest self-sufficient urban units akin to nuclear families in the larger society. Thus, improvements to the neighborhood form can improve livability, maximize walkability, and better the overall life quality of city inhabitants, which, in turn, advance a city’s overall environmental and social sustainability profile. The physical form (arrangement) and density of neighborhoods are primarily shaped by land ownership regulations, transportation, and communication means. The development of smart technology, especially in the fields of transportation and communication, has led to the reconsideration of some of sustainable urban form concepts such as neighborhoods. Low-density cities like Doha, Qatar can become both sustainable and livable, creating the basis for a sustainable city. First, this paper presents a critical review of Doha’s neighborhoods and advanced transport and communication technologies, in addition to the integration of these technologies with the physical form of neighborhoods. Then, the paper discusses the influences of such technologies on the future sustainability of the city of Doha and its neighborhoods. This investigation is based on a Delphi study to address the characteristics of a neighborhood and to identify two paradigms of ‘good’ design practice.

Core neurological examination items for neurology clerks: A modified Delphi study with a grass-roots approach

BACKGROUND

With the evolution of treatments for neurological diseases, the contents of core neurological examinations (NEs) for medical students may need to be modified. We aimed to establish a consensus on the core NE items for neurology clerks and compare viewpoints between different groups of panelists.

METHODS

First, a pilot group proposed the core contents of NEs for neurology clerks. The proposed core NE items were then subject to a modified web-based Delphi process using the online software "SurveyMonkey". A total of 30 panelists from different backgrounds (tutors or learners, neurologists or non-neurologists, community hospitals or medical centers, and different academic positions) participated in the modified Delphi process. Each panelist was asked to agree or disagree on the inclusion of each item using a 9-point Likert scale and was encouraged to provide feedback. We also compared viewpoints between different groups of panelists using the Mann-Whitney U test.

RESULTS

Eighty-three items were used for the first round of the Delphi process. Of them, 18 without consensus of being a core NE item for the neurology clerks in the first round and another 14 items suggested by the panelists were further discussed in the second round. Finally, 75 items with different grades were included in the recommended NE items for neurology clerks.

CONCLUSIONS

Our findings provide a reference regarding the core NE items for milestone development for neurology clerkships. We hope that prioritizing the NE items in this order can help medical students to learn NE more efficiently.