A process for systematically reviewing the literature: providing the research evidence for public health nursing interventions

BACKGROUND

Several groups have outlined methodologies for systematic literature reviews of the effectiveness of interventions. The Effective Public Health Practice Project (EPHPP) began in 1998. Its mandate is to provide research evidence to guide and support the Ontario Ministry of Health in outlining minimum requirements for public health services in the province. Also, the project is expected to disseminate the results provincially, nationally, and internationally. Most of the reviews are relevant to public health nursing practice.

AIMS

This article describes four issues related to the systematic literature reviews of the effectiveness of public health nursing interventions: (1) the process of systematically reviewing the literature, (2) the development of a quality assessment instrument, (3) the results of the EPHPP to date, and (4) some results of the dissemination strategies used.

METHODS

The eight steps of the systematic review process including question formulation, searching and retrieving the literature, establishing relevance criteria, assessing studies for relevance, assessing relevant studies for methodological quality, data extraction and synthesis, writing the report, and dissemination are outlined. Also, the development and assessment of content and construct validity and intrarater reliability of the quality assessment questionnaire used in the process are described.

RESULTS

More than 20 systematic reviews have been completed. Content validity was ascertained by the use of a number of experts to review the questionnaire during its development. Construct validity was demonstrated through comparisons with another highly rated instrument. Intrarater reliability was established using Cohen's Kappa. Dissemination strategies used appear to be effective in that professionals report being aware of the reviews and using them in program planning/policymaking decisions.

CONCLUSIONS

The EPHPP has demonstrated the ability to adapt the most current methods of systematic literature reviews of effectiveness to questions related to public health nursing. Other positive outcomes from the process include the development of a critical mass of public health researchers and practitioners who can actively participate in the process, and the work on dissemination has been successful in attracting external funds. A program of research in this area is being developed.

Updated method guidelines for systematic reviews in the cochrane collaboration back review group

STUDY DESIGN

Descriptive method guidelines.

OBJECTIVES

To help reviewers design, conduct, and report reviews of trials in the field of back and neck pain.

SUMMARY OF BACKGROUND DATA

In 1997, the Cochrane Collaboration Back Review Group published method guidelines for systematic reviews. Since its publication, new methodologic evidence emerged and more experience was acquired in conducting reviews.

METHODS

All reviews and protocols of the Back Review Group were assessed for compliance with the 1997 method guidelines. Also, the most recent version of the Cochrane Handbook (4.1) was checked for new recommendations. In addition, some important topics that were not addressed in the 1997 method guidelines were included (e.g., methods for qualitative analysis, reporting of conclusions, and discussion of clinical relevance of the results). In May 2002, preliminary results were presented and discussed in a workshop. In two rounds, a list of all possible recommendations and the final draft were circulated for comments among the editors of the Back Review Group.

RESULTS

The recommendations are divided in five categories: literature search, inclusion criteria, methodologic quality assessment, data extraction, and data analysis. Each recommendation is classified in minimum criteria and further guidance. Additional recommendations are included regarding assessment of clinical relevance, and reporting of results and conclusions.

CONCLUSIONS

Systematic reviews need to be conducted as carefully as the trials they report and, to achieve full impact, systematic reviews need to meet high methodologic standards.

Critical Analysis of Strategies for Determining Rigor in Qualitative Inquiry

Criteria for determining the trustworthiness of qualitative research were introduced by Guba and Lincoln in the 1980s when they replaced terminology for achieving rigor, reliability, validity, and generalizability with dependability, credibility, and transferability. Strategies for achieving trustworthiness were also introduced. This landmark contribution to qualitative research remains in use today, with only minor modifications in format. Despite the significance of this contribution over the past four decades, the strategies recommended to achieve trustworthiness have not been critically examined. Recommendations for where, why, and how to use these strategies have not been developed, and how well they achieve their intended goal has not been examined. We do not know, for example, what impact these strategies have on the completed research. In this article, I critique these strategies. I recommend that qualitative researchers return to the terminology of social sciences, using rigor, reliability, validity, and generalizability. I then make recommendations for the appropriate use of the strategies recommended to achieve rigor: prolonged engagement, persistent observation, and thick, rich description; inter-rater reliability, negative case analysis; peer review or debriefing; clarifying researcher bias; member checking; external audits; and triangulation.

Key challenges for delivering clinical impact with artificial intelligence

BACKGROUND

Artificial intelligence (AI) research in healthcare is accelerating rapidly, with potential applications being demonstrated across various domains of medicine. However, there are currently limited examples of such techniques being successfully deployed into clinical practice. This article explores the main challenges and limitations of AI in healthcare, and considers the steps required to translate these potentially transformative technologies from research to clinical practice.

MAIN BODY

Key challenges for the translation of AI systems in healthcare include those intrinsic to the science of machine learning, logistical difficulties in implementation, and consideration of the barriers to adoption as well as of the necessary sociocultural or pathway changes. Robust peer-reviewed clinical evaluation as part of randomised controlled trials should be viewed as the gold standard for evidence generation, but conducting these in practice may not always be appropriate or feasible. Performance metrics should aim to capture real clinical applicability and be understandable to intended users. Regulation that balances the pace of innovation with the potential for harm, alongside thoughtful post-market surveillance, is required to ensure that patients are not exposed to dangerous interventions nor deprived of access to beneficial innovations. Mechanisms to enable direct comparisons of AI systems must be developed, including the use of independent, local and representative test sets. Developers of AI algorithms must be vigilant to potential dangers, including dataset shift, accidental fitting of confounders, unintended discriminatory bias, the challenges of generalisation to new populations, and the unintended negative consequences of new algorithms on health outcomes.

CONCLUSION

The safe and timely translation of AI research into clinically validated and appropriately regulated systems that can benefit everyone is challenging. Robust clinical evaluation, using metrics that are intuitive to clinicians and ideally go beyond measures of technical accuracy to include quality of care and patient outcomes, is essential. Further work is required (1) to identify themes of algorithmic bias and unfairness while developing mitigations to address these, (2) to reduce brittleness and improve generalisability, and (3) to develop methods for improved interpretability of machine learning predictions. If these goals can be achieved, the benefits for patients are likely to be transformational.

Unpublished rating scales: a major source of bias in randomised controlled trials of treatments for schizophrenia

BACKGROUND

A recent review suggested an association between using unpublished scales in clinical trials and finding significant results.

AIMS

To determine whether such an association existed in schizophrenia trials.

METHOD

Three hundred trials were randomly selected from the Cochrane Schizophrenia Group's Register. All comparisons between treatment groups and control groups using rating scales were identified. The publication status of each scale was determined and claims of a significant treatment effect were recorded.

RESULTS

Trials were more likely to report that a treatment was superior to control when an unpublished scale was used to make the comparison (relative risk 1.37 (95% CI 1.12-1.68)). This effect increased when a 'gold-standard' definition of treatment superiority was applied (RR 1.94 (95% CI 1.35-2.79)). In non-pharmacological trials, one-third of 'gold-standard' claims of treatment superiority would not have been made if published scales had been used.

CONCLUSIONS

Unpublished scales are a source of bias in schizophrenia trials.

Validation of an index of the quality of review articles

The objective of this study was to assess the validity of an index of the scientific quality of research overviews, the Overview Quality Assessment Questionnaire (OQAQ). Thirty-six published review articles were assessed by 9 judges using the OQAQ. Authors reports of what they had done were compared to OQAQ ratings. The sensibility of the OQAQ was assessed using a 13 item questionnaire. Seven a priori hypotheses were used to assess construct validity. The review articles were drawn from three sampling frames: articles highly rated by criteria external to the study, meta-analyses, and a broad spectrum of medical journals. Three categories of judges were used to assess the articles: research assistants, clinicians with research training and experts in research methodology, with 3 judges in each category. The sensibility of the index was assessed by 15 randomly selected faculty members of the Department of Clinical Epidemiology and Biostatistics at McMaster. Authors' reports of their methods related closely to ratings from corresponding OQAQ items: for each criterion, the mean score was significantly higher for articles for which the authors responses indicated that they had used more rigorous methods. For 10 of the 13 questions used to assess sensibility the mean rating was 5 or greater, indicating general satisfaction with the instrument. The primary shortcoming noted was the need for judgement in applying the index. Six of the 7 hypotheses used to test construct validity held true. The OQAQ is a valid measure of the quality of research overviews.

Chance and consensus in peer review

An experiment in which 150 proposals submitted to the National Science Foundation were evaluated independently by a new set of reviewers indicates that getting a research grant depends to a significant extent on chance. The degree of disagreement within the population of eligible reviewers is such that whether or not a proposal is funded depends in a large proportion of cases upon which reviewers happen to be selected for it. No evidence of systematic bias in the selection of NSF reviewers was found.

Consolidated Health Economic Evaluation Reporting Standards (CHEERS) 2022 Explanation and Elaboration: A Report of the ISPOR CHEERS II Good Practices Task Force

Health economic evaluations are comparative analyses of alternative courses of action in terms of their costs and consequences. The Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement, published in 2013, was created to ensure health economic evaluations are identifiable, interpretable, and useful for decision making. It was intended as guidance to help authors report accurately which health interventions were being compared and in what context, how the evaluation was undertaken, what the findings were, and other details that may aid readers and reviewers in interpretation and use of the study. The new CHEERS 2022 statement replaces the previous CHEERS reporting guidance. It reflects the need for guidance that can be more easily applied to all types of health economic evaluation, new methods and developments in the field, and the increased role of stakeholder involvement including patients and the public. It is also broadly applicable to any form of intervention intended to improve the health of individuals or the population, whether simple or complex, and without regard to context (such as healthcare, public health, education, and social care). This Explanation and Elaboration Report presents the new CHEERS 2022 28-item checklist with recommendations and explanation and examples for each item. The CHEERS 2022 statement is primarily intended for researchers reporting economic evaluations for peer-reviewed journals and the peer reviewers and editors assessing them for publication. Nevertheless, we anticipate familiarity with reporting requirements will be useful for analysts when planning studies. It may also be useful for health technology assessment bodies seeking guidance on reporting, given that there is an increasing emphasis on transparency in decision making.

Prevalence of articles with honorary authors and ghost authors in peer-reviewed medical journals

CONTEXT

Authorship in biomedical publications establishes accountability, responsibility, and credit. Misappropriation of authorship undermines the integrity of the authorship system, but accurate data on its prevalence are limited.

OBJECTIVES

To determine the prevalence of articles with honorary authors (named authors who have not met authorship criteria) and ghost authors (individuals not named as authors but who contributed substantially to the work) in peer-reviewed medical journals and to identify journal characteristics and article types associated with such authorship misappropriation.

DESIGN

Mailed, self-administered, confidential survey.

PARTICIPANTS

A total of 809 corresponding authors (1179 surveyed, 69% response rate) of articles published in 1996 in 3 peer-reviewed, large-circulation general medical journals (Annals of Internal Medicine, JAMA, and The New England Journal of Medicine) and 3 peer-reviewed, smaller-circulation journals that publish supplements (American Journal of Cardiology, American Journal of Medicine, and American Journal of Obstetrics and Gynecology).

MAIN OUTCOME MEASURES

Prevalence of articles with honorary authors and ghost authors, as reported by corresponding authors.

RESULTS

Of the 809 articles, 492 were original research reports, 240 were reviews and articles not reporting original data, and 77 were editorials. A total of 156 articles (1 9%) had evidence of honorary authors (range, 11%-25% among journals); 93 articles (11%) had evidence of ghost authors (range, 7%-16% among journals); and 13 articles (2%) had evidence of both. The prevalence of articles with honorary authors was greater among review articles than research articles (odds ratio [OR], 1.8; 95% confidence interval [CI], 1.2-2.6) but did not differ significantly between large-circulation and smaller-circulation journals (OR, 1.4; 95% CI, 0.96-2.03). Compared with similar-type articles in large-circulation journals, articles with ghost authors in smaller-circulation journals were more likely to be reviews (OR, 4.2; 95% CI, 1.5-13.5) and less likely to be research articles (OR, 0.49; 95% CI, 0.27-0.88).

CONCLUSION

A substantial proportion of articles in peer-reviewed medical journals demonstrate evidence of honorary authors or ghost authors.

Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals

CONTEXT

Review articles are important sources of information to help guide decisions by clinicians, patients, and other decision makers. Ideally, reviews should include strategies to minimize bias and to maximize precision and be reported so explicitly that any interested reader would be able to replicate them.

OBJECTIVE

To compare the methodological and reporting aspects of systematic reviews and meta-analyses published by the Cochrane Collaboration with those published in paper-based journals indexed in MEDLINE.

DATA SOURCES

The Cochrane Library, issue 2 of 1995, and a search of MEDLINE restricted to 1995.

STUDY SELECTION

All 36 completed reviews published in the Cochrane Database of Systematic Reviews and a randomly selected sample of 39 meta-analyses or systematic reviews published in journals indexed by MEDLINE in 1995.

DATA EXTRACTION

Number of authors, trials, and patients; trial sources; inclusion and exclusion criteria; language restrictions; primary outcome; trial quality assessment; heterogeneity testing; and effect estimates. Updating by 1997 was evaluated.

RESULTS

Reviews found in MEDLINE included more authors (median, 3 vs 2; P<.001), more trials (median, 13.5 vs 5; P<.001), and more patients (median, 1280 vs 528; P<.001) than Cochrane reviews. More Cochrane reviews, however, included a description of the inclusion and exclusion criteria (35/36 vs 18/39; P<.001) and assessed trial quality (36/36 vs 12/39; P<.001). No Cochrane reviews had language restrictions (0/36 vs 7/39; P<.01). There were no differences in sources of trials, heterogeneity testing, or description of effect estimates. By June 1997, 18 of 36 Cochrane reviews had been updated vs 1 of 39 reviews listed in MEDLINE.

CONCLUSIONS

Cochrane reviews appear to have greater methodological rigor and are more frequently updated than systematic reviews or meta-analyses published in paper-based journals.

Prevalence of post-COVID-19 symptoms in hospitalized and non-hospitalized COVID-19 survivors: A systematic review and meta-analysis

BACKGROUND

Single studies support the presence of several post-COVID-19 symptoms; however, no meta-analysis differentiating hospitalized and non-hospitalized patients has been published to date. This meta-analysis analyses the prevalence of post-COVID-19 symptoms in hospitalized and non-hospitalized patients recovered from COVID-19 .

METHODS

MEDLINE, CINAHL, PubMed, EMBASE, and Web of Science databases, as well as medRxiv and bioRxiv preprint servers were searched up to March 15, 2021. Peer-reviewed studies or preprints reporting data on post-COVID-19 symptoms collected by personal, telephonic or electronic interview were included. Methodological quality of the studies was assessed using the Newcastle-Ottawa Scale. We used a random-effects models for meta-analytical pooled prevalence of each post-COVID-19 symptom, and I² statistics for heterogeneity. Data synthesis was categorized at 30, 60, and ≥90 days after .

RESULTS

From 15,577 studies identified, 29 peer-reviewed studies and 4 preprints met inclusion criteria. The sample included 15,244 hospitalized and 9011 non-hospitalized patients. The methodological quality of most studies was fair. The results showed that 63.2, 71.9 and 45.9% of the sample exhibited ≥one post-COVID-19 symptom at 30, 60, or ≥90days after onset/hospitalization. Fatigue and dyspnea were the most prevalent symptoms with a pooled prevalence ranging from 35 to 60% depending on the follow-up. Other post-COVID-19 symptoms included cough (20-25%), anosmia (10-20%), ageusia (15-20%) or joint pain (15-20%). Time trend analysis revealed a decreased prevalence 30days after with an increase after 60days .

CONCLUSION

This meta-analysis shows that post-COVID-19 symptoms are present in more than 60% of patients infected by SARS-CoV‑2. Fatigue and dyspnea were the most prevalent post-COVID-19 symptoms, particularly 60 and ≥90 days after.

Standards for distribution models in biodiversity assessments

Demand for models in biodiversity assessments is rising, but which models are adequate for the task? We propose a set of best-practice standards and detailed guidelines enabling scoring of studies based on species distribution models for use in biodiversity assessments. We reviewed and scored 400 modeling studies over the past 20 years using the proposed standards and guidelines. We detected low model adequacy overall, but with a marked tendency of improvement over time in model building and, to a lesser degree, in biological data and model evaluation. We argue that implementation of agreed-upon standards for models in biodiversity assessments would promote transparency and repeatability, eventually leading to higher quality of the models and the inferences used in assessments. We encourage broad community participation toward the expansion and ongoing development of the proposed standards and guidelines.

Global reach of ageism on older persons' health: A systematic review

OBJECTIVE

Although there is anecdotal evidence of ageism occurring at both the structural level (in which societal institutions reinforce systematic bias against older persons) and individual level (in which older persons take in the negative views of aging of their culture), previous systematic reviews have not examined how both levels simultaneously influence health. Thus, the impact of ageism may be underestimated. We hypothesized that a comprehensive systematic review would reveal that these ageism levels adversely impact the health of older persons across geography, health outcomes, and time.

METHOD

A literature search was performed using 14 databases with no restrictions on region, language, and publication type. The systematic search yielded 13,691 papers for screening, 638 for full review, and 422 studies for analyses. Sensitivity analyses that adjusted for sample size and study quality were conducted using standardized tools. The study protocol is registered (PROSPERO CRD42018090857).

RESULTS

Ageism led to significantly worse health outcomes in 95.5% of the studies and 74.0% of the 1,159 ageism-health associations examined. The studies reported ageism effects in all 45 countries, 11 health domains, and 25 years studied, with the prevalence of significant findings increasing over time (p < .0001). A greater prevalence of significant ageism-health findings was found in less-developed countries than more-developed countries (p = .0002). Older persons who were less educated were particularly likely to experience adverse health effects of ageism. Evidence of ageism was found across the age, sex, and race/ethnicity of the targeters (i.e., persons perpetrating ageism).

CONCLUSION

The current analysis which included over 7 million participants is the most comprehensive review of health consequences of ageism to date. Considering that the analysis revealed that the detrimental impact of ageism on older persons' health has been occurring simultaneously at the structural and individual level in five continents, our systematic review demonstrates the pernicious reach of ageism.

Checklists for review articles

Preparing a review entails many judgments. The focus of the review must be decided. Studies that are relevant to the focus of the review must be identified, selected for inclusion and critically appraised. Information must be collected and synthesised from the relevant studies, and conclusions must be drawn. Checklists can help prevent important errors in this process. Reviewers, editors, content experts, and users of reviews all have a role to play in improving the quality of published reviews and promoting the appropriate use of reviews by decisionmakers. It is essential that both providers and users appraise the validity of review articles.

Ethical principles and standards for the conduct of human and animal biological rhythm research

Most research papers published in Chronobiology International report the findings of investigations conducted on laboratory animals and human beings. The Journal, its editors and the publication committee endorse the compliance of investigators to the principles of the Declaration of Helsinki of the World Medical Association relating to the conduct of ethical research on human beings and the Guide for the Care and Use of Laboratory Animals of the Institute for Laboratory Animal Research of the National Research Council relating to the conduct of ethical research on laboratory and other animals. Chronobiology International requires that submitted manuscripts reporting the findings of human and animal research conform to the respective policy and mandates of the Declaration of Helsinki and the Guide for the Care and Use of Laboratory Animals. The peer review of manuscripts will thus include judgment of whether or not the involved research methods conform to the standards of good research practice. This article outlines the basic expectations for the methods of human and animal biological rhythm research, both from the perspective of the fundamental criteria necessary for quality chronobiology investigation and from the perspective of humane and ethical research on human beings and animals.

Accuracy of adolescent self-report of height and weight in assessing overweight status: a literature review

OBJECTIVE

To examine the accuracy of self-reported height and weight data to classify adolescent overweight status. Self-reported height and weight are commonly used with minimal consideration of accuracy.

DATA SOURCES

Eleven studies (4 nationally representative, 7 convenience sample or locally based).

STUDY SELECTION

Peer-reviewed articles of studies conducted in the United States that compared self-reported and directly measured height, weight, and/or body mass index data to classify overweight among adolescents.

MAIN EXPOSURES

Self-reported and directly measured height and weight.

MAIN OUTCOME MEASURES

Overweight prevalence; missing data, bias, and accuracy.

RESULTS

Studies varied in examination of bias. Sensitivity of self-reported data for classification of overweight ranged from 55% to 76% (4 of 4 studies). Overweight prevalence was -0.4% to -17.7% lower when body mass index was based on self-reported data vs directly measured data (5 of 5 studies). Females underestimated weight more than males (ranges, -4.0 to -1.0 kg vs -2.6 to 1.5 kg, respectively) (9 of 9 studies); overweight individuals underestimated weight more than nonoverweight individuals (6 of 6 studies). Missing self-reported data ranged from 0% to 23% (9 of 9 studies). There was inadequate information on bias by age and race/ethnicity.

CONCLUSIONS

Self-reported data are valuable if the only source of data. However, self-reported data underestimate overweight prevalence and there is bias by sex and weight status. Lower sensitivities of self-reported data indicate that one-fourth to one-half of those overweight would be missed. Other potential biases in self-reported data, such as across subgroups, need further clarification. The feasibility of collecting directly measured height and weight data on a state/community level should be explored because directly measured data are more accurate.

The quality of medical care

This article classifies the major approaches to the assessment of the process and outcomes of medical care. The apparent need to safeguard and enhance the quality of care has led to the institution of mechanisms that subject care to constant review so that deficiencies may be found and corrected. The article reviews the developments that led to the involvement of the federal government in this activity through its sponsorship of professional standards review organizations (PSRO's). The major features of the PSRO's are described and their possible effects discussed. It is too early to say how the PSRO's will fare, but should they fail to accomplish their objectives the pressure for more radical solutions will be difficult to resist.

A Guideline for Reporting Mediation Analyses of Randomized Trials and Observational Studies: The AGReMA Statement

Importance

Mediation analyses of randomized trials and observational studies can generate evidence about the mechanisms by which interventions and exposures may influence health outcomes. Publications of mediation analyses are increasing, but the quality of their reporting is suboptimal.

Objective

To develop international, consensus-based guidance for the reporting of mediation analyses of randomized trials and observational studies (A Guideline for Reporting Mediation Analyses; AGReMA).

Design, Setting, and Participants

The AGReMA statement was developed using the Enhancing Quality and Transparency of Health Research (EQUATOR) methodological framework for developing reporting guidelines. The guideline development process included (1) an overview of systematic reviews to assess the need for a reporting guideline; (2) review of systematic reviews of relevant evidence on reporting mediation analyses; (3) conducting a Delphi survey with panel members that included methodologists, statisticians, clinical trialists, epidemiologists, psychologists, applied clinical researchers, clinicians, implementation scientists, evidence synthesis experts, representatives from the EQUATOR Network, and journal editors (n = 19; June-November 2019); (4) having a consensus meeting (n = 15; April 28-29, 2020); and (5) conducting a 4-week external review and pilot test that included methodologists and potential users of AGReMA (n = 21; November 2020).

Results

A previously reported overview of 54 systematic reviews of mediation studies demonstrated the need for a reporting guideline. Thirty-three potential reporting items were identified from 3 systematic reviews of mediation studies. Over 3 rounds, the Delphi panelists ranked the importance of these items, provided 60 qualitative comments for item refinement and prioritization, and suggested new items for consideration. All items were reviewed during a 2-day consensus meeting and participants agreed on a 25-item AGReMA statement for studies in which mediation analyses are the primary focus and a 9-item short-form AGReMA statement for studies in which mediation analyses are a secondary focus. These checklists were externally reviewed and pilot tested by 21 expert methodologists and potential users, which led to minor adjustments and consolidation of the checklists.

Conclusions and Relevance

The AGReMA statement provides recommendations for reporting primary and secondary mediation analyses of randomized trials and observational studies. Improved reporting of studies that use mediation analyses could facilitate peer review and help produce publications that are complete, accurate, transparent, and reproducible.

The PAR Index (Peer Assessment Rating): methods to determine outcome of orthodontic treatment in terms of improvement and standards

In orthodontics it is important to objectively assess whether a worthwhile improvement has been achieved in terms of overall alignment and occlusion for an individual patient or the greater proportion of a practitioner's caseload. An objective measure is described that has been validated against the subjective opinions of 74 dentists. Using the weighted PAR Index it was revealed that at least a 30 per cent reduction in PAR score is required for a case to be considered as 'improved' and a change of 22 PAR points to bring about 'great improvement'. For a practitioner to demonstrate high standards the proportion of an individual's case load falling in the 'worse or no different' category should be negligible and the mean reduction should be as high as possible (e.g. greater than 70 per cent). If the mean percentage reduction in PAR score is high and the proportion of cases that have been 'greatly improved' is also high, this indicates that the practitioner is treating a great proportion of cases with a clear need for treatment to a high standard.

Physician ratings of appropriate indications for six medical and surgical procedures

We convened three panels of physicians to rate the appropriateness of a large number of indications for performing a total of six medical and surgical procedures. The panels followed a modified Delphi process. Panelists separately assigned initial ratings, then met in Santa Monica, California where they received reports showing their initial ratings and the distribution of the other panelists ratings. They discussed the indications and revised the indications lists, then individually assigned final ratings. There was generally better agreement on the final ratings than on the initial ratings. Based on reasonable criteria for agreement and disagreement, and excluding one outlying procedure, the panelists agreed on ratings for 42 to 56 per cent of the indications, and disagreed on 11 to 29 per cent.