Validity of a Structured Method of Selecting Abstracts for a Plastic Surgical Scientific Meeting

Criteria for selection and classification of studies in medical events

OBJECTIVE

The aim of this study was to evaluate the impact of study methodology and evaluation type on the selection of studies during the presentation of scientific events.

METHODS

A prospective, observational, transversal approach was applied to a cohort of studies that were submitted for presentation at the 2021 Brazilian Breast Cancer Symposium. Three forms of criteria (CR) were presented. CR1 was based on six criteria (method, ethics, design, originality, promotion, and social contribution); CR2 graded the studies from 0 to 10 for each study, and CR3 was based on five criteria (presentation, method, originality, scientific knowledge, and social contribution). To evaluate the item correlation, Cronbach's alpha and factorial analysis were performed. For the evaluation of differences between the tests, we used the Kruskal-Wallis and post-hoc Dunn tests. To determine the differences in the study classifications, we used the Friedman test and Namenyi's all-pairs comparisons.

RESULTS

A total of 122 studies were evaluated. There was also a good correlation with the items concerning criterion 1 (α=0.730) and 3 (α=0.937). Evaluating CR1 methodology, study design and social contribution (p=0.741) represents the main factor and CR3 methodology, and the scientific contribution (p=0.994) represents the main factor. The Kruskal-Wallis test showed differences in the results (p<0.001) for all the criteria that were used [CR1-CR2 (p<0.001), CR1-CR3 (p<0.001), and CR2-CR3 (p=0.004)]. The Friedman test showed differences in the ranking of the studies (p<0.001) for all studies (p<0.01).

CONCLUSION

Methodologies that use multiple criteria show good correlation and should be taken into account when ranking the best studies.

How the ESPU grades clinical abstracts

The ability to consistently review abstracts in an unbiased and objective fashion is a skill that most academics hope to master. However, robust standardized rating systems are sparse, with most scientific boards leaving the task of rating abstracts poorly defined and at the whim of the reviewer. In an effort to bring consistency to this process, in 2013, the ESPU board adopted an abstract rating system that has been previously used in the field of plastic surgery and orthopedics. (van der Steen et al., 2004; Poolman et al., 2007). The aim of this manuscript is to outline this practice.

How do Medical Societies Select Science for Conference Presentation? How Should They?

Introduction

Nothing has been published to describe the practices of medical societies in choosing abstracts for presentations at their annual meetings. We surveyed medical societies to determine their practices, and also present a theoretical analysis of the topic.

Methods

We contacted a convenience sample of large U.S. medical conferences, and determined their approach to choosing abstracts. We obtained information from web sites, telephone, and email. Our theoretical analysis compares values-based and empirical approaches for scoring system development.

Results

We contacted 32 societies and obtained data on 28 (response rate 88%). We excluded one upon learning that research was not presented at its annual meeting, leaving 27 for analysis. Only 2 (7%) made their abstract scoring process available to submitters. Reviews were blinded in most societies (21;78%), and all but one asked reviewers to recuse themselves for conflict of interest (96%). All required ≥3 reviewers. Of the 24 providing information on how scores were generated, 21 (88%) reported using a single gestalt score, and three used a combined score created from pooled domain-specific sub-scores. We present a framework for societies to use in choosing abstracts, and demonstrate its application in the development of a new scoring system.

Conclusions

Most medical societies use subjective, gestalt methods to select research for presentation at their annual meetings and do not disclose to submitters the details of how abstracts are chosen. We present a new scoring system that is transparent to submitters and reviewers alike with an accompanying statement of values and ground rules. We discuss the challenges faced in selecting abstracts for a large scientific meeting and share the values and practical considerations that undergird the new system.

Decreasing the ceiling effect in assessing meeting quality at an academic professional meeting

BACKGROUND

The psychometric properties of evaluations at academic meetings have not been well studied.

PURPOSE

To explore the ceiling effect in the evaluation of quality of a professional meeting and whether a change in the scale labels would decrease the ceiling effect.

METHODS

Cross-sectional study at two national meetings (2009-2010), attendees completed the evaluation on paper forms or online (5-point Likert scale).

RESULTS

Of 1,064 evaluations, the mean session ratings was higher among respondents to the paper version in 2009 (4.2; 95% confidence interval [CI], 4.1 to 4.3) as compared to online responders in 2009 (3.0; 95% CI, 2.9 to 3.1) or online responders in 2010 (3.0; 95% CI, 2.9 to 3.1)(p < 0.001).

CONCLUSION

A ceiling effect was present in the evaluation of an academic meeting. A change in the evaluation scale labels decreased the ceiling effect and increased evaluation variability.

Selecting the best clinical vignettes for academic meetings: should the scoring tool criteria be modified?

BACKGROUND

The performance of scoring tools to select clinical vignettes for presentation at academic meetings has never been assessed.

OBJECTIVE

To measure the psychometric properties of two scoring tools used to select clinical vignettes and to determine which elements are most helpful.

DESIGN

Prospective observational study.

PARTICIPANTS

Participants submitting clinical vignette abstracts, Society of General Internal Medicine annual meetings (2006-2007).

MAIN MEASURES

The 2006 scoring tool had three criteria (clarity, significance, and relevance) with brief general descriptors. The 2007 modified tool had five criteria (clarity, significance, relevance, teaching value, and overall assessment) with more detailed descriptors.

KEY RESULTS

A total of 938 clinical vignette abstracts were submitted (484 in 2006; 454 in 2007); 59.5% (n=288) were accepted for presentation. Cronbach's alpha was 0.81 for the 2006 three-item tool and 0.95 for the 2007 modified five-item tool. Simplifying the five-item 2007 tool to three items (relevance, teaching value, overall assessment) yielded a Cronbach's alpha of 0.95. The agreement between the number of clinical vignettes accepted for presentation (2007) using the average score of the five items with the number that would have been accepted using the simplified three items (relevance, teaching value, overall assessment) was almost perfect, with kappa 0.89 (95% confidence interval, 0.85 to 0.93).

CONCLUSIONS

Both scoring tools performed well, but a simplified tool with three items (relevance, teaching value, and overall assessment) and detailed descriptors was optimal; the simplified tool could improve the reviewer efficiency and quality of clinical vignettes presented at national meetings.

Assessment of abstracts submitted to the annual scientific meeting of the Royal Australian and New Zealand College of Radiologists

The process for selecting abstracts submitted for presentation at annual scientific meetings should ensure both the quality of these meetings and fairness to prospective presenters. The aim of the present study was to review the assessment of radiation oncology abstracts submitted for oral presentation to the 2004 Royal Australian and New Zealand College of Radiologists annual scientific meeting. Selection criteria were developed that were primarily focused on the subjective aspects of abstract quality. All research abstracts were reviewed blindly by five individual reviewers (four radiation oncologists and a statistician), scoring each abstract in five categories. The scores of three reviewers were used to select the top 30 general and top eight trainee entries. For comparison, cluster analysis using the scores of all five reviewers was used to group papers into two ranks. There was a strong correlation in total scores for each paper, between all reviewers. Similarly, the study design subscale was strongly correlated between all reviewers. Abstracts belonging to the first-rank cluster were generally selected. Most trainee entries would have been successful in being accepted into the general programme. The selection process described appears feasible and fair and may improve the quality of meetings.

Peer review interrater concordance of scientific abstracts: a study of anesthesiology subspecialty and component societies

Abstracts presented at anesthesiology subspeciality and component society meetings are chosen by peer review. We assessed this process by examining selection criteria and determining interrater concordance. For the societies studied, the level of reviewer agreement ranged from poor to moderate, i.e., slightly better than by chance alone. We hypothesize that having clearer evaluation criteria, scoring systems with interval scales, and assessment based on quality can strengthen the peer review process.

Full Publication of Papers Presented at the 1995 through 1999 European Association of Plastic Surgeons Annual Scientific Meetings: A Systemic Bibliometric Analysis

From the multitude of oral presentations at major medical meetings, the most informative and highest-quality studies make it to full publication in peer-reviewed journals. The rate of publication may be regarded as an indicator of the scientific level of the meeting. Study of the publication rates of consecutive annual meetings allows for the evaluation of the consistency of the scientific level of these meetings and for comparison with publication rates of other meetings in the same field of interest. To grade how useful any publication is to other authors, one can furthermore measure how frequently they cite it in their own publications. Finally, the time lag between oral presentation and full publication is of importance to both its authors and the audience at the meeting. The main objectives of this study were to determine the publication rate of papers of various fields of interest as presented at five consecutive annual meetings of the European Association of Plastic Surgeons (EURAPS) and the time lag between these presentations and their publication. The authors compared their overall findings to those reported for other surgical specialties. Moreover, they identified and classified the journals in which the full publications appeared as an indicator of the scientific value of the meeting. They conclude that a greater than average number of papers presented at the 1995 through 1999 annual EURAPS meetings went on to full publication in peer-reviewed journals. Among these journals, Plastic and Reconstructive Surgery was the best source for information presented at the meetings. Although approximately 90 percent of the publications appeared before 3 years had passed after a meeting, additional publications may be expected to appear even more than 6 years after the meeting. Given the high publication rate and the high average normalized impact factor of the journals in which the presentations appeared, the five studied EURAPS meetings overall had high scientific value.