The Weight of Surgical Knowledge: Navigating Information Overload

The Operating and Anesthetic Reference Assistant (OARA): A fine-tuned large language model for resident teaching

OBJECTIVE

This study aimed to fine-tune a large language model (LLM) for domain-specific text generation in surgical and anesthesia residency education.

SUMMARY BACKGROUND DATA

With growing interest in artificial intelligence (AI) for medical training, the potential of LLMs to transform residency education is explored.

METHODS

The 7-billion parameter base model "Vicuna v1.5" was trained on 266,342 lines of text from 821 peer-reviewed documents. We evaluated the model with 150 surgical or anesthesia queries and assessed accuracy, token count, and inference speed across various reasoning tasks. Tests of significance were conducted using ANOVA and chi-square analysis.

RESULTS

Our model achieved 65.3% accuracy, excelling in surgical case-based tasks. We found no significant difference in accuracy between knowledge domains (P=0.081), though longer response generation demonstrated poorer accuracy, with significant accuracy variation based on output length (P ​= ​0.002).

CONCLUSIONS

LLMs show potential in enhancing residency education. Our model's efficiency and task-specific accuracy highlights such promise, though limits in parameter count diminishes accuracy of longer response generation. Our findings showcase how AI may be integrated effectively within future residency training.

American Black Authorship Has Decreased Across All Clinical Specialties Despite an Increasing Number of Black Physicians Between 1990 and 2020 in the USA

PURPOSE

Many diversity, equity, and inclusion initiatives assume that attainment of a racially diverse healthcare workforce will translate to increased diversity elsewhere in the healthcare system (e.g., leadership roles or academic authorship). We sought to investigate these trends over time by examining the evolution of physician demographics in the USA, in concert with demographic changes in US authorship in US medical journals from 1990 to 2020 across 25 specialties.

METHODS

We reviewed all articles indexed in PubMed, with a primary author affiliation located in the USA and limited to journals based in the USA, relative to the proportion of medical professionals in the CMS National Provider Registry. We employed a previously peer-reviewed/validated algorithm called "averaging-of-proportions" that probabilistically predicts racial identity from surname using the US Census to assess the relationship between diversity among medical professionals and diversity in medical journal authorship.

RESULTS

Data reveals a sharp disconnect between the demographic breakdown of physicians and authors. Despite an increase in the number of Black physicians (from 8.5% in 2005 to 9.1% in 2020), there has been a decrease in Black early-career authorship from 7.2% in 1990 to 5.8% in 2020. The percentage of Black early-career authors across all specialties in 2020 is lower than the average per specialty in 1990. Similar trends were noted for Black senior authorship, decreasing from 7.6% in 1990 to 6.2% in 2020, as well as a plateau in Hispanic authorship over the same time interval despite an increasing number of Hispanic physicians.

CONCLUSION

Modest advances in physician diversity have not translated to increased diversity in academic authorship. Increasing diversity requires initiatives focused beyond recruitment of underrepresented minorities to medical schools or residencies.

Modeling the Adoption of 5760 Clinical Studies Into Practice Across 5 Surgical Specialties

INTRODUCTION

No studies exist that explore the factors that influence the process of synthesizing new knowledge into perioperative standards of care and the operating room. We sought to model the adoption of clinical research into surgical practice and identify modifiable factors influencing the latency of this translation.

METHODS

We created a data set comprised of all UpToDate articles between 2011 and 2020, sampled at 3-mo intervals, to explore how research is incorporated at the point-of-care (POC)-studying 5760 new references from 204 journals across five surgical specialties, compared to all uncited articles published during the same interval. UpToDate authors serve as specialty curators of the vast surgical literature, with an audience of more than a million clinicians in over 180 countries across 3200 institutions. Unlike society guidelines, UpToDate also provides the necessary granularity to quantify the time in bringing research to the bedside. Our main outcomes are citation rates and time-to-citation, split by specialty, journal, article type, and topics. We also model the influence of impact factor, geography, and funding and, finally, propose new impact indices to help with prioritizing surgical literature.

RESULTS

We highlight variation in adoption of clinical research by specialty. We show, despite representing a lower quality of evidence, surgical case reports are one of the most cited article types. Furthermore, most clinical trials (94%-100%) in surgical journals are never incorporated into POC reference lists. While few, pragmatic trials were the most likely to be cited of any article type in any surgical specialty (40%). Journal impact factor did not correlate with time-to-citation or proportion of articles cited in three of five surgical specialties, suggesting differences in how specialties synthesize/value research from specialty journals. Our two metrics, the Clinical Relevancy and Immediacy Indices, were defined to capture this impact/relevance to surgical practice. Of the five surgical subspecialties, gynecology references were >5-fold more likely to get cited, had a larger fraction of higher quality evidence incorporated, and demonstrated more success with POC adoption of practice guidelines. We also quantified the cost of translating research to surgical practice per specialty and generated maps that highlight institutions successful in translating research to the POC. The higher expenditure of National Institutes of Health funding in gynecology may reflect the cost of higher quality research per citation.

CONCLUSIONS

Understanding translational latency is the first step to exposing blocks that slow the adoption of research into everyday surgical practice and to understanding why increasing research funding has not yielded comparative gains in surgical outcomes. Our approach reveals new methods to monitoring the efficiency of research investments and evaluating the efficacy of policies influencing the translation of research to surgical practice.

Challenges in closing the gap between evidence and practice: International survey of institutional surgical stabilization of rib fractures guidelines

BACKGROUND

Surgical stabilization of rib fractures (SSRF) has gained increasing interest over the past decade, yet few candidates who could benefit from SSRF undergo operative management. We conducted an international survey of institutional SSRF guidelines comparing congruence between practice and contemporary evidence. We hypothesized that few guidelines reflect comprehensive evidence to facilitate standardized patient selection, operation, and postoperative management.

METHODS

A request for institutional rib fracture guidelines was distributed from the Chest Wall Injury Society. Surgical stabilization of rib fractures-specific guideline contents were extracted using a priori-designed extraction sheets and compared against 28 SSRF evidence-based recommendations outlined by a panel of 14 international experts. Fisher's exact test compared the proportion of strong and weak evidence-based recommendations specified within a majority of institutional guidelines to evaluate whether strength of evidence is associated with implementation.

RESULTS

A total of 36 institutions from 3 countries submitted institutional rib fracture management guidelines, among which 30 had SSRF-specific guidance. Twenty-eight guidelines (93%) listed at least one injury pattern criteria as an indication for SSRF, while 22 (73%) listed pain and 21 (70%) listed impaired respiratory function as other indications. Quantitative pain and respiratory function impairment thresholds that warrant SSRF varied across institutions. Few guidelines specified nonacute indications for SSRF or perioperative considerations. Seven guidelines (23%) detailed postoperative management but recommended timing and interval for follow-up varied. Overall, only 3 of the 28 evidence-based SSRF recommendations were specified within a majority of institutional practice guidelines. There was no statistically significant association ( p = 0.99) between the strength of recommendation and implementation within institutional guidelines.

CONCLUSION

Institutional SSRF guidelines do not reflect the totality of evidence available in contemporary literature. Guidelines are especially important for emerging interventions to ensure standardized care delivery and minimize low-value care. Consensus effort is needed to facilitate adoption and dissemination of evidence-based SSRF practices.

LEVEL OF EVIDENCE

Therapeutic/Care Management; Level IV.

Bibliometric analysis of the classic cited papers in the American Journal of Surgery: Citation recapitulates surgical history

BACKGROUND

We performed a bibliometric analysis of the American Journal of Surgery (AJS) to identify, characterize and place within a historical context, its published classic cited papers (CCPs).

METHODS

Bibliometric data from papers published in the AJS between January 1, 1945, and December 31, 2021 was extracted from the Web of Science database. Analysis was performed utilizing Bibliometrix and VOSViewer software.

RESULTS

27,070 papers were published in the AJS over the study period. There were 16 CCPs, including 5 Top CCPs, identified. Review of the Top CCPs reveals that they are based on careful clinical observations, innovation and generally build on prior published work. Top CCPs usually are specific to a particular diagnosis or a commonly performed procedure, as such papers frequently present a scoring or classification system, or important details related to new operative approaches or techniques.

CONCLUSIONS

Bibliometric study of the AJS has allowed for identification, characterization and appreciation of many of the key changes that have occurred in the discipline throughout the history of modern surgery.