Postoperative Opioid Prescribing via Rule-Based Guidelines Derived from In-Hospital Consumption: An Assessment of Efficacy Based on Postdischarge Opioid Use

BACKGROUND

Many institutions have developed operation-specific guidelines for opioid prescribing. These guidelines rarely incorporate in-hospital opioid consumption, which is highly correlated with consumption. We compare outcomes of several patient-centered approaches to prescribing that are derived from in-hospital consumption, including several experimental, rule-based prescribing guidelines and our current institutional guideline.

STUDY DESIGN

We performed a retrospective, cohort study of all adults undergoing surgery at a single-academic medical center. Several rule-based guidelines, derived from in-hospital consumption (quantity of opioids consumed within 24 hours of discharge), were used to specify the theoretical quantity of opioid prescribed on discharge. The efficacy of the experimental guidelines was compared with 3 references: an approximation of our institution's tailored prescribing guideline; prescribing all patients the typical quantity of opioids consumed for patients undergoing the same operation; and a representative rule-based, tiered framework. For each scenario, we calculated the penalized residual sum of squares (reflecting the composite deviation from actual patient consumption, with 15% penalty for overprescribing) and the proportion of opioids consumed relative to prescribed.

RESULTS

A total of 1,048 patients met inclusion criteria. Mean (SD) and median (interquartile range [IQR]) quantity of opioids consumed within 24 hours of discharge were 11.2 (26.9) morphine milligram equivalents and 0 (0 to 15) morphine milligram equivalents. Median (IQR) postdischarge consumption was 16 (0 to 150) morphine milligram equivalents. Our institutional guideline and the previously validated rule-based guideline outperform alternate approaches, with median (IQR) differences in prescribed vs consumed opioids of 0 (-60 to 27.25) and 37.5 (-37.5 to 37.5), respectively, corresponding to penalized residual sum of squares of 39,817,602 and 38,336,895, respectively.

CONCLUSIONS

Rather than relying on fixed quantities for defined operations, rule-based guidelines offer a simple yet effective method for tailoring opioid prescribing to in-hospital consumption.

An operational guide to translational clinical machine learning in academic medical centers

Few published data science tools are ever translated from academia to real-world clinical settings for which they were intended. One dimension of this problem is the software engineering task of turning published academic projects into tools that are usable at the bedside. Given the complexity of the data ecosystem in large health systems, this task often represents a significant barrier to the real-world deployment of data science tools for prospective piloting and evaluation. Many information technology companies have created Machine Learning Operations (MLOps) teams to help with such tasks at scale, but the low penetration of home-grown data science tools in regular clinical practice precludes the formation of such teams in healthcare organizations. Based on experiences deploying data science tools at two large academic medical centers (Beth Israel Deaconess Medical Center, Boston, MA; Mayo Clinic, Rochester, MN), we propose a strategy to facilitate this transition from academic product to operational tool, defining the responsibilities of the principal investigator, data scientist, machine learning engineer, health system IT administrator, and clinician end-user throughout the process. We first enumerate the technical resources and stakeholders needed to prepare for model deployment. We then propose an approach to planning how the final product will work from data extraction and analysis to visualization of model outputs. Finally, we describe how the team should execute on this plan. We hope to guide health systems aiming to deploy minimum viable data science tools and realize their value in clinical practice.

Evaluating capabilities of large language models: Performance of GPT-4 on surgical knowledge assessments

BACKGROUND

Artificial intelligence has the potential to dramatically alter health care by enhancing how we diagnose and treat disease. One promising artificial intelligence model is ChatGPT, a general-purpose large language model trained by OpenAI. ChatGPT has shown human-level performance on several professional and academic benchmarks. We sought to evaluate its performance on surgical knowledge questions and assess the stability of this performance on repeat queries.

METHODS

We evaluated the performance of ChatGPT-4 on questions from the Surgical Council on Resident Education question bank and a second commonly used surgical knowledge assessment, referred to as Data-B. Questions were entered in 2 formats: open-ended and multiple-choice. ChatGPT outputs were assessed for accuracy and insights by surgeon evaluators. We categorized reasons for model errors and the stability of performance on repeat queries.

RESULTS

A total of 167 Surgical Council on Resident Education and 112 Data-B questions were presented to the ChatGPT interface. ChatGPT correctly answered 71.3% and 67.9% of multiple choice and 47.9% and 66.1% of open-ended questions for Surgical Council on Resident Education and Data-B, respectively. For both open-ended and multiple-choice questions, approximately two-thirds of ChatGPT responses contained nonobvious insights. Common reasons for incorrect responses included inaccurate information in a complex question (n = 16, 36.4%), inaccurate information in a fact-based question (n = 11, 25.0%), and accurate information with circumstantial discrepancy (n = 6, 13.6%). Upon repeat query, the answer selected by ChatGPT varied for 36.4% of questions answered incorrectly on the first query; the response accuracy changed for 6/16 (37.5%) questions.

CONCLUSION

Consistent with findings in other academic and professional domains, we demonstrate near or above human-level performance of ChatGPT on surgical knowledge questions from 2 widely used question banks. ChatGPT performed better on multiple-choice than open-ended questions, prompting questions regarding its potential for clinical application. Unique to this study, we demonstrate inconsistency in ChatGPT responses on repeat queries. This finding warrants future consideration including efforts at training large language models to provide the safe and consistent responses required for clinical application. Despite near or above human-level performance on question banks and given these observations, it is unclear whether large language models such as ChatGPT are able to safely assist clinicians in providing care.

Mobilizing data during a crisis: Building rapid evidence pipelines using multi-institutional real world data

The COVID-19 pandemic generated tremendous interest in using real world data (RWD). Many consortia across the public and private sectors formed in 2020 with the goal of rapidly producing high-quality evidence from RWD to guide medical decision-making, public health priorities, and more. Experiences were gathered from five large consortia on rapid multi-institutional evidence generation during the COVID-19 pandemic. Insights have been compiled across five dimensions: consortium composition, governance structure and alignment of priorities, data sharing, data analysis, and evidence dissemination. The purpose of this piece is to offer guidance on building large-scale multi-institutional RWD analysis pipelines for future public health issues. The composition of each consortium was largely influenced by existing collaborations. A central set of priorities for evidence generation guided each consortium, however different approaches to governance emerged. Challenges surrounding limited access to clinical data due to various contributors were overcome in unique ways. While all consortia used different methods to construct and analyze patient cohorts ranging from centralized to federated approaches, all proved effective for generating meaningful real-world evidence. Actionable recommendations for clinical practice and public health agencies were made from translating insights from consortium analyses. Each consortium was successful in rapidly answering questions about COVID-19 diagnosis and treatment despite all taking slightly different approaches to data sharing and analysis. Leveraging RWD, leveraged in a manner that applies scientific rigor and transparency, can complement higher-level evidence and serve as an important adjunct to clinical trials to quickly guide policy and critical care, especially for a pandemic response.

An Institutional Curriculum for Opioid Prescribing Education: Outcomes From 2017 to 2022

INTRODUCTION

Prescription opioids, including those prescribed after surgery, have greatly contributed to the US opioid epidemic. Educating opioid prescribers is a crucial component of ensuring the safe use of opioids among surgical patients.

METHODS

An annual opioid prescribing education curriculum was implemented among new surgical prescribers at our institution between 2017 and 2022. The curriculum includes a single 75-min session which is comprised of several components: pain medications (dosing, indications, and contraindications); patients at high risk for uncontrolled pain and/or opioid misuse or abuse; patient monitoring and care plans; and state and federal regulations. Participants were asked to complete an opioid knowledge assessment before and after the didactic session.

RESULTS

Presession and postsession assessments were completed by 197 (89.6%) prescribers. Across the five studied years, the median presession score was 54.5%. This increased to 63.6% after completion of the curriculum, representing a median relative knowledge increase of 18.2%. The median relative improvement was greatest for preinterns and interns (18.2% for both groups); smaller improvements were observed for postgraduate year 2-5 residents (9.1%) and advanced practice providers (9.1%). On a scale of 1 to 10 (with 5 being comfortable), median (interquartile range) self-reported comfort in prescribing opioids increased from 3 (2-5) before education to 5 (4-6) after education (P < 0.001).

CONCLUSIONS

Each year, the curriculum substantially improved provider knowledge of and comfort in opioid prescribing. Despite increased national awareness of the opioid epidemic and increasing institutional initiatives to improve opioid prescribing practices, there was a sustained knowledge and comfort gap among new surgical prescribers. The observed effects of our opioid education curriculum highlight the value of a simple and efficient educational initiative.

Use of Open Science Practices in Surgical Journals

This cross-sectional study assesses the level of adoption of 5 new tools that promote high quality and transparency in surgical research.

Comparing Rationale for Opioid Prescribing Decisions after Surgery with Subsequent Patient Consumption: A Survey of the Highest Quartile of Prescribers

BACKGROUND

Opioid prescribing patterns, including those after surgery, have been implicated as a significant contributor to the US opioid crisis. A plethora of interventions-from nudges to reminders-have been deployed to improve prescribing behavior, but reasons for persistent outlier behavior are often unknown.

STUDY DESIGN

Our institution employs multiple prescribing resources and a near real-time, feedback-based intervention to promote appropriate opioid prescribing. Since 2019, an automated system has emailed providers when a prescription exceeds the 75th percentile of typical opioid consumption for a given procedure-as defined by institutional data collection. Emails include population consumption metrics and an optional survey on rationale for prescribing. Responses were analyzed to understand why providers choose to prescribe atypically large discharge opioid prescriptions. We then compared provider prescriptions against patient consumption.

RESULTS

During the study period, 10,672 eligible postsurgical patients were discharged; 2,013 prescriptions (29.4% of opioid prescriptions) exceeded our institutional guideline. Surveys were completed by outlier prescribers for 414 (20.6%) encounters. Among patients where both consumption data and prescribing rationale surveys were available, 35.2% did not consume any opioids after discharge and 21.5% consumed <50% of their prescription. Only 93 (39.9%) patients receiving outlier prescriptions were outlier consumers. Most common reasons for prescribing outlier amounts were attending preference (34%) and prescriber analysis of patient characteristics (34%).

CONCLUSIONS

The top quartile of opioid prescriptions did not align with, and often far exceeded, patient postdischarge opioid consumption. Providers cite assessment of patient characteristics as a common driver of decision-making, but this did not align with patient usage for approximately 50% of patients.

Clinical prediction tool pitfalls and considerations: Data and algorithms

In recent years, many surgical prediction models have been developed and published to augment surgeon decision-making, predict postoperative patient trajectories, and more. Collectively underlying all of these models is a wide variety of data sources and algorithms. Each data set and algorithm has its unique strengths, weaknesses, and type of prediction task for which it is best suited. The purpose of this piece is to highlight important characteristics of common data sources and algorithms used in surgical prediction model development so that future researchers interested in developing models of their own may be able to critically evaluate them and select the optimal ones for their study.

How surgeons use risk calculators and non-clinical factors for informed consent and shared decision making: A qualitative study

BACKGROUND

The discussion of risks, benefits, and alternatives to surgery with patients is a defining component of informed consent. As shared-decision making has become central to surgeon-patient communication, risk calculators have emerged as a tool to aid communication and decision-making. To optimize informed consent, it is necessary to understand how surgeons assess and communicate risk, and the role of risk calculators in this process.

METHODS

We conducted interviews with 13 surgeons from two institutions to understand how surgeons assess risk, the role of risk calculators in decision-making, and how surgeons approach risk communication during informed consent. We performed a qualitative analysis of interviews based on SRQR guidelines.

RESULTS

Our analysis yielded insights regarding (a) the landscape and approach to obtaining surgical consent; (b) detailed perceptions regarding the value and design of assessing and communicating risk; and (c) practical considerations regarding the future of personalized risk communication in decision-making. Above all, we found that non-clinical factors such as health and risk literacy are changing how surgeons assess and communicate risk, which diverges from traditional risk calculators.

CONCLUSION

Principally, we found that surgeons incorporate a range of clinical and non-clinical factors to risk stratify patients and determine how to optimally frame and discuss risk with individual patients. We observed that surgeons' perception of risk communication, and the importance of eliciting patient preferences to direct shared-decision making, did not consistently align with patient priorities. This study underscored criticisms of risk calculators and novel decision-aids - which must be addressed prior to greater adoption.

Evaluating Capabilities of Large Language Models: Performance of GPT4 on Surgical Knowledge Assessments

Background

Artificial intelligence (AI) has the potential to dramatically alter healthcare by enhancing how we diagnosis and treat disease. One promising AI model is ChatGPT, a large general-purpose language model trained by OpenAI. The chat interface has shown robust, human-level performance on several professional and academic benchmarks. We sought to probe its performance and stability over time on surgical case questions.

Methods

We evaluated the performance of ChatGPT-4 on two surgical knowledge assessments: the Surgical Council on Resident Education (SCORE) and a second commonly used knowledge assessment, referred to as Data-B. Questions were entered in two formats: open-ended and multiple choice. ChatGPT output were assessed for accuracy and insights by surgeon evaluators. We categorized reasons for model errors and the stability of performance on repeat encounters.

Results

A total of 167 SCORE and 112 Data-B questions were presented to the ChatGPT interface. ChatGPT correctly answered 71% and 68% of multiple-choice SCORE and Data-B questions, respectively. For both open-ended and multiple-choice questions, approximately two-thirds of ChatGPT responses contained non-obvious insights. Common reasons for inaccurate responses included: inaccurate information in a complex question (n=16, 36.4%); inaccurate information in fact-based question (n=11, 25.0%); and accurate information with circumstantial discrepancy (n=6, 13.6%). Upon repeat query, the answer selected by ChatGPT varied for 36.4% of inaccurate questions; the response accuracy changed for 6/16 questions.

Conclusion

Consistent with prior findings, we demonstrate robust near or above human-level performance of ChatGPT within the surgical domain. Unique to this study, we demonstrate a substantial inconsistency in ChatGPT responses with repeat query. This finding warrants future consideration and presents an opportunity to further train these models to provide safe and consistent responses. Without mental and/or conceptual models, it is unclear whether language models such as ChatGPT would be able to safely assist clinicians in providing care.

Artificial Intelligence-enabled Decision Support in Surgery: State-of-the-art and Future Directions

OBJECTIVE

To summarize state-of-the-art artificial intelligence-enabled decision support in surgery and to quantify deficiencies in scientific rigor and reporting.

BACKGROUND

To positively affect surgical care, decision-support models must exceed current reporting guideline requirements by performing external and real-time validation, enrolling adequate sample sizes, reporting model precision, assessing performance across vulnerable populations, and achieving clinical implementation; the degree to which published models meet these criteria is unknown.

METHODS

Embase, PubMed, and MEDLINE databases were searched from their inception to September 21, 2022 for articles describing artificial intelligence-enabled decision support in surgery that uses preoperative or intraoperative data elements to predict complications within 90 days of surgery. Scientific rigor and reporting criteria were assessed and reported according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews guidelines.

RESULTS

Sample size ranged from 163-2,882,526, with 8/36 articles (22.2%) featuring sample sizes of less than 2000; 7 of these 8 articles (87.5%) had below-average (<0.83) area under the receiver operating characteristic or accuracy. Overall, 29 articles (80.6%) performed internal validation only, 5 (13.8%) performed external validation, and 2 (5.6%) performed real-time validation. Twenty-three articles (63.9%) reported precision. No articles reported performance across sociodemographic categories. Thirteen articles (36.1%) presented a framework that could be used for clinical implementation; none assessed clinical implementation efficacy.

CONCLUSIONS

Artificial intelligence-enabled decision support in surgery is limited by reliance on internal validation, small sample sizes that risk overfitting and sacrifice predictive performance, and failure to report confidence intervals, precision, equity analyses, and clinical implementation. Researchers should strive to improve scientific quality.

Quantifying the Prognostic Value of Preoperative Surgeon Intuition: Comparing Surgeon Intuition and Clinical Risk Prediction as Derived from the American College of Surgeons NSQIP Risk Calculator

BACKGROUND

Surgical risk prediction models traditionally use patient attributes and measures of physiology to generate predictions about postoperative outcomes. However, the surgeon's assessment of the patient may be a valuable predictor, given the surgeon's ability to detect and incorporate factors that existing models cannot capture. We compare the predictive utility of surgeon intuition and a risk calculator derived from the American College of Surgeons (ACS) NSQIP.

STUDY DESIGN

From January 10, 2021 to January 9, 2022, surgeons were surveyed immediately before performing surgery to assess their perception of a patient's risk of developing any postoperative complication. Clinical data were abstracted from ACS NSQIP. Both sources of data were independently used to build models to predict the likelihood of a patient experiencing any 30-day postoperative complication as defined by ACS NSQIP.

RESULTS

Preoperative surgeon assessment was obtained for 216 patients. NSQIP data were available for 9,182 patients who underwent general surgery (January 1, 2017 to January 9, 2022). A binomial regression model trained on clinical data alone had an area under the receiver operating characteristic curve (AUC) of 0.83 (95% CI 0.80 to 0.85) in predicting any complication. A model trained on only preoperative surgeon intuition had an AUC of 0.70 (95% CI 0.63 to 0.78). A model trained on surgeon intuition and a subset of clinical predictors had an AUC of 0.83 (95% CI 0.77 to 0.89).

CONCLUSIONS

Preoperative surgeon intuition alone is an independent predictor of patient outcomes; however, a risk calculator derived from ACS NSQIP is a more robust predictor of postoperative complication. Combining intuition and clinical data did not strengthen prediction.

The digital transformation of surgery

Rapid advances in digital technology and artificial intelligence in recent years have already begun to transform many industries, and are beginning to make headway into healthcare. There is tremendous potential for new digital technologies to improve the care of surgical patients. In this piece, we highlight work being done to advance surgical care using machine learning, computer vision, wearable devices, remote patient monitoring, and virtual and augmented reality. We describe ways these technologies can be used to improve the practice of surgery, and discuss opportunities and challenges to their widespread adoption and use in operating rooms and at the bedside.

Open science practices in research published in surgical journals: A cross-sectional study

Open science practices are research tools used to improve research quality and transparency. These practices have been used by researchers in various medical fields, though the usage of these practices in the surgical research ecosystem has not been quantified. In this work, we studied the use of open science practices in general surgery journals. Eight of the highest-ranked general surgery journals by SJR2 were chosen and their author guidelines were reviewed. From each journal, 30 articles published between January 1, 2019 and August 11, 2021 were randomly chosen and analyzed. Five open science practices were measured (preprint publication prior to peer-reviewed publication, use of Equator guidelines, study protocol preregistration prior to peer-reviewed publication, published peer review, and public accessibility of data, methods, and/or code). Across all 240 articles, 82 (34%) used one or more open science practices. Articles in the International Journal of Surgery showed greatest use of open science practices, with a mean of 1.6 open science practices compared to 0.36 across the other journals (p<.001). Adoption of open science practices in surgical research remains low, and further work is needed to increase utilization of these tools.

The role of digital technology in surgical home hospital programs

Home hospital (HH), a care delivery model of providing hospital-grade care to patients in their homes, has become increasingly common in medical settings, though surgical uptake has been limited. HH programs have been shown to be safe and effective in a variety of medical contexts, with increased usage of this care pathway during the COVID-19 pandemic. Though surgical patients have unique clinical considerations, surgical Home Hospital (SHH) programs may have important benefits for this population. Various technologies exist for the delivery of hospital care in the home, such as clinical risk prediction models and remote patient monitoring platforms. Here, we use institutional experiences at Brigham and Women's Hospital (BWH) to discuss the utility of technology in enabling SHH programs and highlight current limitations. Additionally, we comment on the importance of data interoperability, access for all patients, and clinical workflow design in successfully implementing SHH programs.

Appraising the Quality of Development and Reporting in Surgical Prediction Models

This cross-sectional study uses the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis reporting guideline to assess 120 published studies about surgical prediction models.

Trends in Medical Management of Moderately to Severely Active Ulcerative Colitis: A Nationwide Retrospective Analysis

BACKGROUND

With an increasing number of therapeutic options available for the management of ulcerative colitis (UC), the variability in treatment and prescribing patterns is not well known. While recent guidelines have provided updates on how these therapeutic options should be used, patterns of long-term use of these drugs over the past 2 decades remain unclear.

METHODS

We analyzed a retrospective, nationwide cohort of more than 1.7 million prescriptions for trends in prescribing behaviors and to evaluate practices suggested in guidelines relating to ordering biologics, step-up therapy, and combination therapy. The primary outcome was 30-day steroid-free remission and secondary outcomes included hospitalization, cost, and additional steroid usage. A pipeline was created to identify cohorts of patients under active UC medical management grouped by prescribing strategies to evaluate comparative outcomes between strategies. Cox proportional hazards and multivariate regression models were utilized to assess postexposure outcomes and adjust for confounders.

RESULTS

Among 6 major drug categories, we noted major baseline differences in patient characteristics at first exposure corresponding to disease activity. We noted earlier use of biologics in patient trajectories (762 days earlier relative to UC diagnosis, 2018 vs 2008; P < .001) and greater overall use of biologics over time (2.53× more in 2018 vs 2008; P < .00001) . Among biologic-naive patients, adalimumab was associated with slightly lower rates of remission compared with infliximab or vedolizumab (odds ratio, 0.92; P < .005). Comparisons of patients with early biologic initiation to patients who transitioned to biologics from 5-aminosalicylic acid suggest lower steroid consumption for early biologic initiation (-761 mg prednisone; P < .001). Combination thiopurine-biologic therapy was associated with higher odds of remission compared with biologic monotherapy (odds ratio, 1.36; P = .01).

CONCLUSIONS

As biologic drugs have become increasingly available for UC management, they have increasingly been used at earlier stages of disease management. Large-scale analyses of prescribing behaviors provide evidence supporting early use of biologics compared with step-up therapy and use of thiopurine and biologic combination therapy.

Association of Postsurgical Opioid Refills for Patients With Risk of Opioid Misuse and Chronic Opioid Use Among Family Members

IMPORTANCE

The US health care system is experiencing a sharp increase in opioid-related adverse events and spending, and opioid overprescription may be a key factor in this crisis. Ambient opioid exposure within households is one of the known major dangers of overprescription.

OBJECTIVE

To quantify the association between the postsurgical initiation of prescription opioid use in opioid-naive patients and the subsequent prescription opioid misuse and chronic opioid use among opioid-naive family members.

DESIGN, SETTING, AND PARTICIPANTS

This cohort study was conducted using administrative data from the database of a US commercial insurance provider with more than 35 million covered individuals. Participants included pairs of patients who underwent surgery from January 1, 2008, to December 31, 2016, and their family members within the same household. Data were analyzed from January 1 to November 30, 2018.

EXPOSURES

Duration of opioid exposure and refills of opioid prescriptions received by patients after surgery.

MAIN OUTCOMES AND MEASURES

Risk of opioid misuse and chronic opioid use in family members were calculated using inverse probability weighted Cox proportional hazards regression models.

RESULTS

The final cohort included 843 531 pairs of patients and family members. Most pairs included female patients (445 456 [52.8%]) and male family members (442 992 [52.5%]), and a plurality of pairs included patients in the 45 to 54 years age group (249 369 [29.6%]) and family members in the 15 to 24 years age group (313 707 [37.2%]). A total of 3894 opioid misuse events (0.5%) and 7485 chronic opioid use events (0.9%) occurred in family members. In adjusted models, each additional opioid prescription refill for the patient was associated with a 19.2% (95% CI, 14.5%-24.0%) increase in hazard of opioid misuse in family members. The risk of opioid misuse appeared to increase only in households in which the patient obtained refills. Family members in households with any refill had a 32.9% (95% CI, 22.7%-43.8%) increased adjusted hazard of opioid misuse. When patients became chronic opioid users, the hazard ratio for opioid misuse among family members was 2.52 (95% CI, 1.68-3.80), and similar patterns were found for chronic opioid use.

CONCLUSIONS AND RELEVANCE

This cohort study found that opioid exposure was a household risk. Family members of a patient who received opioid prescription refills after surgery had an increased risk of opioid misuse and chronic opioid use.

Distinguishing Admissions Specifically for COVID-19 From Incidental SARS-CoV-2 Admissions: National Retrospective Electronic Health Record Study

BACKGROUND

Admissions are generally classified as COVID-19 hospitalizations if the patient has a positive SARS-CoV-2 polymerase chain reaction (PCR) test. However, because 35% of SARS-CoV-2 infections are asymptomatic, patients admitted for unrelated indications with an incidentally positive test could be misclassified as a COVID-19 hospitalization. Electronic health record (EHR)-based studies have been unable to distinguish between a hospitalization specifically for COVID-19 versus an incidental SARS-CoV-2 hospitalization. Although the need to improve classification of COVID-19 versus incidental SARS-CoV-2 is well understood, the magnitude of the problems has only been characterized in small, single-center studies. Furthermore, there have been no peer-reviewed studies evaluating methods for improving classification.

OBJECTIVE

The aims of this study are to, first, quantify the frequency of incidental hospitalizations over the first 15 months of the pandemic in multiple hospital systems in the United States and, second, to apply electronic phenotyping techniques to automatically improve COVID-19 hospitalization classification.

METHODS

From a retrospective EHR-based cohort in 4 US health care systems in Massachusetts, Pennsylvania, and Illinois, a random sample of 1123 SARS-CoV-2 PCR-positive patients hospitalized from March 2020 to August 2021 was manually chart-reviewed and classified as "admitted with COVID-19" (incidental) versus specifically admitted for COVID-19 ("for COVID-19"). EHR-based phenotyping was used to find feature sets to filter out incidental admissions.

RESULTS

EHR-based phenotyped feature sets filtered out incidental admissions, which occurred in an average of 26% of hospitalizations (although this varied widely over time, from 0% to 75%). The top site-specific feature sets had 79%-99% specificity with 62%-75% sensitivity, while the best-performing across-site feature sets had 71%-94% specificity with 69%-81% sensitivity.

CONCLUSIONS

A large proportion of SARS-CoV-2 PCR-positive admissions were incidental. Straightforward EHR-based phenotypes differentiated admissions, which is important to assure accurate public health reporting and research.

Distinguishing Admissions Specifically for COVID-19 from Incidental SARS-CoV-2 Admissions: A National EHR Research Consortium Study

Admissions are generally classified as COVID-19 hospitalizations if the patient has a positive SARS-CoV-2 polymerase chain reaction (PCR) test. However, because 35% of SARS-CoV-2 infections are asymptomatic, patients admitted for unrelated indications with an incidentally positive test could be misclassified as a COVID-19 hospitalization. EHR-based studies have been unable to distinguish between a hospitalization specifically for COVID-19 versus an incidental SARS-CoV-2 hospitalization. From a retrospective EHR-based cohort in four US healthcare systems, a random sample of 1,123 SARS-CoV-2 PCR-positive patients hospitalized between 3/2020â€"8/2021 was manually chart-reviewed and classified as admitted-with-COVID-19 (incidental) vs. specifically admitted for COVID-19 (for-COVID-19). EHR-based phenotyped feature sets filtered out incidental admissions, which occurred in 26%. The top site-specific feature sets had 79-99% specificity with 62-75% sensitivity, while the best performing across-site feature set had 71-94% specificity with 69-81% sensitivity. A large proportion of SARS-CoV-2 PCR-positive admissions were incidental. Straightforward EHR-based phenotypes differentiated admissions, which is important to assure accurate public health reporting and research.

The Impact of Concurrent Multi-Service Coverage on Quality and Safety in Trauma Care

BACKGROUND

At many trauma centers in the United States, one acute care surgeon is responsible for overnight coverage of both the emergency general surgery (EGS) and trauma services. The impact of this scheduling phenomenon on the quality and safety of trauma care has not been studied.

METHODS

Overnight (12:00 AM to 7:00 AM) trauma admissions to an academic Level 1 trauma center from 2013-2015 were studied after the institution adopted this scheduling phenomenon. Admissions were divided into two groups based on whether the admitting surgeon covered only the trauma service, or both the trauma and EGS services ("multi-service coverage"). Four major outcomes (e.g., mortality and complications), six quality metrics (e.g., time to first OR visit and unplanned transfers to the ICU), and procedural utilization patterns were compared.

RESULTS

A total of 1046 admissions were included. There were no differences in any major outcomes between the two exposure groups, including any National Trauma Data Bank-defined complication (OR 1.1, 95% CI 0.8-1.5, P= 0.5). Quality metrics dependent on the admitting surgeon remained unchanged, including attending presence at the highest-level trauma activations within 15 min of arrival (93% versus 86%, P= 0.07) and time to urgent operative intervention (68 min versus 82 min, P= 0.9). There were no differences in the number of laboratory and imaging studies (4.1 versus 4.1, P= 0.9) or bedside interventions (1.8 versus 2.1, P= 0.4) performed per patient by the admitting surgeon. Multivariate logistic regression did not identify multi-service coverage as an independent risk factor for adverse patient outcomes or quality metrics.

CONCLUSIONS

Trauma admissions under a surgeon covering multiple services simultaneously had similar outcomes, quality metrics, and procedural utilization patterns compared to trauma admissions under surgeons covering only the trauma service. Despite concerns that multiple-service coverage may overburden one acute care surgeon, time-dependent quality metrics and studies done during the initial workup of trauma patients remained unchanged. These findings suggest that simultaneous trauma and EGS service coverage by one acute care surgeon does not adversely impact trauma patient care.

Deploying digital health tools within large, complex health systems: key considerations for adoption and implementation

In recent years, the number of digital health tools with the potential to significantly improve delivery of healthcare services has grown tremendously. However, the use of these tools in large, complex health systems remains comparatively limited. The adoption and implementation of digital health tools at an enterprise level is a challenge; few strategies exist to help tools cross the chasm from clinical validation to integration within the workflows of a large health system. Many previously proposed frameworks for digital health implementation are difficult to operationalize in these dynamic organizations. In this piece, we put forth nine dimensions along which clinically validated digital health tools should be examined by health systems prior to adoption, and propose strategies for selecting digital health tools and planning for implementation in this setting. By evaluating prospective tools along these dimensions, health systems can evaluate which existing digital health solutions are worthy of adoption, ensure they have sufficient resources for deployment and long-term use, and devise a strategic plan for implementation.

Cultural adaptation: a framework for addressing an often-overlooked dimension of digital health accessibility

Relatively little is known about how to make digital health tools accessible to different populations from a cultural standpoint. Alignment with cultural values and communication styles may affect these tools’ ability to diagnose and treat various conditions. In this Editorial, we highlight the findings of recent work to make digital tools for mental health more culturally accessible, and propose ways to advance this area of study.

Program Director Perceptions of the General Surgery Milestones Project

OBJECTIVE

As a result of the Milestones Project, all Accreditation Council for Graduate Medical Education accredited training programs now use an evaluation framework based on outcomes in 6 core competencies. Despite their widespread use, the Milestones have not been broadly evaluated. This study sought to examine program director (PD) perceptions of the Milestones Project.

DESIGN, SETTING, AND PARTICIPANTS

A national survey of general surgery PDs distributed between January and March of 2016.

RESULTS

A total of 132 surgical PDs responded to the survey (60% response rate). Positive perceptions included value for education (55%) and evaluation of resident performance (58%), as well as ability of Milestones to provide unbiased feedback (55%) and to identify areas of resident deficiency (58%). Meanwhile, time input and the ability of Milestones to discriminate underperforming programs were less likely to be rated positively (25% and 21%, respectively). Half of PDs felt that the Milestones were an improvement over their previous evaluation system (55%).

CONCLUSIONS

Using the Milestones as competency-based, developmental outcomes measures, surgical PDs reported perceived benefits for education and objectivity in the evaluation of resident performance. The overall response to the Milestones was generally favorable, and most PDs would not return to their previous evaluation systems. To improve future iterations of the Milestones, many PDs expressed a desire for customization of the Milestones' content and structure to allow for programmatic differences.

Electronic Communication of Protected Health Information: Privacy, Security, and HIPAA Compliance

PURPOSE

Technology has enhanced modern health care delivery, particularly through accessibility to health information and ease of communication with tools like mobile device messaging (texting). However, text messaging has created new risks for breach of protected health information (PHI). In the current study, we sought to evaluate hand surgeons' knowledge and compliance with privacy and security standards for electronic communication by text message.

METHODS

A cross-sectional survey of the American Society for Surgery of the Hand membership was conducted in March and April 2016. Descriptive and inferential statistical analyses were performed of composite results as well as relevant subgroup analyses.

RESULTS

A total of 409 responses were obtained (11% response rate). Although 63% of surgeons reported that they believe that text messaging does not meet Health Insurance Portability and Accountability Act of 1996 security standards, only 37% reported they do not use text messages to communicate PHI. Younger surgeons and respondents who believed that their texting was compliant were statistically significantly more like to report messaging of PHI (odds ratio, 1.59 and 1.22, respectively).

DISCUSSION

A majority of hand surgeons in this study reported the use of text messaging to communicate PHI. Of note, neither the Health Insurance Portability and Accountability Act of 1996 statute nor US Department of Health and Human Services specifically prohibits this form of electronic communication. To be compliant, surgeons, practices, and institutions need to take reasonable security precautions to prevent breach of privacy with electronic communication.

CLINICAL RELEVANCE

Communication of clinical information by text message is not prohibited under Health Insurance Portability and Accountability Act of 1996, but surgeons should use appropriate safeguards to prevent breach when using this form of communication.

The Impact of the 2011 Accreditation Council for Graduate Medical Education Duty Hour Reform on Quality and Safety in Trauma Care

BACKGROUND

In 2011, the ACGME limited duty hours for residents. Although studies evaluating the 2011 policy have not shown improvements in general measures of morbidity or mortality, these outcomes might not reflect changes in specialty-specific practice patterns and secondary quality measures.

STUDY DESIGN

All trauma admissions from July 2009 through June 2013 at an academic Level I trauma center were evaluated for 5 primary outcomes (eg, mortality and length of stay), and 10 secondary quality measures and practice patterns (eg, operating room [OR] visits). All variables were compared before and after the reform (July 1, 2011). Piecewise regression was used to study temporal trends in quality.

RESULTS

There were 11,740 admissions studied. The reform was not strongly associated with changes in any primary outcomes except length of stay (7.98 to 7.36 days; p = 0.01). However, many secondary quality metrics changed. The total number of OR and bedside procedures per admission (6.72 to 7.34; p < 0.001) and OR visits per admission (0.76 to 0.91; p < 0.001) were higher in the post-reform group, representing an additional 9,559 procedures and 1,584 OR visits. Use of minor bedside procedures, such as laboratory and imaging studies, increased most significantly.

CONCLUSIONS

Although most major outcomes were unaffected, quality of care might have changed after the reform. Indeed, a consistent change in resource use patterns was manifested by substantial post-reform increases in measures such as bedside procedures and OR visits. No secondary quality measures exhibited improvements strongly associated with the reform. Several factors, including attending oversight, might have insulated major outcomes from change. Our findings show that some less-commonly studied quality metrics related to costs of care changed after the 2011 reform at our institution.