Measuring and controlling medical record abstraction (MRA) error rates in an observational study

Infant Feeding and Weight Trajectories in the Eat, Sleep, Console Trial: A Secondary Analysis of a Randomized Clinical Trial

Importance

Infants with neonatal opioid withdrawal syndrome (NOWS) cared for with the Eat, Sleep, Console (ESC) care approach receive less pharmacologic treatment and have shorter hospital stays compared to usual care with the Finnegan Neonatal Abstinence Scoring Tool, but the effects of these approaches on feeding and weight are unknown.

Objective

To evaluate feeding practices and weight trajectories in infants cared for with ESC vs usual care.

Design, Setting, and Participants

ESC-NOW is a cluster randomized trial of infants with NOWS born at 36 weeks' gestation or later at 26 US hospitals from September 2020 to March 2022. Each site transitioned from usual care to ESC (the study intervention) at a randomized time. Feeding was per site practice and not specified by the intervention. Feeding and weight outcomes were assessed at hospital discharge.

Intervention

ESC vs usual care.

Main Outcomes and Measures

Outcomes include prospectively identified secondary end points related to feeding and weight. z Scores were used for growth to account for corrected gestational age at the time of measurement. All analyses were intention to treat and adjusted for study design. Maternal/infant characteristics were included in adjusted models.

Results

The analyses included 1305 infants (702 in usual care and 603 in ESC; mean [SD] gestational age, 38.6 [1.3] weeks; 655 [50.2%] male and 650 [49.8%] female). Baseline demographic characteristics were similar between groups. The proportion of breastfed infants was higher in the ESC group (52.7% vs 41.7%; absolute difference, 11%; 95% CI, 1.0-20.9). A higher proportion of infants cared for with ESC received exclusive breast milk (15.1% vs 6.7%; absolute difference, 8.4%; 95% CI, 0.9-5.8) or any breast milk (38.8% vs 27.4%; absolute difference, 11.4%; 95% CI, 0.2-23.1) and were directly breastfeeding at discharge (35.2% vs 19.5%; absolute difference, 15.7%; 95% CI, 4.1-27.3). There was no difference in proportion of infants with weight loss greater than 10% or maximum percentage weight loss, although infants cared for with ESC had a lower weight z score on day of life 3 (-1.08 vs -1.01; absolute difference, 0.07; 95% CI, 0.02-0.12). When pharmacologic treatment was added into the model, no breastfeeding outcomes were statistically significant.

Conclusions and Relevance

In this study, infants cared for with ESC were more likely to initiate and continue breastfeeding and had no difference in percentage weight loss. The improvement in breastfeeding with ESC may be driven by reduction in pharmacologic treatment and provision of effective nonpharmacologic care.

Trial Registration

ClinicalTrials.gov Identifier: NCT04057820.

Influence of Eat, Sleep, and Console on Infants Pharmacologically Treated for Opioid Withdrawal: A Post Hoc Subgroup Analysis of the ESC-NOW Randomized Clinical Trial

Importance

The function-based eat, sleep, console (ESC) care approach substantially reduces the proportion of infants who receive pharmacologic treatment for neonatal opioid withdrawal syndrome (NOWS). This reduction has led to concerns for increased postnatal opioid exposure in infants who receive pharmacologic treatment. However, the effect of the ESC care approach on hospital outcomes for infants pharmacologically treated for NOWS is currently unknown.

Objective

To evaluate differences in opioid exposure and total length of hospital stay (LOS) for pharmacologically treated infants managed with the ESC care approach vs usual care with the Finnegan tool.

Design, Setting, and Participants

This post hoc subgroup analysis involved infants pharmacologically treated in ESC-NOW, a stepped-wedge cluster randomized clinical trial conducted at 26 US hospitals. Hospitals maintained pretrial practices for pharmacologic treatment, including opioid type, scheduled opioid dosing, and use of adjuvant medications. Infants were born at 36 weeks' gestation or later, had evidence of antenatal opioid exposure, and received opioid treatment for NOWS between September 2020 and March 2022. Data were analyzed from November 2022 to January 2024.

Exposure

Opioid treatment for NOWS and the ESC care approach.

Main Outcomes and Measures

For each outcome (total opioid exposure, peak opioid dose, time from birth to initiation of first opioid dose, length of opioid treatment, and LOS), we used generalized linear mixed models to adjust for the stepped-wedge design and maternal and infant characteristics.

Results

In the ESC-NOW trial, 463 of 1305 infants were pharmacologically treated (143/603 [23.7%] in the ESC care approach group and 320/702 [45.6%] in the usual care group). Mean total opioid exposure was lower in the ESC care approach group with an absolute difference of 4.1 morphine milligram equivalents per kilogram (MME/kg) (95% CI, 1.3-7.0) when compared with usual care (4.8 MME/kg vs 8.9 MME/kg, respectively; P = .001). Mean time from birth to initiation of pharmacologic treatment was 22.4 hours (95% CI, 7.1-37.7) longer with the ESC care approach vs usual care (75.4 vs 53.0 hours, respectively; P = .002). No significant difference in mean peak opioid dose was observed between groups (ESC care approach, 0.147 MME/kg, vs usual care, 0.126 MME/kg). The mean length of treatment was 6.3 days shorter (95% CI, 3.0-9.6) in the ESC care approach group vs usual care group (11.8 vs 18.1 days, respectively; P < .001), and mean LOS was 6.2 days shorter (95% CI, 3.0-9.4) with the ESC care approach than with usual care (16.7 vs 22.9 days, respectively; P < .001).

Conclusion and Relevance

When compared with usual care, the ESC care approach was associated with less opioid exposure and shorter LOS for infants pharmacologically treated for NOWS. The ESC care approach was not associated with a higher peak opioid dose, although pharmacologic treatment was typically initiated later.

Trial Registration

ClinicalTrials.gov Identifier: NCT04057820.

Validation of a Zero-shot Learning Natural Language Processing Tool to Facilitate Data Abstraction for Urologic Research

BACKGROUND

Urologic research often requires data abstraction from unstructured text contained within the electronic health record. A number of natural language processing (NLP) tools have been developed to aid with this time-consuming task; however, the generalizability of these tools is typically limited by the need for task-specific training.

OBJECTIVE

To describe the development and validation of a zero-shot learning NLP tool to facilitate data abstraction from unstructured text for use in downstream urologic research.

DESIGN, SETTING, AND PARTICIPANTS

An NLP tool based on the GPT-3.5 model from OpenAI was developed and compared with three physicians for time to task completion and accuracy for abstracting 14 unique variables from a set of 199 deidentified radical prostatectomy pathology reports. The reports were processed in vectorized and scanned formats to establish the impact of optical character recognition on data abstraction.

INTERVENTION

A zero-shot learning NLP tool for data abstraction.

OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS

The tool was compared with the human abstractors in terms of superiority for data abstraction speed and noninferiority for accuracy.

RESULTS AND LIMITATIONS

The human abstractors required a median (interquartile range) of 93 s (72-122 s) per report for data abstraction, whereas the software required a median of 12 s (10-15 s) for the vectorized reports and 15 s (13-17 s) for the scanned reports (p < 0.001 for all paired comparisons). The accuracies of the three human abstractors were 94.7% (95% confidence interval [CI], 93.8-95.5%), 97.8% (95% CI, 97.2-98.3%), and 96.4% (95% CI, 95.6-97%) for the combined set of 2786 data points. The tool had accuracy of 94.2% (95% CI, 93.3-94.9%) for the vectorized reports and was noninferior to the human abstractors at a margin of -10% (α = 0.025). The tool had slightly lower accuracy of 88.7% (95% CI 87.5-89.9%) for the scanned reports, making it noninferior to two of three human abstractors.

CONCLUSIONS

The developed zero-shot learning NLP tool offers urologic researchers a highly generalizable and accurate method for data abstraction from unstructured text. An open access version of the tool is available for immediate use by the urologic community.

PATIENT SUMMARY

In this report, we describe the design and validation of an artificial intelligence tool for abstracting discrete data from unstructured notes contained within the electronic medical record. This freely available tool, which is based on the GPT-3.5 technology from OpenAI, is intended to facilitate research and scientific discovery by the urologic community.

Scalable and interpretable alternative to chart review for phenotype evaluation using standardized structured data from electronic health records

OBJECTIVES

Chart review as the current gold standard for phenotype evaluation cannot support observational research on electronic health records and claims data sources at scale. We aimed to evaluate the ability of structured data to support efficient and interpretable phenotype evaluation as an alternative to chart review.

MATERIALS AND METHODS

We developed Knowledge-Enhanced Electronic Profile Review (KEEPER) as a phenotype evaluation tool that extracts patient's structured data elements relevant to a phenotype and presents them in a standardized fashion following clinical reasoning principles. We evaluated its performance (interrater agreement, intermethod agreement, accuracy, and review time) compared to manual chart review for 4 conditions using randomized 2-period, 2-sequence crossover design.

RESULTS

Case ascertainment with KEEPER was twice as fast compared to manual chart review. 88.1% of the patients were classified concordantly using charts and KEEPER, but agreement varied depending on the condition. Missing data and differences in interpretation accounted for most of the discrepancies. Pairs of clinicians agreed in case ascertainment in 91.2% of the cases when using KEEPER compared to 76.3% when using charts. Patient classification aligned with the gold standard in 88.1% and 86.9% of the cases respectively.

CONCLUSION

Structured data can be used for efficient and interpretable phenotype evaluation if they are limited to relevant subset and organized according to the clinical reasoning principles. A system that implements these principles can achieve noninferior performance compared to chart review at a fraction of time.

Natural Language Processing Can Automate Extraction of Barrett's Esophagus Endoscopy Quality Metrics

Objectives

To develop an automated natural language processing (NLP) method for extracting high-fidelity Barrett's Esophagus (BE) endoscopic surveillance and treatment data from the electronic health record (EHR).

Methods

Patients who underwent BE-related endoscopies between 2016 and 2020 at a single medical center were randomly assigned to a development or validation set. Those not aged 40 to 80 and those without confirmed BE were excluded. For each patient, free text pathology reports and structured procedure data were obtained. Gastroenterologists assigned ground truth labels. An NLP method leveraging MetaMap Lite generated endoscopy-level diagnosis and treatment data. Performance metrics were assessed for this data. The NLP methodology was then adapted to label key endoscopic eradication therapy (EET)-related endoscopy events and thereby facilitate calculation of patient-level pre-EET diagnosis, endotherapy time, and time to CE-IM.

Results

99 patients (377 endoscopies) and 115 patients (399 endoscopies) were included in the development and validation sets respectively. When assigning high-fidelity labels to the validation set, NLP achieved high performance (recall: 0.976, precision: 0.970, accuracy: 0.985, and F1-score: 0.972). 77 patients initiated EET and underwent 554 endoscopies. Key EET-related clinical event labels had high accuracy (EET start: 0.974, CE-D: 1.00, and CE-IM: 1.00), facilitating extraction of pre-treatment diagnosis, endotherapy time, and time to CE-IM.

Conclusions

High-fidelity BE endoscopic surveillance and treatment data can be extracted from routine EHR data using our automated, transparent NLP method. This method produces high-level clinical datasets for clinical research and quality metric assessment.