Machine learning to predict notes for chart review in the oncology setting: a proof of concept strategy for improving clinician note-writing

OBJECTIVE

Leverage electronic health record (EHR) audit logs to develop a machine learning (ML) model that predicts which notes a clinician wants to review when seeing oncology patients.

MATERIALS AND METHODS

We trained logistic regression models using note metadata and a Term Frequency Inverse Document Frequency (TF-IDF) text representation. We evaluated performance with precision, recall, F1, AUC, and a clinical qualitative assessment.

RESULTS

The metadata only model achieved an AUC 0.930 and the metadata and TF-IDF model an AUC 0.937. Qualitative assessment revealed a need for better text representation and to further customize predictions for the user.

DISCUSSION

Our model effectively surfaces the top 10 notes a clinician wants to review when seeing an oncology patient. Further studies can characterize different types of clinician users and better tailor the task for different care settings.

CONCLUSION

EHR audit logs can provide important relevance data for training ML models that assist with note-writing in the oncology setting.

Multimodal Representation Learning via Maximization of Local Mutual Information

We propose and demonstrate a representation learning approach by maximizing the mutual information between local features of images and text. The goal of this approach is to learn useful image representations by taking advantage of the rich information contained in the free text that describes the findings in the image. Our method trains image and text encoders by encouraging the resulting representations to exhibit high local mutual information. We make use of recent advances in mutual information estimation with neural network discriminators. We argue that the sum of local mutual information is typically a lower bound on the global mutual information. Our experimental results in the downstream image classification tasks demonstrate the advantages of using local features for image-text representation learning. Our code is available at: https://github.com/RayRuizhiLiao/mutual_info_img_txt.

A visual representation of microbiological culture data improves comprehension: a randomized controlled trial

OBJECTIVE

While the judicious use of antibiotics takes past microbiological culture results into consideration, this data's typical format in the electronic health record (EHR) may be unwieldy when incorporated into clinical decision-making. We hypothesize that a visual representation of sensitivities may aid in their comprehension.

MATERIALS AND METHODS

A prospective parallel unblinded randomized controlled trial was undertaken at an academic urban tertiary care center. Providers managing emergency department (ED) patients receiving antibiotics and having previous culture sensitivity testing were included. Providers were randomly selected to use standard EHR functionality or a visual representation of patients' past culture data as they answered questions about previous sensitivities. Concordance between provider responses and past cultures was assessed using the kappa statistic. Providers were surveyed about their decision-making and the usability of the tool using Likert scales.

RESULTS

518 ED encounters were screened from 3/5/2018 to 9/30/18, with providers from 144 visits enrolled and analyzed in the intervention arm and 129 in the control arm. Providers using the visualization tool had a kappa of 0.69 (95% CI: 0.65-0.73) when asked about past culture results while the control group had a kappa of 0.16 (95% CI: 0.12-0.20). Providers using the tool expressed improved understanding of previous cultures and found the tool easy to use (P < .001). Secondary outcomes showed no differences in prescribing practices.

CONCLUSION

A visual representation of culture sensitivities improves comprehension when compared to standard text-based representations.

Secondary Use of COVID-19 Symptom Incidence Among Hospital Employees as an Example of Syndromic Surveillance of Hospital Admissions Within 7 Days

IMPORTANCE

Alternative methods for hospital occupancy forecasting, essential information in hospital crisis planning, are necessary in a novel pandemic when traditional data sources such as disease testing are limited.

OBJECTIVE

To determine whether mandatory daily employee symptom attestation data can be used as syndromic surveillance to estimate COVID-19 hospitalizations in the communities where employees live.

DESIGN, SETTING, AND PARTICIPANTS

This cohort study was conducted from April 2, 2020, to November 4, 2020, at a large academic hospital network of 10 hospitals accounting for a total of 2384 beds and 136 000 discharges in New England. The participants included 6841 employees who worked on-site at hospital 1 and lived in the 10 hospitals' service areas.

EXPOSURE

Daily employee self-reported symptoms were collected using an automated text messaging system from a single hospital.

MAIN OUTCOMES AND MEASURES

Mean absolute error (MAE) and weighted mean absolute percentage error (MAPE) of 7-day forecasts of daily COVID-19 hospital census at each hospital.

RESULTS

Among 6841 employees living within the 10 hospitals' service areas, 5120 (74.8%) were female individuals and 3884 (56.8%) were White individuals; the mean (SD) age was 40.8 (13.6) years, and the mean (SD) time of service was 8.8 (10.4) years. The study model had a MAE of 6.9 patients with COVID-19 and a weighted MAPE of 1.5% for hospitalizations for the entire hospital network. The individual hospitals had an MAE that ranged from 0.9 to 4.5 patients (weighted MAPE ranged from 2.1% to 16.1%). For context, the mean network all-cause occupancy was 1286 during this period, so an error of 6.9 is only 0.5% of the network mean occupancy. Operationally, this level of error was negligible to the incident command center. At hospital 1, a doubling of the number of employees reporting symptoms (which corresponded to 4 additional employees reporting symptoms at the mean for hospital 1) was associated with a 5% increase in COVID-19 hospitalizations at hospital 1 in 7 days (regression coefficient, 0.05; 95% CI, 0.02-0.07; P < .001).

CONCLUSIONS AND RELEVANCE

This cohort study found that a real-time employee health attestation tool used at a single hospital could be used to estimate subsequent hospitalizations in 7 days at hospitals throughout a larger hospital network in New England.

A Novel Method to Measure Venular Perivascular Spaces in Patients with MS on 7T MRI

In MS, inflammatory cells accumulate within the perivascular spaces of acute and chronic lesions. Reliance on perivascular spaces as biomarkers for MS remains uncertain because various studies have reported inconsistencies in perivascular space anatomy. Distinguishing between venular and arteriolar perivascular spaces is pathophysiologically relevant in MS. In this pilot study, we leverage susceptibility-weighted imaging at 7T to better identify perivascular spaces of venular distribution on corresponding high-resolution T2 images.

Deep Learning to Quantify Pulmonary Edema in Chest Radiographs

Purpose

To develop a machine learning model to classify the severity grades of pulmonary edema on chest radiographs.

Materials and Methods

In this retrospective study, 369 071 chest radiographs and associated radiology reports from 64 581 patients (mean age, 51.71 years; 54.51% women) from the MIMIC-CXR chest radiograph dataset were included. This dataset was split into patients with and without congestive heart failure (CHF). Pulmonary edema severity labels from the associated radiology reports were extracted from patients with CHF as four different ordinal levels: 0, no edema; 1, vascular congestion; 2, interstitial edema; and 3, alveolar edema. Deep learning models were developed using two approaches: a semisupervised model using a variational autoencoder and a pretrained supervised learning model using a dense neural network. Receiver operating characteristic curve analysis was performed on both models.

Results

The area under the receiver operating characteristic curve (AUC) for differentiating alveolar edema from no edema was 0.99 for the semisupervised model and 0.87 for the pretrained models. Performance of the algorithm was inversely related to the difficulty in categorizing milder states of pulmonary edema (shown as AUCs for semisupervised model and pretrained model, respectively): 2 versus 0, 0.88 and 0.81; 1 versus 0, 0.79 and 0.66; 3 versus 1, 0.93 and 0.82; 2 versus 1, 0.69 and 0.73; and 3 versus 2, 0.88 and 0.63.

Conclusion

Deep learning models were trained on a large chest radiograph dataset and could grade the severity of pulmonary edema on chest radiographs with high performance.Supplemental material is available for this article.See also the commentary by Auffermann in this issue.© RSNA, 2021.

Development and validation of a pancreatic cancer risk model for the general population using electronic health records: An observational study

AIM

Pancreatic ductal adenocarcinoma (PDAC) is often diagnosed at a late, incurable stage. We sought to determine whether individuals at high risk of developing PDAC could be identified early using routinely collected data.

METHODS

Electronic health record (EHR) databases from two independent hospitals in Boston, Massachusetts, providing inpatient, outpatient, and emergency care, from 1979 through 2017, were used with case-control matching. PDAC cases were selected using International Classification of Diseases 9/10 codes and validated with tumour registries. A data-driven feature selection approach was used to develop neural networks and L2-regularised logistic regression (LR) models on training data (594 cases, 100,787 controls) and compared with a published model based on hand-selected diagnoses ('baseline'). Model performance was validated on an external database (408 cases, 160,185 controls). Three prediction lead times (180, 270 and 365 days) were considered.

RESULTS

The LR model had the best performance, with an area under the curve (AUC) of 0.71 (confidence interval [CI]: 0.67-0.76) for the training set, and AUC 0.68 (CI: 0.65-0.71) for the validation set, 365 days before diagnosis. Data-driven feature selection improved results over 'baseline' (AUC = 0.55; CI: 0.52-0.58). The LR model flags 2692 (CI 2592-2791) of 156,485 as high risk, 365 days in advance, identifying 25 (CI: 16-36) cancer patients. Risk stratification showed that the high-risk group presented a cancer rate 3 to 5 times the prevalence in our data set.

CONCLUSION

A simple EHR model, based on diagnoses, can identify high-risk individuals for PDAC up to one year in advance. This inexpensive, systematic approach may serve as the first sieve for selection of individuals for PDAC screening programs.

Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessment

We propose and demonstrate a novel machine learning algorithm that assesses pulmonary edema severity from chest radiographs. While large publicly available datasets of chest radiographs and free-text radiology reports exist, only limited numerical edema severity labels can be extracted from radiology reports. This is a significant challenge in learning such models for image classification. To take advantage of the rich information present in the radiology reports, we develop a neural network model that is trained on both images and free-text to assess pulmonary edema severity from chest radiographs at inference time. Our experimental results suggest that the joint image-text representation learning improves the performance of pulmonary edema assessment compared to a supervised model trained on images only. We also show the use of the text for explaining the image classification by the joint model. To the best of our knowledge, our approach is the first to leverage free-text radiology reports for improving the image model performance in this application. Our code is available at: https://github.com/RayRuizhiLiao/joint_chestxray.

Risk of mortality and cardiopulmonary arrest in critical patients presenting to the emergency department using machine learning and natural language processing

Emergency department triage is the first point in time when a patient's acuity level is determined. The time to assign a priority at triage is short and it is vital to accurately stratify patients at this stage, since under-triage can lead to increased morbidity, mortality and costs. Our aim was to present a model that can assist healthcare professionals in triage decision making, namely in the stratification of patients through the risk prediction of a composite critical outcome-mortality and cardiopulmonary arrest. Our study cohort consisted of 235826 adult patients triaged at a Portuguese Emergency Department from 2012 to 2016. Patients were assigned to emergent, very urgent or urgent priorities of the Manchester Triage System (MTS). Demographics, clinical variables routinely collected at triage and the patients' chief complaint were used. Logistic regression, random forests and extreme gradient boosting were developed using all available variables. The term frequency-inverse document frequency (TF-IDF) natural language processing weighting factor was applied to vectorize the chief complaint. Stratified random sampling was used to split the data into train (70%) and test (30%) data sets. Ten-fold cross validation was performed in train to optimize model hyper-parameters. The performance obtained with the best model was compared against the reference model-a regularized logistic regression trained using only triage priorities. Extreme gradient boosting exhibited good calibration properties and yielded areas under the receiver operating characteristic and precision-recall curves of 0.96 (95% CI 0.95-0.97) and 0.31 (95% CI 0.26-0.36), respectively. The predictors ranked with higher importance by this model were the Glasgow coma score, the patients' age, pulse oximetry and arrival mode. Compared to the reference, the extreme gradient boosting model using clinical variables and the chief complaint presented higher recall for patients assigned MTS-3 and can identify those who are at risk of the composite outcome.

Derivation and validation of a machine learning record linkage algorithm between emergency medical services and the emergency department

OBJECTIVE

Linking emergency medical services (EMS) electronic patient care reports (ePCRs) to emergency department (ED) records can provide clinicians access to vital information that can alter management. It can also create rich databases for research and quality improvement. Unfortunately, previous attempts at ePCR and ED record linkage have had limited success. In this study, we use supervised machine learning to derive and validate an automated record linkage algorithm between EMS ePCRs and ED records.

MATERIALS AND METHODS

All consecutive ePCRs from a single EMS provider between June 2013 and June 2015 were included. A primary reviewer matched ePCRs to a list of ED patients to create a gold standard. Age, gender, last name, first name, social security number, and date of birth were extracted. Data were randomly split into 80% training and 20% test datasets. We derived missing indicators, identical indicators, edit distances, and percent differences. A multivariate logistic regression model was trained using 5-fold cross-validation, using label k-fold, L2 regularization, and class reweighting.

RESULTS

A total of 14 032 ePCRs were included in the study. Interrater reliability between the primary and secondary reviewer had a kappa of 0.9. The algorithm had a sensitivity of 99.4%, a positive predictive value of 99.9%, and an area under the receiver-operating characteristic curve of 0.99 in both the training and test datasets. Date-of-birth match had the highest odds ratio of 16.9, followed by last name match (10.6). Social security number match had an odds ratio of 3.8.

CONCLUSIONS

We were able to successfully derive and validate a record linkage algorithm from a single EMS ePCR provider to our hospital EMR.

Robustly Extracting Medical Knowledge from EHRs: A Case Study of Learning a Health Knowledge Graph

Increasingly large electronic health records (EHRs) provide an opportunity to algorithmically learn medical knowledge. In one prominent example, a causal health knowledge graph could learn relationships between diseases and symptoms and then serve as a diagnostic tool to be refined with additional clinical input. Prior research has demonstrated the ability to construct such a graph from over 270,000 emergency department patient visits. In this work, we describe methods to evaluate a health knowledge graph for robustness. Moving beyond precision and recall, we analyze for which diseases and for which patients the graph is most accurate. We identify sample size and unmeasured confounders as major sources of error in the health knowledge graph. We introduce a method to leverage non-linear functions in building the causal graph to better understand existing model assumptions. Finally, to assess model generalizability, we extend to a larger set of complete patient visits within a hospital system. We conclude with a discussion on how to robustly extract medical knowledge from EHRs.

MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports

Chest radiography is an extremely powerful imaging modality, allowing for a detailed inspection of a patient's chest, but requires specialized training for proper interpretation. With the advent of high performance general purpose computer vision algorithms, the accurate automated analysis of chest radiographs is becoming increasingly of interest to researchers. Here we describe MIMIC-CXR, a large dataset of 227,835 imaging studies for 65,379 patients presenting to the Beth Israel Deaconess Medical Center Emergency Department between 2011-2016. Each imaging study can contain one or more images, usually a frontal view and a lateral view. A total of 377,110 images are available in the dataset. Studies are made available with a semi-structured free-text radiology report that describes the radiological findings of the images, written by a practicing radiologist contemporaneously during routine clinical care. All images and reports have been de-identified to protect patient privacy. The dataset is made freely available to facilitate and encourage a wide range of research in computer vision, natural language processing, and clinical data mining.

Assessment of Unintentional Duplicate Orders by Emergency Department Clinicians Before and After Implementation of a Visual Aid in the Electronic Health Record Ordering System

IMPORTANCE

Electronic health records allow teams of clinicians to simultaneously care for patients, but an unintended consequence is the potential for duplicate orders of tests and medications.

OBJECTIVE

To determine whether a simple visual aid is associated with a reduction in duplicate ordering of tests and medications.

DESIGN, SETTING, AND PARTICIPANTS

This cohort study used an interrupted time series model to analyze 184 694 consecutive patients who visited the emergency department (ED) of an academic hospital with 55 000 ED visits annually. Patient visits occurred 1 year before and after each intervention, as follows: for laboratory orders, from August 13, 2012, to August 13, 2014; for medication orders, from February 3, 2013, to February 3, 2015; and for radiology orders, from December 12, 2013, to December 12, 2015. Data were analyzed from April to September 2019.

EXPOSURE

If an order had previously been placed during the ED visit, a red highlight appeared around the checkbox of that order in the computerized provider order entry system.

MAIN OUTCOMES AND MEASURES

Number of unintentional duplicate laboratory, medication, and radiology orders.

RESULTS

A total of 184 694 patients (mean [SD] age, 51.6 [20.8] years; age range, 0-113.0 years; 99 735 [54.0%] women) who visited the ED were analyzed over the 3 overlapping study periods. After deployment of a noninterruptive nudge in electronic health records, there was an associated 49% decrease in the rate of unintentional duplicate orders for laboratory tests (incidence rate ratio, 0.51; 95% CI, 0.45-0.59), from 4485 to 2731 orders, and an associated 40% decrease in unintentional duplicate orders of radiology tests (incidence rate ratio, 0.60; 95% CI, 0.44-0.82), from 956 to 782 orders. There was not a statistically significant change in unintentional duplicate orders of medications (incidence rate ratio, 1.17; 95% CI, 0.52-2.61), which increased from 225 to 287 orders. The nudge eliminated an estimated 17 936 clicks in our electronic health record.

CONCLUSIONS AND RELEVANCE

In this interrupted time series cohort study, passive visual cues that provided just-in-time decision support were associated with reductions in unintentional duplicate orders for laboratory and radiology tests but not in unintentional duplicate medication orders.

Consensus Development of a Modern Ontology of Emergency Department Presenting Problems-The Hierarchical Presenting Problem Ontology (HaPPy)

OBJECTIVE

Numerous attempts have been made to create a standardized "presenting problem" or "chief complaint" list to characterize the nature of an emergency department visit. Previous attempts have failed to gain widespread adoption as they were not freely shareable or did not contain the right level of specificity, structure, and clinical relevance to gain acceptance by the larger emergency medicine community. Using real-world data, we constructed a presenting problem list that addresses these challenges.

MATERIALS AND METHODS

We prospectively captured the presenting problems for 180,424 consecutive emergency department patient visits at an urban, academic, Level I trauma center in the Boston metro area. No patients were excluded. We used a consensus process to iteratively derive our system using real-world data. We used the first 70% of consecutive visits to derive our ontology, followed by a 6-month washout period, and the remaining 30% for validation. All concepts were mapped to Systematized Nomenclature of Medicine-Clinical Terms (SNOMED CT).

RESULTS

Our system consists of a polyhierarchical ontology containing 692 unique concepts, 2,118 synonyms, and 30,613 nonvisible descriptions to correct misspellings and nonstandard terminology. Our ontology successfully captured structured data for 95.9% of visits in our validation data set.

DISCUSSION AND CONCLUSION

We present the HierArchical Presenting Problem ontologY (HaPPy). This ontology was empirically derived and then iteratively validated by an expert consensus panel. HaPPy contains 692 presenting problem concepts, each concept being mapped to SNOMED CT. This freely sharable ontology can help to facilitate presenting problem-based quality metrics, research, and patient care.

Risk of Intracranial Hemorrhage in Ground-level Fall With Antiplatelet or Anticoagulant Agents

OBJECTIVES

Anticoagulant and antiplatelet medications are known to increase the risk and severity of traumatic intracranial hemorrhage (tICH), even with minor head trauma. Most studies on bleeding propensity with head trauma are retrospective, are based on trauma registries, or include heterogeneous mechanisms of injury. The goal of this study was to determine the rate of tICH from only a common low-acuity mechanism of injury, that of a ground-level fall, in patients taking one or more of the following antiplatelet or anticoagulant medications: aspirin, warfarin, prasugrel, ticagrelor, dabigatran, rivaroxaban, apixaban, or enoxaparin.

METHODS

This was a prospective cohort study conducted at a Level I tertiary care trauma center of consecutive patients meeting the inclusion criteria of a ground-level fall with head trauma as affirmed by the treating clinician, a computed tomography (CT) head obtained, and taking and one of the above antiplatelet or anticoagulants. Patients were identified prospectively through electronic screening with confirmatory chart review. Emergency department charts were abstracted without subsequent knowledge of the hospital course. Patients transferred with a known abnormal CT head were excluded. Primary outcome was rate of tICH on initial CT head. Rates with 95% confidence intervals (CIs) were compared.

RESULTS

Over 30 months, we enrolled 939 subjects. The mean ± SD age was 78.3 ± 11.9 years and 44.6% were male. There were a total of 33 patients with tICH (3.5%, 95% CI = 2.5%-4.9%). Antiplatelets had a rate of tICH of 4.3% (95% CI = 3.0%-6.2%) compared to anticoagulants with a rate of 1.7% (95% CI = 0.4%-4.5%). Aspirin without other agents had an tICH rate of 4.6% (95% CI = 3.2%-6.6%); of these, 81.5% were taking low-dose 81 mg aspirin. Two patients received a craniotomy (one taking aspirin, one taking warfarin). There were four deaths (three taking aspirin, one taking warfarin). Most (72.7%) subjects with tICH were discharged home or to a rehabilitation facility. There were no tICH in 31 subjects taking a direct oral anticoagulant. CIs were overlapping for the groups.

CONCLUSION

There is a low incidence of clinically significant tICH with a ground-level fall in head trauma in patients taking an anticoagulant or antiplatelet medication. There was no statistical difference in rate of tICH between antiplatelet and anticoagulants, which is unanticipated and counterintuitive as most literature and teaching suggests a higher rate with anticoagulants. A larger data set is needed to determine if small differences between the groups exist.

Learning a Health Knowledge Graph from Electronic Medical Records

Demand for clinical decision support systems in medicine and self-diagnostic symptom checkers has substantially increased in recent years. Existing platforms rely on knowledge bases manually compiled through a labor-intensive process or automatically derived using simple pairwise statistics. This study explored an automated process to learn high quality knowledge bases linking diseases and symptoms directly from electronic medical records. Medical concepts were extracted from 273,174 de-identified patient records and maximum likelihood estimation of three probabilistic models was used to automatically construct knowledge graphs: logistic regression, naive Bayes classifier and a Bayesian network using noisy OR gates. A graph of disease-symptom relationships was elicited from the learned parameters and the constructed knowledge graphs were evaluated and validated, with permission, against Google’s manually-constructed knowledge graph and against expert physician opinions. Our study shows that direct and automated construction of high quality health knowledge graphs from medical records using rudimentary concept extraction is feasible. The noisy OR model produces a high quality knowledge graph reaching precision of 0.85 for a recall of 0.6 in the clinical evaluation. Noisy OR significantly outperforms all tested models across evaluation frameworks (p < 0.01).

Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning

Objective

To demonstrate the incremental benefit of using free text data in addition to vital sign and demographic data to identify patients with suspected infection in the emergency department.

Methods

This was a retrospective, observational cohort study performed at a tertiary academic teaching hospital. All consecutive ED patient visits between 12/17/08 and 2/17/13 were included. No patients were excluded. The primary outcome measure was infection diagnosed in the emergency department defined as a patient having an infection related ED ICD-9-CM discharge diagnosis. Patients were randomly allocated to train (64%), validate (20%), and test (16%) data sets. After preprocessing the free text using bigram and negation detection, we built four models to predict infection, incrementally adding vital signs, chief complaint, and free text nursing assessment. We used two different methods to represent free text: a bag of words model and a topic model. We then used a support vector machine to build the prediction model. We calculated the area under the receiver operating characteristic curve to compare the discriminatory power of each model.

Results

A total of 230,936 patient visits were included in the study. Approximately 14% of patients had the primary outcome of diagnosed infection. The area under the ROC curve (AUC) for the vitals model, which used only vital signs and demographic data, was 0.67 for the training data set, 0.67 for the validation data set, and 0.67 (95% CI 0.65–0.69) for the test data set. The AUC for the chief complaint model which also included demographic and vital sign data was 0.84 for the training data set, 0.83 for the validation data set, and 0.83 (95% CI 0.81–0.84) for the test data set. The best performing methods made use of all of the free text. In particular, the AUC for the bag-of-words model was 0.89 for training data set, 0.86 for the validation data set, and 0.86 (95% CI 0.85–0.87) for the test data set. The AUC for the topic model was 0.86 for the training data set, 0.86 for the validation data set, and 0.85 (95% CI 0.84–0.86) for the test data set.

Conclusion

Compared to previous work that only used structured data such as vital signs and demographic information, utilizing free text drastically improves the discriminatory ability (increase in AUC from 0.67 to 0.86) of identifying infection.

The Integration of Electronic Medical Student Evaluations Into an Emergency Department Tracking System is Associated With Increased Quality and Quantity of Evaluations

BACKGROUND

Medical student evaluations are essential for determining clerkship grades. Electronic evaluations have various advantages compared to paper evaluations, such as increased ease of collection, asynchronous reporting, and decreased likelihood of becoming lost.

OBJECTIVES

To determine whether electronic medical student evaluations (EMSEs) provide more evaluations and content when compared to paper shift card evaluations.

METHODS

This before and after cohort study was conducted over a 2.5-year period at an academic hospital affiliated with a medical school and emergency medicine residency program. EMSEs replaced the paper shift evaluations that had previously been used halfway through the study period. A random sample of the free text comments on both paper and EMSEs were blindly judged by medical student clerkship directors for their helpfulness and usefulness. Logistic regression was used to test for any relationship between quality and quantity of words.

RESULTS

A total of 135 paper evaluations for 30 students and then 570 EMSEs for 62 students were collected. An average of 4.8 (standard deviation [SD] 3.2) evaluations were completed per student using the paper version compared to 9.0 (SD 3.8) evaluations completed per student electronically (p < 0.001). There was an average of 8.8 (SD 8.5) words of free text evaluation on paper evaluations when compared to 22.5 (SD 28.4) words for EMSEs (p < 0.001). A statistically significant (p < 0.02) association between quality of an evaluation and the word count existed.

CONCLUSIONS

EMSEs that were integrated into the emergency department tracking system significantly increased the number of evaluations completed compared to paper evaluations. In addition, the EMSEs captured more "helpful/useful" information about the individual students as evidenced by the longer free text entries per evaluation.

Electronic medical record phenotyping using the anchor and learn framework

Background Electronic medical records (EMRs) hold a tremendous amount of information about patients that is relevant to determining the optimal approach to patient care. As medicine becomes increasingly precise, a patient’s electronic medical record phenotype will play an important role in triggering clinical decision support systems that can deliver personalized recommendations in real time. Learning with anchors presents a method of efficiently learning statistically driven phenotypes with minimal manual intervention.

Materials and Methods We developed a phenotype library that uses both structured and unstructured data from the EMR to represent patients for real-time clinical decision support. Eight of the phenotypes were evaluated using retrospective EMR data on emergency department patients using a set of prospectively gathered gold standard labels.

Results We built a phenotype library with 42 publicly available phenotype definitions. Using information from triage time, the phenotype classifiers have an area under the ROC curve (AUC) of infection 0.89, cancer 0.88, immunosuppressed 0.85, septic shock 0.93, nursing home 0.87, anticoagulated 0.83, cardiac etiology 0.89, and pneumonia 0.90. Using information available at the time of disposition from the emergency department, the AUC values are infection 0.91, cancer 0.95, immunosuppressed 0.90, septic shock 0.97, nursing home 0.91, anticoagulated 0.94, cardiac etiology 0.92, and pneumonia 0.97.

Discussion The resulting phenotypes are interpretable and fast to build, and perform comparably to statistically learned phenotypes developed with 5000 manually labeled patients.

Conclusion Learning with anchors is an attractive option for building a large public repository of phenotype definitions that can be used for a range of health IT applications, including real-time decision support.

Using Anchors to Estimate Clinical State without Labeled Data

We present a novel framework for learning to estimate and predict clinical state variables without labeled data. The resulting models can used for electronic phenotyping, triggering clinical decision support, and cohort selection. The framework relies on key observations which we characterize and term "anchor variables". By specifying anchor variables, an expert encodes a certain amount of domain knowledge about the problem while the rest of learning proceeds in an unsupervised manner. The ability to build anchors upon standardized ontologies and the framework's ability to learn from unlabeled data promote generalizability across institutions. We additionally develop a user interface to enable experts to choose anchor variables in an informed manner. The framework is applied to electronic medical record-based phenotyping to enable real-time decision support in the emergency department. We validate the learned models using a prospectively gathered set of gold-standard responses from emergency physicians for nine clinically relevant variables.

Evaluating how electronic charting affects resident productivity

Electronic medical records (EMRs) are becoming standard to improve the communication of information and longevity of patient records. Using an EMR in the emergency department (ED) could potentially slow residents evaluating patients. We evaluated how introducing an EMR affected resident productivity in an academic ED. We retrospectively studied first year emergency medicine residents from a large, academic, tertiary care center before-and-after the institution of an EMR on July 1st, 2010. No residents from the 2009-2010 class used the EMR, while all of the 2010-2011 residents used the EMR. We performed univariate and multivariate analyses using productivity, measured in patients per hour (pt/hr), as the primary outcome. A mixed-model multivariate regression, stratified by acuity zone, was created incorporating EMR and other possible confounders: admissions, signouts, daily ED volume, and days after July 1st for each shift. The study was granted IRB waiver of informed. We reviewed 2,405 shifts: 1,259 shifts before and 1,146 shifts after EMR implementation. When using the EMR, the univariate analysis estimated a 0.084 pt/hr increase in the high acuity zone (p = 0.1317) and 0.029 pt/hr decrease (p = 0.7085) in the low acuity zone. The multivariate regression estimated a 0.038 pt/hr increase (p = 0.3413) in the high acuity zone and a 0.009 pt/hr increase (p = 0.9049) in the low acuity zone with the EMR. Despite the expectation that electronic charting is detrimental to resident productivity, our analyses do not suggest a significant relationship between resident productivity and using the EMR.

Prospective pilot study of a tablet computer in an Emergency Department

BACKGROUND

The recent availability of low-cost tablet computers can facilitate bedside information retrieval by clinicians.

OBJECTIVE

To evaluate the effect of physician tablet use in the Emergency Department.

DESIGN

Prospective cohort study comparing physician workstation usage with and without a tablet.

SETTING

55,000 visits/year Level 1 Emergency Department at a tertiary academic teaching hospital.

PARTICIPANTS

13 emergency physicians (7 Attendings, 4 EM3s, and 2 EM1s) worked a total of 168 scheduled shifts (130 without and 38 with tablets) during the study period.

INTERVENTION

Physician use of a tablet computer while delivering direct patient care in the Emergency Department.

MAIN OUTCOME MEASURES

The primary outcome measure was the time spent using the Emergency Department Information System (EDIS) at a computer workstation per shift. The secondary outcome measure was the number of EDIS logins at a computer workstation per shift.

RESULTS

Clinician use of a tablet was associated with a 38min (17-59) decrease in time spent per shift using the EDIS at a computer workstation (p<0.001) after adjusting for clinical role, location, and shift length. The number of logins was also associated with a 5-login (2.2-7.9) decrease per shift (p<0.001) after adjusting for other covariates.

CONCLUSION

Clinical use of a tablet computer was associated with a reduction in the number of times physicians logged into a computer workstation and a reduction in the amount of time they spent there using the EDIS. The presumed benefit is that decreasing time at a computer workstation increases physician availability at the bedside. However, this association will require further investigation.