Cybersecurity features of digital medical devices: an analysis of FDA product summaries

OBJECTIVES

To more clearly define the landscape of digital medical devices subject to US Food and Drug Administration (FDA) oversight, this analysis leverages publicly available regulatory documents to characterise the prevalence and trends of software and cybersecurity features in regulated medical devices.

DESIGN

We analysed data from publicly available FDA product summaries to understand the frequency and recent time trends of inclusion of software and cybersecurity content in publicly available product information.

SETTING

The full set of regulated medical devices, approved over the years 2002-2016 included in the FDA's 510(k) and premarket approval databases.

PRIMARY AND SECONDARY OUTCOME MEASURES

The primary outcome was the share of devices containing software that included cybersecurity content in their product summaries. Secondary outcomes were differences in these shares (a) over time and (b) across regulatory areas.

RESULTS

Among regulated devices, 13.79% were identified as including software. Among these products, only 2.13% had product summaries that included cybersecurity content over the period studied. The overall share of devices including cybersecurity content was higher in recent years, growing from an average of 1.4% in the first decade of our sample to 5.5% in 2015 and 2016, the most recent years included. The share of devices including cybersecurity content also varied across regulatory areas from a low of 0% to a high of 22.2%.

CONCLUSIONS

To ensure the safest possible healthcare delivery environment for patients and hospitals, regulators and manufacturers should work together to make the software and cybersecurity content of new medical devices more easily accessible.

An Internet of Things Buttons to Measure and Respond to Restroom Cleanliness in a Hospital Setting: Descriptive Study

Background

Restroom cleanliness is an important factor in hospital quality. Due to its dynamic process, it can be difficult to detect the presence of dirty restrooms that need to be cleaned. Using an Internet of Things (IoT) button can permit users to designate restrooms that need cleaning and in turn, allow prompt response from housekeeping to maintain real-time restroom cleanliness.

Objective

This study aimed to describe the deployment of an IoT button–based notification system to measure hospital restroom cleanliness reporting system usage and qualitative feedback from housekeeping staff on IoT button use.

Methods

We deployed IoT buttons in 16 hospital restrooms. Over an 8-month period, housekeeping staff received real-time notifications and responded to button presses for restroom cleaning. All button presses were recorded. We reported average button usage by hospital area, time of day, and day of week. We also conducted interviews with housekeeping supervisors and staff to understand their acceptance of and experience with the system.

Results

Over 8 months, 1920 requests to clean restrooms in the main hospital lobby and satellite buildings were received. The hospital lobby IoT buttons received over half (N=1055, 55%) of requests for cleaning. Most requests occurred in afternoon hours from 3 PM to midnight. Requests for cleaning remained stable throughout the work week with fewer requests occurring over weekends. IoT button use was sustained throughout the study period. Interviews with housekeeping supervisors and staff demonstrated acceptance of the IoT buttons; actual use was centered around asynchronous communication between supervisors and staff in response to requests to clean restrooms.

Conclusions

An IoT button system is a feasible method to generate on-demand request for restroom cleaning that is easy to deploy and that users will consistently engage with. Data from this system have the potential to enable responsive scheduling for restroom service and anticipate periods of high restroom utilization in a hospital.

The opportunities and shortcomings of using big data and national databases for sarcoma research

The rarity and heterogeneity of sarcomas make performing appropriately powered studies challenging and magnify the significance of large databases in sarcoma research. Established large tumor registries and population-based databases have become increasingly relevant for answering clinical questions regarding sarcoma incidence, treatment patterns, and outcomes. However, the validity of large databases has been questioned and scrutinized because of the inaccuracy and wide variability of coding practices and the absence of clinically relevant variables. In addition, the utilization of large databases for the study of rare cancers such as sarcoma may be particularly challenging because of the known limitations of administrative data and poor overall data quality. Currently, there are several large national cancer databases, including the Surveillance, Epidemiology, and End Results database, the National Cancer Data Base of the American College of Surgeons and the American Cancer Society, and the National Program of Cancer Registries of the Centers for Disease Control and Prevention. These databases are often used for sarcoma research, but they are limited by their dependence on administrative or billing data, the lack of agreement between chart abstractors on diagnosis codes, and the use of preexisting documented hospital diagnosis codes for tumor registries, which lead to a significant underestimation of sarcomas in large data sets. Current and future initiatives to improve databases and big data applications for sarcoma research include increasing the utilization of sarcoma-specific registries and encouraging national initiatives to expand on real-world, evidence-based data sets.

An electronic notification system for improving patient flow in the emergency department

Healthcare providers make time-sensitive care decisions based on EHR data. As systems of record, the EHR is often not configured to optimally surface timely information. For patients awaiting admission, infection control concerns that potentially require private rooms can prolong stays in the Emergency Department. We aim to determine if an event-based notification platform connected with a commercial EHR can help prioritize timely information and improve patient flow in the emergency department. We undertook a pre-post analysis for patients being admitted from the emergency room who were tested for influenza. We used a primary outcome of mean time from negative test result to inpatient transfer. The median time decreased by 27%, from 4.1 hours to 3.0 hours. The distribution of transfer times pre and post-intervention were significantly different with a p-value of <0.001. Our findings support the use of event-based notification systems to improve patient flow in the emergency department.

Assessment of Employee Susceptibility to Phishing Attacks at US Health Care Institutions

IMPORTANCE

Cybersecurity is an increasingly important threat to health care delivery, and email phishing is a major attack vector against hospital employees.

OBJECTIVE

To describe the practice of phishing simulation and the extent to which health care employees are vulnerable to phishing simulations.

DESIGN, SETTING, AND PARTICIPANTS

Retrospective, multicenter quality improvement study of a convenience sample of 6 geographically dispersed US health care institutions that ran phishing simulations from August 1, 2011, through April 10, 2018. The specific institutions are anonymized herein for security and privacy concerns.

EXPOSURES

Simulated phishing emails received by employees at US health care institutions.

MAIN OUTCOMES AND MEASURES

Date of phishing campaign, campaign number, number of emails sent, number of emails clicked, and email content. Emails were classified into 3 categories (office related, personal, or information technology related).

RESULTS

The final study sample included 6 anonymized US health care institutions, 95 simulated phishing campaigns, and 2 971 945 emails, 422 062 of which were clicked (14.2%). The median institutional click rates for campaigns ranged from 7.4% (interquartile range [IQR], 5.8%-9.6%) to 30.7% (IQR, 25.2%-34.4%), with an overall median click rate of 16.7% (IQR, 8.3%-24.2%) across all campaigns and institutions. In the regression model, repeated phishing campaigns were associated with decreased odds of clicking on a subsequent phishing email (adjusted OR, 0.511; 95% CI, 0.382-0.685 for 6-10 campaigns; adjusted OR, 0.335; 95% CI, 0.282-0.398 for >10 campaigns).

CONCLUSIONS AND RELEVANCE

Among a sample of US health care institutions that sent phishing simulations, almost 1 in 7 simulated emails sent were clicked on by employees. Increasing campaigns were associated with decreased odds of clicking on a phishing email, suggesting a potential benefit of phishing simulation and awareness. With cyberattacks increasing against US health care systems, these click rates represent a major cybersecurity risk for hospitals.

Characteristics of the National Applicant Pool for Clinical Informatics Fellowships (2016-2017)

We conducted a national study to assess the numbers and diversity of applicants for 2016 and 2017 clinical informatics fellowship positions. In each year, we collected data on the number of applications that programs received from candidates who were ultimately successful vs. unsuccessful. In 2017, we also conducted an anonymous applicant survey. Successful candidates applied to an average of 4.2 and 5.5 programs for 2016 and 2017, respectively. In the survey, unsuccessful candidates reported applying to fewer programs. Assuming unsuccessful candidates submitted between 2-5 applications each, the total applicant pool numbered 42-69 for 2016 (competing for 24 positions) and 52-85 for 2017 (competing for 30 positions). Among survey respondents (n=33), 24% were female, 1 was black and none were Hispanic. We conclude that greater efforts are needed to enhance interest in clinical informatics among medical students and residents, particularly among women and members of underrepresented minority groups.

Assessing Documentation of Critical Imaging Result Follow-up Recommendations in Emergency Department Discharge Instructions

To facilitate follow-up of critical test results across transitions in patient care settings, we implemented an electronic discharge module that enabled care providers to include follow-up recommendations in the discharge instructions. We assessed the impact of this module on documentation of follow-up recommendations for critical imaging findings in Emergency Department (ED) discharge instructions. We studied 240 patients with critical imaging findings discharged from the ED before (n = 80) and after (n = 160) implementation of the module. We manually reviewed hand-written forms and electronic discharge instructions to determine if follow-up recommendations were documented. Follow-up recommendations in ED discharge instructions increased from 60.0% (48/80) to 73.8% (118/160) post-module implementation (p = 0.03), a relative increase of 23%. There was no significant change in the rate of documented critical imaging findings in the discharge instructions (77.5% [62/80] before the intervention and 76.9% [123/160] after the intervention; p = 0.91). Implementation of a discharge module was associated with increased documentation of critical imaging finding follow-up recommendations in ED discharge instructions. However, one in four patients still did not receive adequate follow-up recommendations, suggesting further opportunities for performance improvement exist.

Analysis of Errors in Dictated Clinical Documents Assisted by Speech Recognition Software and Professional Transcriptionists

IMPORTANCE

Accurate clinical documentation is critical to health care quality and safety. Dictation services supported by speech recognition (SR) technology and professional medical transcriptionists are widely used by US clinicians. However, the quality of SR-assisted documentation has not been thoroughly studied.

OBJECTIVE

To identify and analyze errors at each stage of the SR-assisted dictation process.

DESIGN SETTING AND PARTICIPANTS

This cross-sectional study collected a stratified random sample of 217 notes (83 office notes, 75 discharge summaries, and 59 operative notes) dictated by 144 physicians between January 1 and December 31, 2016, at 2 health care organizations using Dragon Medical 360 | eScription (Nuance). Errors were annotated in the SR engine-generated document (SR), the medical transcriptionist-edited document (MT), and the physician's signed note (SN). Each document was compared with a criterion standard created from the original audio recordings and medical record review.

MAIN OUTCOMES AND MEASURES

Error rate; mean errors per document; error frequency by general type (eg, deletion), semantic type (eg, medication), and clinical significance; and variations by physician characteristics, note type, and institution.

RESULTS

Among the 217 notes, there were 144 unique dictating physicians: 44 female (30.6%) and 10 unknown sex (6.9%). Mean (SD) physician age was 52 (12.5) years (median [range] age, 54 [28-80] years). Among 121 physicians for whom specialty information was available (84.0%), 35 specialties were represented, including 45 surgeons (37.2%), 30 internists (24.8%), and 46 others (38.0%). The error rate in SR notes was 7.4% (ie, 7.4 errors per 100 words). It decreased to 0.4% after transcriptionist review and 0.3% in SNs. Overall, 96.3% of SR notes, 58.1% of MT notes, and 42.4% of SNs contained errors. Deletions were most common (34.7%), then insertions (27.0%). Among errors at the SR, MT, and SN stages, 15.8%, 26.9%, and 25.9%, respectively, involved clinical information, and 5.7%, 8.9%, and 6.4%, respectively, were clinically significant. Discharge summaries had higher mean SR error rates than other types (8.9% vs 6.6%; difference, 2.3%; 95% CI, 1.0%-3.6%; P < .001). Surgeons' SR notes had lower mean error rates than other physicians' (6.0% vs 8.1%; difference, 2.2%; 95% CI, 0.8%-3.5%; P = .002). One institution had a higher mean SR error rate (7.6% vs 6.6%; difference, 1.0%; 95% CI, -0.2% to 2.8%; P = .10) but lower mean MT and SN error rates (0.3% vs 0.7%; difference, -0.3%; 95% CI, -0.63% to -0.04%; P = .03 and 0.2% vs 0.6%; difference, -0.4%; 95% CI, -0.7% to -0.2%; P = .003).

CONCLUSIONS AND RELEVANCE

Seven in 100 words in SR-generated documents contain errors; many errors involve clinical information. That most errors are corrected before notes are signed demonstrates the importance of manual review, quality assurance, and auditing.

A Novel Mobile App and Population Management System to Manage Rheumatoid Arthritis Flares: Protocol for a Randomized Controlled Trial

Background

Rheumatoid arthritis flares have a profound effect on patients, causing pain and disability. However, flares often occur between regularly scheduled health care provider visits and are, therefore, difficult to monitor and manage. We sought to develop a mobile phone app combined with a population management system to help track RA flares between visits.

Objective

The objective of this study is to implement the mobile app plus the population management system to monitor rheumatoid arthritis disease activity between scheduled health care provider visits over a period of 6 months.

Methods

This is a randomized controlled trial that lasts for 6 months for each participant. We aim to recruit 190 patients, randomized 50:50 to the intervention group versus the control group. The intervention group will be assigned the mobile app and be prompted to answer daily questionnaires sent to their mobile devices. Both groups will be assigned a population manager, who will communicate with the participants via telephone at 6 weeks and 18 weeks. The population manager will also communicate with the participants in the intervention group if their responses indicate a sustained increase in rheumatoid arthritis disease activity. To assess patient satisfaction, the primary outcomes will be scores on the Treatment Satisfaction Questionnaire for Medication as well as the Perceived Efficacy in Patient-Physician Interactions questionnaire at 6 months. To determine the effect of the mobile app on rheumatoid arthritis disease activity, the primary outcome will be the Clinical Disease Activity Index at 6 months.

Results

The trial started in November 2016, and an estimated 2.5 years will be necessary to complete the study. Study results are expected to be published by the end of 2019.

Conclusions

The completion of this study will provide important data regarding the following: (1) the assessment of validated outcome measures to assess rheumatoid arthritis disease activity with a mobile app between routinely scheduled health care provider visits, (2) patient engagement in monitoring their condition, and (3) communication between patients and health care providers through the population management system.

Trial Registration

ClinicalTrials.gov NCT02822521, http://clinicaltrials.gov/ct2/show/NCT02822521 (Archived by WebCite at http://www.webcitation.org/6xed3kGPd)

Core Components for a Clinically Integrated mHealth App for Asthma Symptom Monitoring

Background mHealth apps may be useful tools for supporting chronic disease management. Objective Our aim was to apply user-centered design principles to efficiently identify core components for an mHealth-based asthma symptom–monitoring intervention using patient-reported outcomes (PROs). Methods We iteratively combined principles of qualitative research, user-centered design, and “gamification” to understand patients' and providers' needs, develop and refine intervention components, develop prototypes, and create a usable mobile app to integrate with clinical workflows. We identified anticipated benefits and burdens for stakeholders. Results We conducted 19 individual design sessions with nine adult patients and seven clinicians from an academic medical center (some were included multiple times). We identified four core intervention components: (1) Invitation—patients are invited by their physicians. (2) Symptom checks—patients receive weekly five-item questionnaires via the app with 48 hours to respond. Depending on symptoms, patients may be given the option to request a call from a nurse or receive one automatically. (3) Patient review—in the app, patients can view their self-reported data graphically. (4) In-person visit—physicians have access to patient-reported symptoms in the electronic health record (EHR) where they can review them before in-person visits. As there is currently no location in the EHR where physicians would consistently notice these data, recording a recent note was the best option. Benefits to patients may include helping decide when to call their provider and facilitating shared decision making. Benefits to providers may include saving time discussing symptoms. Provider organizations may need to pay nurses extra, but those costs may be offset by reduced visits and hospitalizations. Conclusion Recent systematic reviews show inconsistent outcomes and little insight into functionalities required for mHealth asthma interventions, highlighting the need for systematic intervention design. We identified specific features for adoption and engagement that meet the stated needs of users for asthma symptom monitoring.

A Custom-Developed Emergency Department Provider Electronic Documentation System Reduces Operational Efficiency

STUDY OBJECTIVE

Electronic health record implementation can improve care, but may also adversely affect emergency department (ED) efficiency. We examine how a custom, ED provider, electronic documentation system (eDoc), which replaced paper documentation, affects operational performance.

METHODS

We analyzed retrospective operational data for 1-year periods before and after eDoc implementation in a single ED. We computed daily operational statistics, reflecting 60,870 pre- and 59,337 postimplementation patient encounters. The prespecified primary outcome was daily mean length of stay; secondary outcomes were daily mean length of stay for admitted and discharged patients and daily mean arrival time to disposition for admitted patients. We used a prespecified multiple regression model to identify differences in outcomes while controlling for prespecified confounding variables.

RESULTS

The unadjusted change in length of stay was 8.4 minutes; unadjusted changes in secondary outcomes were length of stay for admitted patients 11.4 minutes, length of stay for discharged patients 1.8 minutes, and time to disposition 1.8 minutes. With a prespecified regression analysis to control for variations in operational characteristics, there were significant increases in length of stay (6.3 minutes [95% confidence interval 3.5 to 9.1 minutes]) and length of stay for discharged patients (5.1 minutes [95% confidence interval 1.9 to 8.3 minutes]). There was no statistically significant change in length of stay for admitted patients or time to disposition.

CONCLUSION

In our single-center study, the isolated implementation of eDoc was associated with increases in overall and discharge length of stay. Our findings suggest that a custom-designed electronic provider documentation may negatively affect ED throughput. Strategies to mitigate these effects, such as reducing documentation requirements or adding clinical staff, scribes, or voice recognition, would be a valuable area of future research.

Catalyzing healthcare transformation with digital health: Performance indicators and lessons learned from a Digital Health Innovation Group

Despite considerable investment in digital health (DH) companies and a growing DH ecosystem, there are multiple challenges to testing and implementing innovative solutions. Health systems have recognized the potential of DH and have formed DH innovation centers. However, limited information is available on DH innovation center processes, best practices, or outcomes. This case report describes a DH innovation center process that can be replicated across health systems and defines and benchmarks process indicators to assess DH innovation center performance. The Brigham and Women's Hospital's Digital Health Innovation Group (DHIG) accelerates DH innovations from idea to pilot safely and efficiently using a structured process. Fifty-four DH innovations were accelerated by the DHIG process between July 2014 and December 2016. In order to measure effectiveness of the DHIG process, key process indicators were defined as 1) number of solutions that completed each DHIG phase and 2) length of time to complete each phase. Twenty-three DH innovations progressed to pilot stage and 13 innovations were terminated after barriers to pilot implementation were identified by the DHIG process. For 4 DH solutions that executed a pilot, the average time for innovations to proceed from DHIG intake to pilot initiation was 9 months. Overall, the DHIG is a reproducible process that addresses key roadblocks in DH innovation within health systems. To our knowledge, this is the first report to describe DH innovation process indicators and results within an academic health system. Therefore, there is no published data to compare our results with the results of other DH innovation centers. Standardized data collection and indicator reporting could allow benchmark comparisons across institutions. Additional opportunities exist for the validation of DH solution effectiveness and for translational support from pilot to implementation. These are critical steps to advance DH technologies and effectively leverage the DH ecosystem to transform healthcare.

The Benefits and Challenges of an Interfaced Electronic Health Record and Laboratory Information System: Effects on Laboratory Processes

Context.—

A recent government regulation incentivizes implementation of an electronic health record (EHR) with computerized order entry and structured results display. Many institutions have also chosen to interface their EHR with their laboratory information system (LIS).

Objective.—

To determine the impact of an interfaced EHR-LIS on laboratory processes.

Design.—

We analyzed several different processes before and after implementation of an interfaced EHR-LIS: the turnaround time, the number of stat specimens received, venipunctures per patient per day, preanalytic errors in phlebotomy, the number of add-on tests using a new electronic process, and the number of wrong test codes ordered. Data were gathered through the LIS and/or EHR.

Results.—

The turnaround time for potassium and hematocrit decreased significantly (P = .047 and P = .004, respectively). The number of stat orders also decreased significantly, from 40% to 7% for potassium and hematocrit, respectively (P &lt; .001 for both). Even though the average number of inpatient venipunctures per day increased from 1.38 to 1.62 (P &lt; .001), the average number of preanalytic errors per month decreased from 2.24 to 0.16 per 1000 specimens (P &lt; .001). Overall there was a 16% increase in add-on tests. The number of wrong test codes ordered was high and it was challenging for providers to correctly order some common tests.

Conclusions.—

An interfaced EHR-LIS significantly improved within-laboratory turnaround time and decreased stat requests and preanalytic phlebotomy errors. Despite increasing the number of add-on requests, an electronic add-on process increased efficiency and improved provider satisfaction. Laboratories implementing an interfaced EHR-LIS should be cautious of its effects on test ordering and patient venipunctures per day.

Significant Reduction in Preanalytical Errors for Nonphlebotomy Blood Draws After Implementation of a Novel Integrated Specimen Collection Module

OBJECTIVES

Most preanalytical errors at our institution occur during nonphlebotomy blood draws. We implemented an electronic health record (EHR), interfaced the EHR to the laboratory information system, and designed a new specimen collection module. We studied the effects of the new system on nonphlebotomy preanalytical errors.

METHODS

We used an electronic database of preanalytical errors and calculated the number and type of the most common errors in the emergency department (ED) and inpatient nursing for 3-month periods before (August-October 2014) and after (August-October 2015) implementation. The level of staff compliance with the new system was also assessed.

RESULTS

The average monthly preanalytical errors decreased significantly from 7.95 to 1.45 per 1,000 specimens in the ED (P < 0001) and 11.75 to 3.25 per 1,000 specimens in inpatient nursing (P < 0001). The rate of decrease was similar for mislabeled, unlabeled, wrong specimen received and no specimen received errors. Most residual errors (80% in the ED and 67% in inpatient nursing) occurred when providers did not use the new system as designed.

CONCLUSIONS

Implementation of a customized specimen collection module led to a significant reduction in preanalytical errors. Improved compliance with the system may lead to further reductions in error rates.

Usability of the Massachusetts Prescription Drug Monitoring Program in the Emergency Department: A Mixed-methods Study

OBJECTIVES

Prescription drug monitoring programs (PDMPs) are underutilized, despite evidence showing that they may reduce the epidemic of opioid-related addiction, diversion, and overdose. We evaluated the usability of the Massachusetts (MA) PDMP by emergency medicine providers (EPs), as a system's usability may affect how often it is used.

METHODS

This was a mixed-methods study of 17 EPs. We compared the time and number of clicks required to review one patient's record in the PDMP to three other commonly performed computer-based tasks in the emergency department (ED: ordering a computed tomography [CT] scan, writing a prescription, and searching a medication history service integrated within the electronic medical record [EMR]). We performed semistructured interviews and analyzed participant comments and responses regarding their experience using the MA PDMP.

RESULTS

The PDMP task took a longer time to complete (mean = 4.22 minutes) and greater number of mouse clicks to complete (mean = 50.3 clicks) than the three other tasks (CT-pulmonary embolism = 1.42 minutes, 24.8 clicks; prescription = 1.30 minutes, 19.5 clicks; SureScripts = 1.45 minutes, 9.5 clicks). Qualitative analysis yielded four main themes about PDMP usability, three negative and one positive: 1) difficulty accessing the PDMP, 2) cumbersome acquiring patient medication history information within the PDMP, 3) nonintuitive display of patient medication history information within the PDMP, and 4) overall perceived value of the PDMP despite an inefficient interface.

CONCLUSIONS

The complicated processes of gaining access to, logging in, and using the MA PDMP are barriers to preventing its more frequent use. All states should evaluate the PDMP usability in multiple practice settings including the ED and work to improve provider enrollment, login procedures, patient information input, prescription data display, and ultimately, PDMP data integration into EMRs.

Interrater Reliability of mHealth App Rating Measures: Analysis of Top Depression and Smoking Cessation Apps

Background

There are over 165,000 mHealth apps currently available to patients, but few have undergone an external quality review. Furthermore, no standardized review method exists, and little has been done to examine the consistency of the evaluation systems themselves.

Objective

We sought to determine which measures for evaluating the quality of mHealth apps have the greatest interrater reliability.

Methods

We identified 22 measures for evaluating the quality of apps from the literature. A panel of 6 reviewers reviewed the top 10 depression apps and 10 smoking cessation apps from the Apple iTunes App Store on these measures. Krippendorff’s alpha was calculated for each of the measures and reported by app category and in aggregate.

Results

The measure for interactiveness and feedback was found to have the greatest overall interrater reliability (alpha=.69). Presence of password protection (alpha=.65), whether the app was uploaded by a health care agency (alpha=.63), the number of consumer ratings (alpha=.59), and several other measures had moderate interrater reliability (alphas>.5). There was the least agreement over whether apps had errors or performance issues (alpha=.15), stated advertising policies (alpha=.16), and were easy to use (alpha=.18). There were substantial differences in the interrater reliabilities of a number of measures when they were applied to depression versus smoking apps.

Conclusions

We found wide variation in the interrater reliability of measures used to evaluate apps, and some measures are more robust across categories of apps than others. The measures with the highest degree of interrater reliability tended to be those that involved the least rater discretion. Clinical quality measures such as effectiveness, ease of use, and performance had relatively poor interrater reliability. Subsequent research is needed to determine consistent means for evaluating the performance of apps. Patients and clinicians should consider conducting their own assessments of apps, in conjunction with evaluating information from reviews.

The effect of electronic health record implementation on community emergency department operational measures of performance

STUDY OBJECTIVE

We study the effect of an emergency department (ED) electronic health record implementation on the operational metrics of a diverse group of community EDs.

METHODS

We performed a retrospective before/after analysis of 23 EDs from a single management group that experienced ED electronic health record implementation (with the majority of electronic health records optimized specifically for ED use). We obtained electronic data for 4 length of stay measures (arrival to provider, admitted, discharged, and overall length of stay) and 4 measures of operational characteristics (left before treatment complete, significant returns, overall patient satisfaction, and provider efficiency). We compared the 6-month "baseline" period immediately before implementation with a "steady-state" period commencing 6 months after implementation for all 8 metrics.

RESULTS

For the length of stay measures, there were no differences in the arrival-to-provider interval (difference of -0.02 hours; 95% confidence interval [CI] of difference -0.12 to 0.08), admitted length of stay (difference of 0.10 hours; 95% CI of difference -0.17 to 0.37), discharged length of stay (difference of 0.07 hours; 95% CI of difference -0.07 to 0.22), and overall length of stay (difference of 0.11 hours; 95% CI of difference -0.04 to 0.27). For operational characteristics, there were no differences in the percentage who left before treatment was complete (difference of 0.24%; 95% CI of difference -0.47% to 0.95%), significant returns (difference of -0.04%; 95% CI of difference -0.48% to 0.39%), overall percentile patient satisfaction (difference of -0.02%; 95% CI of difference -2.35% to 2.30%), and provider efficiency (difference of -0.05 patients/hour; 95% CI of difference -0.11 to 0.02).

CONCLUSION

There is no meaningful difference in 8 measures of operational performance for community EDs experiencing optimized ED electronic health record implementation between a baseline and steady-state period.