Quality of Layperson CPR Instructions From Artificial Intelligence Voice Assistants

This case series study evaluates responses from 4 artificial intelligence voice assistance on CPR questions from laypersons.

A Pilot of Digital Whiteboards for Improving Patient Satisfaction in the Emergency Department: Nonrandomized Controlled Trial

BACKGROUND

Electronic paper (E-paper) screens use electrophoretic ink to provide paper-like low-power displays with advanced networking capabilities that may potentially serve as an alternative to traditional whiteboards and television display screens in hospital settings. E-paper may be leveraged in the emergency department (ED) to facilitate communication. Providing ED patient status updates on E-paper screens could improve patient satisfaction and overall experience and provide more equitable access to their health information.

OBJECTIVE

We aimed to pilot a patient-facing digital whiteboard using E-paper to display relevant orienting and clinical information in real time to ED patients. We also sought to assess patients' satisfaction after our intervention and understand our patients' overall perception of the impact of the digital whiteboards on their stay.

METHODS

We deployed a 41-inch E-paper digital whiteboard in 4 rooms in an urban, tertiary care, and academic ED and enrolled 110 patients to understand and evaluate their experience. Participants completed a modified Hospital Consumer Assessment of Health Care Provider and Systems satisfaction questionnaire about their ED stay. We compared responses to a matched control group of patients triaged to ED rooms without digital whiteboards. We designed the digital whiteboard based on iterative feedback from various departmental stakeholders. After establishing IT infrastructure to support the project, we enrolled patients on a convenience basis into a control and an intervention (digital whiteboard) group. Enrollees were given a baseline survey to evaluate their comfort with technology and an exit survey to evaluate their opinions of the digital whiteboard and overall ED satisfaction. Statistical analysis was performed to compare baseline characteristics as well as satisfaction.

RESULTS

After the successful prototyping and implementation of 4 digital whiteboards, we screened 471 patients for inclusion. We enrolled 110 patients, and 50 patients in each group (control and intervention) completed the study protocol. Age, gender, and racial and ethnic composition were similar between groups. We saw significant increases in satisfaction on postvisit surveys when patients were asked about communication regarding delays (P=.03) and what to do after discharge (P=.02). We found that patients in the intervention group were more likely to recommend the facility to family and friends (P=.04). Additionally, 96% (48/50) stated that they preferred a room with a digital whiteboard, and 70% (35/50) found the intervention "quite a bit" or "extremely" helpful in understanding their ED stay.

CONCLUSIONS

Digital whiteboards are a feasible and acceptable method of displaying patient-facing data in the ED. Our pilot suggested that E-paper screens coupled with relevant, real-time clinical data and packaged together as a digital whiteboard may positively impact patient satisfaction and the perception of the facility during ED visits. Further study is needed to fully understand the impact on patient satisfaction and experience.

TRIAL REGISTRATION

ClinicalTrials.gov NCT04497922; https://clinicaltrials.gov/ct2/show/NCT04497922.

Acceptance of a computer vision facilitated protocol to measure adherence to face mask use: a single-site, observational cohort study among hospital staff

OBJECTIVES

Mask adherence continues to be a critical public health measure to prevent transmission of aerosol pathogens, such as SARS-CoV-2. We aimed to develop and deploy a computer vision algorithm to provide real-time feedback of mask wearing among staff in a hospital.

DESIGN

Single-site, observational cohort study.

SETTING

An urban, academic hospital in Boston, Massachusetts, USA.

PARTICIPANTS

We enrolled adult hospital staff entering the hospital at a key ingress point.

INTERVENTIONS

Consenting participants entering the hospital were invited to experience the computer vision mask detection system. Key aspects of the detection algorithm and feedback were described to participants, who then completed a quantitative assessment to understand their perceptions and acceptance of interacting with the system to detect their mask adherence.

OUTCOME MEASURES

Primary outcomes were willingness to interact with the mask system, and the degree of comfort participants felt in interacting with a public facing computer vision mask algorithm.

RESULTS

One hundred and eleven participants with mean age 40 (SD15.5) were enrolled in the study. Males (47.7%) and females (52.3%) were equally represented, and the majority identified as white (N=54, 49%). Most participants (N=97, 87.3%) reported acceptance of the system and most participants (N=84, 75.7%) were accepting of deployment of the system to reinforce mask adherence in public places. One third of participants (N=36) felt that a public facing computer vision system would be an intrusion into personal privacy.Public-facing computer vision software to detect and provide feedback around mask adherence may be acceptable in the hospital setting. Similar systems may be considered for deployment in locations where mask adherence is important.

Development and Testing of an Electronic Health Record‐Integrated Patient‐Reported Outcome Application and Intervention to Improve Efficiency of Rheumatoid Arthritis Care

Objective

Many patients with rheumatoid arthritis (RA) have difficulty finding clinicians to treat them because of workforce shortages. We developed an app to address this problem by improving care efficiency. The app collects patient‐reported outcomes (PROs) and can be used to inform visit timing, potentially reducing the volume of low‐value visits. We describe the development process, intervention design, and planned study for testing the app.

Methods

We employed user‐centered design, interviewing patients and clinicians, to develop the app. To improve visit efficiency, symptom tracking logic alerts clinicians to PRO trends: worsening PROs generate alerts suggesting an earlier visit, and stable or improving PROs generate notifications that scheduled visits could be delayed. An interrupted time‐series analysis with a nonrandomized control population will allow assessment of the impact of the app on visit frequency.

Results

Patient interviews identified several of the following needs for effective app and intervention design: the importance of a simple user interface facilitating rapid answering of PROs, the availability of condensed summary information with links to more in‐depth answers to common questions regarding RA, and the need for clinicians to discuss the PRO data during visits with patients. Clinician interviews identified the following user needs: PRO data must be easy to view and use during the clinical workflow, and there should be reduced interval visits when PROs are trending worse. Some clinicians believed visits could be delayed for patients with stable PROs, whereas others raised concerns.

Conclusion

PRO apps may improve care efficiency in rheumatology. Formal evaluation of an integrated PRO RA app is forthcoming.

Deploying digital health tools within large, complex health systems: key considerations for adoption and implementation

In recent years, the number of digital health tools with the potential to significantly improve delivery of healthcare services has grown tremendously. However, the use of these tools in large, complex health systems remains comparatively limited. The adoption and implementation of digital health tools at an enterprise level is a challenge; few strategies exist to help tools cross the chasm from clinical validation to integration within the workflows of a large health system. Many previously proposed frameworks for digital health implementation are difficult to operationalize in these dynamic organizations. In this piece, we put forth nine dimensions along which clinically validated digital health tools should be examined by health systems prior to adoption, and propose strategies for selecting digital health tools and planning for implementation in this setting. By evaluating prospective tools along these dimensions, health systems can evaluate which existing digital health solutions are worthy of adoption, ensure they have sufficient resources for deployment and long-term use, and devise a strategic plan for implementation.

Real-World Analysis of Off-Label Use of Molecularly Targeted Therapy in a Large Academic Medical Center Cohort

PURPOSE

The primary objective of this study is to quantify the use of off-label molecularly targeted therapy and describe the clinical situations in which off-label targeted therapy are used. A key secondary objective is to report the outcomes of patients treated with off-label use of targeted therapy.

PATIENTS AND METHODS

We searched the electronic health record between 2000 and 2020 at our center to characterize the volume, clinical settings, and outcomes associated with off-label use of targeted therapies in different types of solid tumors.

RESULTS

Among 46,712 patients who received targeted therapies, we identified 119 instances of off-label use of targeted therapy. Colon cancer was the most common cancer type to receive off-label targeted therapy in 18 patients (15.1%), followed by 13 with non-small-cell lung cancer (10.9%), eight with cholangiocarcinoma (6.7%), and seven with glioblastoma (5.9%). The most frequent molecular rationale for off-label therapy came from a comprehensive next-generation sequencing test (53.7%). The most frequently mutated gene that provided the rationale for targeted therapy was BRAF (20.1%), with BRAFV600E being the most common molecular alteration overall (15.1%). The median duration of off-label targeted therapy was 3.58 months, and the overall survival of treated patients was 7.59 months. There were 37 patients (31.1%) treated for longer than 6 months, 23 patients (19.3%) who survived ≥ 2 years, and 13 patients who were still on therapy as of June 2020.

CONCLUSION

In this large cohort study of patients with solid tumors, off-label use of targeted therapy was uncommon. With that said, a notable proportion of patients had treatment durations ≥ 6 months and survivals of ≥ 2 years.

Assessing Patient Experience and Orientation in the Emergency Department with Virtual Windows

Patients have benefitted from increasingly sophisticated diagnostic and therapeutic innovations over the years. However, the design of the physical hospital environment has garnered less attention. This may negatively impact a patient's experience and health. In areas of the hospital, such as the emergency department (ED), patients may spend hours, or even days, in a windowless environment. Studies have highlighted the importance of natural light and imagery, as they are essential in providing important stimuli to regulate circadian rhythm and orientation, and to mitigate the onset of certain medical conditions. In hospital locations where standard windows may be infeasible, the use of a virtual window may simulate the benefits of an actual window. In this pilot study, we assessed patient experience and orientation with virtual windows in the ED. We demonstrated that virtual windows are an acceptable technology that may improve patient experience and orientation.

Virtual Team Rounding: A Cross-Specialty Inpatient Care Staffing Program to Manage COVID-19 Surges

PROBLEM

The SARS-CoV-2 (COVID-19) pandemic presented numerous challenges to inpatient care, including overtaxed inpatient medicine services, surges in patient censuses, disrupted patient care and educational activities for trainees, underused providers in certain specialties, and personal protective equipment shortages and new requirements for physical distancing. In March 2020, as the COVID-19 surge began, an interdisciplinary group of administrators, providers, and trainees at Brigham and Women's Hospital created an inpatient virtual staffing model called the Virtual Team Rounding Program (VTRP).

APPROACH

The conceptual framework guiding VTRP development was rapid-cycle innovation. The VTRP was designed iteratively using feedback from residents, physician assistants, attendings, and administrators from March to June 2020. The VTRP trained and deployed a diverse set of providers across specialties as "virtual rounders" to support inpatient teams by joining and participating in rounds via videoconference and completing documentation tasks during and after rounds. The program was rapidly scaled up from March to June 2020.

OUTCOMES

In a survey of inpatient providers at the end of the pilot phase, 10/10 (100%) respondents reported they were getting either "a lot" or "a little" benefit from the VTRP and did not find the addition of the virtual rounder burdensome. During the scaling phase, the program grew to support 24 teams. In a survey at the end of the contraction phase, 117/187 (62.6%) inpatient providers who worked with a virtual rounder felt the rounder saved them time. VTRP leadership collaboratively and iteratively developed best practices for challenges encountered during implementation.

NEXT STEPS

Virtual rounding provides a valuable extension of inpatient teams to manage COVID-19 surges. Future work will quantitatively and qualitatively assess the impact of the VTRP on inpatient provider satisfaction and well-being, virtual rounders' experiences, and patient care outcomes.

Association of Self-reported High-Risk Allergy History With Allergy Symptoms After COVID-19 Vaccination

IMPORTANCE

Allergic history in individuals with confirmed anaphylaxis to a messenger RNA (mRNA) COVID-19 vaccine is common. However, the risk factors for allergy symptoms after receiving the vaccine are unknown.

OBJECTIVE

To assess the association between self-reported history of high-risk allergy and self-reported allergic reactions after mRNA COVID-19 vaccination of health care employees.

DESIGN, SETTING, AND PARTICIPANTS

This cohort study obtained demographic, medical, and vaccine administration data of employees of Mass General Brigham from the institutional electronic health record. Employees who received at least 1 dose of an mRNA COVID-19 vaccine between December 14, 2020, and February 1, 2021, and who completed at least 1 postvaccination symptom survey in the 3 days after vaccination were included.

EXPOSURES

Self-reported history of high-risk allergy, defined as a previous severe allergic reaction to a vaccine, an injectable medication, or other allergen.

MAIN OUTCOMES AND MEASURES

The primary outcome was 1 or more self-reported allergic reactions in the first 3 days after dose 1 or dose 2 of an mRNA COVID-19 vaccine. Multivariable log binomial regression was used to assess the association between allergic reactions and high-risk allergy status.

RESULTS

A total of 52 998 health care employees (mean [SD] age, 42 [14] years; 38 167 women [72.0%]) were included in the cohort, of whom 51 706 (97.6%) received 2 doses of an mRNA COVID-19 vaccine and 474 (0.9%) reported a history of high-risk allergy. Individuals with vs without a history of high-risk allergy reported more allergic reactions after receiving dose 1 or 2 of the vaccine (11.6% [n = 55] vs 4.7% [n = 2461]). In the adjusted model, a history of high-risk allergy was associated with an increased risk of allergic reactions (adjusted relative risk [aRR], 2.46; 95% CI, 1.92-3.16), with risk being highest for hives (aRR, 3.81; 95% CI, 2.33-6.22) and angioedema (aRR, 4.36; 95% CI, 2.52-7.54).

CONCLUSIONS AND RELEVANCE

This cohort study found that self-reported history of high-risk allergy was associated with an increased risk of self-reported allergic reactions within 3 days of mRNA COVID-19 vaccination. However, reported allergy symptoms did not impede the completion of the 2-dose vaccine protocol among a cohort of eligible health care employees, supporting the overall safety of mRNA COVID-19 vaccine.

Landscape Analysis of Oncology Mobile Health Applications

PURPOSE

More than 325,000 mobile health (mhealth) applications (apps) have been developed. We sought to describe the state of oncology-specific apps and to highlight areas of strength and opportunities for future development.

METHODS

We searched for oncology apps in the Apple iOS and Google Play app stores in January 2020. Apps were classified by English language support, date of last update, downloads, intended audience, intended purpose, and developer type.

RESULTS

We identified 794 oncology-specific, English language applications; only 257 (32%) met basic recency standards and were considered evaluable. Of evaluable apps, almost half (47%) were found in the Medical Store Category and the majority were free (88%). The most common intended audience was health care professionals (45%), with 28% being geared toward the general public and 27% being intended for patients. The intended function was education for 36%, clinical decision support for 19.5%, and patient support for 18%. Only 23% of education apps and 40% of clinical decision support apps reported any formal app content review process. Web developers created 61.5% of apps, scientific societies created 10%, and hospitals or health care organizations created just 6%. Of 54 studies that used mobile apps in oncology identified by a recent meta-analysis, only two could be matched to commercially available apps from our study, suggesting a substantial divide between investigation and product dissemination.

CONCLUSION

Relatively few oncology-related apps exist in the commercial marketplace, up-to-date apps are uncommon, and there is a notable absence of key oncology stakeholders in app development. Meaningful development opportunities exist.

Electronic Paper Displays in Hospital Operations: Proposal for Deployment and Implementation

BACKGROUND

Display signage is ubiquitous and essential in hospitals to serve several clerical, operational, and clinical functions, including displaying notices, providing directions, and presenting clinical information. These functions improve efficiency and patient engagement, reduce errors, and enhance the continuity of care. Over time, signage has evolved from analog approaches such as whiteboards and handwritten notices to digital displays such as liquid crystal displays, light emitting diodes, and, now, electronic ink displays. Electronic ink displays are paper-like displays that are not backlit and show content by aligning microencapsulated color beads in response to an applied electric current. Power is only required to generate content and not to retain it. These displays are very readable, with low eye strain; minimize the emission of blue light; require minimal power; and can be driven by several data sources, ranging from virtual servers to electronic health record systems. These attributes make adapting electronic ink displays to hospitals an ideal use case.

OBJECTIVE

In this paper, we aimed to outline the use of signage and displays in hospitals with a focus on electronic ink displays. We aimed to assess the advantages and limitations of using these displays in hospitals and outline the various public-facing and patient-facing applications of electronic ink displays. Finally, we aimed to discuss the technological considerations and an implementation framework that must be followed when adopting and deploying electronic ink displays.

METHODS

The public-facing applications of electronic ink displays include signage and way-finders, timetables for shared workspaces, and noticeboards and bulletin boards. The clinical display applications may be smaller form factors such as door signs or bedside cards. The larger, ≥40-inch form factors may be used within patient rooms or at clinical command centers as a digital whiteboard to display general information, patient and clinician information, and care plans. In all these applications, such displays could replace analog whiteboards, noticeboards, and even other digital screens.

RESULTS

We are conducting pilot research projects to delineate best use cases and practices in adopting electronic ink displays in clinical settings. This will entail liaising with key stakeholders, gathering objective logistical and feasibility data, and, ultimately, quantifying and describing the effect on clinical care and patient satisfaction.

CONCLUSIONS

There are several use cases in a clinical setting that may lend themselves perfectly to electronic ink display use. The main considerations to be studied in this adoption are network connectivity, content management, privacy and security robustness, and detailed comparison with existing modalities. Electronic ink displays offer a superior opportunity to future-proof existing practices. There is a need for theoretical considerations and real-world testing to determine if the advantages outweigh the limitations of electronic ink displays.

Alert-based computerized decision support for high-risk hospitalized patients with atrial fibrillation not prescribed anticoagulation: a randomized, controlled trial (AF-ALERT)

AIMS

Despite widely available risk stratification tools, safe and effective anticoagulant options, and guideline recommendations, anticoagulation for stroke prevention in atrial fibrillation (AF) is underprescribed. We created and evaluated an alert-based computerized decision support (CDS) strategy to increase anticoagulation prescription in hospitalized AF patients at high risk for stroke.

METHODS AND RESULTS

We enrolled 458 patients (CHA2DS2-VASc score ≥1) with AF who were not prescribed anticoagulant therapy and were hospitalized at Brigham and Women's Hospital. Patients were randomly allocated, according to Attending Physician of record, to intervention (alert-based CDS) vs. control (no notification). The primary efficacy outcome was the frequency of anticoagulant prescription. The CDS tool assigned 248 patients to the alert group and 210 to the control group. Patients in the alert group were more likely to be prescribed anticoagulation during the hospitalization (25.8% vs. 9.5%, P < 0.0001), at discharge (23.8% vs. 12.9%, P = 0.003), and at 90 days (27.7% vs. 17.1%, P = 0.007). The alert reduced the odds of a composite outcome of death, myocardial infarction (MI), cerebrovascular event, and systemic embolic event at 90 days [11.3% vs. 21.9%, P = 0.002; odds ratio (OR) 0.45; 95% confidence interval (CI) 0.27-0.76]. The alert reduced the odds of MI at 90 days by 87% (1.2% vs. 8.6%, P = 0.0002; OR 0.13; 95% CI 0.04-0.45) and cerebrovascular events or systemic embolism at 90 days by 88% (0% vs. 2.4%, P = 0.02; OR 0.12; 95% CI 0.0-0.91).

CONCLUSION

An alert-based CDS strategy increased anticoagulation in high-risk hospitalized AF patients and reduced major adverse cardiovascular events, including MI and stroke.

CLINICALTRIALS.GOV IDENTIFIER

NCT02339493.

Transmitting Device Identifiers of Implants From the Point of Care to Insurers: A Demonstration Project

BACKGROUND

For implanted devices, an effective postmarket surveillance system does not exist. For medications, the Food and Drug Administration's Sentinel Initiative plays that role, relying mainly on drug codes in insurance claims. Unique device identifiers (UDIs) could play an analogous role for implants, but there is no mandate for providers to include UDIs in claims or for payers to record them. Objections have been raised to incorporating UDIs into claims based on a potential burden on providers.

METHODS

To assess this purported barrier, we modified information systems at 2 provider-payer dyads to allow for the transmission of UDI data from provider to payer. In addition, to illustrate the potential benefit of including device data in claims, we used our data to compare rates of 90-day adverse events after implantation using the electronic health record (EHR) alone with the EHR plus claims.

RESULTS

The software system modifications were modest and performed as designed. Moreover, the level of difficulty of their development and implementation was comparable to that associated with a typical new release of an existing system. In addition, our data demonstrated the ability of claims-based data plus EHR data to reveal a larger percentage of postprocedure adverse events than data from EHRs alone.

CONCLUSIONS

Modifying information systems to allow for the transmission of UDI data from providers to payers should not impose a substantial burden on either. Implementation of a postmarket surveillance system based on such data in claims will require, however, the development of a system analogous to Sentinel.

Association of Display of Patient Photographs in the Electronic Health Record With Wrong-Patient Order Entry Errors

IMPORTANCE

Wrong-patient order entry (WPOE) errors have a high potential for harm; these errors are particularly frequent wherever workflows are complex and multitasking and interruptions are common, such as in the emergency department (ED). Previous research shows that interruptive solutions, such as electronic patient verification forms or alerts, can reduce these types of errors but may be time-consuming and cause alert fatigue.

OBJECTIVE

To evaluate whether the use of noninterruptive display of patient photographs in the banner of the electronic health record (EHR) is associated with a decreased rate of WPOE errors.

DESIGN, SETTING, AND PARTICIPANTS

In this cohort study, data collected as part of care for patients visiting the ED of a large tertiary academic urban hospital in Boston, Massachusetts, between July 1, 2017, and June 31, 2019, were analyzed.

EXPOSURES

In a quality improvement initiative, the ED staff encouraged patients to have their photographs taken by informing them of the intended safety impact.

MAIN OUTCOMES AND MEASURES

The rate of WPOE errors (measured using the retract-and-reorder method) for orders placed when the patient's photograph was displayed in the banner of the EHR vs the rate for patients without a photograph displayed. The primary analysis focused on orders placed in the ED; a secondary analysis included orders placed in any care setting.

RESULTS

A total of 2 558 746 orders were placed for 71 851 unique patients (mean [SD] age, 49.2 [19.1] years; 42 677 (59.4%) female; 55 109 (76.7%) non-Hispanic). The risk of WPOE errors was significantly lower when the patient's photograph was displayed in the EHR (odds ratio, 0.72; 95% CI, 0.57-0.89). After this risk was adjusted for potential confounders using multivariable logistic regression, the effect size remained essentially the same (odds ratio, 0.57; 95% CI, 0.52-0.61). Risk of error was significantly lower in patients with higher acuity levels and among patients whose race was documented as White.

CONCLUSIONS AND RELEVANCE

This cohort study suggests that displaying patient photographs in the EHR provides decision support functionality for enhancing patient identification and reducing WPOE errors while being noninterruptive with minimal risk of alert fatigue. Successful implementation of such a program in an ED setting involves a modest financial investment and requires appropriate engagement of patients and staff.

Digital triage: Novel strategies for population health management in response to the COVID-19 pandemic

The COVID-19 pandemic has created unique challenges for the U.S. healthcare system due to the staggering mismatch between healthcare system capacity and patient demand. The healthcare industry has been a relatively slow adopter of digital innovation due to the conventional belief that humans need to be at the center of healthcare delivery tasks. However, in the setting of the COVID-19 pandemic, artificial intelligence (AI) may be used to carry out specific tasks such as pre-hospital triage and enable clinicians to deliver care at scale. Recognizing that the majority of COVID-19 cases are mild and do not require hospitalization, Partners HealthCare (now Mass General Brigham) implemented a digitally-automated pre-hospital triage solution to direct patients to the appropriate care setting before they showed up at the emergency department and clinics, which would otherwise consume resources, expose other patients and staff to potential viral transmission, and further exacerbate supply-and-demand mismatching. Although the use of AI has been well-established in other industries to optimize supply and demand matching, the introduction of AI to perform tasks remotely that were traditionally performed in-person by clinical staff represents a significant milestone in healthcare operations strategy.

Usage Patterns of a Web-Based Palliative Care Content Platform (PalliCOVID) During the COVID-19 Pandemic

CONTEXT

The COVID-19 pandemic has highlighted the essential role of palliative care to support the delivery of compassionate, goal-concordant patient care. We created the Web-based application, PalliCOVID (https://pallicovid.app/), in April 2020 to provide all clinicians with convenient access to palliative care resources and support. PalliCOVID features evidence-based clinical guidelines, educational content, and institutional protocols related to palliative care for COVID-19 patients. It is a publicly available resource accessible from any mobile device or desktop computer that provides clinicians with access to palliative care guidance across a variety of care settings, including the emergency department, hospital ward, intensive care unit, and primary care practice.

OBJECTIVE

The primary objective of this study was to evaluate usage patterns of PalliCOVID to understand user behavior in relation to this palliative care content platform during the period of the local peak of COVID-19 infection in Massachusetts.

METHODS

We retrospectively analyzed deidentified usage data collected by Google Analytics from the first day of PalliCOVID's launch on April 7, 2020, until May 1, 2020, the time period that encompassed the local peak of the COVID-19 surge in Massachusetts. User access data were collected and summarized by using Google Analytics software that had been integrated into the PalliCOVID Web application.

RESULTS

A total of 2042 users accessed PalliCOVID and viewed 4637 pages from April 7 to May 1, 2020. Users spent an average of 2 minutes and 6 seconds per session. Eighty-one percent of users were first-time visitors, while the remaining 19% were return visitors. Most users accessed PalliCOVID from the United States (87%), with a large proportion of users coming from Boston and the surrounding cities (32% of overall users).

CONCLUSIONS

PalliCOVID is one example of a scalable digital health solution that can bring palliative care resources to frontline clinicians. Analysis of PalliCOVID usage patterns has the potential to inform the improvement of the platform to better meet the needs of its user base and guide future dissemination strategies. The quantitative data presented here, although informative about user behavior, should be supplemented with future qualitative research to further define the impact of this tool and extend our ability to deliver clinical care that is compassionate, rational, and well-aligned with patients' values and goals.

Unscheduled Radiologic Examination Orders in the Electronic Health Record: A Novel Resource for Targeting Ambulatory Diagnostic Errors in Radiology

PURPOSE

The aim of this study was to assess the prevalence of unscheduled radiologic examination orders in an electronic health record, and the proportion of unscheduled orders that are clinically necessary, by modality.

METHODS

This retrospective study was conducted from January to October 2016 at an academic institution. All unscheduled radiologic examination orders were retrieved for seven modalities (CT, MR, ultrasound, obstetric ultrasound, bone densitometry, mammography, and fluoroscopy). After excluding duplicates, 100 randomly selected orders from each modality were assigned to two physician reviewers who classified their clinical necessity, with 10% overlap. Interannotator agreement was assessed using κ statistics, the percentage of clinically necessary unscheduled orders was compared, and χ² analysis was used to assess differences by modality.

RESULTS

A total 494,503 radiologic examination orders were placed during the study period. After exclusions, 33,546 unscheduled orders were identified, 7% of all radiologic examination orders. Among 700 reviewed unscheduled orders, agreement was substantial (κ = 0.63). Eighty-seven percent of bone densitometric examinations and sixty-five percent of mammographic studies were considered clinically necessary, primarily for follow-up management. The majority of orders in each modality were clinically necessary, except for CT, obstetric ultrasound, and fluoroscopy (P < .0001).

CONCLUSIONS

Large numbers of radiologic examination orders remain unscheduled in the electronic health record. A substantial portion are clinically necessary, representing potential delays in executing documented provider care plans. Clinically unnecessary unscheduled orders may inadvertently be scheduled and performed. Identifying and performing clinically necessary unscheduled radiologic examination orders may help reduce diagnostic errors related to diagnosis and treatment delays and enhance patient safety, while eliminating clinically unnecessary unscheduled orders will help avoid unneeded testing.

Surgical Informatics: Defining the Role of Informatics in the Current Surgical Training Paradigm

OBJECTIVE

Clinical Informatics is a large part of the current surgical practice, yet, there is no formal curriculum to teach residents about informatics. The purpose of this article is to describe the field of clinical informatics, describe the potential of informatics to improve surgical care, and propose a framework to train surgical informaticians.

DESIGN

This is a perspective summarizing the role of clinical informatics in current surgical training.

CONCLUSIONS

The manuscript concisely discusses the potential impact of novel technologies and information systems on surgical care delivery. We demonstrate the importance of informatics training for surgeons and describe the relatively new fellowship program as well as the various pathways to obtain board certification in clinical informatics.

The Utility and Feasibility of Extending Beyond Traditional Patient Descriptions in Daily Practice

OBJECTIVES/HYPOTHESIS

Institutional and national programs have endorsed the use of validated instruments in daily practice, but their utility for frontline clinicians is unknown. Our first objective was thus to determine if routinely obtained patient descriptions accurately predict disease-specific validated instrument results. Our second objective was to assess the feasibility of deploying an electronic system to obtain and document patient-reported descriptions of health status.

STUDY DESIGN

Prospective assessment of consecutive eligible ambulatory care patients.

METHODS

For the first objective, patients presenting with the same chief complaint at consecutive visits provided concurrent routine symptom descriptions and validated instrument responses. To determine whether patients' routine descriptions predicted scores from disease-specific instruments, receiver operator characteristic (ROC) curves, sensitivity/specificity, Spearman ρ correlation, and regression models were utilized. For the second objective, feasibility assessment focused on percent with successful response capture, time to questionnaire completion, and impact on physician time needed to obtain and document patient histories.

RESULTS

For the first objective, among patients with hearing loss, eustachian tube dysfunction, nasal obstruction, and chronic rhinosinusitis, the areas under the ROC curves were 0.38 to 0.83. Spearman ρ coefficients were 0.25 to 0.46. Sensitivity and specificity ranged from 15.9% to 87.8%, with correct classification in 42.3% to 55.2% of cases. For the second objective, there was a 91% completion rate. Completion times were 7:18 to 12:37 for new patients and 3:23 to 6:41 for established patients. Physician time savings were estimated at 11 minutes per patient.

CONCLUSIONS

Using validated instruments and electronic data collection systems in daily practice has practical and clinical implications. These topics warrant further study.

LEVEL OF EVIDENCE

2c Laryngoscope, 130:S1-S13, 2020.

Cybersecurity features of digital medical devices: an analysis of FDA product summaries

OBJECTIVES

To more clearly define the landscape of digital medical devices subject to US Food and Drug Administration (FDA) oversight, this analysis leverages publicly available regulatory documents to characterise the prevalence and trends of software and cybersecurity features in regulated medical devices.

DESIGN

We analysed data from publicly available FDA product summaries to understand the frequency and recent time trends of inclusion of software and cybersecurity content in publicly available product information.

SETTING

The full set of regulated medical devices, approved over the years 2002-2016 included in the FDA's 510(k) and premarket approval databases.

PRIMARY AND SECONDARY OUTCOME MEASURES

The primary outcome was the share of devices containing software that included cybersecurity content in their product summaries. Secondary outcomes were differences in these shares (a) over time and (b) across regulatory areas.

RESULTS

Among regulated devices, 13.79% were identified as including software. Among these products, only 2.13% had product summaries that included cybersecurity content over the period studied. The overall share of devices including cybersecurity content was higher in recent years, growing from an average of 1.4% in the first decade of our sample to 5.5% in 2015 and 2016, the most recent years included. The share of devices including cybersecurity content also varied across regulatory areas from a low of 0% to a high of 22.2%.

CONCLUSIONS

To ensure the safest possible healthcare delivery environment for patients and hospitals, regulators and manufacturers should work together to make the software and cybersecurity content of new medical devices more easily accessible.

An Internet of Things Buttons to Measure and Respond to Restroom Cleanliness in a Hospital Setting: Descriptive Study

Background

Restroom cleanliness is an important factor in hospital quality. Due to its dynamic process, it can be difficult to detect the presence of dirty restrooms that need to be cleaned. Using an Internet of Things (IoT) button can permit users to designate restrooms that need cleaning and in turn, allow prompt response from housekeeping to maintain real-time restroom cleanliness.

Objective

This study aimed to describe the deployment of an IoT button–based notification system to measure hospital restroom cleanliness reporting system usage and qualitative feedback from housekeeping staff on IoT button use.

Methods

We deployed IoT buttons in 16 hospital restrooms. Over an 8-month period, housekeeping staff received real-time notifications and responded to button presses for restroom cleaning. All button presses were recorded. We reported average button usage by hospital area, time of day, and day of week. We also conducted interviews with housekeeping supervisors and staff to understand their acceptance of and experience with the system.

Results

Over 8 months, 1920 requests to clean restrooms in the main hospital lobby and satellite buildings were received. The hospital lobby IoT buttons received over half (N=1055, 55%) of requests for cleaning. Most requests occurred in afternoon hours from 3 PM to midnight. Requests for cleaning remained stable throughout the work week with fewer requests occurring over weekends. IoT button use was sustained throughout the study period. Interviews with housekeeping supervisors and staff demonstrated acceptance of the IoT buttons; actual use was centered around asynchronous communication between supervisors and staff in response to requests to clean restrooms.

Conclusions

An IoT button system is a feasible method to generate on-demand request for restroom cleaning that is easy to deploy and that users will consistently engage with. Data from this system have the potential to enable responsive scheduling for restroom service and anticipate periods of high restroom utilization in a hospital.

The opportunities and shortcomings of using big data and national databases for sarcoma research

The rarity and heterogeneity of sarcomas make performing appropriately powered studies challenging and magnify the significance of large databases in sarcoma research. Established large tumor registries and population-based databases have become increasingly relevant for answering clinical questions regarding sarcoma incidence, treatment patterns, and outcomes. However, the validity of large databases has been questioned and scrutinized because of the inaccuracy and wide variability of coding practices and the absence of clinically relevant variables. In addition, the utilization of large databases for the study of rare cancers such as sarcoma may be particularly challenging because of the known limitations of administrative data and poor overall data quality. Currently, there are several large national cancer databases, including the Surveillance, Epidemiology, and End Results database, the National Cancer Data Base of the American College of Surgeons and the American Cancer Society, and the National Program of Cancer Registries of the Centers for Disease Control and Prevention. These databases are often used for sarcoma research, but they are limited by their dependence on administrative or billing data, the lack of agreement between chart abstractors on diagnosis codes, and the use of preexisting documented hospital diagnosis codes for tumor registries, which lead to a significant underestimation of sarcomas in large data sets. Current and future initiatives to improve databases and big data applications for sarcoma research include increasing the utilization of sarcoma-specific registries and encouraging national initiatives to expand on real-world, evidence-based data sets.

An electronic notification system for improving patient flow in the emergency department

Healthcare providers make time-sensitive care decisions based on EHR data. As systems of record, the EHR is often not configured to optimally surface timely information. For patients awaiting admission, infection control concerns that potentially require private rooms can prolong stays in the Emergency Department. We aim to determine if an event-based notification platform connected with a commercial EHR can help prioritize timely information and improve patient flow in the emergency department. We undertook a pre-post analysis for patients being admitted from the emergency room who were tested for influenza. We used a primary outcome of mean time from negative test result to inpatient transfer. The median time decreased by 27%, from 4.1 hours to 3.0 hours. The distribution of transfer times pre and post-intervention were significantly different with a p-value of <0.001. Our findings support the use of event-based notification systems to improve patient flow in the emergency department.

Assessment of Employee Susceptibility to Phishing Attacks at US Health Care Institutions

IMPORTANCE

Cybersecurity is an increasingly important threat to health care delivery, and email phishing is a major attack vector against hospital employees.

OBJECTIVE

To describe the practice of phishing simulation and the extent to which health care employees are vulnerable to phishing simulations.

DESIGN, SETTING, AND PARTICIPANTS

Retrospective, multicenter quality improvement study of a convenience sample of 6 geographically dispersed US health care institutions that ran phishing simulations from August 1, 2011, through April 10, 2018. The specific institutions are anonymized herein for security and privacy concerns.

EXPOSURES

Simulated phishing emails received by employees at US health care institutions.

MAIN OUTCOMES AND MEASURES

Date of phishing campaign, campaign number, number of emails sent, number of emails clicked, and email content. Emails were classified into 3 categories (office related, personal, or information technology related).

RESULTS

The final study sample included 6 anonymized US health care institutions, 95 simulated phishing campaigns, and 2 971 945 emails, 422 062 of which were clicked (14.2%). The median institutional click rates for campaigns ranged from 7.4% (interquartile range [IQR], 5.8%-9.6%) to 30.7% (IQR, 25.2%-34.4%), with an overall median click rate of 16.7% (IQR, 8.3%-24.2%) across all campaigns and institutions. In the regression model, repeated phishing campaigns were associated with decreased odds of clicking on a subsequent phishing email (adjusted OR, 0.511; 95% CI, 0.382-0.685 for 6-10 campaigns; adjusted OR, 0.335; 95% CI, 0.282-0.398 for >10 campaigns).

CONCLUSIONS AND RELEVANCE

Among a sample of US health care institutions that sent phishing simulations, almost 1 in 7 simulated emails sent were clicked on by employees. Increasing campaigns were associated with decreased odds of clicking on a phishing email, suggesting a potential benefit of phishing simulation and awareness. With cyberattacks increasing against US health care systems, these click rates represent a major cybersecurity risk for hospitals.

Characteristics of the National Applicant Pool for Clinical Informatics Fellowships (2016-2017)

We conducted a national study to assess the numbers and diversity of applicants for 2016 and 2017 clinical informatics fellowship positions. In each year, we collected data on the number of applications that programs received from candidates who were ultimately successful vs. unsuccessful. In 2017, we also conducted an anonymous applicant survey. Successful candidates applied to an average of 4.2 and 5.5 programs for 2016 and 2017, respectively. In the survey, unsuccessful candidates reported applying to fewer programs. Assuming unsuccessful candidates submitted between 2-5 applications each, the total applicant pool numbered 42-69 for 2016 (competing for 24 positions) and 52-85 for 2017 (competing for 30 positions). Among survey respondents (n=33), 24% were female, 1 was black and none were Hispanic. We conclude that greater efforts are needed to enhance interest in clinical informatics among medical students and residents, particularly among women and members of underrepresented minority groups.

Assessing Documentation of Critical Imaging Result Follow-up Recommendations in Emergency Department Discharge Instructions

To facilitate follow-up of critical test results across transitions in patient care settings, we implemented an electronic discharge module that enabled care providers to include follow-up recommendations in the discharge instructions. We assessed the impact of this module on documentation of follow-up recommendations for critical imaging findings in Emergency Department (ED) discharge instructions. We studied 240 patients with critical imaging findings discharged from the ED before (n = 80) and after (n = 160) implementation of the module. We manually reviewed hand-written forms and electronic discharge instructions to determine if follow-up recommendations were documented. Follow-up recommendations in ED discharge instructions increased from 60.0% (48/80) to 73.8% (118/160) post-module implementation (p = 0.03), a relative increase of 23%. There was no significant change in the rate of documented critical imaging findings in the discharge instructions (77.5% [62/80] before the intervention and 76.9% [123/160] after the intervention; p = 0.91). Implementation of a discharge module was associated with increased documentation of critical imaging finding follow-up recommendations in ED discharge instructions. However, one in four patients still did not receive adequate follow-up recommendations, suggesting further opportunities for performance improvement exist.

Analysis of Errors in Dictated Clinical Documents Assisted by Speech Recognition Software and Professional Transcriptionists

IMPORTANCE

Accurate clinical documentation is critical to health care quality and safety. Dictation services supported by speech recognition (SR) technology and professional medical transcriptionists are widely used by US clinicians. However, the quality of SR-assisted documentation has not been thoroughly studied.

OBJECTIVE

To identify and analyze errors at each stage of the SR-assisted dictation process.

DESIGN SETTING AND PARTICIPANTS

This cross-sectional study collected a stratified random sample of 217 notes (83 office notes, 75 discharge summaries, and 59 operative notes) dictated by 144 physicians between January 1 and December 31, 2016, at 2 health care organizations using Dragon Medical 360 | eScription (Nuance). Errors were annotated in the SR engine-generated document (SR), the medical transcriptionist-edited document (MT), and the physician's signed note (SN). Each document was compared with a criterion standard created from the original audio recordings and medical record review.

MAIN OUTCOMES AND MEASURES

Error rate; mean errors per document; error frequency by general type (eg, deletion), semantic type (eg, medication), and clinical significance; and variations by physician characteristics, note type, and institution.

RESULTS

Among the 217 notes, there were 144 unique dictating physicians: 44 female (30.6%) and 10 unknown sex (6.9%). Mean (SD) physician age was 52 (12.5) years (median [range] age, 54 [28-80] years). Among 121 physicians for whom specialty information was available (84.0%), 35 specialties were represented, including 45 surgeons (37.2%), 30 internists (24.8%), and 46 others (38.0%). The error rate in SR notes was 7.4% (ie, 7.4 errors per 100 words). It decreased to 0.4% after transcriptionist review and 0.3% in SNs. Overall, 96.3% of SR notes, 58.1% of MT notes, and 42.4% of SNs contained errors. Deletions were most common (34.7%), then insertions (27.0%). Among errors at the SR, MT, and SN stages, 15.8%, 26.9%, and 25.9%, respectively, involved clinical information, and 5.7%, 8.9%, and 6.4%, respectively, were clinically significant. Discharge summaries had higher mean SR error rates than other types (8.9% vs 6.6%; difference, 2.3%; 95% CI, 1.0%-3.6%; P < .001). Surgeons' SR notes had lower mean error rates than other physicians' (6.0% vs 8.1%; difference, 2.2%; 95% CI, 0.8%-3.5%; P = .002). One institution had a higher mean SR error rate (7.6% vs 6.6%; difference, 1.0%; 95% CI, -0.2% to 2.8%; P = .10) but lower mean MT and SN error rates (0.3% vs 0.7%; difference, -0.3%; 95% CI, -0.63% to -0.04%; P = .03 and 0.2% vs 0.6%; difference, -0.4%; 95% CI, -0.7% to -0.2%; P = .003).

CONCLUSIONS AND RELEVANCE

Seven in 100 words in SR-generated documents contain errors; many errors involve clinical information. That most errors are corrected before notes are signed demonstrates the importance of manual review, quality assurance, and auditing.

A Novel Mobile App and Population Management System to Manage Rheumatoid Arthritis Flares: Protocol for a Randomized Controlled Trial

Background

Rheumatoid arthritis flares have a profound effect on patients, causing pain and disability. However, flares often occur between regularly scheduled health care provider visits and are, therefore, difficult to monitor and manage. We sought to develop a mobile phone app combined with a population management system to help track RA flares between visits.

Objective

The objective of this study is to implement the mobile app plus the population management system to monitor rheumatoid arthritis disease activity between scheduled health care provider visits over a period of 6 months.

Methods

This is a randomized controlled trial that lasts for 6 months for each participant. We aim to recruit 190 patients, randomized 50:50 to the intervention group versus the control group. The intervention group will be assigned the mobile app and be prompted to answer daily questionnaires sent to their mobile devices. Both groups will be assigned a population manager, who will communicate with the participants via telephone at 6 weeks and 18 weeks. The population manager will also communicate with the participants in the intervention group if their responses indicate a sustained increase in rheumatoid arthritis disease activity. To assess patient satisfaction, the primary outcomes will be scores on the Treatment Satisfaction Questionnaire for Medication as well as the Perceived Efficacy in Patient-Physician Interactions questionnaire at 6 months. To determine the effect of the mobile app on rheumatoid arthritis disease activity, the primary outcome will be the Clinical Disease Activity Index at 6 months.

Results

The trial started in November 2016, and an estimated 2.5 years will be necessary to complete the study. Study results are expected to be published by the end of 2019.

Conclusions

The completion of this study will provide important data regarding the following: (1) the assessment of validated outcome measures to assess rheumatoid arthritis disease activity with a mobile app between routinely scheduled health care provider visits, (2) patient engagement in monitoring their condition, and (3) communication between patients and health care providers through the population management system.

Trial Registration

ClinicalTrials.gov NCT02822521, http://clinicaltrials.gov/ct2/show/NCT02822521 (Archived by WebCite at http://www.webcitation.org/6xed3kGPd)

Core Components for a Clinically Integrated mHealth App for Asthma Symptom Monitoring

Background mHealth apps may be useful tools for supporting chronic disease management. Objective Our aim was to apply user-centered design principles to efficiently identify core components for an mHealth-based asthma symptom–monitoring intervention using patient-reported outcomes (PROs). Methods We iteratively combined principles of qualitative research, user-centered design, and “gamification” to understand patients' and providers' needs, develop and refine intervention components, develop prototypes, and create a usable mobile app to integrate with clinical workflows. We identified anticipated benefits and burdens for stakeholders. Results We conducted 19 individual design sessions with nine adult patients and seven clinicians from an academic medical center (some were included multiple times). We identified four core intervention components: (1) Invitation—patients are invited by their physicians. (2) Symptom checks—patients receive weekly five-item questionnaires via the app with 48 hours to respond. Depending on symptoms, patients may be given the option to request a call from a nurse or receive one automatically. (3) Patient review—in the app, patients can view their self-reported data graphically. (4) In-person visit—physicians have access to patient-reported symptoms in the electronic health record (EHR) where they can review them before in-person visits. As there is currently no location in the EHR where physicians would consistently notice these data, recording a recent note was the best option. Benefits to patients may include helping decide when to call their provider and facilitating shared decision making. Benefits to providers may include saving time discussing symptoms. Provider organizations may need to pay nurses extra, but those costs may be offset by reduced visits and hospitalizations. Conclusion Recent systematic reviews show inconsistent outcomes and little insight into functionalities required for mHealth asthma interventions, highlighting the need for systematic intervention design. We identified specific features for adoption and engagement that meet the stated needs of users for asthma symptom monitoring.

Catalyzing healthcare transformation with digital health: Performance indicators and lessons learned from a Digital Health Innovation Group

Despite considerable investment in digital health (DH) companies and a growing DH ecosystem, there are multiple challenges to testing and implementing innovative solutions. Health systems have recognized the potential of DH and have formed DH innovation centers. However, limited information is available on DH innovation center processes, best practices, or outcomes. This case report describes a DH innovation center process that can be replicated across health systems and defines and benchmarks process indicators to assess DH innovation center performance. The Brigham and Women's Hospital's Digital Health Innovation Group (DHIG) accelerates DH innovations from idea to pilot safely and efficiently using a structured process. Fifty-four DH innovations were accelerated by the DHIG process between July 2014 and December 2016. In order to measure effectiveness of the DHIG process, key process indicators were defined as 1) number of solutions that completed each DHIG phase and 2) length of time to complete each phase. Twenty-three DH innovations progressed to pilot stage and 13 innovations were terminated after barriers to pilot implementation were identified by the DHIG process. For 4 DH solutions that executed a pilot, the average time for innovations to proceed from DHIG intake to pilot initiation was 9 months. Overall, the DHIG is a reproducible process that addresses key roadblocks in DH innovation within health systems. To our knowledge, this is the first report to describe DH innovation process indicators and results within an academic health system. Therefore, there is no published data to compare our results with the results of other DH innovation centers. Standardized data collection and indicator reporting could allow benchmark comparisons across institutions. Additional opportunities exist for the validation of DH solution effectiveness and for translational support from pilot to implementation. These are critical steps to advance DH technologies and effectively leverage the DH ecosystem to transform healthcare.

A Custom-Developed Emergency Department Provider Electronic Documentation System Reduces Operational Efficiency

STUDY OBJECTIVE

Electronic health record implementation can improve care, but may also adversely affect emergency department (ED) efficiency. We examine how a custom, ED provider, electronic documentation system (eDoc), which replaced paper documentation, affects operational performance.

METHODS

We analyzed retrospective operational data for 1-year periods before and after eDoc implementation in a single ED. We computed daily operational statistics, reflecting 60,870 pre- and 59,337 postimplementation patient encounters. The prespecified primary outcome was daily mean length of stay; secondary outcomes were daily mean length of stay for admitted and discharged patients and daily mean arrival time to disposition for admitted patients. We used a prespecified multiple regression model to identify differences in outcomes while controlling for prespecified confounding variables.

RESULTS

The unadjusted change in length of stay was 8.4 minutes; unadjusted changes in secondary outcomes were length of stay for admitted patients 11.4 minutes, length of stay for discharged patients 1.8 minutes, and time to disposition 1.8 minutes. With a prespecified regression analysis to control for variations in operational characteristics, there were significant increases in length of stay (6.3 minutes [95% confidence interval 3.5 to 9.1 minutes]) and length of stay for discharged patients (5.1 minutes [95% confidence interval 1.9 to 8.3 minutes]). There was no statistically significant change in length of stay for admitted patients or time to disposition.

CONCLUSION

In our single-center study, the isolated implementation of eDoc was associated with increases in overall and discharge length of stay. Our findings suggest that a custom-designed electronic provider documentation may negatively affect ED throughput. Strategies to mitigate these effects, such as reducing documentation requirements or adding clinical staff, scribes, or voice recognition, would be a valuable area of future research.

The Benefits and Challenges of an Interfaced Electronic Health Record and Laboratory Information System: Effects on Laboratory Processes

Context.—

A recent government regulation incentivizes implementation of an electronic health record (EHR) with computerized order entry and structured results display. Many institutions have also chosen to interface their EHR with their laboratory information system (LIS).

Objective.—

To determine the impact of an interfaced EHR-LIS on laboratory processes.

Design.—

We analyzed several different processes before and after implementation of an interfaced EHR-LIS: the turnaround time, the number of stat specimens received, venipunctures per patient per day, preanalytic errors in phlebotomy, the number of add-on tests using a new electronic process, and the number of wrong test codes ordered. Data were gathered through the LIS and/or EHR.

Results.—

The turnaround time for potassium and hematocrit decreased significantly (P = .047 and P = .004, respectively). The number of stat orders also decreased significantly, from 40% to 7% for potassium and hematocrit, respectively (P &lt; .001 for both). Even though the average number of inpatient venipunctures per day increased from 1.38 to 1.62 (P &lt; .001), the average number of preanalytic errors per month decreased from 2.24 to 0.16 per 1000 specimens (P &lt; .001). Overall there was a 16% increase in add-on tests. The number of wrong test codes ordered was high and it was challenging for providers to correctly order some common tests.

Conclusions.—

An interfaced EHR-LIS significantly improved within-laboratory turnaround time and decreased stat requests and preanalytic phlebotomy errors. Despite increasing the number of add-on requests, an electronic add-on process increased efficiency and improved provider satisfaction. Laboratories implementing an interfaced EHR-LIS should be cautious of its effects on test ordering and patient venipunctures per day.

Significant Reduction in Preanalytical Errors for Nonphlebotomy Blood Draws After Implementation of a Novel Integrated Specimen Collection Module

OBJECTIVES

Most preanalytical errors at our institution occur during nonphlebotomy blood draws. We implemented an electronic health record (EHR), interfaced the EHR to the laboratory information system, and designed a new specimen collection module. We studied the effects of the new system on nonphlebotomy preanalytical errors.

METHODS

We used an electronic database of preanalytical errors and calculated the number and type of the most common errors in the emergency department (ED) and inpatient nursing for 3-month periods before (August-October 2014) and after (August-October 2015) implementation. The level of staff compliance with the new system was also assessed.

RESULTS

The average monthly preanalytical errors decreased significantly from 7.95 to 1.45 per 1,000 specimens in the ED (P < 0001) and 11.75 to 3.25 per 1,000 specimens in inpatient nursing (P < 0001). The rate of decrease was similar for mislabeled, unlabeled, wrong specimen received and no specimen received errors. Most residual errors (80% in the ED and 67% in inpatient nursing) occurred when providers did not use the new system as designed.

CONCLUSIONS

Implementation of a customized specimen collection module led to a significant reduction in preanalytical errors. Improved compliance with the system may lead to further reductions in error rates.

Usability of the Massachusetts Prescription Drug Monitoring Program in the Emergency Department: A Mixed-methods Study

OBJECTIVES

Prescription drug monitoring programs (PDMPs) are underutilized, despite evidence showing that they may reduce the epidemic of opioid-related addiction, diversion, and overdose. We evaluated the usability of the Massachusetts (MA) PDMP by emergency medicine providers (EPs), as a system's usability may affect how often it is used.

METHODS

This was a mixed-methods study of 17 EPs. We compared the time and number of clicks required to review one patient's record in the PDMP to three other commonly performed computer-based tasks in the emergency department (ED: ordering a computed tomography [CT] scan, writing a prescription, and searching a medication history service integrated within the electronic medical record [EMR]). We performed semistructured interviews and analyzed participant comments and responses regarding their experience using the MA PDMP.

RESULTS

The PDMP task took a longer time to complete (mean = 4.22 minutes) and greater number of mouse clicks to complete (mean = 50.3 clicks) than the three other tasks (CT-pulmonary embolism = 1.42 minutes, 24.8 clicks; prescription = 1.30 minutes, 19.5 clicks; SureScripts = 1.45 minutes, 9.5 clicks). Qualitative analysis yielded four main themes about PDMP usability, three negative and one positive: 1) difficulty accessing the PDMP, 2) cumbersome acquiring patient medication history information within the PDMP, 3) nonintuitive display of patient medication history information within the PDMP, and 4) overall perceived value of the PDMP despite an inefficient interface.

CONCLUSIONS

The complicated processes of gaining access to, logging in, and using the MA PDMP are barriers to preventing its more frequent use. All states should evaluate the PDMP usability in multiple practice settings including the ED and work to improve provider enrollment, login procedures, patient information input, prescription data display, and ultimately, PDMP data integration into EMRs.

Interrater Reliability of mHealth App Rating Measures: Analysis of Top Depression and Smoking Cessation Apps

Background

There are over 165,000 mHealth apps currently available to patients, but few have undergone an external quality review. Furthermore, no standardized review method exists, and little has been done to examine the consistency of the evaluation systems themselves.

Objective

We sought to determine which measures for evaluating the quality of mHealth apps have the greatest interrater reliability.

Methods

We identified 22 measures for evaluating the quality of apps from the literature. A panel of 6 reviewers reviewed the top 10 depression apps and 10 smoking cessation apps from the Apple iTunes App Store on these measures. Krippendorff’s alpha was calculated for each of the measures and reported by app category and in aggregate.

Results

The measure for interactiveness and feedback was found to have the greatest overall interrater reliability (alpha=.69). Presence of password protection (alpha=.65), whether the app was uploaded by a health care agency (alpha=.63), the number of consumer ratings (alpha=.59), and several other measures had moderate interrater reliability (alphas>.5). There was the least agreement over whether apps had errors or performance issues (alpha=.15), stated advertising policies (alpha=.16), and were easy to use (alpha=.18). There were substantial differences in the interrater reliabilities of a number of measures when they were applied to depression versus smoking apps.

Conclusions

We found wide variation in the interrater reliability of measures used to evaluate apps, and some measures are more robust across categories of apps than others. The measures with the highest degree of interrater reliability tended to be those that involved the least rater discretion. Clinical quality measures such as effectiveness, ease of use, and performance had relatively poor interrater reliability. Subsequent research is needed to determine consistent means for evaluating the performance of apps. Patients and clinicians should consider conducting their own assessments of apps, in conjunction with evaluating information from reviews.

The effect of electronic health record implementation on community emergency department operational measures of performance

STUDY OBJECTIVE

We study the effect of an emergency department (ED) electronic health record implementation on the operational metrics of a diverse group of community EDs.

METHODS

We performed a retrospective before/after analysis of 23 EDs from a single management group that experienced ED electronic health record implementation (with the majority of electronic health records optimized specifically for ED use). We obtained electronic data for 4 length of stay measures (arrival to provider, admitted, discharged, and overall length of stay) and 4 measures of operational characteristics (left before treatment complete, significant returns, overall patient satisfaction, and provider efficiency). We compared the 6-month "baseline" period immediately before implementation with a "steady-state" period commencing 6 months after implementation for all 8 metrics.

RESULTS

For the length of stay measures, there were no differences in the arrival-to-provider interval (difference of -0.02 hours; 95% confidence interval [CI] of difference -0.12 to 0.08), admitted length of stay (difference of 0.10 hours; 95% CI of difference -0.17 to 0.37), discharged length of stay (difference of 0.07 hours; 95% CI of difference -0.07 to 0.22), and overall length of stay (difference of 0.11 hours; 95% CI of difference -0.04 to 0.27). For operational characteristics, there were no differences in the percentage who left before treatment was complete (difference of 0.24%; 95% CI of difference -0.47% to 0.95%), significant returns (difference of -0.04%; 95% CI of difference -0.48% to 0.39%), overall percentile patient satisfaction (difference of -0.02%; 95% CI of difference -2.35% to 2.30%), and provider efficiency (difference of -0.05 patients/hour; 95% CI of difference -0.11 to 0.02).

CONCLUSION

There is no meaningful difference in 8 measures of operational performance for community EDs experiencing optimized ED electronic health record implementation between a baseline and steady-state period.