Challenges and Paradoxes of Human Factors in Health Technology Design

Novel Note Templates to Enhance Signal and Reduce Noise in Medical Documentation: a Prospective Improvement Study

BACKGROUND

The introduction of electronic workflows has allowed for the flow of raw un-contextualized clinical data into medical documentation. As a result, many electronic notes have become replete of "noise" and deplete of clinically significant "signals". There is an urgent need to develop and implement innovative approaches in electronic clinical documentation that improve note quality and reduce unnecessary bloating.

OBJECTIVE

To describe the development and impact of a novel set of templates designed to change the flow of information in medical documentation.

METHODS

This is a multi-hospital nonrandomized prospective improvement study conducted on the Inpatient General Internal Medicine Service across three hospital campuses at the New York University (NYU) Langone Health System. A group of physician leaders representing each campus met biweekly for six months. The output of these meetings included 1) a conceptualization of the note bloat problem as a dysfunction in information flow 2) a set of guiding principles for organizational documentation improvement 3) the design and build of novel electronic templates that reduced the flow of extraneous information into provider notes by providing link outs to best practice data visualizations and 4) a documentation improvement curriculum for inpatient medicine providers. Prior to go-live, pragmatic usability testing was performed with the new progress note template, and the overall user experience measured using the System Usability Scale (SUS). Primary outcomes measures after go-live include template utilization rate and note length in characters.

RESULTS

In usability testing amongst 22 medicine providers, the new progress note template averaged a usability score of 90.6/100 on the System Usability Scale. 77% of providers strongly agreed that the new template was easy to use. 68% strongly agreed that they would like to use the template frequently. In the three months after template implementation, General Internal Medicine providers wrote 65% of all inpatient notes with the new templates. During this period of time the organization saw a 46%, 47%, and 32% reduction in note length for general medicine progress notes, consults, and H&Ps, respectively, when compared to a baseline measurement period prior to interventions.

CONCLUSIONS

A bundled intervention that included deployment of novel templates for inpatient general medicine providers significantly reduced average note length on the clinical service. Templates designed to reduce the flow of extraneous information into provider notes performed well during usability testing, and these templates were rapidly adopted across all hospital campuses. Further research is needed to assess the impact of novel templates on note quality, provider efficiency and patient outcomes.

CLINICALTRIAL

Pretransplant Patient Education in Solid-organ Transplant: A Narrative Review

Education for pretransplant, solid-organ recipient candidates aims to improve knowledge and understanding about the transplant process, outcomes, and potential complications to support informed, shared decision-making to reduce fears and anxieties about transplant, inform expectations, and facilitate adjustment to posttransplant life. In this review, we summarize novel pretransplant initiatives and approaches to educate solid-organ transplant recipient candidates. First, we review approaches that may be common to all solid-organ transplants, then we summarize interventions specific to kidney, liver, lung, and heart transplant. We describe evidence that emphasizes the need for multidisciplinary approaches to transplant education. We also summarize initiatives that consider online (eHealth) and mobile (mHealth) solutions. Finally, we highlight education initiatives that support racialized or otherwise marginalized communities to improve equitable access to solid-organ transplant. A considerable amount of work has been done in solid-organ transplant since the early 2000s with promising results. However, many studies on education for pretransplant recipient candidates involve relatively small samples and nonrandomized designs and focus on short-term surrogate outcomes. Overall, many of these studies have a high risk of bias. Frequently, interventions assessed are not well characterized or they are combined with administrative and data-driven initiatives into multifaceted interventions, which makes it difficult to assess the impact of the education component on outcomes. In the future, well-designed studies rigorously assessing well-defined surrogate and clinical outcomes will be needed to evaluate the impact of many promising initiatives.

Extended Reality for Enhanced Telehealth During and Beyond COVID-19: Viewpoint

The COVID-19 pandemic caused widespread challenges and revealed vulnerabilities across global health care systems. In response, many health care providers turned to telehealth solutions, which have been widely embraced and are likely to become standard for modern care. Immersive extended reality (XR) technologies have the potential to enhance telehealth with greater acceptability, engagement, and presence. However, numerous technical, logistic, and clinical barriers remain to the incorporation of XR technology into telehealth practice. COVID-19 may accelerate the union of XR and telehealth as researchers explore novel solutions to close social distances. In this viewpoint, we highlight research demonstrations of XR telehealth during the COVID-19 pandemic and discuss future directions to make XR the next evolution of remote health care.

Home-Based Pediatric Palliative Care and Electronic Health: Systematic Mixed Methods Review

Background

Children and families in pediatric palliative care depend on close contact with health care personnel, and electronic health (eHealth) is suggested to support care at home by facilitating their remote interactions.

Objective

This study aimed to identify and review the use of eHealth to communicate and support home-based pediatric palliative care and appraise the methodological quality of the published research.

Methods

We conducted a convergent, systematic mixed methods review and searched Medical Literature Analysis and Retrieval System Online (Medline), EMBASE, PsycINFO, Cochrane Library, Cumulative Index to Nursing and Allied Health Literature (CINAHL), Web of Science, and Scopus for eligible papers. Studies evaluating 2-way communication technology for palliative care for children aged ≤18 years and applying quantitative, qualitative, or mixed methods from 2012 to 2018 were eligible for inclusion. Quantitative and qualitative studies were equally valued during the search, screening, extraction, and analysis. Quantitative data were transformed into qualitative data and analyzed using a thematic analysis. Overall, 2 independent researchers methodologically appraised all included studies.

Results

We identified 1277 citations. Only 7 papers were eligible for review. Evaluating eHealth interventions in pediatric palliative care poses specific methodological and ethical challenges. eHealth to facilitate remote pediatric palliative care was acknowledged both as an intrusion and as a support at home. Reluctance toward eHealth was mainly identified among professionals.

Conclusions

The strengths of the conclusions are limited by the studies’ methodological challenges. Despite the limitless possibilities held by new technologies, research on eHealth in home-based pediatric palliative care is scarce. The affected children and families appeared to hold positive attitudes toward eHealth, although their views were less apparent compared with those of the professionals.

Trial Registration

PROSPERO CRD42018119051; https://tinyurl.com/rtsw5ky

Opportunities and Challenges of Telehealth in Remote Communities: Case Study of the Yukon Telehealth System

Background

Telehealth has been shown to improve access to health care and to reduce costs to the patient and health care system, especially for patients living in rural settings. However, unique challenges arise when implementing telehealth in remote communities.

Objective

The study aimed to explore the current use, challenges, and opportunities of the Yukon Telehealth System. The lessons learned from this study were used to determine important factors to consider when attempting to advance and expand telehealth programs in remote communities.

Methods

A mixed methods approach was used to evaluate the Yukon Telehealth System and to determine possible future advances. Quantitative data were obtained through usage logs. Web-based questionnaires were administered to nurses in each of the 14 Yukon community health centers outside of Whitehorse and patients who had used telehealth. Qualitative data included focus groups and semistructured interviews with 36 telehealth stakeholders.

Results

Since 2008, there has been a consistent number of telehealth sessions of about 1000 per year, with clinical care as the main use (69.06% [759/1099] of all sessions in 2015). From the questionnaire (11 community nurses and 10 patients) and the interview data, there was a consensus among the clinicians and patients that the system provided timely access and cost savings from reduced travel. However, they believed that it was underutilized, and the equipment was outdated. The following 4 factors were identified, which should be considered when trying to advance and expand a telehealth program: (1) patient and clinician buy-in: past telehealth experiences (eg, negative clinician experiences with outdated technology) should be considered when advancing the system. Expansion of services in orthopedics, dermatology, and psychiatry were found to be particularly feasible and beneficial in Yukon; (2) workflow: the use and scheduling of telehealth should be streamlined and automated as much as possible to reduce dependencies on the single Yukon telehealth coordinator; (3) access to telehealth technology: clinicians and patients should have easy access to up-to-date telehealth technology. The use of consumer products, such as mobile technology, should be leveraged as appropriate; and (4) infrastructure: the required human resources and technology need to be established when expanding and advancing telehealth.

Conclusions

While clinicians and patients had generally positive perceptions of the Yukon Telehealth System, there was consensus that it was underutilized. Many opportunities exist to expand the types of telehealth services and the number of telehealth sessions, including the expansion of services in several new specialty areas, updating telehealth equipment to streamline workflows and increase convenience and uptake, and integrating novel technologies. The identified barriers and recommendations from this evaluation can be applied to the development and expansion of telehealth in other remote communities to realize telehealth’s potential for providing efficient, safe, convenient, and cost-effective care delivery.

Development and Implementation of the Portable Operating Room Tracker App With Vital Signs Streaming Infrastructure: Operational Feasibility Study

Background

In the perioperative environment, a multidisciplinary clinical team continually observes and evaluates patient information. However, data availability may be restricted to certain locations, cognitive workload may be high, and team communication may be constrained by availability and priorities. We developed the remote Portable Operating Room Tracker app (the telePORT app) to improve information exchange and communication between anesthesia team members. The telePORT app combines a real-time feed of waveforms and vital signs from the operating rooms with messaging, help request, and reminder features.

Objective

The aim of this paper is to describe the development of the app and the back-end infrastructure required to extract monitoring data, facilitate data exchange and ensure privacy and safety, which includes results from clinical feasibility testing.

Methods

telePORT’s client user interface was developed using user-centered design principles and workflow observations. The server architecture involves network-based data extraction and data processing. Baseline user workload was assessed using step counters and communication logs. Clinical feasibility testing analyzed device usage over 11 months.

Results

telePORT was more commonly used for help requests (approximately 4.5/day) than messaging between team members (approximately 1/day). Passive operating room monitoring was frequently utilized (34% of screen visits). Intermittent loss of wireless connectivity was a major barrier to adoption (decline of 0.3%/day).

Conclusions

The underlying server infrastructure was repurposed for real-time streaming of vital signs and their collection for research and quality improvement. Day-to-day activities of the anesthesia team can be supported by a mobile app that integrates real-time data from all operating rooms.

Understanding the Situated Roles of Electronic Medical Record Systems to Enable Redesign: Mixed Methods Study

BACKGROUND

Redesigning electronic medical record (EMR) systems is needed to improve their usability and usefulness. Similar to other artifacts, EMR systems can evolve with time and exhibit situated roles. Situated roles refer to the ways in which a system is appropriated by its users, that is, the unintended ways the users engage with, relate to, and perceive the system in its context of use. These situated roles are usually unknown to the designers as they emerge and evolve as a response by the users to a contextual need or constraint. Understanding the system's situated roles can expose the unarticulated needs of the users and enable redesign opportunities.

OBJECTIVE

This study aimed to find EMR redesign opportunities by understanding the situated roles of EMR systems in prenatal care settings.

METHODS

We conducted a field-based observational study at a Japanese prenatal care clinic. We observed 3 obstetricians and 6 midwives providing prenatal care to 37 pregnant women. We looked at how the EMR system is used during the checkups. We analyzed the observational data following a thematic analysis approach and identified the situated roles of the EMR system. Finally, we administered a survey to 5 obstetricians and 10 midwives to validate our results and understand the attitudes of the prenatal care staff regarding the situated roles of the EMR system.

RESULTS

We identified 10 distinct situated roles that EMR systems play in prenatal care settings. Among them, 4 roles were regarded as favorable as most users wanted to experience them more frequently, and 4 roles were regarded as unfavorable as most users wanted to experience them less frequently; 2 ambivalent roles highlighted the providers' reluctance to document sensitive psychosocial information in the EMR and their use of the EMR system as an accomplice to pause communication during the checkups. To improve the usability and usefulness of EMR systems, designers can amplify the favorable roles and minimize the unfavorable roles. Our results also showed that obstetricians and midwives may have different experiences, wants, and priorities regarding the use of the EMR system.

CONCLUSIONS

Currently, EMR systems are mainly viewed as tools that support the clinical workflow. Redesigning EMR systems is needed to amplify their roles as communication support tools. Our results provided multiple EMR redesign opportunities to improve the usability and usefulness of EMR systems in prenatal care. Designers can use the results to guide their EMR redesign activities and align them with the users' wants and priorities. The biggest challenge is to redesign EMR systems in a way that amplifies their favorable roles for all the stakeholders concurrently.

A Patient-Centered Mobile Health System That Supports Asthma Self-Management (breathe): Design, Development, and Utilization

Background

Uncontrolled asthma poses substantial negative personal and health system impacts. Web-based technologies, including smartphones, are novel means to enable evidence-based care and improve patient outcomes.

Objective

The aim of this study was to design, develop, and assess the utilization of an asthma collaborative self-management (CSM) platform (breathe) using content based on international evidence-based clinical guidelines.

Methods

We designed and developed breathe as a Web-based mobile health (mHealth) platform accessible on smartphones, tablets, or desktop with user-centered design methods and International Organization for Standardization–certified quality development processes. Moreover, breathe was envisioned as a multifunctional, CSM mHealth platform, with content based on international clinical practice guidelines and compliant with national privacy and security specifications. The system enabled CSM (patient, provider, and breathe) and self-monitoring of asthma patients through (1) assessment of asthma control, (2) real-time access to a dynamic asthma action plan, (3) access to real-time environmental conditions, and (4) risk-reduction messaging. The data collection protocol collected user data for 12 months, with clinic visits at baseline and 6 and 12 months. Utilization outcomes included user interactions with the platform, user impressions, self-reported medication use, asthma symptom profile, reported peak flow measurement, and the delivery and impact of email reminders.

Results

We enrolled 138 patients with a mean age of 45.3 years to receive the breathe intervention. Majority were female (100/138, 72.5%), had a smartphone (92/138, 66.7%), and had a mean Asthma Control Test score of 18.3 (SD 4.9). A majority reported that breathe helped in the management of their asthma. Moreover, breathe scored 71.1 (SD 18.9) on the System Usability Scale. Overall, 123 patients had complete usage analytics datasets. The platform sent 7.96 reminder emails per patient per week (pppw), patients accessed breathe 3.08 times, journaled symptoms 2.56 times, reported medication usage 0.30 times, and reported peak flow measurements 0.92 times pppw. Furthermore, breathe calculated patients’ action plan zone of control 2.72 times pppw, with patients being in the green (well-controlled) zone in 47.71% (8300/17,396) of the total calculations. Usage analysis showed that 67.5% (83/123) of the participants used the app at week 4 and only 57.7% (71/123) by week 45. Physician visits, email reminders, and aged 50 years and above were associated with higher utilization.

Conclusions

Individuals with asthma reported good usability and high satisfaction levels, reacted to breathe notifications, and had confidence in the platform’s assessment of asthma control. Strong utilization was seen at the intervention’s initiation, followed by a rapid reduction in use. Patient reminders, physician visits, and being aged 50 years and above were associated with higher utilization.

Trial Registration

ClinicalTrials.gov NCT01964469; https://clinicaltrials.gov/ct2/show/NCT01964469

An Analytics Platform to Evaluate Effective Engagement With Pediatric Mobile Health Apps: Design, Development, and Formative Evaluation

BACKGROUND

Mobile health (mHealth) apps for pediatric chronic conditions are growing in availability and challenge investigators to conduct rigorous evaluations that keep pace with mHealth innovation. Traditional research methods are poorly suited to operationalize the agile, iterative trials required to evidence and optimize these digitally mediated interventions.

OBJECTIVE

We sought to contribute a resource to support the quantification, analysis, and visualization of analytic indicators of effective engagement with mHealth apps for chronic conditions.

METHODS

We applied user-centered design methods to design and develop an Analytics Platform to Evaluate Effective Engagement (APEEE) with consumer mHealth apps for chronic conditions and implemented the platform to analyze both retrospective and prospective data generated from a smartphone-based pain self-management app called iCanCope for young people with chronic pain.

RESULTS

Through APEEE, we were able to automate the process of defining, operationalizing, and evaluating effective engagement with iCanCope. Configuring the platform to integrate with the app was feasible and provided investigators with a resource to consolidate, analyze, and visualize engagement data generated by participants in real time. Preliminary efforts to evaluate APEEE showed that investigators perceived the platform to be an acceptable evaluative resource and were satisfied with its design, functionality, and performance. Investigators saw potential in APEEE to accelerate and augment evidence generation and expressed enthusiasm for adopting the platform to support their evaluative practice once fully implemented.

CONCLUSIONS

Dynamic, real-time analytic platforms may provide investigators with a powerful means to characterize the breadth and depth of mHealth app engagement required to achieve intended health outcomes. Successful implementation of APEEE into evaluative practice may contribute to the realization of effective and evidence-based mHealth care.

Clinician-Driven Design of VitalPAD-An Intelligent Monitoring and Communication Device to Improve Patient Safety in the Intensive Care Unit

The pediatric intensive care unit (ICU) is a complex environment, in which a multidisciplinary team of clinicians (registered nurses, respiratory therapists, and physicians) continually observe and evaluate patient information. Data are provided by multiple, and often physically separated sources, cognitive workload is high, and team communication can be challenging. Our aim is to combine information from multiple monitoring and therapeutic devices in a mobile application, the VitalPAD, to improve the efficiency of clinical decision-making, communication, and thereby patient safety. We observed individual ICU clinicians, multidisciplinary rounds, and handover procedures for 54 h to identify data needs, workflow, and existing cognitive aid use and limitations. A prototype was developed using an iterative participatory design approach; usability testing, including general and task-specific feedback, was obtained from 15 clinicians. Features included map overviews of the ICU showing clinician assignment, patient status, and respiratory support; patient vital signs; a photo-documentation option for arterial blood gas results; and team communication and reminder functions. Clinicians reported the prototype to be an intuitive display of vital parameters and relevant alerts and reminders, as well as a user-friendly communication tool. Future work includes implementation of a prototype, which will be evaluated under simulation and real-world conditions, with the aim of providing ICU staff with a monitoring device that will improve their daily work, communication, and decision-making capacity. Mobile monitoring of vital signs and therapy parameters might help improve patient safety in wards with single-patient rooms and likely has applications in many acute and critical care settings.

What to Build for Middle-Agers to Come? Attractive and Necessary Functions of Exercise-Promotion Mobile Phone Apps: A Cross-Sectional Study

BACKGROUND

Physical activity is important for middle-agers to maintain health both in middle age and in old age. Although thousands of exercise-promotion mobile phone apps are available for download, current literature offers little understanding regarding which design features can enhance middle-aged adults' quality perception toward exercise-promotion apps and which factor may influence such perception.

OBJECTIVES

The aims of this study were to understand (1) which design features of exercise-promotion apps can enhance quality perception of middle-agers, (2) whether their needs are matched by current functions offered in app stores, and (3) whether physical activity (PA) and mobile phone self-efficacy (MPSE) influence quality perception.

METHODS

A total of 105 middle-agers participated and filled out three scales: the International Physical Activity Questionnaire (IPAQ), the MPSE scale, and the need for design features questionnaire. The design features were developed based on the Coventry, Aberdeen, and London-Refined (CALO-RE) taxonomy. Following the Kano quality model, the need for design features questionnaire asked participants to classify design features into five categories: attractive, one-dimensional, must-be, indifferent, and reverse. The quality categorization was conducted based on a voting approach and the categorization results were compared with the findings of a prevalence study to realize whether needs match current availability. In total, 52 multinomial logistic regression models were analyzed to evaluate the effects of PA level and MPSE on quality perception of design features.

RESULTS

The Kano analysis on the total sample revealed that visual demonstration of exercise instructions is the only attractive design feature, whereas the other 51 design features were perceived with indifference. Although examining quality perception by PA level, 21 features are recommended to low level, 6 features to medium level, but none to high-level PA. In contrast, high-level MPSE is recommended with 14 design features, medium level with 6 features, whereas low-level participants are recommended with 1 feature. The analysis suggests that the implementation of demanded features could be low, as the average prevalence of demanded design features is 20% (4.3/21). Surprisingly, social comparison and social support, most implemented features in current apps, were categorized into the indifferent category. The magnitude of effect is larger for MPSE because it effects quality perception of more design features than PA. Delving into the 52 regression models revealed that high MPSE more likely induces attractive or one- dimensional categorization, suggesting the importance of technological self-efficacy on eHealth care promotion.

CONCLUSIONS

This study is the first to propose middle-agers' needs in relation to mobile phone exercise-promotion. In addition to the tailor-made recommendations, suggestions are offered to app designers to enhance the performance of persuasive features. An interesting finding on change of quality perception attributed to MPSE is proposed as future research.

Negotiating Tensions Between Theory and Design in the Development of Mailings for People Recovering From Acute Coronary Syndrome

Background

Taking all recommended secondary prevention cardiac medications and fully participating in a formal cardiac rehabilitation program significantly reduces mortality and morbidity in the year following a heart attack. However, many people who have had a heart attack stop taking some or all of their recommended medications prematurely and many do not complete a formal cardiac rehabilitation program.

Objective

The objective of our study was to develop a user-centered, theory-based, scalable intervention of printed educational materials to encourage and support people who have had a heart attack to use recommended secondary prevention cardiac treatments.

Methods

Prior to the design process, we conducted theory-based interviews and surveys with patients who had had a heart attack to identify key determinants of secondary prevention behaviors. Our interdisciplinary research team then partnered with a patient advisor and design firm to undertake an iterative, theory-informed, user-centered design process to operationalize techniques to address these determinants. User-centered design requires considering users’ needs, goals, strengths, limitations, context, and intuitive processes; designing prototypes adapted to users accordingly; observing how potential users respond to the prototype; and using those data to refine the design. To accomplish these tasks, we conducted user research to develop personas (archetypes of potential users), developed a preliminary prototype using behavior change theory to map behavior change techniques to identified determinants of medication adherence, and conducted 2 design cycles, testing materials via think-aloud and semistructured interviews with a total of 11 users (10 patients who had experienced a heart attack and 1 caregiver). We recruited participants at a single cardiac clinic using purposive sampling informed by our personas. We recorded sessions with users and extracted key themes from transcripts. We held interdisciplinary team discussions to interpret findings in the context of relevant theory-based evidence and iteratively adapted the intervention accordingly.

Results

Through our iterative development and testing, we identified 3 key tensions: (1) evidence from theory-based studies versus users’ feelings, (2) informative versus persuasive communication, and (3) logistical constraints for the intervention versus users’ desires or preferences. We addressed these by (1) identifying root causes for users’ feelings and addressing those to better incorporate theory- and evidence-based features, (2) accepting that our intervention was ethically justified in being persuasive, and (3) making changes to the intervention where possible, such as attempting to match imagery in the materials to patients’ self-images.

Conclusions

Theory-informed interventions must be operationalized in ways that fit with user needs. Tensions between users’ desires or preferences and health care system goals and constraints must be identified and addressed to the greatest extent possible. A cluster randomized controlled trial of the final intervention is currently underway.

Beyond the Randomized Controlled Trial: A Review of Alternatives in mHealth Clinical Trial Methods

Background

Randomized controlled trials (RCTs) have long been considered the primary research study design capable of eliciting causal relationships between health interventions and consequent outcomes. However, with a prolonged duration from recruitment to publication, high-cost trial implementation, and a rigid trial protocol, RCTs are perceived as an impractical evaluation methodology for most mHealth apps.

Objective

Given the recent development of alternative evaluation methodologies and tools to automate mHealth research, we sought to determine the breadth of these methods and the extent that they were being used in clinical trials.

Methods

We conducted a review of the ClinicalTrials.gov registry to identify and examine current clinical trials involving mHealth apps and retrieved relevant trials registered between November 2014 and November 2015.

Results

Of the 137 trials identified, 71 were found to meet inclusion criteria. The majority used a randomized controlled trial design (80%, 57/71). Study designs included 36 two-group pretest-posttest control group comparisons (51%, 36/71), 16 posttest-only control group comparisons (23%, 16/71), 7 one-group pretest-posttest designs (10%, 7/71), 2 one-shot case study designs (3%, 2/71), and 2 static-group comparisons (3%, 2/71). A total of 17 trials included a qualitative component to their methodology (24%, 17/71). Complete trial data collection required 20 months on average to complete (mean 21, SD 12). For trials with a total duration of 2 years or more (31%, 22/71), the average time from recruitment to complete data collection (mean 35 months, SD 10) was 2 years longer than the average time required to collect primary data (mean 11, SD 8). Trials had a moderate sample size of 112 participants. Two trials were conducted online (3%, 2/71) and 7 trials collected data continuously (10%, 7/68). Onsite study implementation was heavily favored (97%, 69/71). Trials with four data collection points had a longer study duration than trials with two data collection points: F_4,56=3.2, P=.021, η²=0.18. Single-blinded trials had a longer data collection period compared to open trials: F_2,58=3.8, P=.028, η²=0.12. Academic sponsorship was the most common form of trial funding (73%, 52/71). Trials with academic sponsorship had a longer study duration compared to industry sponsorship: F_2,61=3.7, P=.030, η²=0.11. Combined, data collection frequency, study masking, sample size, and study sponsorship accounted for 32.6% of the variance in study duration: F_4,55=6.6, P<.01, adjusted r²=.33. Only 7 trials had been completed at the time this retrospective review was conducted (10%, 7/71).

Conclusions

mHealth evaluation methodology has not deviated from common methods, despite the need for more relevant and timely evaluations. There is a need for clinical evaluation to keep pace with the level of innovation of mHealth if it is to have meaningful impact in informing payers, providers, policy makers, and patients.

The usability of ventilators: a comparative evaluation of use safety and user experience

Background

The design complexity of critical care ventilators (CCVs) can lead to use errors and patient harm. In this study, we present the results of a comparison of four CCVs from market leaders, using a rigorous methodology for the evaluation of use safety and user experience of medical devices.

Methods

We carried out a comparative usability study of four CCVs: Hamilton G5, Puritan Bennett 980, Maquet SERVO-U, and Dräger Evita V500. Forty-eight critical care respiratory therapists participated in this fully counterbalanced, repeated measures study. Participants completed seven clinical scenarios composed of 16 tasks on each ventilator.

Use safety was measured by percentage of tasks with use errors or close calls (UE/CCs). User experience was measured by system usability and workload metrics, using the Post-Study System Usability Questionnaire (PSSUQ) and the National Aeronautics and Space Administration Task Load Index (NASA-TLX).

Results

Nine of 18 post hoc contrasts between pairs of ventilators were significant after Bonferroni correction, with effect sizes between 0.4 and 1.09 (Cohen’s d). There were significantly fewer UE/CCs with SERVO-U when compared to G5 (p = 0.044) and V500 (p = 0.020). Participants reported higher system usability for G5 when compared to PB980 (p = 0.035) and higher system usability for SERVO-U when compared to G5 (p < 0.001), PB980 (p < 0.001), and V500 (p < 0.001). Participants reported lower workload for G5 when compared to PB980 (p < 0.001) and lower workload for SERVO-U when compared to PB980 (p < 0.001) and V500 (p < 0.001). G5 scored better on two of nine possible comparisons; SERVO-U scored better on seven of nine possible comparisons. Aspects influencing participants’ performance and perception include the low sensitivity of G5’s touchscreen and the positive effect from the quality of SERVO-U’s user interface design.

Conclusions

This study provides empirical evidence of how four ventilators from market leaders compare and highlights the importance of medical technology design. Within the boundaries of this study, we can infer that SERVO-U demonstrated the highest levels of use safety and user experience, followed by G5. Based on qualitative data, differences in outcomes could be explained by interaction design, quality of hardware components used in manufacturing, and influence of consumer product technology on users’ expectations.

Electronic supplementary material

The online version of this article (doi:10.1186/s13054-016-1431-1) contains supplementary material, which is available to authorized users.