An evaluation of patients' experienced usability of a diabetes mHealth system using a multi-method approach

Protocol for a usability and pilot implementation study of a digital medical device to assess pain in non-verbal people with dementia in Portuguese residential care facilities

Objective

People living with moderate to severe dementia (PLWD) are often unable to self-report pain. This matter is of particular concern given that up to 80% experience chronic pain. Mistreated or untreated pain in PLWD is associated with symptoms such as agitation and aggression, and unnecessary use of antipsychotic agents. Further, it can also engender mental burden in formal caregivers. The PainChek® App, a regulatory cleared class I medical device, enables the assessment and monitoring of pain in people who cannot verbalise it, such as those with moderate to severe dementia. To date there are no data on the real-world use of the PainChek® App in Portugal. To address this gap, we report the protocol of a pilot study, which combines usability evaluation and implementation research.

Methods

Usability evaluation of the PainChek® (Portuguese) App will be guided by the ISO framework, focused on effectiveness, efficiency and user satisfaction. Implementation research will combine qualitative interviews to inform the implementation process, a longitudinal study of formal caregivers' psychological variables, implementation outcomes, plus qualitative interviews to explore the 'hows' and 'whys' of implementation. The NASSS framework will be used as an implementation framework, together with the COM-B model and the Theoretical Domains Framework.

Results

The usability and implementation studies have received ethics approval from the Egas Moniz Ethics Committee, under numbers 1367 and 64/24, respectively.

Conclusion

This study is expected to inform the scale-up of the PainChek® (Portuguese) App in real-world settings and establish a foundation for a larger effectiveness and implementation study.

Evaluating the usability of Iran's national comprehensive health information system: a think-aloud study to uncover usability problems in the recording of childcare data

INTRODUCTION

Health information systems play a crucial role in the delivery of efficient and effective healthcare. Poor usability is one of the reasons for their lack of acceptance and low usage by users. The aim of this study was to identify the usability problems of a national comprehensive health information system using the concurrent think-aloud method in the recording of childcare data.

METHODS

A descriptive cross-sectional study was conducted in the health centers of Kashan University of Medical Sciences, Iran, in 2020. Ten healthcare providers as system's users were purposively selected to evaluate the system. To identify problems, a concurrent think-aloud evaluation was conducted. Two administrators of the system designed scenarios for ten childcare data recording tasks. By analysing the recorded files, usability problems were identified. The severity of the problems was then determined with the help of the users and problems were assigned to usability attributes based on their impact on the user.

RESULTS

A total of 68 unique problems were identified in the system, of which 47.1% were rated as catastrophic problems. The participants assigned 47 problems (69%) to the user satisfaction attribute and 45 problems (66%) to the efficiency attribute; they also did not assign any problems to the effectiveness attribute.

CONCLUSION

The problems identified in the national comprehensive health information system using the think-aloud method were rated as major and catastrophic, which indicates poor usability of this system. Therefore, resolving the system problems will help increase user satisfaction and system efficiency, allowing more time to be spent on patient care and parent's education as well as improving overall quality of care.

Co-producing an online patient public community research hub: a qualitative study exploring the perspectives of national institute for health research (NIHR) research champions in England

BACKGROUND

Patient and Public Involvement and Engagement (PPIE) should be embedded as part of researchers' everyday practice. However, this can be challenging. Creating a digital presence for PPIE as part of Higher Education Institutes' (HEIs) infrastructure may be one way of supporting this. This can support how information is made available to patients and members of the public, but relatively little is known about how HEIs can best do this. Our aim was to develop a university website for patients and members of the public to learn about ways to get actively involved in research and be able to access the results of health and social care research.

METHODS

This project involved working as partners with five National Institute for Health and Care Research (NIHR) Research Champions. NIHR Research Champions are volunteers who raise awareness and share experiences about health and social care research. Content of a prototype Patient Public Community Research Hub website was co-produced with the Research Champions, and then 15 NIHR Research Champions from across England were asked for their views about the website.

FINDINGS

The information collected told us that the Patient Public Community Research Hub was viewed as being beneficial for increasing visibility of PPIE opportunities and sharing the findings of studies though needs further work: to make the information more user-friendly; to improve the methods for directing people to the site and to create new ways of connecting with people. It provides a foundation for further co-development and evaluation. A set of recommendations has been developed that may be of benefit to other HEIs and organisations who are committed to working with patients and members of the public.

An mHealth Application in German Health Care System: Importance of User Participation in the Development Process

This paper addresses the challenges and solutions in developing a holistic prevention mobile health application (mHealth app) for Germany's healthcare sector. Despite Germany's lag in healthcare digitalization, the app aims to enhance primary prevention in physical activity, nutrition, and stress management. A significant focus is on user participation and usability to counter the prevalent issue of user attrition in mHealth applications, as described by Eysenbach's 'law of attrition'. The development process, conducted in a scientific and university context, faces constraints like limited budgets and external service providers. The study firstly presents the structure and functionality of the app for people with statutory health insurance in Germany and secondly the implementation of user participation through a usability study. User participation is executed via usability tests, particularly the think-aloud method, where users verbalize their thoughts while using the app. This approach has proven effective in identifying and resolving usability issues, although some user feedback could not be implemented due to cost-benefit considerations. The implementation of this study into the development process was able to show that user participation, facilitated by methods like think-aloud, is vital for developing mHealth apps. Especially in health prevention, where long-term engagement is a challenge. The findings highlight the importance of allocating time and resources for user participation in the development of mHealth applications.

Feasibility and Usability of EnergyPoints: A Mobile Health App to Guide Acupressure Use for Cancer Symptom Management

OBJECTIVE

To examine the feasibility and usability of EnergyPoints™, an innovative mobile health app that teaches and guides people with cancer to implement daily acupressure to self-manage their fatigue and sleep disturbances.

METHODS AND INTERVENTION

The study used an integrated agile, human-centered approach. Adults (age 18 years and over) with cancer experiencing at least moderate fatigue, and living in the Greater New York City community, were recruited from social media, patient advocacy groups, and referrals. Twenty participants (in 3 sprints of 3, 5, and 12) were video-recorded thinking aloud while using the app for the first time. They then used the app at home to self-administer acupressure (twice daily for 1 week) while continuously wearing a fitness tracker. Each participant completed an exit interview and modified Computer System Usability Questionnaire post-participation.

RESULTS

Participants were ages 40 to 76 years and 65% female; 65% were non-Hispanic white. Mean pass rates per ritual exceeded 80%. Users completed (totally or partially) greater than 90% of stimulating acupressure and 70% of relaxing acupressure rituals. Sprint 3 SPs totally completed at least 1 ritual 87% of the time. The majority agreed or strongly agreed the app was easy to use (90%), easy to learn (85%), easy to understand (75%), and effective in helping perform self-acupressure (85%). In an analysis of ease of completing 5 key tasks, all successfully completed the tasks; 3 users required some assistance. Of 654 usability statements, those coded as personal experience/context (197), content related to acupressure learning (105), and content related to the onboarding/profile (71) were most frequent. The design team integrated recommendations into the app before the next sprint.

CONCLUSIONS

Findings supported feasibility and usability, as well as acceptability, and led to significant alterations and improvements. EnergyPoints™ offers an opportunity to mainstream acupressure and help cancer survivors self-manage their symptoms.

The Teach-ABI Professional Development Module for Educators About Pediatric Acquired Brain Injury: Mixed Method Usability Study

BACKGROUND

Acquired brain injury (ABI) is a leading cause of death and disability in children and can lead to lasting cognitive, physical, and psychosocial outcomes that affect school performance. Students with an ABI experience challenges returning to school due in part to lack of educator support and ABI awareness. A lack of knowledge and training contribute to educators feeling unprepared to support students with ABI. Teach-ABI, an online professional development module, was created to enhance educators' ABI knowledge and awareness to best support students. Using a case-based approach, Teach-ABI explains what an ABI is, identifies challenges for students with ABI in the classroom, discusses the importance of an individualized approach to supporting students with ABI, and describes how to support a student with an ABI in the classroom.

OBJECTIVE

This study aims to assess the usability of and satisfaction with Teach-ABI by elementary school educators. The following questions were explored: (1) Can elementary school teachers use and navigate Teach-ABI?, (2) Are the content and features of Teach-ABI satisfactory?, and (3) What modifications are needed to improve Teach-ABI?

METHODS

Elementary school educators currently employed or in training to be employed in Ontario elementary schools were recruited. Using Zoom, individual online meetings with a research team member were held, where educators actively reviewed Teach-ABI. Module usability was evaluated through qualitative analysis of think-aloud data and semistructured interviews, direct observation, user success rate during task completion, and the System Usability Scale (SUS) scores. The usability benchmark selected was 70% of participants performing more than half of module tasks independently.

RESULTS

A total of 8 female educators participated in the study. Educators were classroom (n=7) and preservice (n=1) teachers from public (n=7) and private (n=1) school boards. In terms of task performance, more than 85% of participants (ie, 7/8) independently completed 10 out of 11 tasks and 100% of participants independently completed 7 out of 11 tasks, demonstrating achievement of the module usability goal. The average overall SUS score was 86.25, suggesting a high satisfaction level with the perceived usability of Teach-ABI. Overall, participants found Teach-ABI content valuable, useful, and aligned with the realities of their profession. Participants appreciated the visual design, organization, and varying use of education strategies within Teach-ABI. Opportunities for enhancement included broadening content case examples of students with ABI and enhancing the accessibility of the content.

CONCLUSIONS

Validated usability measures combined with qualitative methodology revealed educators' high level of satisfaction with the design, content, and navigation of Teach-ABI. Educators engaged with the module as active participants in knowledge construction, as they reflected, questioned, and connected content to their experiences and knowledge. This study established strong usability and satisfaction with Teach-ABI and demonstrated the importance of usability testing in building online professional development modules.

Learning Designers as Expert Evaluators of Usability: Understanding Their Potential Contribution to Improving the Universality of Interface Design for Health Resources

User-based evaluation by end users is an essential step in designing useful interfaces. Inspection methods can offer an alternate approach when end-user recruitment is problematic. A Learning Designers' usability scholarship could offer usability evaluation expertise adjunct to multidisciplinary teams in academic settings. The feasibility of Learning Designers as 'expert evaluators' is assessed within this study. Two groups, healthcare professionals and Learning Designers, applied a hybrid evaluation method to generate usability feedback from a palliative care toolkit prototype. Expert data were compared to end-user errors detected from usability testing. Interface errors were categorised, meta-aggregated and severity calculated. The analysis found that reviewers detected N = 333 errors, with N = 167 uniquely occurring within the interface. Learning Designers identified errors at greater frequencies (60.66% total interface errors, mean (M) = 28.86 per expert) than other evaluator groups (healthcare professionals 23.12%, M = 19.25 and end users 16.22%, M = 9.0). Patterns in severity and error types were also observed between reviewer groups. The findings suggest that Learning Designers are skilled in detecting interface errors, which benefits developers assessing usability when access to end users is limited. Whilst not offering rich narrative feedback generated by user-based evaluations, Learning Designers complement healthcare professionals' content-specific knowledge as a 'composite expert reviewer' with the ability to generate meaningful feedback to shape digital health interfaces.

App Design Features Important for Diabetes Self-management as Determined by the Self-Determination Theory on Motivation: Content Analysis of Survey Responses From Adults Requiring Insulin Therapy

BACKGROUND

Using a diabetes app can improve glycemic control; however, the use of diabetes apps is low, possibly due to design issues that affect patient motivation.

OBJECTIVE

This study aimed to describes how adults with diabetes requiring insulin perceive diabetes apps based on 3 key psychological needs (competence, autonomy, and connectivity) described by the Self-Determination Theory (SDT) on motivation.

METHODS

This was a qualitative analysis of data collected during a crossover randomized laboratory trial (N=92) testing 2 diabetes apps. Data sources included (1) observations during app testing and (2) survey responses on desired app features. Guided by the SDT, coding categories included app functions that could address psychological needs for motivation in self-management: competence, autonomy, and connectivity.

RESULTS

Patients described design features that addressed needs for competence, autonomy, and connectivity. To promote competence, electronic data recording and analysis should help patients track and understand blood glucose (BG) results necessary for planning behavior changes. To promote autonomy, BG trend analysis should empower patients to set safe and practical personalized behavioral goals based on time and the day of the week. To promote connectivity, app email or messaging function could share data reports and communicate with others on self-management advice. Additional themes that emerged are the top general app designs to promote positive user experience: patient-friendly; automatic features of data upload; voice recognition to eliminate typing data; alert or reminder on self-management activities; and app interactivity of a sound, message, or emoji change in response to keeping or not keeping BG in the target range.

CONCLUSIONS

The application of the SDT was useful in identifying motivational app designs that address the psychological needs of competence, autonomy, and connectivity. User-centered design concepts, such as being patient-friendly, differ from the SDT because patients need a positive user experience (ie, a technology need). Patients want engaging diabetes apps that go beyond data input and output. Apps should be easy to use, provide personalized analysis reports, be interactive to affirm positive behaviors, facilitate data sharing, and support patient-clinician communication.

Learning experience design of an mHealth self-management intervention for adolescents with type 1 diabetes

Type 1 diabetes (T1D) is a lifelong and chronic condition that can cause severely compromised health. The T1D treatment regimen is complex, and is a particular challenge for adolescents, who frequently experience a number of treatment adherence barriers (e.g., forgetfulness, planning and organizational challenges, stress). Diabetes Journey is a gamified mHealth program designed to improve T1D self-management through a specific focus on decreasing adherence barriers and improving executive functioning skills for adolescents. Grounded in situativity theory and guided by a sociotechnical-pedagogical usability framework, Diabetes Journey was designed, developed, and evaluated using a learning experience design approach. This approach applied design thinking methods within a Successive Approximation Model design process. Iterative design and formative evaluation were conducted across three design phases, and improvements were implemented following each phase. Findings from the user testing phase indicate Diabetes Journey is a user-friendly mHealth program with high usability that holds promise for enhancing adolescents' T1D self-management. Implications for future designers and researchers are discussed regarding the social dimension of the sociotechnical-pedagogical usability framework. An extension to the framework is proposed to extend the social dimension to include socio-cultural and contextual considerations when designing mHealth applications. Consideration of the pedagogical and sociocultural dimensions of learning is imperative when developing psychoeducational interventions.

Public perceptions of a healthy lifestyle change conversational agent in Singapore: A qualitative study

OBJECTIVE

Conversational agents (CAs) are increasingly used for the delivery of healthy lifestyle behaviour interventions. This qualitative study aimed to explore the barriers and facilitators to participants' usage of a healthy lifestyle change CA and collect their views on areas for its improvement.

METHODS

Twenty participants were recruited from a convenience sample of users interacting with a CA promoting healthy lifestyle changes to the general population in Singapore. This CA, Precilla, educated users on healthy living, specifically: diet, exercise, sleep and stress; for four weeks. The volunteers participated in semi-structured interviews where an interview guide was used, with questions on acceptability, satisfaction and critical appraisal of the CA. Interviews were transcribed and analysed in parallel by two researchers using thematic content analysis.

RESULTS

Four main themes were identified: (1) enjoyable and acceptable experiences, (2) suboptimal experience(s), (3) alterations to Precilla for enhanced interaction and (4) suggestions for the future. Enjoyable experiences referenced the CA's friendly personality and important content that motivated a positive change to their lifestyle. Some participants were less satisfied and found the content to be too simple or sometimes, the messages too lengthy.

CONCLUSIONS

Participants suggested that in the future, CAs should provide regularly updated content on healthy living, specifically pre-diabetes. Multiple answer options should also be provided for more personalisation along with links to external resources to help improve users' health literacy. Further recommendations include a necessity for a user-centered approach in CA development, employment of engagement strategies, use of a delivery platform most familiar to the target population and stratified message timings to suit the population and purpose of CA. Translating the health CAs to languages relevant to the target group could also enable wider reach and applicability.

Designing, Developing, Evaluating, and Implementing a Smartphone-Delivered, Rule-Based Conversational Agent (DISCOVER): Development of a Conceptual Framework

BACKGROUND

Conversational agents (CAs), also known as chatbots, are computer programs that simulate human conversations by using predetermined rule-based responses or artificial intelligence algorithms. They are increasingly used in health care, particularly via smartphones. There is, at present, no conceptual framework guiding the development of smartphone-based, rule-based CAs in health care. To fill this gap, we propose structured and tailored guidance for their design, development, evaluation, and implementation.

OBJECTIVE

The aim of this study was to develop a conceptual framework for the design, evaluation, and implementation of smartphone-delivered, rule-based, goal-oriented, and text-based CAs for health care.

METHODS

We followed the approach by Jabareen, which was based on the grounded theory method, to develop this conceptual framework. We performed 2 literature reviews focusing on health care CAs and conceptual frameworks for the development of mobile health interventions. We identified, named, categorized, integrated, and synthesized the information retrieved from the literature reviews to develop the conceptual framework. We then applied this framework by developing a CA and testing it in a feasibility study.

RESULTS

The Designing, Developing, Evaluating, and Implementing a Smartphone-Delivered, Rule-Based Conversational Agent (DISCOVER) conceptual framework includes 8 iterative steps grouped into 3 stages, as follows: design, comprising defining the goal, creating an identity, assembling the team, and selecting the delivery interface; development, including developing the content and building the conversation flow; and the evaluation and implementation of the CA. They were complemented by 2 cross-cutting considerations-user-centered design and privacy and security-that were relevant at all stages. This conceptual framework was successfully applied in the development of a CA to support lifestyle changes and prevent type 2 diabetes.

CONCLUSIONS

Drawing on published evidence, the DISCOVER conceptual framework provides a step-by-step guide for developing rule-based, smartphone-delivered CAs. Further evaluation of this framework in diverse health care areas and settings and for a variety of users is needed to demonstrate its validity. Future research should aim to explore the use of CAs to deliver health care interventions, including behavior change and potential privacy and safety concerns.

The Usability of a Smartphone-Based Fall Risk Assessment App for Adult Wheelchair Users: Observational Study

Background

Individuals who use wheelchairs and scooters rarely undergo fall risk screening. Mobile health technology is a possible avenue to provide fall risk assessment. The promise of this approach is dependent upon its usability.

Objective

We aimed to determine the usability of a fall risk mobile health app and identify key technology development insights for aging adults who use wheeled devices.

Methods

Two rounds (with 5 participants in each round) of usability testing utilizing an iterative design-evaluation process were performed. Participants completed use of the custom-designed fall risk app, Steady-Wheels. To quantify fall risk, the app led participants through 12 demographic questions and 3 progressively more challenging seated balance tasks. Once completed, participants shared insights on the app’s usability through semistructured interviews and completion of the Systematic Usability Scale. Testing sessions were recorded and transcribed. Codes were identified within the transcriptions to create themes. Average Systematic Usability Scale scores were calculated for each round.

Results

The first round of testing yielded 2 main themes: ease of use and flexibility of design. Systematic Usability Scale scores ranged from 72.5 to 97.5 with a mean score of 84.5 (SD 11.4). After modifications were made, the second round of testing yielded 2 new themes: app layout and clarity of instruction. Systematic Usability Scale scores improved in the second iteration and ranged from 87.5 to 97.5 with a mean score of 91.9 (SD 4.3).

Conclusions

The mobile health app, Steady-Wheels, has excellent usability and the potential to provide adult wheeled device users with an easy-to-use, remote fall risk assessment tool. Characteristics that promoted usability were guided navigation, large text and radio buttons, clear and brief instructions accompanied by representative illustrations, and simple error recovery. Intuitive fall risk reporting was achieved through the presentation of a single number located on a color-coordinated continuum that delineated low, medium, and high risk.

Barriers and Facilitators of Using eHealth to Support Gestational Diabetes Mellitus Self-Management (GDM): A Systematic Literature Review of Perceptions of Healthcare Professionals and Women with GDM (Preprint)

Background

Gestational diabetes mellitus (GDM) is one of the most common medical complications during pregnancy. eHealth technologies are proving to be successful in supporting the self-management of medical conditions. Digital technologies have the potential to improve GDM self-management.

Objective

The primary objective of this systematic literature review was to identify the views of health professionals (HPs) and women with GDM regarding the use of eHealth for GDM self-management. The secondary objective was to investigate the usability and user satisfaction levels when using these technologies.

Methods

Following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) approach, the search included primary papers in English on the evaluation of technology to support self-management of GDM from January 2008 to September 2021 using MEDLINE, CINAHL, Embase, ACM, and IEEE databases. The lists of references from previous systematic literature reviews, which were related to technology and GDM, were also examined for primary studies. Papers with qualitative, quantitative, and mixed methodologies were included and evaluated. The selected papers were assessed for quality using the Cochrane Collaboration tool, National Institute for Health and Care Excellence clinical guidelines, Critical Appraisal Skills Programme Qualitative Checklist, and McGill University Mixed Methods Appraisal Tool. NVivo (QSR International) was used to extract qualitative data, which were subjected to thematic analysis. Narrative synthesis was used to analyze the quantitative data.

Results

A total of 26 papers were included in the review. Of these, 19% (5/26) of studies used quantitative research methodologies, 19% (5/26) used qualitative methods, and 62% (16/26) used mixed methods. In all, 4 themes were identified from the qualitative data: the benefits of using technology, engagement with people via technology, the usability of technology, and discouragement factors for the use of technology. The thematic analysis revealed a vast scope of challenges and facilitators in the use of GDM self-management systems. The challenges included usability aspects of the system, technical problems, data privacy, lack of emotional support, the accuracy of reported data, and adoption of the system by HPs. Convenience, improved GDM self-management, peer support, increased motivation, increased independence, and consistent monitoring were facilitators to use these technologies. Quantitative data showed that there is potential for improving the usability of the GDM self-management systems. It also showed that convenience, usefulness, increasing motivation for GDM self-management, helping with GDM self-management, and being monitored by HPs were facilitators to use the GDM self-management systems.

Conclusions

This novel systematic literature review shows that HPs and women with GDM encountered some challenges in using GDM self-management systems. The usability of GDM systems was the primary challenge derived from qualitative and quantitative results, with convenience, consistent monitoring, and optimization of GDM self-management emerging as important facilitators.

Searching for Sustainability in Health Systems: Toward a Multidisciplinary Evaluation of Mobile Health Innovations

Mobile health (mHealth) innovations are considered by governments as game changers toward more sustainable health systems. The existing literature focuses on the clinical aspects of mHealth but lacks an integrated framework on its sustainability. The foundational idea for this paper is to include disciplinary complementarities into a multi-dimensional vision to evaluate the non-clinical aspects of mHealth innovations. We performed a targeted literature review to find how the sustainability of mHealth innovations was appraised in each discipline. We found that each discipline considers a different outcome of interest and adopts different time horizons and perspectives for the evaluation. This article reflects on how the sustainability of mHealth innovation can be assessed at both the level of the device itself as well as the level of the health system. We identify some of the challenges ahead of researchers working on mobile health innovations in contributing to shaping a more sustainable health system.

User-Centered Design of A Novel Risk Prediction Behavior Change Tool Augmented With an Artificial Intelligence Engine (MyDiabetesIQ): A Sociotechnical Systems Approach

Background

Diabetes and its complications account for 10% of annual health care spending in the United Kingdom. Digital health care interventions (DHIs) can provide scalable care, fostering diabetes self-management and reducing the risk of complications. Tailorability (providing personalized interventions) and usability are key to DHI engagement/effectiveness. User-centered design of DHIs (aligning features to end users’ needs) can generate more usable interventions, avoiding unintended consequences and improving user engagement.

Objective

MyDiabetesIQ (MDIQ) is an artificial intelligence engine intended to predict users’ diabetes complications risk. It will underpin a user interface in which users will alter lifestyle parameters to see the impact on their future risks. MDIQ will link to an existing DHI, My Diabetes My Way (MDMW). We describe the user-centered design of the user interface of MDIQ as informed by human factors engineering.

Methods

Current users of MDMW were invited to take part in focus groups to gather their insights about users being shown their likelihood of developing diabetes-related complications and any risks they perceived from using MDIQ. Findings from focus groups informed the development of a prototype MDIQ interface, which was then user-tested through the “think aloud” method, in which users speak aloud about their thoughts/impressions while performing prescribed tasks. Focus group and think aloud transcripts were analyzed thematically, using a combination of inductive and deductive analysis. For think aloud data, a sociotechnical model was used as a framework for thematic analysis.

Results

Focus group participants (n=8) felt that some users could become anxious when shown their future complications risks. They highlighted the importance of easy navigation, jargon avoidance, and the use of positive/encouraging language. User testing of the prototype site through think aloud sessions (n=7) highlighted several usability issues. Issues included confusing visual cues and confusion over whether user-updated information fed back to health care teams. Some issues could be compounded for users with limited digital skills. Results from the focus groups and think aloud workshops were used in the development of a live MDIQ platform.

Conclusions

Acting on the input of end users at each iterative stage of a digital tool’s development can help to prioritize users throughout the design process, ensuring the alignment of DHI features with user needs. The use of the sociotechnical framework encouraged the consideration of interactions between different sociotechnical dimensions in finding solutions to issues, for example, avoiding the exclusion of users with limited digital skills. Based on user feedback, the tool could scaffold good goal setting, allowing users to balance their palatable future complications risk against acceptable lifestyle changes. Optimal control of diabetes relies heavily on self-management. Tools such as MDMW/ MDIQ can offer personalized support for self-management alongside access to users’ electronic health records, potentially helping to delay or reduce long-term complications, thereby providing significant reductions in health care costs.

Development and Usability Evaluation of a Telemedicine System for Management and Monitoring of Patients with Diabetic Foot

Objectives: The aim of the present study was to develop and evaluate the usability of a telemedicine system for management and monitoring of patients with diabetic foot.Methods: This study was conducted in four phases. In the first phase, the information needs and characteristics required to design the telemedicine system were identified based on a literature review. Then, in a two-stage Delphi survey, 15 experts approved the identified information needs and characteristics. The prototype telemedicine system was then designed. In the third phase, system usability was evaluated through a semi-structured interview. In the fourth phase, users’ satisfaction with the designed system was analyzed.Results: Out of 115 information needs and required characteristics, 95 were considered in the system design. Eight main pages for enabling patient-physician interactions and physician-physician interactions, monitoring the patient and controlling the disease process, providing medical consultation, and prescribing medications were considered. In the third phase, 26 distinct problems were identified. However, 75% of the participants were very satisfied with the system.Conclusions: This study presents an attempt to design and evaluate a telemedicine system for the management and monitoring of patients with diabetic foot. In this system, patients receiving medical services or physicians who encounter rare cases can send the complete medical history, clinical test results, and videos and images related to the foot to specialist physicians. After examining the medical history or images and videos, the physician can provide the necessary medication prescriptions and laboratory tests or other recommendations.

InCoPE-App: Study protocol to examine usability and effectiveness of an individualized, tablet-based multidomain exercise program for institutionalized people with dementia delivered by nursing assistants. (Preprint)

Background

The COVID-19 pandemic has had drastic consequences on everyday life in nursing homes. Limited personnel resources and modified hygiene and safety measures (eg, no external exercise instructors, no group settings) have often led to interrupted physical exercise treatments. As a consequence, people with dementia benefiting from individualized exercise programs are affected by the pandemic’s impact.

Objective

Our goal is to develop an easily applicable mobile application (Individualized Cognitive and Physical Exercise [InCoPE] app) allowing nursing assistants to test cognitive function and physical performance and subsequently train people with dementia through a multidomain, individualized exercise program.

Methods

We will evaluate the usability and effectiveness of the InCoPE-App by applying a mixed method design. Nursing assistants will use the InCoPE-App for 18 weeks to assess the cognitive function and physical performance of 44 people with dementia every 3 weeks and apply the individualized exercise program. We will record overall usability using questionnaires (eg, Post-Study System Usability and ISONORM 9241/10), log events, and interviews. Perceived hedonic and pragmatic quality will be assessed using the AttrakDiff questionnaire. Effectiveness will be evaluated by considering changes in quality of life as well as cognitive function and physical performance between before and after the program.

Results

Enrollment into the study will be completed in the first half of 2022. We expect an improvement in the quality of life of people with dementia accompanied by improvements in cognitive function and physical performance. The usability of the InCoPE-App is expected to be rated well by nursing assistants.

Conclusions

To date, there is no scientifically evaluated app available that enables nursing assistants without expertise in sports science to deliver an individualized exercise program among people with dementia. A highly usable and effective InCoPE-App allows nursing assistants to test cognitive function and physical performance of people with dementia and, based thereon, select and deliver an appropriate individualized exercise program based on the cognitive and physical status of an individual, even in times of a pandemic.

Trial Registration

German Register of Clinical Trials DRKS00024069; https://www.drks.de/drks_web/navigate.do?navigationId=trial.HTML&TRIAL_ID=DRKS00024069

International Registered Report Identifier (IRRID)

DERR1-10.2196/36247

Evaluating the Usability of mHealth Applications on Type 2 Diabetes Mellitus Using Various MCDM Methods

The recent developments in the IT world have brought several changes in the medical industry. This research work focuses on few mHealth applications that work on the management of type 2 diabetes mellitus (T2DM) by the patients on their own. Looking into the present doctor-to-patient ratio in our country (1:1700 as per a Times of India report in 2021), it is very essential to develop self-management mHealth applications. Thus, there is a need to ensure simple and user-friendly mHealth applications to improve customer satisfaction. The goal of this study is to assess and appraise the usability and effectiveness of existing T2DM-focused mHealth applications. TOPSIS, VIKOR, and PROMETHEE II are three multi-criteria decision-making (MCDM) approaches considered in the proposed work for the evaluation of the usability of five existing T2DM mHealth applications, which include Glucose Buddy, mySugr, Diabetes: M, Blood Glucose Tracker, and OneTouch Reveal. The methodology used in the research work is a questionnaire-based evaluation that focuses on certain attributes and sub-attributes, identified based on the features of mHealth applications. CRITIC methodology is used for obtaining the attribute weights, which give the priority of the attributes. The resulting analysis signifies our proposed research by ranking the mHealth applications based on usability and customer satisfaction.

Usability Evaluation of a Smartphone Medication Reminder Application in Patients Treated With Short-term Antibiotic

Medication adherence is poor in persons with chronic disease, especially in those with multiple chronic diseases, one of which is a psychological disorder. Social support, medication education, and external reminders have been identified as facilitators of adherence. Mobile health applications have the potential to enhance adherence; however, it is unknown if publicly available applications are user-friendly and useful. We aimed to examine the usability and feasibility of the "MediSafe" medication reminder application in adults with diarrhea-predominant irritable bowel syndrome undergoing short-term antibiotic therapy and a "Medfriend" from their social support network (N = 14). A mixed-methods study was conducted. All patient participants used the MediSafe application daily for 14 days. Ease of use, ease of learning, and satisfaction scales were rated highest by both patient participants and Medfriends, whereas usefulness was rated lowest by both groups, with Medfriends' usefulness rating significantly lower than that of patient participants. Telephone interviews identified patient participants found the application instrumental in facilitating medication adherence, and Medfriends viewed themselves as active participants in the patient participants' care. The MediSafe medication reminder application is easy to use and accepted by both patients and their designated Medfriend. The MediSafe is instrumental in facilitating short-term antibiotic adherence and social support engagement.

Application of the National Institute for Health and Care Excellence Evidence Standards Framework for Digital Health Technologies in Assessing Mobile-Delivered Technologies for the Self-Management of Type 2 Diabetes Mellitus: Scoping Review

BACKGROUND

There is a growing role of digital health technologies (DHTs) in the management of chronic health conditions, specifically type 2 diabetes. It is increasingly important that health technologies meet the evidence standards for health care settings. In 2019, the National Institute for Health and Care Excellence (NICE) published the NICE Evidence Standards Framework for DHTs. This provides guidance for evaluating the effectiveness and economic value of DHTs in health care settings in the United Kingdom.

OBJECTIVE

The aim of this study is to assess whether scientific articles on DHTs for the self-management of type 2 diabetes mellitus report the evidence suggested for implementation in clinical practice, as described in the NICE Evidence Standards Framework for DHTs.

METHODS

We performed a scoping review of published articles and searched 5 databases to identify systematic reviews and primary studies of mobile device-delivered DHTs that provide self-management support for adults with type 2 diabetes mellitus. The evidence reported within articles was assessed against standards described in the NICE framework.

RESULTS

The database search yielded 715 systematic reviews, of which, 45 were relevant and together included 59 eligible primary studies. Within these, there were 39 unique technologies. Using the NICE framework, 13 technologies met best practice standards, 3 met minimum standards only, and 23 technologies did not meet minimum standards.

CONCLUSIONS

On the assessment of peer-reviewed publications, over half of the identified DHTs did not appear to meet the minimum evidence standards recommended by the NICE framework. The most common reasons for studies of DHTs not meeting these evidence standards included the absence of a comparator group, no previous justification of sample size, no measurable improvement in condition-related outcomes, and a lack of statistical data analysis. This report provides information that will enable researchers and digital health developers to address these limitations when designing, delivering, and reporting digital health technology research in the future.

Determining the Intellectual Structure and Academic Trends of Smart Home Health Care Research: Coword and Topic Analyses

Background

With the rapid development of information and communication technologies, smart homes are being investigated as effective solutions for home health care. The increasing academic attention on smart home health care has primarily been on the development and application of smart home technologies. However, comprehensive studies examining the general landscape of diverse research areas for smart home health care are still lacking.

Objective

This study aims to determine the intellectual structure of smart home health care in a time series by conducting a coword analysis and topic analysis. Specifically, it investigates (1) the intellectual basis of smart home health care through overall academic status, (2) the intellectual foci through influential keywords and their evolutions, and (3) intellectual trends through primary topics and their evolutions.

Methods

Analyses were conducted in 5 steps: (1) data retrieval from article databases (Web of Science, Scopus, and PubMed) and the initial dataset preparation of 6080 abstracts from the year 2000 to the first half of 2019; (2) data preprocessing and refinement extraction of 25,563 words; (3) a descriptive analysis of the overall academic status and period division (ie, 4 stages of 3-year blocks); (4) coword analysis based on word co-occurrence networks for the intellectual foci; and (5) topic analysis for the intellectual trends based on latent Dirichlet allocation (LDA) topic modeling, word-topic networks, and researcher workshops.

Results

First, regarding the intellectual basis of smart home health care, recent academic interest and predominant journals and research domains were verified. Second, to determine the intellectual foci, primary keywords were identified and classified according to the degree of their centrality values. Third, 5 themes pertaining to the topic evolution emerged: (1) the diversification of smart home health care research topics; (2) the shift from technology-oriented research to technological convergence research; (3) the expansion of application areas and system functionality of smart home health care; (4) the increased focus on system usability, such as service design and experiences; and (5) the recent adaptation of the latest technologies in health care. Based on these findings, the pattern of technology diffusion in smart home health care research was determined as the adaptation of technologies, the proliferation of application areas, and an extension into system design and service experiences.

Conclusions

The research findings provide academic and practical value in 3 aspects. First, they promote a comprehensive understanding of the smart home health care domain by identifying its multifaceted intellectual structure in a time series. Second, they can help clinicians discern the development and dispersion level of their respective disciplines. Third, the pattern of technology diffusion in smart home health care could help scholars comprehend current and future research trends and identify research opportunities based on upcoming research waves of newly adapted technologies in smart home health care.

The Self-management Smartphone Application for Cancer Survivors, ReLive: Development and Usability Testing

ReLive is a nursing theory-driven and evidence-based smartphone application that aims to promote self-management among cancer survivors. It has been designed to display symptom measurement results in different traffic light colors, depending on the severity of a user's symptoms (eg, severe symptoms are presented in red). Therefore, it is easy for users to draw inferences about changes in their symptoms. Further, users can simultaneously set several physical activity goals and monitor their performance. Social support, self-efficacy, and quality of life of a user can also be monitored regularly. This study investigated the usability of this application. An iterative formative test, including a cognitive walkthrough and face-to-face interviews, was conducted. Participants were seven individuals with a diagnosis of chronic myeloid leukemia. The ease of use and understanding, acceptability, and usefulness of the application were evaluated. The results revealed that the participants had evaluated ReLive positively. This program could be used as an intervention to deliver health information and manage their performance. Further research is needed to assess the application's effects on self-management among survivors of various types of cancers.

Performance of a web-based application measuring spot quality in dried blood spot sampling

Background The dried blood spot (DBS) method allows patients and researchers to collect blood on a sampling card using a skin-prick. An important issue in the application of DBSs is that samples for therapeutic drug monitoring are frequently rejected because of poor spot quality, leading to delayed monitoring or missing data. We describe the development and performance of a web-based application (app), accessible on smartphones, tablets or desktops, capable of assessing DBS quality at the time of sampling by means of analyzing a picture of the DBS. Methods The performance of the app was compared to the judgment of experienced laboratory technicians for samples obtained in a trained and untrained setting. A robustness- and user test were performed. Results In a trained setting the app yielded an adequate decision in 90.0% of the cases with 4.1% false negatives (insufficient quality DBSs incorrectly not rejected) and 5.9% false positives (sufficient quality DBSs incorrectly rejected). In an untrained setting this was 87.4% with 5.5% false negatives and 7.1% false positives. A patient user test resulted in a system usability score of 74 out of 100 with a median time of 1 min and 45 s to use the app. Robustness testing showed a repeatability of 84%. Using the app in a trained and untrained setting improves the amount of sufficient quality samples from 80% to 95.9% and 42.2% to 87.9%, respectively. Conclusions The app can be used in trained and untrained setting to decrease the amount of insufficient quality DBS samples.

Untold Stories in User-Centered Design of Mobile Health: Practical Challenges and Strategies Learned From the Design and Evaluation of an App for Older Adults With Heart Failure

Background

User-centered design (UCD) is a powerful framework for creating useful, easy-to-use, and satisfying mobile health (mHealth) apps. However, the literature seldom reports the practical challenges of implementing UCD, particularly in the field of mHealth.

Objective

This study aims to characterize the practical challenges encountered and propose strategies when implementing UCD for mHealth.

Methods

Our multidisciplinary team implemented a UCD process to design and evaluate a mobile app for older adults with heart failure. During and after this process, we documented the challenges the team encountered and the strategies they used or considered using to address those challenges.

Results

We identified 12 challenges, 3 about UCD as a whole and 9 across the UCD stages of formative research, design, and evaluation. Challenges included the timing of stakeholder involvement, overcoming designers’ assumptions, adapting methods to end users, and managing heterogeneity among stakeholders. To address these challenges, practical recommendations are provided to UCD researchers and practitioners.

Conclusions

UCD is a gold standard approach that is increasingly adopted for mHealth projects. Although UCD methods are well-described and easily accessible, practical challenges and strategies for implementing them are underreported. To improve the implementation of UCD for mHealth, we must tell and learn from these traditionally untold stories.

Usability Evaluation of Four Top-Rated Commercially Available Diabetes Apps for Adults With Type 2 Diabetes

Despite the many diabetes applications available, the rate of use is low, which may be associated with design issues. This study examined app usability compliance with heuristic design principles, guided by the Self-determination Theory on motivation. Four top-rated commercially available apps (Glucose Buddy, MyNetDiary, mySugr, and OnTrack) were tested for data recording, blood glucose analysis, and data sharing important for diabetes competence, autonomy, and connection with a healthcare provider. Four clinicians rated each app's compliance with Nielsen's 10 principles and its usability using the System Usability Scale. All four apps lacked one task function related to diabetes care competence or autonomy. Experts ranked app usability rated with the System Usability Scale: OnTrack (61) and Glucose Buddy (60) as a "D" and MyNetDairy (41) and mySugr (15) as an "F." A total of 314 heuristic violations were identified. The heuristic principle violated most frequently was "Help and Documentation" (n = 50), followed by "Error Prevention" (n = 45) and "Aesthetic and Minimalist Design" (n = 43). Four top-rated diabetes apps have "marginally acceptable" to "completely unacceptable." Future diabetes app design should target patient motivation and incorporate key heuristic design principles by providing tutorials with a help function, eliminating error-prone operations, and providing enhanced graphical or screen views.

The development of My Care Hub Mobile-Phone App to Support Self-Management in Australians with Type 1 or Type 2 Diabetes

Non-adherence to self-management poses a serious risk to diabetes complications. Digital behavioural change interventions have the potential to provide education and motivate users to regularly engage with self-management of diabetes. This paper describes the development of My Care Hub mobile phone application (app) aimed at supporting self-management in people with type 1 or type 2 diabetes. The development of My Care Hub involved a comprehensive process of healthy behavioural change identification, end users’ needs, expert consensus, data security and privacy considerations. The app translation was a highly iterative process accompanied by usability testing and design modification. The app development process included: (1) behaviour change strategy selection; (2) users’ involvement; (3) expert advisory involvement; (4) data security and privacy considerations; (5) design creation and output translation into a smartphone app and (6) two usability testings of the app prototype version. The app features include self-management activities documentation, analytics, personalized and generalized messages for diabetes self-management as well as carbohydrate components of common foods in Australia. Twelve respondents provided feedback on the usability of the app. Initially, a simplification of the documentation features of the app was identified as a need to improve usability. Overall, results indicated good user satisfaction rate.

Technology usage and barriers to the use of behavioral intervention technologies in adolescents and young adults with spina bifida

PURPOSE

The majority of behavioral intervention technologies (BITs) have been designed and targeted towards the general population (i.e., typically-developing individuals); thus, little is known about the use of BITs to aid those with special needs, such as youth with disabilities. The current study assessed adolescents and young adults with spina bifida (AYA-SB) for: 1) their technology usage, and 2) anticipated barriers to using technology to help manage their health.

METHODS

AYA-SB completed a survey of their media and technology usage. A card sorting task that ranked and grouped anticipated barriers to using a mobile app to manage health was also completed. Ranked means, standard deviations, and the number of times a barrier was discarded were used to interpret sample rankings.

RESULTS

AYA-SB reported less frequent technology and media use than the general population. However, differences emerged by age, with young adults endorsing higher usage than their younger counterparts. Top concerns focused on usability, accessibility, safety, personal barriers due to lack of engagement, technological functioning, privacy, and efficacy.

CONCLUSIONS

AYA-SB appear to be selective users of technology. It is therefore critical that the design of BITs address their concerns, specifically aiming to have high usability, accessibility, and engagement.

Development of a self-guided web-based exercise intervention (SPIN) to treat shoulder pain in people living with spinal cord injury: protocol of a mixed methods study

INTRODUCTION

Chronic shoulder pain is common after spinal cord injury (SCI) and limits community mobility. This leads to loss of independence and reduced quality of life. Evidence suggests that exercises can help reduce shoulder pain. However, cost, expertise and transport barriers frequently limit access to treatment services. The objective of this study is to develop an evidence-based, acceptable, usable and persuasive self-guided web-based exercise intervention to treat shoulder pain in people living with SCI.

METHODS AND ANALYSIS

An iterative and phased person-based approach (PBA) will capture users' perspectives on usability and acceptability to develop guiding principles that will shape the design of the intervention. The intervention will be based on key elements identified through participant input and from evidence identified through systematic and narrative reviews, to ensure the intervention addresses participants' needs and increase the likelihood of uptake. The prototype will be iteratively refined through focus groups and think-aloud sessions. Review data will be synthesised drawing on systematic and narrative review conventions. Qualitative data will be analysed using conventional content analysis (planning phase) and directed content analysis (development phase) to inform intervention design and refinement.

ETHICS AND DISSEMINATION

Ethical approval has been granted by the Auckland University of Technology Ethics Committee (AUTEC) in Auckland, New Zealand. The results of the study will be published in a peer-reviewed journal and presented at relevant national and international conferences. A summary of findings will be presented to key stakeholder groups. We will progress to a definitive trial should the findings from this intervention development study indicate the intervention is acceptable and usable.

An approach to revising mHealth interventions for children and families: A case example in sickle cell disease

Frameworks for developing mHealth interventions that are informed by the end-user and improve accessibility are necessary to ensure interventions meet the needs of the intended population and advance the science of health behavior change. The approach described in this paper addresses a gap in the mHealth development literature by describing a step-by-step method for evaluating and revising the mHealth interventions for health behavior change in child-parent dyads. Furthermore, this approach introduces the Website Analysis and MeasureMent Inventory framework as an initial coding structure for analyzing qualitative data to gauge appeal and enhance engagement of intervention for end-users. This method uses specific considerations for child-parent dyads and details an example in the development and refinement of a mobile application for self-management in children with sickle cell disease. This approach is translatable to populations of children with other chronic conditions and to other health behaviors.

A Mobile Phone App for the Self-Management of Pediatric Concussion: Development and Usability Testing

BACKGROUND

Concussion is a common injury among Canadian children and adolescents that leads to a range of neurobehavioral deficits. However, noticeable gaps continue to exist in the management of pediatric concussion, with poor health outcomes associated with the inadequate application of best practice guidelines.

OBJECTIVE

The aim of this study was to describe the development and assess the usability of a mobile phone app to aid youth in the self-management of concussion. A secondary objective was to assess the usefulness of the app.

METHODS

An agile user-centered design approach was used to develop the technology, followed by a formative lab-based usability study for assessment and improvement proposals. Youths aged 10 to 18 years with a history of concussion and health care professionals involved in concussion management were recruited. This study included participants performing 12 tasks with the mobile phone app while using the think aloud protocol and the administration of the System Usability Scale (SUS), posttest questionnaire, and a semistructured interview.

RESULTS

A mobile phone app prototype called NeuroCare, an easily accessible pediatric concussion management intervention that provides easy access to expert-informed concussion management strategies and helps guide youth in self-managing and tracking their concussion recovery, was developed. A total of 7 youths aged between 10 and 18 years with a history of concussion and 7 health care professionals were recruited. The mean SUS score was 81.9, mean task success rates were greater than 90% for 92% (11/12) of the tasks, 92% (11/12) of tasks had a total error frequency of less than 11 errors, and mean task completion times were less than 2 min for 100% of the tasks.

CONCLUSIONS

Results suggest that participants rated this app as highly usable, acceptable to users, and that it may be useful in helping youth self-manage concussion.

Usability Challenges for Health and Wellness Mobile Apps: Mixed-Methods Study Among mHealth Experts and Consumers

BACKGROUND

By 2019, there will be an estimated 4.68 billion mobile phone users globally. This increase comes with an unprecedented proliferation in mobile apps, a plug-and-play product positioned to improve lives in innumerable ways. Within this landscape, medical apps will see a 41% compounded annual growth rate between 2015 and 2020, but paradoxically, prevailing evidence indicates declining downloads of such apps and decreasing "stickiness" with the intended end users.

OBJECTIVE

As usability is a prerequisite for success of health and wellness mobile apps, this paper aims to provide insights and suggestions for improving usability experience of the mobile health (mHealth) app by exploring the degree of alignment between mHealth insiders and consumers.

METHODS

Usability-related major themes were selected from over 20 mHealth app development studies. The list of themes, grouped into 5 categories using the Nielsen usability model, was then used as a framework to identify and classify the responses from mHealth expert (insider) interviews. Responses from the qualitative phase were integrated into some questions for a quantitative consumer survey. Subsequently, categorical data from qualitative mHealth insider interviews and numerical data from a quantitative consumer survey were compared in order to identify common usability themes and areas of divergence.

RESULTS

Of the 5 usability attributes described in Nielsen model,  Satisfaction ranked as the top attribute for both mHealth insiders and consumers. Satisfaction refers to user likability, comfort, and pleasure. The consumer survey yielded 451 responses. Out of 9 mHealth insiders' top concerns, 5 were similar to those of the consumers. On the other hand, consumers did not grade themes such as Intuitiveness as important, which was deemed vital by mHealth insiders. Other concerns of the consumers include in-app charges and advertisements.

CONCLUSIONS

This study supports and contributes to the existing pool of mixed-research studies. Strengthening the connectivity between suppliers and users (through the designed research tool) will help increase uptake of mHealth apps. In a holistic manner, this will have a positive overall outcome for the mHealth app ecosystem.

Say, What Is on Your Mind? Surgeons' Evaluations of Realism and Usability of a Virtual Reality Vertebroplasty Simulator

BACKGROUND

Virtual reality (VR)-based simulations offer rich opportunities for surgical skill training and assessment of surgical novices and experts. A structured evaluation and validation process of such training and assessment tools is necessary for effective surgical learning environments.

OBJECTIVE

To develop and apply a classification system of surgeon-reported experience during operation of a VR vertebroplasty simulator.

METHODS

A group of orthopedic, trauma surgeons and neurosurgeons (n = 13) with various levels of expertise performed on a VR vertebroplasty simulator. We established a mixed-methods design using think-aloud protocols, senior surgical expert evaluations, performance metrics, and a post-simulation questionnaire. Verbal content was systematically analyzed using structured qualitative content analysis. We established a category system for classification of surgeons' verbal evaluations during the simulation. Furthermore, we evaluated intraoperative performance metrics and explored potential associations with surgeons' characteristics and simulator evaluation.

RESULTS

Overall, 244 comments on realism and usability of the vertebroplasty simulator were collected. This included positive and negative remarks, questions, and specific suggestions for improvement. Further findings included surgeons' approval of the realism and usability of the simulator and the observation that the haptic feedback of the VR patient's anatomy requires further improvement. Surgeon-reported evaluations were not associated with performance decrements.

DISCUSSION

This study is the first to apply think-aloud protocols for evaluation of a surgical VR-based simulator. A novel classification approach is introduced that can be used to classify surgeons' verbalized experiences during simulator use. Our lessons learned may be valuable for future research with similar methodological approach.

A Web-Based Interactive Tool to Reduce Childhood Obesity Risk in Urban Minority Youth: Usability Testing Study

Background

Childhood obesity is a serious public health issue among minority youth in the United States. Technology-enhanced approaches can be effective for promoting healthy behavior change.

Objective

The purpose of this study was to test the usability of prototypes of a Web-based interactive tool promoting healthy dietary behaviors to reduce childhood obesity risk in urban minority youth. The Web-based tool comprised a manga-style comic with interactive features (eg, sound effects, clickable pop-ups), tailored messaging, and goal setting, and was optimized for use on tablet devices.

Methods

Latino and black/African American children ages 9 to 13 years were recruited to participate in two rounds of usability testing. A modified think-aloud method was utilized. Self-reported surveys and field notes were collected. Audio recordings and field notes from usability testing sessions were systematically reviewed by extracting and coding user feedback as either positive comments or usability or negative issues. The quantitative data from self-reported questionnaires were analyzed using descriptive statistics.

Results

Twelve children (four female; eight black/African American) with a mean age of 10.92 (SD 1.16) years participated. Testing highlighted overall positive experiences with the Web-based interactive tool, especially related to storyline, sound effects, and color schemes. Specific usability issues were classified into six themes: appearance, content, special effects, storyline, terminology, and navigation. Changes to the Web-based tool after round 1 included adding a navigation guide, making clickable icons more visible, improving graphic designs, and fixing programming errors. In round 2 of testing (after modifications to the Web-based tool were incorporated), many of the usability issues that were identified in round 1 did not emerge.

Conclusions

Results of testing will inform further development and finalization of the tool, which will be tested using a two-group pilot randomized study, with the goal of reducing childhood obesity risk in minority, low-income youth.

Preferences for Health Information Technologies Among US Adults: Analysis of the Health Information National Trends Survey

BACKGROUND

Emerging health technologies are increasingly being used in health care for communication, data collection, patient monitoring, education, and facilitating adherence to chronic disease management. However, there is a lack of studies on differences in the preference for using information exchange technologies between patients with chronic and nonchronic diseases and factors affecting these differences.

OBJECTIVE

The purpose of this paper is to understand the preferences and use of information technology for information exchange among a nationally representative sample of adults with and without 3 chronic disease conditions (ie, cardiovascular disease [CVD], diabetes, and hypertension) and to assess whether these preferences differ according to varying demographic variables.

METHODS

We utilized data from the 2012 and 2014 iteration of the Health Information National Trends Survey (N=7307). We used multiple logistic regressions, adjusting for relevant demographic covariates, to identify the independent factors associated with lower odds of using health information technology (HIT), thus, identifying targets for awareness. Analyses were weighted for the US population and adjusted for the sociodemographic variables of age, gender, race, and US census region.

RESULTS

Of 7307 participants, 3529 reported CVD, diabetes, or hypertension. In the unadjusted models, individuals with diabetes, CVD, or hypertension were more likely to report using email to exchange medical information with their provider and less likely to not use any of the technology in health information exchange, as well as more likely to say it was not important for them to access personal medical information electronically. In the unadjusted model, additional significant odds ratio (OR) values were observed. However, after adjustment, most relationships regarding the use and interest in exchanging information with the provider were no longer significant. In the adjusted model, individuals with CVD, diabetes, or hypertension were more likely to access Web-based personal health information through a website or app. Furthermore, we assessed adjusted ORs for demographic variables. Those aged >65 years and Hispanic people were more likely to report no use of email to exchange medical information with their provider. Minorities (Hispanic, non-Hispanic black, and Asian people) were less likely to indicate they had no interest in exchanging general health tips with a provider electronically.

CONCLUSIONS

The analysis did not show any significant association among those with comorbidities and their proclivity toward health information, possibly implying that HIT-related interventions, particularly design of information technologies, should focus more on demographic factors, including race, age, and region, than on comorbidities or chronic disease status to increase the likelihood of use. Future research is needed to understand and explore more patient-friendly use and design of information technologies, which can be utilized by diverse age, race, and education or health literacy groups efficiently to further bridge the patient-provider communication gap.

In-hospital usability and feasibility evaluation of Panda, an app for the management of pain in children at home

BACKGROUND

Postoperative pain in children is often poorly managed at home, leading to slower functional recovery, poor oral intake, sleep disturbances, and behavioral changes. Panda is a smartphone application (app) designed to support parents in assessing their child's pain and managing medications.

AIMS

The aim of this study was to evaluate the Panda app's usability and feasibility in hospital prior to testing the app at home.

METHODS

The study comprised two phases. Phase I evaluated Panda's usability with nurses, parents, and adolescents using simulated scenarios. Usability was measured by task completion rate, user error rates, and the Computer Systems Usability Questionnaire. Phase II evaluated Panda's feasibility by observing parents/guardians of pediatric patients using the app on the postsurgical ward. Feasibility was measured using response frequency and delay following app notifications from an audit trail of app function, and parental satisfaction from an interview. Feedback was used to guide iterative app improvements.

RESULTS

In Phase I, 13 nurses, 12 parents, and 5 adolescents evaluated the app. A total of 103 usability issues were identified, analyzed, and addressed. In Phase II, 29 parents responded to a total of 151 app notifications, with 84% responding within 1 hour in the final round of testing; 93% of participants reported the app was easy to use, and rated the app with a median [interquartile range] Computer Systems Usability Questionnaire score of 2 [1-4]. Significant barriers to use included lack of flexibility in the medication scheduling, low volume of alert sounds, and the extra time spent on medication safety checks.

CONCLUSION

Panda's usability was improved and its feasibility demonstrated in the controlled hospital environment. The next step is to evaluate its feasibility for use at home.

A Patient-Facing Diabetes Dashboard Embedded in a Patient Web Portal: Design Sprint and Usability Testing

Background

Health apps and Web-based interventions designed for patients with diabetes offer novel and scalable approaches to engage patients and improve outcomes. However, careful attention to the design and usability of these apps and Web-based interventions is essential to reduce the barriers to engagement and maximize use.

Objective

The aim of this study was to apply design sprint methodology paired with mixed-methods, task-based usability testing to design and evaluate an innovative, patient-facing diabetes dashboard embedded in an existing patient portal and integrated into an electronic health record.

Methods

We applied a 5-day design sprint methodology developed by Google Ventures (Alphabet Inc, Mountain View, CA) to create our initial dashboard prototype. We identified recommended strategies from the literature for using patient-facing technologies to enhance patient activation and designed a dashboard functionality to match each strategy. We then conducted a mixed-methods, task-based usability assessment of dashboard prototypes with individual patients. Measures included validated metrics of task performance on 5 common and standardized tasks, semistructured interviews, and a validated usability satisfaction questionnaire. After each round of usability testing, we revised the dashboard prototype in response to usability findings before the next round of testing until the majority of participants successfully completed tasks, expressed high satisfaction, and identified no new usability concerns (ie, stop criterion was met).

Results

The sample (N=14) comprised 5 patients in round 1, 3 patients in round 2, and 6 patients in round 3, at which point we reached our stop criterion. The participants’ mean age was 63 years (range 45-78 years), 57% (8/14) were female, and 50% (7/14) were white. Our design sprint yielded an initial patient-facing diabetes dashboard prototype that displayed and summarized 5 measures of patients’ diabetes health status (eg, hemoglobin A_1c). The dashboard used graphics to visualize and summarize health data and reinforce understanding, incorporated motivational strategies (eg, social comparisons and gamification), and provided educational resources and secure-messaging capability. More than 80% of participants were able to successfully complete all 5 tasks using the final prototype. Interviews revealed usability concerns with design, the efficiency of use, and content and terminology, which led to improvements. Overall satisfaction (0=worst and 7=best) improved from the initial to the final prototype (mean 5.8, SD 0.4 vs mean 6.7, SD 0.5).

Conclusions

Our results demonstrate the utility of the design sprint methodology paired with mixed-methods, task-based usability testing to efficiently and effectively design a patient-facing, Web-based diabetes dashboard that is satisfying for patients to use.

Aging barriers influencing mobile health usability for older adults: A literature based framework (MOLD-US)

BACKGROUND

With the growing population of older adults as a potential user group of mHealth, the need increases for mHealth interventions to address specific aging characteristics of older adults. The existence of aging barriers to computer use is widely acknowledged. Yet, usability studies show that mHealth still fails to be appropriately designed for older adults and their expectations. To enhance designs of mHealth aimed at older adult populations, it is essential to gain insight into aging barriers that impact the usability of mHealth as experienced by these adults.

OBJECTIVES

This study aims to synthesize literature on aging barriers to digital (health) computer use, and explain, map and visualize these barriers in relation to the usability of mHealth by means of a framework.

METHODS

We performed a scoping review to synthesize and summarize reported physical and functional age barriers in relation to digital (mobile) health applications use. Aging barriers reported in the literature were mapped onto usability aspects categorized by Nielsen to explain their influence on user experience of mHealth. A framework (MOLD-US) was developed summarizing the evidence on the influence of aging barriers on mHealth use experienced by older adults.

RESULTS

Four key categories of aging barriers influencing usability of mHealth were identified: cognition, motivation, physical ability and perception. Effective and satisfactory use of mHealth by older adults is complicated by cognition and motivation barriers. Physical ability and perceptual barriers further increase the risk of user errors and fail to notice important interaction tasks. Complexities of medical conditions, such as diminished eye sight related to diabetes or deteriorated motor skills as a result of rheumatism, can cause errors in user interaction.

CONCLUSIONS

This research provides a novel framework for the exploration of aging barriers and their causes influencing mHealth usability in older adults. This framework allows for further systematic empirical testing and analysis of mHealth usability issues, as it enables results to be classified and interpreted based on impediments intrinsic to usability issues experienced by older adults. Importantly, the paper identifies a key need for future research on motivational barriers impeding mhealth use of older adults. More insights are needed in particular to disaggregating normal age related functional changes from specific medical conditions that influence experienced usefulness of mHealth by these adults.

Multi-method laboratory user evaluation of an actionable clinical performance information system: Implications for usability and patient safety

INTRODUCTION

Electronic audit and feedback (e-A&F) systems are used worldwide for care quality improvement. They measure health professionals' performance against clinical guidelines, and some systems suggest improvement actions. However, little is known about optimal interface designs for e-A&F, in particular how to present suggested actions for improvement. We developed a novel theory-informed system for primary care (the Performance Improvement plaN GeneratoR; PINGR) that covers the four principal interface components: clinical performance summaries; patient lists; detailed patient-level information; and suggested actions. As far as we are aware, this is the first report of an e-A&F system with all four interface components.

OBJECTIVES

(1) Use a combination of quantitative and qualitative methods to evaluate the usability of PINGR with target end-users; (2) refine existing design recommendations for e-A&F systems; (3) determine the implications of these recommendations for patient safety.

METHODS

We recruited seven primary care physicians to perform seven tasks with PINGR, during which we measured on-screen behaviour and eye movements. Participants subsequently completed usability questionnaires, and were interviewed in-depth. Data were integrated to: gain a more complete understanding of usability issues; enhance and explain each other's findings; and triangulate results to increase validity.

RESULTS

Participants committed a median of 10 errors (range 8-21) when using PINGR's interface, and completed a median of five out of seven tasks (range 4-7). Errors violated six usability heuristics: clear response options; perceptual grouping and data relationships; representational formats; unambiguous description; visually distinct screens for confusable items; and workflow integration. Eye movement analysis revealed the integration of components largely supported effective user workflow, although the modular design of clinical performance summaries unnecessarily increased cognitive load. Interviews and questionnaires revealed PINGR is user-friendly, and that improved information prioritisation could further promote useful user action.

CONCLUSIONS

Comparing our results with the wider usability literature we refine a previously published set of interface design recommendations for e-A&F. The implications for patient safety are significant regarding: user engagement; actionability; and information prioritisation. Our results also support adopting multi-method approaches in usability studies to maximise issue discovery and the credibility of findings.

Usability Evaluation of Electronic Health Record System around Clinical Notes Usage-An Ethnographic Study

Background A substantial gap exists between current Electronic Health Record (EHR) usability and potential optimal usability. One of the fundamental reasons for this discrepancy is poor incorporation of a User-Centered Design (UCD) approach during the Graphical User Interface (GUI) development process. Objective To evaluate usability strengths and weaknesses of two widely implemented EHR GUIs for critical clinical notes usage tasks. Methods Twelve Internal Medicine resident physicians interacting with one of the two EHR systems (System-1 at Location-A and System-2 at Location-B) were observed by two usability evaluators employing an ethnographic approach. User comments and observer findings were analyzed for two critical tasks: (1) clinical notes entry and (2) related information-seeking tasks. Data were analyzed from two standpoints: (1) usability references categorized by usability evaluators as positive, negative, or equivocal and (2) usability impact of each feature measured through a 7-point severity rating scale. Findings were also validated by user responses to a post observation questionnaire. Results For clinical notes entry, System-1 surpassed System-2 with more positive (26% vs. 12%) than negative (12% vs. 34%) usability references. Greatest impact features on EHR usability (severity score pertaining to each feature) for clinical notes entry were: autopopulation (6), screen options (5.5), communication (5), copy pasting (4.5), error prevention (4.5), edit ability (4), and dictation and transcription (3.5). Both systems performed equally well on information-seeking tasks and features with greatest impacts on EHR usability were navigation for notes (7) and others (e.g., looking for ancillary data; 5.5). Ethnographic observations were supported by follow-up questionnaire responses. Conclusion This study provides usability-specific insights to inform future, improved, EHR interface that is better aligned with UCD approach.

Usability of a Smartphone Application to Support the Prevention and Early Intervention of Anxiety in Youth

Anxiety disorders are among the most common psychiatric problems in youth, fail to spontaneously remit, and place some youth at risk for additional behavioral and emotional difficulties. Efforts to target anxiety have resulted in evidence-based interventions but the resulting prevention effects are relatively small, often weakening over time. Mobile health (mHealth) tools could be of use to strengthen the effects of anxiety prevention efforts. Although a large number of mHealth apps have been developed, few have been evaluated in terms of usability prior to clinical effectiveness testing. Because usability is one of the main barriers to mHealth usage and adoption, the objective of this research was to evaluate the usability of a smartphone application (app) corresponding to an indicated prevention and early intervention targeting youth anxiety. To accomplish this, 132 children (M age = 9.65; 63% girls) and 45 service providers (M age = 29.13, 87% female) rated our app along five established dimensions of usability (ease of use, ease of learning, quality of support information, satisfaction, and stigma) using a standardized group-based testing protocol. Findings showed that the app was highly and positively rated by both youth and providers, with some variations (lower ratings when errors occurred). Path analyses findings also showed that system understanding was significantly related to greater system satisfaction, but that such relation occurred through the quality of support information offered by the app.

Usability and clinical efficacy of diabetes mobile applications for adults with type 2 diabetes: A systematic review

OBJECTIVES

To assess the usability and clinical effectiveness of diabetes mobile applications (diabetes apps) developed for adults with type 2 diabetes.

METHOD

A systematic review of the usability and effectiveness of diabetes apps was conducted. Searches were performed using MEDLINE, EMBASE, COMPENDEX, and IEEE XPLORE for articles published from January 1, 2011, to January 17, 2017. Search terms included: diabetes, mobile apps, and mobile health (mHealth).

RESULTS

The search yielded 723 abstracts of which seven usability studies and ten clinical effectiveness studies met the inclusion criteria from 20 publications. Usability, as measured by satisfaction ratings from experts and patients, ranged from 38% to 80%. Usability problem ratings ranged from moderate to catastrophic. Top usability problems are multi-steps task, limited functionality and interaction, and difficult system navigation. Clinical effectiveness, measured by reductions in HbA1c, ranged from 0.15% to 1.9%.

CONCLUSION

Despite meager satisfaction ratings and major usability problems, there is some limited evidence supporting the effectiveness of diabetes apps to improve glycemic control for adults with type 2 diabetes. Findings strongly suggest that efforts to improve user satisfaction, incorporate established principles of health behavior change, and match apps to user characteristics will increase the therapeutic impact of diabetes apps.

Heart Failure: Self-care to Success: Development and evaluation of a program toolkit

The Heart Failure: Self-care to Success toolkit was developed to assist NPs in empowering patients with heart failure (HF) to improve individual self-care behaviors. This article details the evolution of this toolkit for NPs, its effectiveness with patients with HF, and recommendations for future research and dissemination strategies.

Processes and Recommendations for Creating mHealth Apps for Low-Income Populations

BACKGROUND

Mobile health (mHealth) apps have shown to improve health indicators, but concerns remain about the inclusion of populations from low- and medium-income countries (LMIC) in these new technologies. Atrial fibrillation (AF) is a chronic condition with a challenging management. Previous studies have shown socioeconomic differences in the prescription of anticoagulant treatment and shared decision strategies are encouraged to achieve better outcomes. mHealth can aid both doctors and patients in this matter.

OBJECTIVE

We describe the development of an mHealth app (aFib) idealized to aid shared decision between doctor and patient about anticoagulation prophylaxis in AF in a low-income and low-literacy population in Brazil. On the basis of our research, we suggest the processes to be followed when developing mHealth apps in this context.

METHODS

A multidisciplinary team collected information about the target population and its needs and detected the best opportunity to insert the app in their current health care. Literature about the subject was reviewed and important data were selected to be delivered through good navigability, easy terminology, and friendly design. The app was evaluated in a multimethod setting.

RESULTS

The steps suggested to develop an mHealth app target to LMIC are: (1) characterize the problem and the target user, (2) review the literature, (3) translate information to knowledge, (4) protect information, and (5) evaluate usability and efficacy.

CONCLUSIONS

We expect that these recommendations can guide the development of new mHealth apps in LMIC, on a scientific basis.

Mixed-Methods Research in Diabetes Management via Mobile Health Technologies: A Scoping Review

BACKGROUND

Considering the increasing incidence and prevalence of diabetes worldwide and the high level of patient involvement it requires, diabetes self-management is a serious issue. The use of mobile health (mHealth) in diabetes self-management has increased, but so far research has not provided sufficient information about the uses and effectiveness of mHealth-based interventions. Alternative study designs and more rigorous methodologies are needed. Mixed-methods designs may be particularly useful because both diabetes self-management and mHealth studies require integrating theoretical and methodological approaches.

OBJECTIVE

This scoping review aimed to examine the extent of the use of mixed-methods research in mHealth-based diabetes management studies. The methodological approaches used to conduct mixed-methods studies were analyzed, and implications for future research are provided.

METHODS

Guided by Arksey and O'Malley's framework, this scoping review implemented a comprehensive search strategy including reviewing electronic databases, key journal searches, Web-based research and knowledge centers, websites, and handsearching reference lists of the studies. The studies focusing on mHealth technologies and diabetes management were included in the review if they were primary research papers published in academic journals and reported using a combination of qualitative and quantitative methods. The key data extracted from the reviewed studies include purpose of mixing, design type, stage of integration, methods of legitimation, and data collection techniques.

RESULTS

The final sample (N=14) included studies focused on the feasibility and usability of mHealth diabetes apps (n=7), behavioral measures related to the mHealth apps (n=6), and challenges of intervention delivery in the mHealth context (n=1). Reviewed studies used advanced forms of mixed-methods designs where integration occurred at multiple points and data were collected using multiple techniques. However, the majority of studies did not identify a specific mixed-methods design or use accepted terminology; nor did they justify using this approach.

CONCLUSIONS

This review provided important insights into the use of mixed methods in studies focused on diabetes management via mHealth technologies. The prominent role of qualitative methods and tailored measures in diabetes self-management studies was confirmed, and the importance of using multiple techniques and approaches in this field was emphasized. This review suggests defining specific mixed-methods questions, using specific legitimation methods, and developing research designs that overcome sampling and other methodological problems in future studies.

Development and Usability of REACH: A Tailored Theory-Based Text Messaging Intervention for Disadvantaged Adults With Type 2 Diabetes

BACKGROUND

Among adults with type 2 diabetes mellitus (T2DM), adherence to recommended self-care activities is suboptimal, especially among racial and ethnic minorities with low income. Self-care nonadherence is associated with having worse glycemic control and diabetes complications. Text messaging interventions are improving the self-care of adults with T2DM, but few have been tested with disadvantaged populations.

OBJECTIVE

To develop Rapid Education/Encouragement And Communications for Health (REACH), a tailored, text messaging intervention to support the self-care adherence of disadvantaged patients with T2DM, based on the Information-Motivation-Behavioral skills model. We then tested REACH's usability to make improvements before evaluating its effects.

METHODS

We developed REACH's content and functionality using an empirical and theory-based approach, findings from a previously pilot-tested intervention, and the expertise of our interdisciplinary research team. We recruited 36 adults with T2DM from Federally Qualified Health Centers to participate in 1 of 3 rounds of usability testing. For 2 weeks, participants received daily text messages assessing and promoting self-care, including tailored messages addressing users' unique barriers to adherence, and weekly text messages with adherence feedback. We analyzed quantitative and qualitative user feedback and system-collected data to improve REACH.

RESULTS

Participants were, on average, 52.4 (SD 9.5) years old, 56% (20/36) female, 63% (22/35) were a racial or ethnic minority, and 67% (22/33) had an income less than US $35,000. About half were taking insulin, and average hemoglobin A1c level was 8.2% (SD 2.2%). We identified issues (eg, user concerns with message phrasing, technical restrictions with responding to assessment messages) and made improvements between testing rounds. Overall, participants favorably rated the ease of understanding (mean 9.6, SD 0.7) and helpfulness (mean 9.3, SD 1.4) of self-care promoting text messages on a scale of 1-10, responded to 96% of assessment text messages, and rated the helpfulness of feedback text messages 8.5 (SD 2.7) on a scale of 1-10. User feedback led to refining our study enrollment process so that users understood the flexibility in message timing and that computers, not people, send the messages. Furthermore, research assistants' feedback on the enrollment process helped improve participants' engagement with study procedures.

CONCLUSIONS

Testing technology-delivered interventions with disadvantaged adults revealed preferences and concerns unique to this population. Through iterative testing and multiple data sources, we identified and responded to users' intervention preferences, technical issues, and shortcomings in our research procedures.