A Review of Usability Evaluation Methods and Their Use for Testing eHealth HIV Interventions

Comparison of evaluation methods for improving the usability of a Spanish mHealth tool

OBJECTIVE

Mobile health (mHealth) technology is now widely used across health conditions and populations. The rigorous development of these tools has yielded improved health outcomes, yet the ideal approach for developing mHealth tools continues to evolve, indicating the need for rigorous usability evaluation methods. This study compares two usability evaluation methods - cognitive interviews and usability assessments employing a think-aloud approach - for adapting an evidence-based mHealth tool from English into Spanish.

METHODS

We conducted cognitive interviews and usability assessments using a think-aloud protocol to evaluate the usability of an HIV mHealth application among 40 Spanish-speaking adults with HIV in New York City, NY, and La Romana, Dominican Republic. The Health IT Usability Evaluation Model (Health-ITUEM) was used to guide the analysis of qualitative data collected from each method.

RESULTS

Participants (N = 40) averaged 43 years old (SD = 12.26; range 20-79), identified primarily Hispanic/Latino (92.5 %), and resided in La Romana (50 %) or New York City (50 %). Both usability evaluation methods yielded similar findings, highlighting learnability and information needs as crucial components of participant feedback for the mHealth application. Cognitive interviews captured participants' perspectives on the app's interface and design. On the other hand, results from usability assessments offered insights into participants' competency while interacting with the mHealth tool.

CONCLUSION

Findings from this study highlight the contributions and limitations of including cognitive interviews and task-based usability assessments using a think-aloud approach in mHealth usability testing. Future research should employ a multi-method approach, incorporating complementary usability evaluation methods and engaging participants in multiple assessments. Using complementary usability evaluation methods may provide a more comprehensive understanding of the usability and participant experience aspects of a mHealth tool compared to using a single usability evaluation method.

Patients' perceptions of use, needs, and preferences related to a telemedicine solution for HIV care in a Norwegian outpatient clinic: a qualitative study

BACKGROUND

Telemedicine in outpatient services for people living with human immunodeficiency virus (PLHIV) was scaled up during the COVID-19 pandemic as services transitioned to remote care. Many studies have reported on the challenges and advantages of telemedicine care during the pandemic. However, there is limited research on the provision of telemedicine human immunodeficiency virus (HIV) care beyond the COVID-19 pandemic, which entails different telemedicine components and focuses on ways to improve the telemedicine experience for patients. This study aimed to explore PLHIV's perceptions of use, needs, and preferences related to a telemedicine solution for HIV care in an outpatient clinic in Norway. The telemedicine solution included a pre-consultation questionnaire, asynchronous digital messages, and video consultation.

METHODS

Qualitative interviews were conducted with 12 PLHIV. The interviews were analysed using thematic analysis.

RESULTS

We identified four main themes that covered the participants' perceptions, needs, and preferences: (1) perceived usability, (2) maintaining confidentiality, (3) accommodating personal preferences, and (4) perceived usefulness. Some participants had difficulty logging into the telemedicine solution. Other participants suggested additional functionalities, such as picture sharing and access to test result. Telemedicine care enabled the avoidance of stigmatising clinic experiences, although a few participants reported concerns about confidentiality and data security. Accommodating personal preferences and needs in terms of the type of consultations (in-person or video) and frequency of visits was essential to the participants. With telemedicine care, participants felt more in control of their own lives, perceiving that it increased their perceived quality of life and saved them both time and money for travelling to the clinic.

CONCLUSIONS

Our study identified several specific needs and preferences related to the assessed technical solution and the provision of current and future telemedicine care services. Nevertheless, the telemedicine solution was perceived as a usable, flexible, and person-centred approach to HIV care, contributing to accommodating the participants' personal preferences. However, healthcare professionals need to ensure that individual requirements and preferences are consistent with evidence-based follow-up and supported by person-centred care. Thus, the practice of shared decision making is important in telemedicine care.

Usability study of SOSteniamoci: An internet-based intervention platform to support informal caregivers in Italy

Background

Providing informal care can be experienced as stressful and lead to caregiver burden. Internet-based interventions, a specific form of eHealth, have proven to be a good option to support informal caregivers. SOSteniamoci, an internet-based intervention already tested in Lithuania, was translated and adapted for Italian caregivers.

Objective

As many novel eHealth solutions have been rejected by end-users due to usability problems, we aimed to evaluate the usability of the adapted platform, using a computer-based prototype.

Methods

The following methods and metrics were applied: 1. task analysis, using audio and video recordings that included three usability metrics: task completion rate, frequency of errors, and frequency of help requests; 2. the system usability scale (SUS); and 3. a semi-structured interview to collect additional data about the system's design and overall satisfaction.

Results

Ten informal caregivers (60% female; age M = 47.8, SD = 15.21) provided insights and suggestions for increasing the usability of the platform. The platform was considered satisfactory, with a mean score on the SUS of 75 (SD = 13.07) out of 100. The task analysis measurements highlighted difficulties in how to log in to the platform, understanding what the intervention is about, and texting the therapist. The same difficulties were also mentioned during the post-experience interview. Thus, improvements were subsequently made to enhance users' experience when navigating the platform. Finally, the platform overall was found to be intuitive and friendly, and the contents were appreciated.

Conclusion

To maintain participants' engagement and prevent drop-out, it is crucial to test the usability of internet-based interventions. Even though the platform proved to be user-friendly, intuitive and easy to use, several enhancements were implemented based on participants' feedback. Thus, the usability of internet-based interventions should be tested, and end-users must be involved in the development process of such solutions.

Patient-centered development of Embrace Pain: an online acceptance and commitment therapy intervention for cancer survivors with chronic painful chemotherapy-induced peripheral neuropathy

BACKGROUND

Around 30% of cancer survivors suffer from chemotherapy-induced peripheral neuropathy (CIPN) ≥6 months after completion of chemotherapy, which comes with limitations in daily functioning and worsened quality of life(QoL). Treatment options are scarce. Our aim was to develop an online self-help intervention based on Acceptance and Commitment Therapy (ACT) to reduce pain interference in cancer survivors experiencing painful chronic CIPN.

MATERIAL AND METHODS

This article applied a patient-centered design process using the Center for eHealth Research (CeHRes) roadmap. User needs were examined using online semi-structured interviews with patients and experts (N = 23). Interviews were transcribed verbatim and analyzed using thematic analysis. Personas were created based on interviews. Intervention content was based on identified user needs and ACT. Content and design were finalized using low-fidelity prototype testing (N = 5), and high-fidelity prototype testing (N = 7).

RESULTS

Patients appreciated and agreed with the elements of ACT, had varying guidance needs, and wanted to have autonomy (e.g., moment and duration of use). Additionally, it was important to be aware that patients have had a life-threatening disease which directly relates to the symptoms they experience. Patients reported to prefer a user-friendly and accessible intervention. Similar points also emerged in the expert interviews. The final intervention, named Embrace Pain, includes six sessions. Session content is based on psychoeducation and all ACT processes. Further interpretation of the intervention (such as quotes, guidance, and multimedia choices) is based on the interviews.

CONCLUSION

This development demonstrated how a patient-centered design process from a theoretical framework can be applied. Theory-driven content was used as the basis of the intervention. Findings show an online ACT intervention designed for cancer survivors with painful chronic CIPN.

Usability of a mobile application for health professionals in home care services: a user-centered approach

The use of mobile devices by healthcare professionals has led to rapid growth in the development of mobile healthcare applications designed to improve healthcare services. This study was conducted to assess the acceptability and usability of a mobile application for health professionals in relation to their work in hospitalization at home. A mixed methods approach was used. Acceptance, included the satisfaction of the professionals, attitudes toward using the application, and intention or willingness to continue using the application. Usability tests were performed in laboratory analyzing five controlled clinical tasks, and the interaction of the participants with the mobile application was based on the six basic facial expressions published by the American Psychological Association. Perceived satisfaction was assessed using the computer system usability questionnaire. Thirty-two participants completed the task scenarios and questionnaire. More than 90 per cent of participants were able to complete the tasks with only some difficult with vital signs. Satisfaction had a score of 6.18/7 (SD: 0.76), and recommendation of the mobile application had a score of 6.21/7 (SD: 0.81). This study showed a significant usability and acceptability of this mobile application, in terms of effectiveness, efficiency, and satisfaction.

Usability evaluation of mHealth apps for elderly individuals: a scoping review

BACKGROUND

Usability is a key factor affecting the acceptance of mobile health applications (mHealth apps) for elderly individuals, but traditional usability evaluation methods may not be suitable for use in this population because of aging barriers. The objectives of this study were to identify, explore, and summarize the current state of the literature on the usability evaluation of mHealth apps for older adults and to incorporate these methods into the appropriate evaluation stage.

METHODS

Electronic searches were conducted in 10 databases. Inclusion criteria were articles focused on the usability evaluation of mHealth apps designed for older adults. The included studies were classified according to the mHealth app usability evaluation framework, and the suitability of evaluation methods for use among the elderly was analyzed.

RESULTS

Ninety-six articles met the inclusion criteria. Research activity increased steeply after 2013 (n = 92). Satisfaction (n = 74) and learnability (n = 60) were the most frequently evaluated critical measures, while memorability (n = 13) was the least evaluated. The ratios of satisfaction, learnability, operability, and understandability measures were significantly related to the different stages of evaluation (P < 0.05). The methods used for usability evaluation were questionnaire (n = 68), interview (n = 36), concurrent thinking aloud (n = 25), performance metrics (n = 25), behavioral observation log (n = 14), screen recording (n = 3), eye tracking (n = 1), retrospective thinking aloud (n = 1), and feedback log (n = 1). Thirty-two studies developed their own evaluation tool to assess unique design features for elderly individuals.

CONCLUSION

In the past five years, the number of studies in the field of usability evaluation of mHealth apps for the elderly has increased rapidly. The mHealth apps are often used as an auxiliary means of self-management to help the elderly manage their wellness and disease. According to the three stages of the mHealth app usability evaluation framework, the critical measures and evaluation methods are inconsistent. Future research should focus on selecting specific critical measures relevant to aging characteristics and adapting usability evaluation methods to elderly individuals by improving traditional tools, introducing automated evaluation tools and optimizing evaluation processes.

Non-empirical ISO 9241-210:2019-based usability evaluation using fuzzy inference analyser: a special issue on interactive social robots

Purpose

The objective of this research was to develop a new and highly accurate approach based on a fuzzy inference system (FIS) for the evaluation of usability based on ISO 9241-210:2019. In this study, a fully automated method of usability evaluation is used for interactive systems with a special look at interactive social robots.

Design/methodology/approach

Fuzzy logic uses as an intelligent computing technique to deal with uncertainty and incomplete data. Here this system is implemented using MATLAB fuzzy toolbox. This system attempted to quantify four criteria that correlate highly with ISO 9241-210:2019 criteria for the evaluation of interactive systems with maximum usability. Also, the system was evaluated with standard cases of computer interactive systems usability evaluation. The system did not need to train various data and to check the rules. Just small data were used to fine-tune the fuzzy sets. The results were compared against experimental usability evaluation with the statistical analysis.

Findings

It is found that there was a high strong linear relation between the FIS usability assessment and System Usability Scale (SUS) based usability assessment, and authors’ new method provides reliable results in the estimation of the usability.

Research limitations/implications

In human-robot systems, human performance plays an important role in the performance of social interactive systems. In the present study, the proposed system has considered all the necessary criteria for designing an interactive system with a high level of user because it is based on ISO 9241-210:2019.

Practical implications

For future research, the system could be expanded with the training of historical data and the production of rules through integrating FIS and neural networks.

Originality/value

This system considered all essential criteria for designing an interactive system with a high level of usability because it is based on ISO 9241-210:2019. For future research, the system could be expanded with the training of historical data and the production of rules through integrating FIS and neural networks.

Agile, Easily Applicable, and Useful eHealth Usability Evaluations: Systematic Review and Expert-Validation

Background  Electronic health (eHealth) usability evaluations of rapidly developed eHealth systems are difficult to accomplish because traditional usability evaluation methods require substantial time in preparation and implementation. This illustrates the growing need for fast, flexible, and cost-effective methods to evaluate the usability of eHealth systems. To address this demand, the present study systematically identified and expert-validated rapidly deployable eHealth usability evaluation methods.

Objective  Identification and prioritization of eHealth usability evaluation methods suitable for agile, easily applicable, and useful eHealth usability evaluations.

Methods  The study design comprised a systematic iterative approach in which expert knowledge was contrasted with findings from literature. Forty-three eHealth usability evaluation methods were systematically identified and assessed regarding their ease of applicability and usefulness through semi-structured expert interviews with 10 European usability experts and systematic literature research. The most appropriate eHealth usability evaluation methods were selected stepwise based on the experts' judgements of their ease of applicability and usefulness.

Results  Of these 43 eHealth usability evaluation methods identified as suitable for agile, easily applicable, and useful eHealth usability evaluations, 10 were recommended by the experts based on their usefulness for rapid eHealth usability evaluations. The three most frequently recommended eHealth usability evaluation methods were Remote User Testing, Expert Review, and Rapid Iterative Test and Evaluation Method. Eleven usability evaluation methods, such as Retrospective Testing, were not recommended for use in rapid eHealth usability evaluations.

Conclusion  We conducted a systematic review and expert-validation to identify rapidly deployable eHealth usability evaluation methods. The comprehensive and evidence-based prioritization of eHealth usability evaluation methods supports faster usability evaluations, and so contributes to the ease-of-use of emerging eHealth systems.

Establishing a usability cut-point for the Health Information Technology Usability Evaluation Scale (Health-ITUES)

OBJECTIVE

The Health Information Technology Usability Evaluation Scale (Health-ITUES) is a validated and reliable instrument to evaluate usability of information technology (IT) tools. In this study, we aimed to establish the optimal cut-point of the Health-ITUES to identify usability of IT tools.

METHODS

Adult participants were recruited to a trial evaluating a mobile app for self-managing HIV. Participants completed the Health-ITUES at the 3- and 6-month follow-up. Health-ITUES is a 20-item questionnaire that assesses four subscales: impact, perceived usefulness, perceived ease of use, and user control. The total score ranged from 1 to 5 and a higher score indicates greater usability. App use was defined as the proportion of activities completed by participants in both study arms. The selection of an optimal cut-point involved a series of multiple linear regression models with 500 bootstrap replications to examine the relationship between the Health-ITUES total score and app use, controlling for potential covariates.

RESULTS

We included 158 participants; mean age was 49.7 years (SD 10.3), 71% were African American/Black, and 72% were non-Hispanic. Mean Health-ITUES total scores at 3 and 6 months were 4.39 (SD 0.75) and 4.43 (SD 0.75), respectively. App use completedby participants from baseline to the 3-month follow-up visits was 0.61 (SD 0.36, range 0-1.72) and from 3-month to the 6-month follow-up visits was 0.51 (SD 0.37). Participants who reported greater Health-ITUES total score completed more activities [β = 0.18, 95% Confidence Interval (CI) 0.10-0.27]. The optimal cut-point of 4.32 (95% CI: 4.25-4.56) yielded the lowest p-value to identify usability of IT tools.

CONCLUSIONS

In this study of adults with HIV, we identified an optimal cut-point of 4.32 on the Health-ITUES total score to define usability. Further studies are needed to validate this cut-point.

Usability Evaluation of the mLab App for Improving Home HIV Testing Behaviors in Youth at Risk of HIV Infection

Improving access to HIV testing among youth at high risk is essential for reaching those who are most at risk for HIV and least likely to access health care services. This study evaluates the usability of mLab, an app with image-processing feature that analyzes photos of OraQuick HIV self-tests and provides real-time, personalized feedback. mLab includes HIV prevention information, testing reminders, and instructions. It was developed through iterative feedback with a youth advisory board (N = 8). The final design underwent heuristic (N = 5) and end-user testing (N = 20). Experts rated mLab following Nielsen's heuristic checklist. End-users used the Health Information Technology Usability Evaluation Scale. While there were some usability problems, overall study participants found mLab useful and user-friendly. This study provides important insights into using a mobile app with imaging for interpreting HIV test results with the goal of improving HIV testing and prevention in populations at high risk.

Examining the Information Systems Success (ISS) of a mobile sexual health app (MyPEEPS Mobile) from the perspective of very young men who have sex with men (YMSM)

OBJECTIVE

The widespread and frequent use of mobile technology among adolescents, including sexual minority adolescents, presents an opportunity for the development of mobile health (mHealth) technology to combat the continuing HIV epidemic among young men who have sex with men (YMSM). We analyzed perceptions of the quality and impact of an HIV prevention mobile app on sexual risk reduction among YMSM.

METHODS

Participants were recruited from a larger randomized controlled trial of the MyPEEPS Mobile app among YMSM aged 13-18 years. Data were collected via semi-structured interviews to assess quality and user satisfaction with MyPEEPS Mobile app using analysis informed by the Information Systems Success framework. Interview data were transcribed verbatim and analyzed using six themes: information quality, net benefit, user satisfaction, product quality, service quality, and health care barriers.

RESULTS

Interviews were conducted with 40 YMSM (45% Hispanic; 80% non-White; 88% non-rural resident; 28% aged 17 years). Participants' responses indicated that information quality was high, reporting that the app information was concise, easy to understand, useful, and relevant to their life. The net benefits were stated as improvements in their decision-making skills, health behaviors, communication skills with partner(s), and increased knowledge of HIV risk. There was general user satisfaction and enjoyment when using the app, although most of the participants did not intend to reuse the app unless new activities were added. Participants expressed that the product quality of the app was good due to its personalization, representation of the LGBTQIA + community, and user-friendly interface. Although no major technical issues were reported, participants suggested that adaption to a native app, rather than a web app, would improve service quality through faster loading speed. Participants also identified some health care barriers that were minimized by app use.

CONCLUSIONS

The MyPEEPS Mobile app is a well received, functional, and entertaining mHealth HIV prevention tool that may improve HIV prevention skills and reduce HIV risk among YMSM.

Usability testing of tablet-based cognitive behavioral intervention application to improve a simple walking activity for older adults with arthritis fatigue

The purpose of this study was to test usability of a tablet-based cognitive behavioral intervention ("Tab-CBI") application. Tab-CBI was designed to improve a simple walking activity for older adults with arthritis fatigue. Two phases of usability evaluation were conducted: phase I with 5 older adults to identify any initial design issues and phase II with 10 older adults to examine if Tab-CBI was seamlessly integrated into their daily living. Participants perceived that Tab-CBI was easy, light, and intuitive to use and helped them walking. They reported two key benefit as a motivator to make more active in walking and improved confidence of self-management of fatigue. They referred video recordings and videoconferencing communications as two preferable features. There were also comments for improvement, for example, a challenge for holding a tablet when videoconferencing communications. Participants comments were incorporated into a final version of Tab-CBI to prepare for a future pilot study.

The most used questionnaires for evaluating telemedicine services

Background

Questionnaires are commonly used tools in telemedicine services that can help to evaluate different aspects. Selecting the ideal questionnaire for this purpose may be challenging for researchers. This study aims to review which questionnaires are used to evaluate telemedicine services in the studies, which are most common, and what aspects of telemedicine evaluation do they capture.

Methods

The PubMed database was searched in August 2020 to retrieve articles. Data extracted from the final list of articles included author/year of publication, journal of publication, type of evaluation, and evaluation questionnaire. Data were analyzed using descriptive statistics.

Results

Fifty-three articles were included in this study. The questionnaire was used for evaluating the satisfaction (49%), usability (34%), acceptance (11.5%), and implementation (2%) of telemedicine services. Among telemedicine specific questionnaires, Telehealth Usability Questionnaire (TUQ) (19%), Telemedicine Satisfaction Questionnaire (TSQ) (13%), and Service User Technology Acceptability Questionnaire (SUTAQ) (5.5%), were respectively most frequently used in the collected articles. Other most used questionnaires generally used for evaluating the users’ satisfaction, usability, and acceptance of technology were Client Satisfaction Questionnaire (CSQ) (5.5%), Questionnaire for User Interaction Satisfaction (QUIS) (5.5%), System Usability Scale (SUS) (5.5%), Patient Satisfaction Questionnaire (PSQ) (5.5%), and Technology Acceptance Model (TAM) (3.5%) respectively.

Conclusion

Employing specifically designed questionnaires or designing a new questionnaire with fewer questions and more comprehensiveness in terms of the issues studied provides a better evaluation. Attention to user needs, end-user acceptance, and implementation processes, along with users' satisfaction and usability evaluation, may optimize telemedicine efforts in the future.