ChatGPT-3.5 System Usability Scale early assessment among Healthcare Workers: Horizons of adoption in medical practice

Artificial intelligence (AI) chatbots, such as ChatGPT, have widely invaded all domains of human life. They have the potential to transform healthcare future. However, their effective implementation hinges on healthcare workers' (HCWs) adoption and perceptions. This study aimed to evaluate HCWs usability of ChatGPT three months post-launch in Saudi Arabia using the System Usability Scale (SUS). A total of 194 HCWs participated in the survey. Forty-seven percent were satisfied with their usage, 57 % expressed moderate to high trust in its ability to generate medical decisions. 58 % expected ChatGPT would improve patients' outcomes, even though 84 % were optimistic of its potential to improve the future of healthcare practice. They expressed possible concerns like recommending harmful medical decisions and medicolegal implications. The overall mean SUS score was 64.52, equivalent to 50 % percentile rank, indicating high marginal acceptability of the system. The strongest positive predictors of high SUS scores were participants' belief in AI chatbot's benefits in medical research, self-rated familiarity with ChatGPT and self-rated computer skills proficiency. Participants' learnability and ease of use score correlated positively but weakly. On the other hand, medical students and interns had significantly high learnability scores compared to others, while ease of use scores correlated very strongly with participants' perception of positive impact of ChatGPT on the future of healthcare practice. Our findings highlight the HCWs' perceived marginal acceptance of ChatGPT at the current stage and their optimism of its potential in supporting them in future practice, especially in the research domain, in addition to humble ambition of its potential to improve patients' outcomes particularly in regard of medical decisions. On the other end, it underscores the need for ongoing efforts to build trust and address ethical and legal concerns of AI implications in healthcare. The study contributes to the growing body of literature on AI chatbots in healthcare, especially addressing its future improvement strategies and provides insights for policymakers and healthcare providers about the potential benefits and challenges of implementing them in their practice.

User Experience Evaluation of a Spinal Surgery Robot: Workload, Usability, and Satisfaction Study

BACKGROUND

Robotic spine surgery has continued to evolve since its US Food and Drug Administration approval in 2004, with products now including real-time video guidance and navigation during surgery. As the market for robotic surgical devices evolves, it is important to consider usability factors.

OBJECTIVE

The primary objective of this study was to determine the user experience of a surgical-assistive robotic device. The secondary objective was to evaluate workload, usability, the After-Scenario Questionnaire (ASQ), and the System Usability Scale (SUS). In addition, this study compares the workload, usability, and satisfaction survey of the device among different occupational groups using the device.

METHODS

Doctors (n=15) and nurses (n=15), the intended users of the surgical assistant robot, participated in the usability evaluation. Participants performed essential scenarios for the surgical assistant robot and provided scenario-specific satisfaction (ASQ), workload (NASA Task Load Index), and usability (SUS) scores.

RESULTS

Both doctors and nurses had task success rates of 85% or higher for each scenario. ASQ results showed that both doctors and nurses were least satisfied with ease of completing the task of registration (group 1: mean 4.73, SD 1.57 and group 2: mean 4.47, SD 1.8), amount of time it took (group 1: mean 4.47, SD 1.63 and group 2: mean 4.40, SD 2.09), and support information satisfaction (group 1: mean 5.13, SD 1.50 and group 2: mean 5.13, SD 1.89). All participants had low workloads, and the overall Task Load Index score had a P value of .77, which is greater than .05. The SUS results showed that the overall usability mean for doctors was 64.17 (SD 16.52) and the mean for nurses was 61.67 (SD 19.18), with a P value of .84, which is greater than .05, indicating no difference between the 2 groups.

CONCLUSIONS

In this study, doctors and nurses evaluated the interaction of the device in a simulated environment, the operating room. By evaluating the use experience and usability of the device with real intended users, we can develop a more effective and convenient user interface.

Patient-Centered Chronic Wound Care Mobile Apps: Systematic Identification, Analysis, and Assessment

Background

The prevalence of chronic wounds is predicted to increase within the aging populations in industrialized countries. Patients experience significant distress due to pain, wound secretions, and the resulting immobilization. As the number of wounds continues to rise, their adequate care becomes increasingly costly in terms of health care resources worldwide. eHealth support systems are being increasingly integrated into patient care. However, to date, no systematic analysis of such apps for chronic wounds has been published.

Objective

The aims of this study were to systematically identify and subjectively assess publicly available German- or English-language mobile apps for patients with chronic wounds, with quality assessments performed by both patients and physicians.

Methods

Two reviewers independently conducted a systematic search and assessment of German- or English-language mobile apps for patients with chronic wounds that were available in the Google Play Store and Apple App Store from April 2022 to May 2022. In total, 3 apps met the inclusion and exclusion criteria and were reviewed independently by 10 physicians using the German Mobile App Rating Scale (MARS) and the System Usability Scale (SUS). The app with the highest mean MARS score was subsequently reviewed by 11 patients with chronic wounds using the German user version of the MARS (uMARS) and the SUS. Additionally, Affinity for Technology Interaction (ATI) scale scores were collected from both patients and physicians.

Results

This study assessed mobile apps for patients with chronic wounds that were selected from a pool of 118 identified apps. Of the 73 apps available in both app stores, 10 were patient oriented. After excluding apps with advertisements or costs, 3 apps were evaluated by 10 physicians. Mean MARS scores ranged from 2.64 (SD 0.65) to 3.88 (SD 0.65) out of 5, and mean SUS scores ranged from 50.75 (SD 27) to 80.5 (SD 17.7) out of 100. WUND APP received the highest mean MARS score (mean 3.88, SD 0.65 out of 5) among physicians. Hence, it was subsequently assessed by 11 patients and achieved a similar rating (uMARS score: mean 3.89, SD 0.4 out of 5). Technical affinity, as measured with the ATI scale, was slightly lower in patients (score: mean 3.62, SD 1.35 out of 6) compared to physicians (score: mean 3.88, SD 1.03 out 6).

Conclusions

The quality ratings from physicians and patients were comparable and indicated mediocre app quality. Technical affinity, as assessed by using the ATI scale, was slightly lower for patients. Adequate apps for patients with chronic wounds remain limited, emphasizing the need for improved app development to meet patient needs. The ATI scale proved valuable for assessing technical affinity among different user groups.

Administrative Dashboard for Monitoring Use of a Web-Based Parent Training Intervention: Usability Study

BACKGROUND

Web-based parent training (PT) programs can strengthen parent-child relationships by equipping caregivers with knowledge and evidence-based strategies to manage behavior. Hybrid facilitation of PT includes facilitator interaction paired with self-administered and web-based PT. Web-based administrative dashboards provide users (eg, administrators, facilitators, and researchers) with an integrated platform to monitor parent progress and activities within a PT program or website. Despite the utility and prevalence of administrative dashboards for web-based behavioral interventions, to our knowledge, no research studies have explored the perspectives and insights of dashboard users to enhance user experience and program delivery.

OBJECTIVE

The purpose of this study is to evaluate the usability of the administrative dashboard (ezDashboard) for the ezParent program, a 6-module web-based PT program for parents of children aged 2-5 years.

METHODS

This study used a descriptive, single-group design with administrators who were overseeing the implementation of the ezParent program and trained facilitators for hybrid ezParent delivery. Participants spent at least 30 minutes reviewing and evaluating the ezDashboard and then completed a survey of their experience with the dashboard. The survey included the validated 10-item System Usability Scale and open-ended questions focusing on user performance, navigation ease, and overall usefulness of the ezDashboard.

RESULTS

Participants (N=15) indicated high usability of the ezDashboard with System Usability Scale scoring a total mean score of 83.5 (SD 16.3). Most participants (n=13, 87%) rated the overall user-friendliness of the ezDashboard as good (n=3, 20%), excellent (n=9, 60%), or best imaginable (n=1, 7%). Open-ended questions revealed the ezDashboard is or would be useful to monitor parent progress and trends in engagement (n=8, 53%) and for reviewing topics for discussion and communicating with parents (n=5, 33%). ezParent administrators (n=4) identified that real-time data for ezParent use helps overall management of program uptake. Suggestions for features to add to the ezDashboard included the ability to track partial progress of program modules (4/14, 29%), total time spent per module (2/14, 14%), and exportable reports (4/14, 29%). Other ideas for improvement included direct messaging capabilities, videoconferencing platform integration, and being able to modify participant account and contact information.

CONCLUSIONS

Results indicate that the ezDashboard is easy to use and provides functional information to facilitators and administrators in delivering ezParent. Qualitative results indicate that integrating suggested features into the ezDashboard may help provide a smoother experience for facilitators, administrators, and ultimately the parents using the program. Providing resources for facilitators and administrators in real time to monitor intervention participants' progress in a program can be helpful in tracking progress and providing facilitated support in tailoring program content and program completion.

Translation and validation of the System Usability Scale to a Dutch version: D-SUS

PURPOSE

The System Usability Scale (SUS) is the most commonly used questionnaire to assess usability of healthcare innovations but is not available in Dutch (D-SUS). This study aims to translate the SUS to Dutch and to determine its internal consistency, test-retest reliability, and construct validity in healthcare innovations focused on rehabilitation technologies.

METHODS

Translation of the SUS was performed according to the WHO recommendations. Fifty-four participants filled out the D-SUS and Dutch Quebec User Evaluation of Satisfaction with assistive Technology (D-QUEST) twice. Internal consistency was assessed by Cronbach's alpha. Test-retest reliability was evaluated by Gwet's agreement coefficient (Gwet's AC2) on item scale, and Pearson correlation coefficient (PCC) for the overall D-SUS scores. Construct validity was assessed with the PCC between the D-SUS and D-QUEST overall scores (Netherlands Trial Register, ID: NL9169).

RESULTS

After translation, Cronbach's alpha was 0.74. Gwet's AC2 was 0.68 and the PCC between the first and second overall D-SUS scores was 0.75. No significant difference in D-SUS score between the two measurements was found. Repeatability coefficient was 18.4. The PCC between the D-SUS and D-QUEST overall scores was 0.49.

CONCLUSIONS

The D-SUS is a valid and reliable tool for usability assessment of healthcare innovations, specifically rehabilitation technologies.

Comparison of German Translations of the System Usability Scale - Which to Take?

The System Usability Scale (SUS) is a reliable tool for usability measurement and evaluation. Since its original language is English, a translation is required before a target group can answer it in their native language. The challenge of translating questionnaires lies in the preservation of its original properties. Different versions of a German SUS have been proposed and are currently in use. Objective of this work is to find and compare available German translations. Four versions were found and compared in terms of the translation process and the exact wording of the translation. Only the version of Gao et al. has been systematically validated, but has an unnatural wording. Although not validated yet, the proposed version of Rummel et al. is a good compromise between wording and methodically clean development. The version of Lohmann and Schäffer is the close runner up, as it may improve the wording at the expense of methodological accuracy. Since the version of Rauer gives no information about its translation process, it is considered least preferred of the four compared translations.

The feasibility of using a digital tool to enhance mobilisation following abdominal cancer surgery-a non-randomised controlled trial

BACKGROUND

Early mobilisation is strongly recommended following abdominal cancer surgery, but evidence on how to structure early mobilisation to improve outcomes and support patient adherence is lacking. Pedatim® (Phystec) is a novel digital tool designed to support mobilisation in hospital settings using prescribed activities and goals on a tablet. The aim of this study was to evaluate the feasibility of the Pedatim tablet to enhance mobilisation following abdominal cancer surgery.

METHODS

In a non-randomised feasibility trial design, participants were recruited between January and May 2022 at Karolinska University Hospital, Sweden. Participants used a Pedatim tablet from postoperative day 1 (POD 1) until hospital discharge. The primary objective was to evaluate process feasibility, regarding recruitment, compliance, and acceptability. Recruitment was measured by percentage of available patients included, eligibility criteria sufficiency, and number of dropouts. Compliance was measured by number of patients using versus not using the board. Acceptability was measured using the System Usability Scale. The secondary objective was to evaluate scientific feasibility, defined as an indication of treatment effects where physical activity was assessed using an activPAL accelerometer. Unforeseen events relating to the tablet were also registered.

RESULTS

Based on predetermined feasibility criteria, the overall study design was determined to be feasible regarding recruitment as 69% accepted participation (n = 20), compliance was 95%, and the acceptability mean score was high (77/100). Eligibility criteria were not feasible as 79% (n = 108) of available patients were excluded. The intervention was determined to be scientifically feasible, mean steps per day increased from 623 (SD 766) to 1823 (SD 1446), and mean sit-to-stand transitions per day increased from 11 (SD 8) to 29 (SD 12) POD 1-4. Technical issues emerged, highlighting the need for available technical support and "user champions" among healthcare professionals on the ward.

CONCLUSIONS

Using the Pedatim tablet to enhance mobilisation following abdominal cancer surgery was deemed feasible, but a randomised controlled trial is needed to determine the tool's effectiveness. The study process was determined to be feasible with revisions of the eligibility criteria needed before a future trial. Involving healthcare professionals and providing available technical support are important for future implementation.

Patients' Experiences of a National Patient Portal and Its Usability: Cross-Sectional Survey Study

BACKGROUND

Patient portals not only provide patients with access to electronic health records (EHRs) and other digital health services, such as prescription renewals, but they can also improve patients' self-management, engagement with health care professionals (HCPs), and care processes. However, these benefits depend on patients' willingness to use patient portals and, ultimately, their experiences with the usefulness and ease of use of the portals.

OBJECTIVE

This study aimed to investigate the perceived usability of a national patient portal and the relationship of patients' very positive and very negative experiences with perceived usability. The study was aimed to be the first step in developing an approach for benchmarking the usability of patient portals in different countries.

METHODS

Data were collected through a web-based survey of the My Kanta patient portal's logged-in patient users in Finland from January 24, 2022, to February 14, 2022. Respondents were asked to rate the usability of the patient portal, and the ratings were used to calculate approximations of the System Usability Scale (SUS) score. Open-ended questions asked the patients about their positive and negative experiences with the patient portal. The statistical analysis included multivariate regression, and the experience narratives were analyzed using inductive content analysis.

RESULTS

Of the 1,262,708 logged-in patient users, 4719 responded to the survey, giving a response rate of 0.37%. The patient portal's usability was rated as good, with a mean SUS score of 74.3 (SD 14.0). Reporting a very positive experience with the portal was positively associated with perceived usability (β=.51; P<.001), whereas reporting a very negative experience was negatively associated with perceived usability (β=-1.28; P<.001). These variables explained 23% of the variation in perceived usability. The information provided and a lack of information were the most common positive and negative experiences. Furthermore, specific functionalities, such as prescription renewal and the ease of using the patient portal, were often mentioned as very positive experiences. The patients also mentioned negative emotions, such as anger and frustration, as part of their very negative experiences.

CONCLUSIONS

The study offers empirical evidence about the significant role of individual experiences when patients are evaluating the usability of patient portals. The results suggest that positive and negative experiences provide relevant information that can be used for improving the patient portal's usability. Usability should be improved so that patients receive information efficiently, easily, and quickly. Respondents would also appreciate interactive features in the patient portal.

Usability Evaluation of the Preoperative ISBAR (Identification, Situation, Background, Assessment, and Recommendation) Desktop Virtual Reality Application: Qualitative Observational Study

BACKGROUND

Systematic communication, such as the ISBAR (identification, situation, background, assessment, recommendation) approach, comprises a generic, transferable nontechnical skill. It can be used during the handover of patients set to undergo surgery and can be practiced in various ways, including virtual reality (VR). VR increasingly has been implemented and valued in nursing education as a positive contribution to teach students about pre- and postoperative nursing. A new nonimmersive 3D learning activity called the Preoperative ISBAR Desktop VR Application has been developed for undergraduate nursing students to learn preoperative handover using the ISBAR approach. However, the usability of this learning activity has not been studied.

OBJECTIVE

This study aimed to investigate how second-year undergraduate nursing students evaluated the usability of the Preoperative ISBAR Desktop VR Application.

METHODS

This was a qualitative study with observation and interviews. The inclusion criteria were undergraduate second-year nursing students of varying ages, gender, and anticipated technological competence. The System Usability Scale (SUS) questionnaire was used to get a score on overall usability.

RESULTS

A total of 9 second-year nursing students aged 22-29 years participated in the study. The average score on the SUS was 83 (range 0-100), which equals a "B" on the graded scale and is excellent for an adjective-grade rating. The students expressed increased motivation to learn while working in self-instructed desktop VR. Still, a few technical difficulties occurred, and some students reported that they experienced some problems comprehending the instructions provided in the application. Long written instructions and a lack of self-pacing built into the application were considered limitations.

CONCLUSIONS

The nursing students found the application to be usable overall, giving it an excellent usability score and noting that the application provided opportunities for active participation, which was motivational and facilitated their perceived learning outcomes. The next version of the application, to be used in a randomized controlled trial, will be upgraded to address technological and comprehension issues.

Electronic Source Data Transcription for Electronic Case Report Forms in China: Validation of the Electronic Source Record Tool in a Real-world Ophthalmology Study

BACKGROUND

As researchers are increasingly interested in real-world studies (RWSs), improving data collection efficiency and data quality has become an important challenge. An electronic source (eSource) generally includes direct capture, collection, and storage of electronic data to simplify clinical research. It can improve data quality and patient safety and reduce clinical trial costs. Although there are already large projects on eSource technology, there is a lack of experience in using eSource technology to implement RWSs. Our team designed and developed an eSource record (ESR) system in China. In a preliminary prospective study, we selected a cosmetic medical device project to evaluate ESR software's effect on data collection and transcription. As the previous case verification was simple, we plan to choose more complicated ophthalmology projects to further evaluate the ESR.

OBJECTIVE

We aimed to evaluate the data transcription efficiency and quality of ESR software in retrospective studies to verify the feasibility of using eSource as an alternative to traditional manual transcription of data in RWS projects.

METHODS

The approved ophthalmic femtosecond laser project was used for ESR case validation. This study compared the efficiency and quality of data transcription between the eSource method using ESR software and the traditional clinical research model of manually transcribing the data. Usability refers to the quality of a user's experience when interacting with products or systems including websites, software, devices, or applications. To evaluate the system availability of ESR, we used the System Usability Scale (SUS). The questionnaire consisted of the following 2 parts: participant information and SUS evaluation of the electronic medical record (EMR), electronic data capture (EDC), and ESR systems. By accessing log data from the EDC system previously used by the research project, all the time spent from the beginning to the end of the study could be counted.

RESULTS

In terms of transcription time cost per field, the eSource method can reduce the time cost by 81.8% (11.2/13.7). Compared with traditional manual data transcription, the eSource method has higher data transcription quality (correct entry rate of 2356/2400, 98.17% vs 47,991/51,424, 93.32%). A total of 15 questionnaires were received with a response rate of 100%. In terms of usability, the average overall SUS scores of the EMR, EDC, and ESR systems were 50.3 (SD 21.9), 51.5 (SD 14.2), and 63.0 (SD 11.3; contract research organization experts: 69.5, SD 11.5; clinicians: 59.8, SD 10.2), respectively. The Cronbach α for the SUS items of the EMR, EDC, and ESR systems were 0.591 (95% CI -0.012 to 0.903), 0.588 (95% CI -0.288 to 0.951), and 0.785 (95% CI 0.576-0.916), respectively.

CONCLUSIONS

In real-world ophthalmology studies, the eSource approach based on the ESR system can replace the traditional clinical research model that relies on the manual transcription of data.

Usability Testing and Technology Acceptance of an mHealth App at the Point of Care During Simulated Pediatric In- and Out-of-Hospital Cardiopulmonary Resuscitations: Study Nested Within 2 Multicenter Randomized Controlled Trials

BACKGROUND

Mobile apps are increasingly being used in various domains of medicine. Few are evidence-based, and their benefits can only be achieved if end users intend to adopt and use them. To date, only a small fraction of mobile apps have published data on their field usability and end user acceptance results, especially in emergency medicine.

OBJECTIVE

This study aims to determine the usability and acceptance of an evidence-based mobile app while safely preparing emergency drugs at the point of care during pediatric in- and out-of-hospital cardiopulmonary resuscitations by frontline caregivers.

METHODS

In 2 multicenter randomized controlled parent trials conducted at 6 pediatric emergency departments from March 1 to December 31, 2017, and 14 emergency medical services from September 3, 2019, to January 21, 2020, the usability and technology acceptance of the PedAMINES (Pediatric Accurate Medication in Emergency Situations) app were evaluated among skilled pediatric emergency nurses and advanced paramedics when preparing continuous infusions of vasoactive drugs and direct intravenous emergency drugs at pediatric dosages during standardized, simulation-based, pediatric in- and out-of-hospital cardiac arrest scenarios, respectively. Usability was measured using the 10-item System Usability Scale. A 26-item technology acceptance self-administered survey (5-point Likert-type scales), adapted from the Unified Theory of Acceptance and Use of Technology model, was used to measure app acceptance and intention to use.

RESULTS

All 100% (128/128) of nurses (crossover trial) and 49.3% (74/150) of paramedics (parallel trial) were assigned to the mobile app. Mean total scores on the System Usability Scale were excellent and reached 89.5 (SD 8.8; 95% CI 88.0-91.1) for nurses and 89.7 (SD 8.7; 95% CI 87.7-91.7) for paramedics. Acceptance of the technology was very good and rated on average >4.5/5 for 5 of the 8 independent constructs evaluated. Only the image construct scored between 3.2 and 3.5 by both participant populations.

CONCLUSIONS

The results provide evidence that dedicated mobile apps can be easy to use and highly accepted at the point of care during in- and out-of-hospital cardiopulmonary resuscitations by frontline emergency caregivers. These findings can contribute to the implementation and valorization of studies aimed at evaluating the usability and acceptance of mobile apps in the field by caregivers, even in critical situations.

TRIAL REGISTRATION

ClinicalTrials.gov NCT03021122; https://clinicaltrials.gov/ct2/show/NCT03021122. ClinicalTrials.gov NCT03921346; https://clinicaltrials.gov/ct2/show/NCT03921346.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR2-10.1186/s13063-019-3726-4.

Designing and Usability testing of a New Prototype Active Footrest for Knee Extension Exercise Among Office Workers

OBJECTIVES

The active factor along with light exercise can reduce static muscle work and increase muscle flexibility and endurance. Accordingly, this study aimed to design and prototype a new active footrest, implemented in conventional sitting workstations and to test its usability among office workers.

METHODS

The steps taken to design and prototype the active footrest were a) selection of an exercise appropriate for goals of ergonomic interventions, b) idea development, c) selection of the best conceptual design, d) design with SOLIDWORKS version 2016 software, and e) fabrication of a prototype. Afterward, the usability of the active footrest prototype was assessed among 20 office workers (10 females) using field data by the System Usability Scale.

RESULTS

Upon the completion of the design steps, the active footrest prototype was prototyped by taking into account design criteria (e.g. functionality, inclusive design, easy application, and reduction of additional body movement), anthropometric data and mechanical properties. The participants rated the usability of the active workstation as 89 ± 8.21 (out of 100) and indicating good usability results.

CONCLUSION

In this study, an active footrest was designed and prototyped to perform knee extension exercise with the capability of being implemented in conventional sitting workstations.

Perceived Electronic Health Record Usability as a Predictor of Task Load and Burnout Among US Physicians: Mediation Analysis

BACKGROUND

Electronic health record (EHR) usability and physician task load both contribute to physician professional burnout. The association between perceived EHR usability and workload has not previously been studied at a national level. Better understanding these interactions could give further information as to the drivers of extraneous task load.

OBJECTIVE

This study aimed to determine the relationship between physician-perceived EHR usability and workload by specialty and evaluate for associations with professional burnout.

METHODS

A secondary analysis of a cross-sectional survey of US physicians from all specialties was conducted from October 2017 to March 2018. Among the 1250 physicians invited to respond to the subsurvey analyzed here, 848 (67.8%) completed it. EHR usability was assessed with the System Usability Scale (SUS; range: 0-100). Provider task load (PTL) was assessed using the mental demand, physical demand, temporal demand, and effort required subscales of the National Aeronautics and Space Administration Task Load Index (range: 0-400). Burnout was measured using the Maslach Burnout Inventory.

RESULTS

The mean scores were 46.1 (SD 22.1) for SUS and 262.5 (SD 71.7) for PTL. On multivariable analysis adjusting for age, gender, relationship status, medical specialty, practice setting, hours worked per week, and number of nights on call per week, physician-rated EHR usability was associated with PTL, with each 1-point increase in SUS score (indicating more favorable) associated with a 0.57-point decrease in PTL score (P<.001). On mediation analysis, higher SUS score was associated with lower PTL score, which was associated with lower odds of burnout.

CONCLUSIONS

A strong association was observed between EHR usability and workload among US physicians, with more favorable usability associated with less workload. Both outcomes were associated with the odds of burnout, with task load acting as a mediator between EHR usability and burnout. Improving EHR usability while decreasing task load has the potential to allow practicing physicians more working memory for medical decision making and patient communication.

Spanish Version of the System Usability Scale for the Assessment of Electronic Tools: Development and Validation

Background

The System Usability Scale (SUS) is a common metric used to assess the usability of a system, and it was initially developed in English. The implementation of electronic systems for clinical counseling (eHealth and mobile health) is increasing worldwide. Therefore, tools are needed to evaluate these applications in the languages and regional contexts in which the electronic tools are developed.

Objective

This study aims to translate, culturally adapt, and validate the original English version of the SUS into a Spanish version.

Methods

The translation process included forward and backward translation. Forward translations were made by 2 native Spanish speakers who spoke English as their second language, and a backward translation was made by a native English speaker. The Spanish SUS questionnaire was validated by 10 experts in mobile app development. The face validity of the questionnaire was tested with 10 mobile phone users, and the reliability testing was conducted among 88 electronic application users.

Results

The content validity index of the new Spanish SUS was good, as indicated by a rating of 0.92 for the relevance of the items. The questionnaire was easy to understand, based on a face validity index of 0.94. The Cronbach α was .812 (95% CI 0.748-0.866; P<.001).

Conclusions

The new Spanish SUS questionnaire is a valid and reliable tool to assess the usability of electronic tools among Spanish-speaking users.

Usability evaluation of an eHealth intervention for family carers of individuals affected by psychosis: A mixed-method study

Background

Existing research suggests that eHealth interventions targeting family carers of individuals with long-term illness offer a promising approach to care delivery. In particular, digital psychoeducational interventions with interactive psychosocial support are well-received with high rates of satisfaction and acceptability. However, development of such interventions for psychosis carers is lacking. We developed a multi-component eHealth intervention specifically for carers of individuals affected by psychosis, called COPe-support (Carers fOr People with Psychosis e-support).

Objective

Using mixed methods to evaluate usability, system heuristics and perceived acceptability, we conducted a usability study to establish the suitability of the intervention prototype for the target user group.

Methods

Twenty-three carers were recruited to the study and participated in a think-aloud test or a remote online trial of the intervention. Qualitative feedback, post-use System Usability Scale (SUS) scores, and real-world usage data collected from the tests were analysed. These were also supplemented with heuristic evaluation data provided by an independent eLearning technology expert.

Results

Participants evaluated the intervention content as useful and helpful, and indicated that the system had satisfactory usability with a mean SUS score of 73%, above the usability quality benchmark threshold. Study results identified some minor usability issues, which were corroborated with the eLearning expert’s heuristic evaluation findings. We used these results to refine the COPe-support intervention.

Conclusions

The usability study with end-users and service providers identified real-life usage and usability issues. The study results helped us refine COPe-support and its delivery strategy before its launch as part of a large-scale clinical trial.

Usability Study of Mainstream Wearable Fitness Devices: Feature Analysis and System Usability Scale Evaluation

Background

Wearable devices have the potential to promote a healthy lifestyle because of their real-time data monitoring capabilities. However, device usability is a critical factor that determines whether they will be adopted on a large scale. Usability studies on wearable devices are still scarce.

Objective

This study aims to compare the functions and attributes of seven mainstream wearable devices and to evaluate their usability.

Methods

The wearable devices selected were the Apple Watch, Samsung Gear S, Fitbit Surge, Jawbone Up3, Mi Band, Huawei Honor B2, and Misfit Shine. A mixed method of feature comparison and a System Usability Scale (SUS) evaluation based on 388 participants was applied; the higher the SUS score, the better the usability of the product.

Results

For features, all devices had step counting, an activity timer, and distance recording functions. The Samsung Gear S had a unique sports track recording feature and the Huawei Honor B2 had a unique wireless earphone. The Apple Watch, Samsung Gear S, Jawbone Up3, and Fitbit Surge could measure heart rate. All the devices were able to monitor sleep, except the Apple Watch. For product characteristics, including attributes such as weight, battery life, price, and 22 functions such as step counting, activity time, activity type identification, sleep monitoring, and expandable new features, we found a very weak negative correlation between the SUS scores and price (r=−.10, P=.03) and devices that support expandable new features (r=−.11, P=.02), and a very weak positive correlation between the SUS scores and devices that support the activity type identification function (r=.11, P=.02). The Huawei Honor B2 received the highest score of mean 67.6 (SD 16.1); the lowest Apple Watch score was only 61.4 (SD 14.7). No significant difference was observed among brands. The SUS score had a moderate positive correlation with the user’s experience (length of time the device was used) (r=.32, P<.001); participants in the medical and health care industries gave a significantly higher score (mean 61.1, SD 17.9 vs mean 68.7, SD 14.5, P=.03).

Conclusions

The functions of wearable devices tend to be homogeneous and usability is similar across various brands. Overall, Mi Band had the lowest price and the lightest weight. Misfit Shine had the longest battery life and most functions, and participants in the medical and health care industries had the best evaluation of wearable devices. The perceived usability of mainstream wearable devices is unsatisfactory and customer loyalty is not high. A consumer’s SUS rating for a wearable device is related to their personal situation instead of the device brand. Device manufacturers should put more effort into developing innovative functions and improving the usability of their products by integrating more cognitive behavior change techniques.

Translation, Cross-Cultural Adaptation, and Validation of the Malay Version of the System Usability Scale Questionnaire for the Assessment of Mobile Apps

Background

A mobile app is a programmed system designed to be used by a target user on a mobile device. The usability of such a system refers not only to the extent to which product can be used to achieve the task that it was designed for, but also its effectiveness and efficiency, as well as user satisfaction. The System Usability Scale is one of the most commonly used questionnaires used to assess the usability of a system. The original 10-item version of System Usability Scale was developed in English and thus needs to be adapted into local languages to assess the usability of a mobile apps developed in other languages.

Objective

The aim of this study is to translate and validate (with cross-cultural adaptation) the English System Usability Scale questionnaire into Malay, the main language spoken in Malaysia. The development of a translated version will allow the usability of mobile apps to be assessed in Malay.

Methods

Forward and backward translation of the questionnaire was conducted by groups of Malay native speakers who spoke English as their second language. The final version was obtained after reconciliation and cross-cultural adaptation. The content of the Malay System Usability Scale questionnaire for mobile apps was validated by 10 experts in mobile app development. The efficacy of the questionnaire was further probed by testing the face validity on 10 mobile phone users, followed by reliability testing involving 54 mobile phone users.

Results

The content validity index was determined to be 0.91, indicating good relevancy of the 10 items used to assess the usability of a mobile app. Calculation of the face validity index resulted in a value of 0.94, therefore indicating that the questionnaire was easily understood by the users. Reliability testing showed a Cronbach alpha value of .85 (95% CI 0.79-0.91) indicating that the translated System Usability Scale questionnaire is a reliable tool for the assessment of usability of a mobile app.

Conclusions

The Malay System Usability Scale questionnaire is a valid and reliable tool to assess the usability of mobile app in Malaysia.

Mobile phone-based interactive voice response as a tool for improving access to healthcare in remote areas in Ghana - an evaluation of user experiences

OBJECTIVES

To investigate and determine the factors that enhanced or constituted barriers to the acceptance of an mHealth system which was piloted in Asante-Akim North District of Ghana to support healthcare of children.

METHODS

Four semi-structured focus group discussions were conducted with a total of 37 mothers. Participants were selected from a study population of mothers who subscribed to a pilot mHealth system which used an interactive voice response (IVR) for its operations. Data were evaluated using qualitative content analysis methods. In addition, a short quantitative questionnaire assessed system's usability (SUS).

RESULTS

Results revealed 10 categories of factors that facilitated user acceptance of the IVR system including quality-of-care experience, health education and empowerment of women. The eight categories of factors identified as barriers to user acceptance included the lack of human interaction, lack of update and training on the electronic advices provided and lack of social integration of the system into the community. The usability (SUS median: 79.3; range: 65-97.5) of the system was rated acceptable.

CONCLUSIONS

The principles of the tested mHealth system could be of interest during infectious disease outbreaks, such as Ebola or Lassa fever, when there might be a special need for disease-specific health information within populations.