Heuristic evaluation and usability testing of a computerized patient-reported outcomes survey for headache sufferers

Methodological Quality of Manuscripts Reporting on the Usability of Mobile Applications for Pain Assessment and Management: A Systematic Review

BACKGROUND

There has been increasing use of mobile mHealth applications, including pain assessment and pain self-management apps. The usability of mHealth applications has vital importance as it affects the quality of apps. Thus, usability assessment with methodological rigor is essential to minimize errors and undesirable consequences, as well as to increase user acceptance.

OBJECTIVE

this study aimed to synthesize and evaluate existing studies on the assessment of the usability of pain-related apps using a newly developed scale.

METHODS

an electronic search was conducted in several databases, combining relevant keywords. Then titles and abstracts were screened against inclusion and exclusion criteria. The eligible studies were retrieved and independently screened for inclusion by two authors. Disagreements were resolved by discussion until consensus was reached.

RESULTS

a total of 31 articles were eligible for inclusion. Quality assessment revealed that most manuscripts did not assess usability using valid instruments or triangulation of methods of usability assessment. Most manuscripts also failed to assess the three domains of usability (effectiveness, efficiency and satisfaction).

CONCLUSIONS

future studies should consider existing guidelines on usability assessment design, development and assessment of pain-related apps.

Using Heuristic Evaluation to Enhance the Visual Display of a Provider Dashboard for Patient-Reported Outcomes

Introduction

With the rising use of patient-reported outcomes (PRO) in clinical practice, there is an increasing need to understand the data visualization needs of clinical teams to support their effective use of PRO data for both individual patient decision making and broader population health applications. A human-centered design (HCD) approach can optimize the visual design of an interactive PRO system.

Including Heuristic Evaluation in the HCD Toolbox

Recent literature regarding the use of HCD to design and develop PRO visualizations demonstrates the benefits of iterative methods that engage representative users who are likely to use the system in the future. However, the literature has not explored the additive value of other HCD methods such as heuristic evaluation, which involves expert examination of the interface with respect to recognized usability principles, the heuristics.

Insights from Using Heuristic Evaluation

Our experience in using heuristic evaluation to enhance the design of a PRO dashboard led to several recommendations to improve the display, accessibility, and interpretability of the dashboard's data. Heuristic evaluation can serve as a complement to HCD methods that directly engage users and thereby enhance usability.

A Modified User-Oriented Heuristic Evaluation of a Mobile Health System for Diabetes Self-management Support

Mobile health platforms offer significant opportunities for improving diabetic self-care, but only if adequate usability exists. Expert evaluations such as heuristic evaluation can provide distinct usability information about systems. The purpose of this study was to complete a usability evaluation of a mobile health system for diabetes patients using a modified heuristic evaluation technique of (1) dual-domain experts (healthcare professionals, usability experts), (2) validated scenarios and user tasks related to patients' self-care, and (3) in-depth severity factor ratings. Experts identified 129 usability problems with 274 heuristic violations for the system. The categories Consistency and Standards dominated at 24.1% (n = 66), followed by Match Between System and Real World at 22.3% (n = 61). Average severity ratings across system views were 2.8 (of 4), with 9.3% (n = 12) rated as catastrophic and 53.5% (n = 69) as major. The large volume of violations with severe ratings indicated clear priorities for redesign. The modified heuristic approach allowed evaluators to identify unique and important issues, including ones related to self-management and patient safety. This article provides a template for one type of expert evaluation adding to the informaticists' toolbox when needing to conduct a fast, resource-efficient and user-oriented heuristic evaluation.

Usability in telemedicine systems-A literature survey

INTRODUCTION

The rapid development of sensors and communication technologies enable the growth of new innovative services in healthcare, such as Telemedicine. An essential ingredient in the development of a telemedicine system and its final acceptance by end users are usability studies. The principles of usability engineering, evaluations and telemedicine are well established, and it may contribute to the adoption and eventually deployment of such systems and services. An in-depth usability analysis, including performance and attitude measures, requires knowledge about available usability techniques, and is depending on the amount of resources. Therefore it is worth investigating how usability methods are applied in developing telemedicine systems. Our hypothesis is: with increasing research and development of telemedicine systems, we expect that various usability methods are more equally employed for different end-user groups and applications.

METHOD

A literature survey was conducted to find telemedicine systems that have been evaluated for usability or ease of use. The elements of the PICO framework were used as a basis for the selection criteria in the literature search. The search was not limited by year. Two independent reviewers screened all search results first by title, and then by abstract for inclusion. Articles were included up to May 2015.

RESULTS

In total, 127 publications were included in this survey. The number of publications on telemedicine systems significantly increased after 2008. Older adults and end-users with cardiovascular conditions were among largest target end-user groups. Remote monitoring systems were found the most, in 90 publications. Questionnaires are the most common means for evaluating telemedicine systems, and were found in 88 publications. Questionnaires are used frequently in studies focusing on cardiovascular diseases, Parkinson's disease and older adult conditions. Interviews are found the most in publications related to stroke. In total 71% of the publications were trial-orientated and the remaining process orientated. An increase in telemedicine research, development and applications is found worldwide, with the majority of publications conducted in America.

DISCUSSION AND CONCLUSION

Monitoring patients in their homes can lead to better healthcare at lower costs which implies an increased demand of new healthcare strategies like telemedicine. We expected that with the increase in telemedicine research and development, a greater range of usability methods would also be employed in the included publications. This is not the case. Researchers employed questionnaires as a preferred usability method for each type of telemedicine system and most end-users. However, in process-orientated studies a greater range of usability evaluations were applied, with fewer differences found in the amount of publications for each evaluation method. Questionnaires enable researchers to evaluate a system quickly on end users, as it requires less expertise on the evaluation method compared to the other methods. They are easily distributed and are customizable. The use of questionnaires is therefore an evaluation method of choice for a variety of telemedicine systems and end-users.

User Interface Problems of a Nationwide Inpatient Information System: A Heuristic Evaluation

INTRODUCTION

While studies have shown that usability evaluation could uncover many design problems of health information systems, the usability of health information systems in developing countries using their native language is poorly studied. The objective of this study was to evaluate the usability of a nationwide inpatient information system used in many academic hospitals in Iran.

MATERIAL AND METHODS

Three trained usability evaluators independently evaluated the system using Nielsen's 10 usability heuristics. The evaluators combined identified problems in a single list and independently rated the severity of the problems. We statistically compared the number and severity of problems identified by HIS experienced and non-experienced evaluators.

RESULTS

A total of 158 usability problems were identified. After removing duplications 99 unique problems were left. The highest mismatch with usability principles was related to "Consistency and standards" heuristic (25%) and the lowest related to "Flexibility and efficiency of use" (4%). The average severity of problems ranged from 2.4 (Major problem) to 3.3 (Catastrophe problem). The experienced evaluator with HIS identified significantly more problems and gave higher severities to problems (p<0.02).

DISCUSSION

Heuristic Evaluation identified a high number of usability problems in a widely used inpatient information system in many academic hospitals. These problems, if remain unsolved, may waste users' and patients' time, increase errors and finally threaten patient's safety. Many of them can be fixed with simple redesign solutions such as using clear labels and better layouts. This study suggests conducting further studies to confirm the findings concerning effect of evaluator experience on the results of Heuristic Evaluation.

Beyond lip service and box ticking: how effective patient engagement is integral to the development and delivery of patient-reported outcomes

BACKGROUND

In the UK, demonstration of patient and public involvement (PPI) is now a funding requirement. Despite advice being available to researchers regarding PPI, levels of engagement are variable. Patient involvement has been at the core of the Leeds Psychosocial Oncology and Clinical Practice Research Group since 2007 when a local Research Advisory Group (RAG) was established. In addition, we work with experienced patient advocates from national groups.

METHODS

The RAG is led by designated researchers who manage and communicate with members. The RAG is invited to twice yearly meetings with the full research team when study findings are disseminated and advice sought. The meetings are also an opportunity to socialise and thank members. Effective partnerships and engagement require good communication, building relationships over time and tailoring involvement to individuals' skills and experience.

RESULTS

Patients have been involved in design, planning new projects and assisting with grant proposals; development, pilot testing of interview strategies and question generation, project steering groups and management teams, development of self-management advice for online patient portals; implementation, extensive beta testing of new questionnaire builder software to enable collection of online patient-reported outcomes (PRO) and study websites, cognitive interviews to develop PRO items; dissemination, co-authorship of papers and presentations, attendance/representation of the group at conferences.

CONCLUSION

The involvement of patient advocates is integral to ensuring PRO development remains patient-centred. Having a co-operative, well-established local PPI group and nationally active patient collaborators has had a rewarding and significant impact on our research programmes.

Acceptance of New Technology: A Usability Test of a Computerized Adaptive Test for Fatigue in Rheumatoid Arthritis

Background

Little is known about the acceptance and usability of computerized adaptive tests (CATs) among patients with rheumatoid arthritis (RA). The main difference between completing a CAT and a traditional questionnaire concerns item presentation. CATs only provide one item at a time on the screen, and skipping forward or backward to review and change already given answers is often not possible.

Objective

The objective of this study was to examine how patients with RA experience a Web-based CAT for fatigue.

Methods

In individual sessions, participants filled in the CAT while thinking aloud, and were subsequently interviewed about their experience with the new instrument. The technology acceptance model (TAM) was used to structure the results.

Results

The participants were 15 patients with RA. They perceived the CAT as clear, brief, and easy to use. They were positive about answering one question per screen, the changing response options, layout, progress bar, and item number. There were 40% (6/15) of the participants that also mentioned that they experienced the completion of the CAT as useful and pleasant, and liked the adaptive test mechanism. However, some participants noted that not all items were applicable to everybody, and that the wordings of questions within the severity dimension were often similar.

Conclusions

Participants perceived the “CAT Fatigue RA” as easy to use, and also its usefulness was expressed. A 2.0 version has been improved according to the participants’ comments, and is currently being used in a validation study before it will be implemented in daily clinical practice. Our results give a first indication that CAT methodology may outperform traditional questionnaires not merely on measurement precision, but also on usability and acceptance valuation.

A new computerized adaptive test advancing the measurement of health-related quality of life (HRQoL) in children: the Kids-CAT

PURPOSE

Assessing health-related quality of life (HRQoL) via Computerized Adaptive Tests (CAT) provides greater measurement precision coupled with a lower test burden compared to conventional tests. Currently, there are no European pediatric HRQoL CATs available. This manuscript aims at describing the development of a HRQoL CAT for children and adolescents: the Kids-CAT, which was developed based on the established KIDSCREEN-27 HRQoL domain structure.

METHODS

The Kids-CAT was developed combining classical test theory and item response theory methods and using large archival data of European KIDSCREEN norm studies (n = 10,577-19,580). Methods were applied in line with the US PROMIS project. Item bank development included the investigation of unidimensionality, local independence, exploration of Differential Item Functioning (DIF), evaluation of Item Response Curves (IRCs), estimation and norming of item parameters as well as first CAT simulations.

RESULTS

The Kids-CAT was successfully built covering five item banks (with 26-46 items each) to measure physical well-being, psychological well-being, parent relations, social support and peers, and school well-being. The Kids-CAT item banks proved excellent psychometric properties: high content validity, unidimensionality, local independence, low DIF, and model conform IRCs. In CAT simulations, seven items were needed to achieve a measurement precision between .8 and .9 (reliability). It has a child-friendly design, is easy accessible online and gives immediate feedback reports of scores.

CONCLUSIONS

The Kids-CAT has the potential to advance pediatric HRQoL measurement by making it less burdensome and enhancing the patient-doctor communication.

An interactive, bilingual touch screen program to promote breastfeeding among Hispanic rural women: usability study

Background

Computer technology can be effectively used to educate patients and improve knowledge and attitudes, leading to healthier behavior. Among rural women, breastfeeding outcomes seem to be worst compared to women living in urban areas. The implementation of a bilingual computer mediated health education program to disseminate information and improve outcomes among users with low literacy levels has proven to be successful.

Objective

The objective of this pilot study was to examine the usability of an interactive, bilingual touch screen computer-based educational program to promote breastfeeding practices among Hispanic women living in rural settings.

Methods

A convenience sample of 10 Hispanic rural women at the Regional West Medical Center (RWMC), Scottsbluff was enrolled during May 2013. Information about this cross-sectional study was made available through the flyers at the RWMC. A brief introduction of the prototype was given and study subjects were then asked to complete a predefined set of tasks by interacting with the prototype. Users were assigned 6 tasks and information was gathered about the time taken to complete the tasks, number of attempts, and if assistance was needed. Notes and test sessions were audiotaped. Usability assessment was performed using the System Usability Scale (SUS).

Results

The mean age of the study participants was 28 years (SD 3.6), the majority of them had 12 or more years of education (90%, 9/10), and 60% (6/10) had breastfed less than 6 months. There were 90% (9/10) of the study participants that had no prior history of taking prenatal classes and 80% (8/10) that did not intend to take any prenatal classes in the future. The average SUS scores were 90 and SD was 10.5. There were three participants that had average SUS scores of 100, followed by scores of 97.5 (1/10), 95 (1/10), 87.5 (1/10), 85 (2/10), 82.5 (1/10), and one participant had a score of 67.5 (1/10). No assistance was needed to complete any of the tasks.

Conclusions

The study participants were able to navigate through the multimedia program with ease and obtain relevant breastfeeding related health information. The interactive, touch screen computer-based breastfeeding program had high acceptance among 10 Hispanic women living in rural settings.

Cognitive testing and readability of an item bank for measuring the impact of headache on health-related quality of life

BACKGROUND

Patient-reported outcomes (PRO) measures should be valid and accessible to a wide audience.

OBJECTIVE

Cognitive item testing and readability studies were conducted to evaluate how adult headache sufferers (N = 9) understood and responded to the Headache Impact Test (HIT™) item bank, a PRO measure for headache that serves as the source of item content for the HIT-6™ (a widely used six-item short-form measure of headache impact with more than 30 language translations), and the Dynamic Health Assessment Headache Impact Test (DYNHA® HIT™) [a computerized adaptive test (CAT) of headache impact].

METHODS

During cognitive interviews, participants were asked to 'think aloud' as they read survey instructions, completed items, and formulated responses. Data analyses evaluated item comprehension, memory recall of relevant information, and decision and response processes; compared various item attributes; and tested shortened item versions.

RESULTS

Survey readability was at the seventh-grade level. Respondents understood most revised items as intended, and found shorter items comparable to longer items with some exceptions. When recall period was included in instructions but not within the items themselves, respondents often expanded the recall period to answer the item. Some response scales (e.g. "Never" to "Always") were more readily understood than others (e.g. "Definitely true" to "Definitely false").

CONCLUSION

Qualitative research can improve the validity and accessibility of PRO measures that are used to monitor health conditions and aid patient-provider communication.