Usability testing of ANSWER: a web-based methotrexate decision aid for patients with rheumatoid arthritis

Development and evaluation of shared decision-making tools in rheumatology: A scoping review

INTRODUCTION

Shared decision-making (SDM) tools are facilitators of decision-making through a collaborative process between patients/caregivers and clinicians. These tools help clinicians understand patient's perspectives and help patients in making informed decisions based on their preferences. Despite their usefulness for both patients and clinicians, SDM tools are not widely implemented in everyday practice. One barrier is the lack of clarity on the development and evaluation processes of these tools. Such processes have not been previously described in the field of rheumatology.

OBJECTIVE

To describe the development and evaluation processes of shared decision-making (SDM) tools used in rheumatology.

METHODS

Bibliographic databases (e.g., EMBASE and CINAHL) were searched for relevant articles. Guidelines for the PRISMA extension for scoping reviews were followed. Studies included were: addressing SDM among adults in rheumatology, focusing on development and/or evaluation of SDM tool, full texts, empirical research, and in the English language.

RESULTS

Of the 2030 records screened, forty-six reports addressing 36 SDM tools were included. Development basis and evaluation measures varied across the studies. The most commonly reported development basis was the International Patient Decision Aids Standards (IPDAS) criteria (19/36, 53 %). Other developmental foundations reported were: The Ottawa Decision Support Framework (ODSF) (6/36, 16 %), Informed Medical Decision Foundation elements (3/36, 8 %), edutainment principles (2/36, 5.5 %), and others (e.g. DISCERN and MARKOV Model) (9/31,29 %). The most commonly used evaluation measures were the Decisional Conflict Scale (18/46, 39 %), acceptability and knowledge (7/46, 15 %), and the preparation for decision-making scale (5/46,11 %).

CONCLUSION

For better quality and wider implementation of such tools, there is a need for detailed, transparent, systematic, and consistent reporting of development methods and evaluation measures. Using established checklists for reporting development and evaluation is encouraged.

Design, Development, and Evaluation of an mHealth App for Reporting of Side Effects During Cytostatic Treatment: Usability Test and Interview Study

BACKGROUND

Using mobile health (mHealth) interventions such as smartphone apps to deliver health services is an opportunity to engage patients more actively in their own treatment. Usability tests allow for the evaluation of a service by testing it out on the relevant users before implementation in clinical practice.

OBJECTIVE

The objective of this study was to design, develop, and evaluate the user interface of an app that would aid patients with cancer in reporting a more comprehensive summary of their side effects.

METHODS

The usability test was conducted by exposing patients with cancer to a prototype of an mHealth app that allowed for reporting of side effects from a chemotherapy regimen. After solving a set of 13 tasks, the test participants completed a system usability scale questionnaire and were interviewed using a semistructured interview guide. The interviews were later transcribed and analyzed.

RESULTS

The 10 test participants had a mean age of 56.5 (SD 7.11) years. The mean total task completion time for the task-solving session was 240.15 (SD 166.78) seconds. The calculated system usability scale score was 92.5. Most participants solved most of the tasks without any major issues. A minority reported having difficulties using apps on smartphones in general. One patient never achieved a meaningful interaction with our app prototype. Most of those who engaged with the app approved of features that calmed them down, made them more empowered, and put them in control. They preferred to report on side effects in a detailed and concise manner. App features that provided specific advice could provoke both fear and rational action.

CONCLUSIONS

The user tests uncovered design flaws that allowed for subsequent refining of an app that has the potential to enhance the safety of patients undergoing home-based chemotherapy. However, a refined version of the app is unlikely to be of value to all patients. Some might not be able to use apps on smartphones in general, or their ability to use apps is impaired because of their disease. This finding should have implications for health care providers' overall design of their follow-up service as the service must allow for all the patients to receive safe treatment whether they can use an mHealth app or not.

Methodologies for Evaluating the Usability of Rehabilitation Technologies Aimed at Supporting Shared Decision-Making: Scoping Review

BACKGROUND

The field of rehabilitation has seen a recent rise in technologies to support shared decision-making (SDM). Usability testing during the design process of SDM technologies is needed to optimize adoption and realize potential benefits. There is variability in how usability is defined and measured. Given the complexity of usability, a thorough examination of the methodologies used to measure usability to develop the SDM technologies used in rehabilitation care is needed.

OBJECTIVE

This scoping review aims to answer the following research questions: which methods and measures have been used to produce knowledge about the usability of rehabilitation technologies aimed at supporting SDM at the different phases of development and implementation? Which parameters of usability have been measured and reported?

METHODS

This review followed the Arksey and O'Malley framework. An electronic search was performed in the Ovid MEDLINE, Embase, CINAHL, and PsycINFO databases from January 2005 up to November 2020. In total, 2 independent reviewers screened all retrieved titles, abstracts, and full texts according to the inclusion criteria and extracted the data. The International Organization for Standardization framework was used to define the scope of usability (effectiveness, efficiency, and satisfaction). The characteristics of the studies were outlined in a descriptive summary. Findings were categorized based on usability parameters, technology interventions, and measures of usability.

RESULTS

A total of 38 articles were included. The most common SDM technologies were web-based aids (15/33, 46%). The usability of SDM technologies was assessed during development, preimplementation, or implementation, using 14 different methods. The most frequent methods were questionnaires (24/38, 63%) and semistructured interviews (16/38, 42%). Satisfaction (27/38, 71%) was the most common usability parameter mapped to types of SDM technologies and usability evaluation methods. User-centered design (9/15, 60%) was the most frequently used technology design framework.

CONCLUSIONS

The results from this scoping review highlight the importance and the complexity of usability evaluation. Although various methods and measures were shown to be used to evaluate the usability of technologies to support SDM in rehabilitation, very few evaluations used in the included studies were found to adequately span the selected usability domains. This review identified gaps in usability evaluation, as most studies (24/38, 63%) relied solely on questionnaires rather than multiple methods, and most questionnaires simply focused on the usability parameter of satisfaction. The consideration of end users (such as patients and clinicians) is of particular importance for the development of technologies to support SDM, as the process of SDM itself aims to improve patient-centered care and integrate both patient and clinician voices into their rehabilitation care.

An Intuitive Risk Communication Tool to Enhance Patient-Provider Partnership in Diabetes Consultation

INTRODUCTION

This technology report introduces an innovative risk communication tool developed to support providers in communicating diabetes-related risks more intuitively to people with type 2 diabetes mellitus (T2DM).

METHODS

The development process involved three main steps: (1) selecting the content and format of the risk message; (2) developing a digital interface; and (3) assessing the usability and usefulness of the tool with clinicians through validated questionnaires.

RESULTS

The tool calculates personalized risk information based on a validated simulation model (United Kingdom Prospective Diabetes Study Outcomes Model 2) and delivers it using more intuitive risk formats, such as "effective heart age" to convey cardiovascular risks. Clinicians reported high scores for the usability and usefulness of the tool, making its adoption in routine care promising.

CONCLUSIONS

Despite increased use of risk calculators in clinical care, this is the first time that such a tool has been developed in the diabetes area. Further studies are needed to confirm the benefits of using this tool on behavioral and health outcomes in T2DM populations.

Evaluation of a prenatal screening decision aid: A mixed methods pilot study

BACKGROUND

We developed a decision aid (DA) to help pregnant women and their partners make informed decisions about prenatal screening for trisomy. We aimed to determine its usefulness for preparing for decision-making and its acceptability among end-users.

METHODS

In this mixed-methods pilot study, we recruited participants in three prenatal care settings in Quebec City. Eligible women were over 18 and more than 16 weeks pregnant or had given birth recently. We asked them about the usefulness of the DA using an interview grid based on the Technology Acceptance Model. We performed descriptive statistics and deductive analysis.

RESULTS

Thirty-nine dyads or individuals participated in the study. Mean usefulness score was 86.2 ± 13. Most participants found the amount of information in the DA just right (79.5%), balanced (89.7%), and very useful (61.5%). They were less satisfied with the presentation and the values worksheet and suggested different values clarification methods.

CONCLUSION

Rigorous pilot tests of DAs with patients are an important stage in their development before the more formal assessments that precede scaling up the DA in clinical practice.

PRACTICE IMPLICATIONS

The next version of the DA will integrate the suggestions of end-users for better decision-making processes about prenatal screening for trisomy.

Proposal for a Framework to Enable Elicitation of Preferences for Clients in Need of Long-Term Care

PURPOSE

Collaborative deliberation comprises personal engagement, recognition of alternative actions, comparative learning, preference elicitation, and preference integration. Collaborative deliberation may be improved by assisting preference elicitation during shared decision-making. This study proposes a framework for preference elicitation to facilitate collaborative deliberation in long-term care consultations.

METHODS

First, a literature overview was conducted comprising current models for the elicitation of preferences in health and social care settings. The models were reviewed and compared. Second, qualitative research was applied to explore those issues that matter most to clients in long-term care. Data were collected from clients in long-term care, comprising 16 interviews, 3 focus groups, 79 client records, and 200 online client reports. The qualitative analysis followed a deductive approach. The results of the literature overview and qualitative research were combined.

RESULTS

Based on the literature overview, five overarching domains of preferences were described: "Health", "Daily life", "Family and friends", "Living conditions", and "Finances". The credibility of these domains was confirmed by qualitative data analysis. During interviews, clients addressed issues that matter in their lives, including a "click" with their care professional, safety, contact with loved ones, and assistance with daily structure and activities. These data were used to determine the content of the domains.

CONCLUSION

A framework for preference elicitation in long-term care is proposed. This framework could be useful for clients and professionals in preference elicitation during collaborative deliberation.

Does Animation Improve Comprehension of Risk Information in Patients with Low Health Literacy? A Randomized Trial

Introduction. Enhanced visual effects, like animation, have the potential to improve comprehension of probabilistic risk information, particularly for those with lower health literacy. We tested the effect of presentation format on comprehension of colorectal cancer (CRC) screening probabilities to identify optimal risk communication strategies. Methods. Participants from a community foodbank and a cancer prevention center were randomized to 1 of 3 CRC screening risk presentations. The presentations used identical content but varied in format: 1) video with animated pictographs, 2) video with static pictographs, and 3) audiobooklet with static pictographs. Participants completed pre- and postpresentation surveys. The primary outcome was knowledge of probability/risk information regarding CRC screening, calculated as total, verbatim, and gist scores. Results. In total, 187 participants completed the study and were included in this analysis. Median age was 58 years (interquartile range [IQR]: 14 years), most participants were women (63%), and almost half had a high school education or less (46%). Approximately one-quarter had inadequate health literacy (Short Test of Functional Health Literacy in Adults marginal/inadequate: 28%; Brief Health Literacy Screener low: 18%), and about half had low numeracy (Subjective Numeracy Scale low: 54%; Graphical Literacy Measure low: 50%). We found no significant differences in total, verbatim, or gist knowledge across presentation formats (all P > 0.05). Discussion. Use of an animated pictograph to communicate risk does not appear to augment or impede knowledge of risk information. Regardless of health literacy level, difficulty understanding pictographs presenting numerical information persists. There may be a benefit to teaching or priming individuals on how to interpret numerical information presented in pictographs before communicating risk using visual methods. Trial Registry: NCT02151032.

The relationship between user interface problems of an admission, discharge and transfer module and usability features: a usability testing method

Background

The admission, discharge and transfer (ADT) module is used in the hospital information system (HIS) for the purposes of managing appointments, patient admission, daily control of hospital beds, planning surgery procedures, keeping up-to-date on patient discharges, and registering patient transfers within or outside the hospital. The present study aimed to evaluate the usability of ADT module of a HIS through usability testing and assess the relationship between the number of user interface problems and usability features (i.e. effectiveness, efficiency, and satisfaction).

Methods

This descriptive analytical study was conducted in Shahid Beheshti hospital in Kashan, Iran, in 2017. The participating users were eight students in their last semester of a Bachelor of Health Information Technology Sciences degree. First, the users were introduced to the module functions in a two-hour session; ten days later, the users were asked to perform scenarios designed based on seven tasks and take notes of the problems encountered in performing each task after it was over. Effectiveness was measured based on the rate of completing the tasks, efficiency based on the time taken to perform each task, and satisfaction based on the users’ answers to a satisfaction questionnaire. The relationship between these three usability features and the number of problems noted was assessed using Spearman’s test in SPSS version 16.

Results

Thirteen unique usability problems were identified from the perspective of the users. Effectiveness was rated as 58.9%, efficiency as 53.3%, and mean user satisfaction as 53.4 ± 10.6. The number of problems in each task had significant relationships to the effectiveness (P = 0.009) and efficiency (P = 0.016) scores. User satisfaction also had a significant relationship with the effectiveness (P = 0.043) but not with the efficiency (P = 0.230) scores.

Conclusions

In the view of the potential users, a HIS, used in more than 200 hospitals in a developing country, has several usability problems in its ADT module and its effectiveness, efficiency, and user satisfaction were not acceptable. The number of usability problems in the HIS user interface affected the effectiveness, efficiency and user satisfaction of the system.

A User-Centered design and usability testing of a web-based medication reconciliation application integrated in an eHealth network

BACKGROUND

Medication discrepancies, which are a threat to patient safety, can be reduced by medication reconciliation (MedRec). MedRec is a complex process that can be supported by the use of information technology and patient engagement. Therefore, the SEAMPAT project aims to develop a MedRec IT platform based on two applications. The application for the professionals is called: the "MedRec app".

OBJECTIVE

In the present study, we aimed to describe the development and usability testing of the MedRec app, reporting results of a three iterations user-centered usability evaluation.

METHODS

We used a three phase iterative user-centered study spread over 16 months. At each phase, the usability evaluation included several methods (observations, questionnaires, and follow-up discussions with participants) to collect quantitative and qualitative data in order to improve the current prototype and evolve to the next prototype.

RESULTS

In total, 48 healthcare professionals (25 general practitioners and 23 hospital clinicians) participated to the MedRec app evaluation. There were 14, 32 and 5 participants for phases 1, 2 and 3 respectively. At each phase, many design modifications were done to strengthen usability. Concerning usability, participants considered the prototypes as an acceptable interface with a median System Usability Score of 73 at phase 2 and 75 at phase 3. Participants emphasized the need for improvements concerning workflow integration, usefulness and interoperability.

CONCLUSION

The MedRec app was perceived as being useful, usable and satisfying. However, further improvements are required in several usability aspects. Our study demonstrates the importance of conducting usability assessments before investing time and resources in a large study evaluating the effect of an eMedRec approach on clinical outcomes. Our findings may also increase the chances of acceptability and sustained use over time by clinicians.

An Online Hand Exercise Intervention for Adults With Rheumatoid Arthritis (mySARAH): Design, Development, and Usability Testing

BACKGROUND

The Strengthening and Stretching for Rheumatoid Arthritis of the Hand (SARAH) program is a tailored, progressive 12-week exercise program for people with hand problems due to rheumatoid arthritis. The program was shown to be clinically and cost-effective in a large clinical trial and is recommended by the UK National Institute for Health and Care Excellence (NICE) guidelines for rheumatoid arthritis in adults.

OBJECTIVE

We have developed an online version of the SARAH program (mySARAH) to make the SARAH program widely accessible to people with rheumatoid arthritis. The purposes of this study were to develop mySARAH and to evaluate and address its usability issues.

METHODS

We developed mySARAH using a three-step process and gaining feedback from patient contributors. After initial development, mySARAH was tested in two iterative usability cycles in nine participants using a simplified think-aloud protocol and self-reported questionnaires. We also evaluated if participants executed the SARAH exercises correctly after watching the exercise videos included on the website.

RESULTS

A preliminary version of mySARAH consisting of six sessions over a 12-week period and delivered via text, exercise videos, images, exercise plan form, exercise calendar, and links to additional information on rheumatoid arthritis was developed. Five participants (1 male; 4 females; median age 64 years) and four participants (four females; median age 64.5 years) took part in the first and second usability testing cycles respectively. Usability issues identified from Cycle 1 such as having a navigation tutorial video and individualised feedback on pain levels were addressed prior to Cycle 2. The need for more instructions to complete the mySARAH patient forms was identified in Cycle 2 and was rectified. Self-reports from both cycles indicated that participants found the program useful and easy to use and were confident in performing the SARAH exercises themselves. Eight of the nine participants correctly demonstrated all the exercises.

CONCLUSIONS

mySARAH is the first online hand exercise intervention for people with rheumatoid arthritis. We actively involved target users in the development and usability evaluation and ensured mySARAH met their needs and preferences.

A Web-Based Training Resource for Therapists to Deliver an Evidence-Based Exercise Program for Rheumatoid Arthritis of the Hand (iSARAH): Design, Development, and Usability Testing

Background

The Strengthening and Stretching for Rheumatoid Arthritis of the Hand (SARAH) is a tailored, progressive exercise program for people having difficulties with wrist and hand function due to rheumatoid arthritis (RA). The program was evaluated in a large-scale clinical trial and was found to improve hand function, was safe to deliver, and was cost-effective. These findings led to the SARAH program being recommended in the UK National Institute for Health and Care Excellence guidelines for the management of adults with RA. To facilitate the uptake of this evidence-based program by clinicians, we proposed a Web-based training program for SARAH (iSARAH) to educate and train physiotherapists and occupational therapists on delivering the SARAH program in their practice. The overall iSARAH implementation project was guided by the 5 phases of the analysis, design, development, implementation, and evaluation (ADDIE) system design model.

Objective

The objective of our study was to conduct the first 3 phases of the model in the development of the iSARAH project.

Methods

Following publication of the trial, the SARAH program materials were made available to therapists to download from the trial website for use in clinical practice. A total of 35 therapists who downloaded these materials completed an online survey to provide feedback on practice trends in prescribing hand exercises for people with RA, perceived barriers and facilitators to using the SARAH program in clinical practice, and their preferences for the content and Web features of iSARAH. The development and design of iSARAH were further guided by a team of multidisciplinary health professionals (n=17) who took part in a half-day development meeting. We developed the preliminary version of iSARAH and tested it among therapists (n=10) to identify and rectify usability issues and to produce the final version.

Results

The major recommendations made by therapists and the multidisciplinary team were having a simple Web design and layout, clear exercise pictures and videos, and compatibility of iSARAH on various browsers and devices. We rectified all usability issues in the preliminary version to develop the final version of iSARAH, which included 4 short modules and additional sections on self-assessment, frequently asked questions, and a resource library.

Conclusions

The use of the ADDIE design model and engagement of end users in the development and evaluation phases have rendered iSARAH a convenient, easy-to-use, and effective Web-based learning resource for therapists on how to deliver the SARAH program. There is also huge potential for adapting iSARAH across different cultures and languages, thus opening more opportunities for wider uptake and application of the SARAH program into practice.

Assessing Implementation Readiness and Success of an e-Resource to Improve Prelicensure Physical Therapy Workforce Capacity to Manage Rheumatoid Arthritis

Study Design Prospective within-subject, cross-sectional, between-group, nested qualitative designs within an implementation science framework. Background Physical therapy is recommended for rheumatoid arthritis (RA) care, yet prelicensure RA curriculum time remains limited. Objectives To determine readiness for, and success of, implementing an e-learning tool, Rheumatoid Arthritis for Physiotherapists e-Learning (RAP-eL), within the prelicensure physical therapy curriculum. Methods All physical therapy students in a 1-year cohort in 2014 had RAP-eL embedded in their curriculum. Rheumatoid Arthritis for Physiotherapists e-Learning is an online platform that delivers RA disease information with translation to clinical practice. Implementation readiness, determined by acceptability of RAP-eL to students, was evaluated using focus groups (n = 23). Implementation success was measured using quantitative data from a previously validated questionnaire, including changes in students' self-reported confidence in knowledge (out of 45) and skills (out of 40) in managing RA after 4 weeks of access to RAP-eL, retention of learning over 14 months, and differences in workforce readiness between students in the cohort who had access to RAP-eL and a historical control cohort. Results Acceptability of RAP-eL was confirmed from qualitative data, demonstrating implementation readiness. Short-term improvements were observed in RA knowledge (mean difference, 16.6; 95% confidence interval [CI]: 15.7, 17.6) and RA skills (mean difference, 14.9; 95% CI: 13.9, 15.9; n = 137). Retention was demonstrated after 14 months (P<.001; n = 62). Students in the 1-year cohort who had RAP-eL embedded in the curriculum scored significantly higher on knowledge (mean difference, 3.6; 95% CI: 1.3, 5.9) and skills (mean difference, 3.3; 95% CI: 0.9, 5.7; n = 62) compared to those without RAP-eL (n = 36). Rheumatoid Arthritis for Physiotherapists e-Learning remains embedded in the curriculum. Conclusion This study demonstrated both readiness and success of the sustainable implementation of RAP-eL within a prelicensure physical therapy curriculum. J Orthop Sports Phys Ther 2017;47(9):652-663. doi:10.2519/jospt.2017.7281.

Use of eHealth technologies to enable the implementation of musculoskeletal Models of Care: Evidence and practice

Musculoskeletal (MSK) conditions are the second leading cause of morbidity-related burden of disease globally. EHealth is a potentially critical factor that enables the implementation of accessible, sustainable and more integrated MSK models of care (MoCs). MoCs serve as a vehicle to drive evidence into policy and practice through changes at a health system, clinician and patient level. The use of eHealth to implement MoCs is intuitive, given the capacity to scale technologies to deliver system and economic efficiencies, to contribute to sustainability, to adapt to low-resource settings and to mitigate access and care disparities. We follow a practice-oriented approach to describing the 'what' and 'how' to harness eHealth in the implementation of MSK MoCs. We focus on the practical application of eHealth technologies across care settings to those MSK conditions contributing most substantially to the burden of disease, including osteoarthritis and inflammatory arthritis, skeletal fragility-associated conditions and persistent MSK pain.

Towards Usable E-Health. A Systematic Review of Usability Questionnaires

BACKGROUND

The use of e-health can lead to several positive outcomes. However, the potential for e-health to improve healthcare is partially dependent on its ease of use. In order to determine the usability for any technology, rigorously developed and appropriate measures must be chosen.

OBJECTIVES

To identify psychometrically tested questionnaires that measure usability of e-health tools, and to appraise their generalizability, attributes coverage, and quality.

METHODS

We conducted a systematic review of studies that measured usability of e-health tools using four databases (Scopus, PubMed, CINAHL, and HAPI). Non-primary research, studies that did not report measures, studies with children or people with cognitive limitations, and studies about assistive devices or medical equipment were systematically excluded. Two authors independently extracted information including: questionnaire name, number of questions, scoring method, item generation, and psychometrics using a data extraction tool with pre-established categories and a quality appraisal scoring table.

RESULTS

Using a broad search strategy, 5,558 potentially relevant papers were identified. After removing duplicates and applying exclusion criteria, 35 articles remained that used 15 unique questionnaires. From the 15 questionnaires, only 5 were general enough to be used across studies. Usability attributes covered by the questionnaires were: learnability (15), efficiency (12), and satisfaction (11). Memorability (1) was the least covered attribute. Quality appraisal showed that face/content (14) and construct (7) validity were the most frequent types of validity assessed. All questionnaires reported reliability measurement. Some questionnaires scored low in the quality appraisal for the following reasons: limited validity testing (7), small sample size (3), no reporting of user centeredness (9) or feasibility estimates of time, effort, and expense (7).

CONCLUSIONS

Existing questionnaires provide a foundation for research on e-health usability. However, future research is needed to broaden the coverage of the usability attributes and psychometric properties of the available questionnaires.

Medication Risk Communication in Rheumatology: Where Are We and Where Do We Go from Here?

This paper reviews findings from recent research examining issues related to the risk communication process within the context of rheumatologic conditions. Five specific questions are addressed. First, to what extent do patients with rheumatic disorders exhibit adequate knowledge relevant to disease and medication risks and risk management? Second, what sources do patients use when attempting to find information about disease and medication risks and risk management? Third, what types of information about disease and medication risks and risk management do rheumatologists provide during routine office visits. Fourth, what governmental regulations attempt to increase consumer access to information about medication risks and risk management? Finally, what other educational resources are available to facilitate patient-provider communication concerning disease and medication risks and risk management? The findings reported identify gaps in patient knowledge concerning medication risks and risk management, highlight areas for improvement, and identify resources that may enhance medication risk communication.

Usability and utility evaluation of the web-based "Should I Start Insulin?" patient decision aid for patients with type 2 diabetes among older people

This study aimed to evaluate the usability (ease of use) and utility (impact on user's decision-making process) of a web-based patient decision aid (PDA) among older-age users. A pragmatic, qualitative research design was used. We recruited patients with type 2 diabetes who were at the point of making a decision about starting insulin from a tertiary teaching hospital in Malaysia in 2014. Computer screen recording software was used to record the website browsing session and in-depth interviews were conducted while playing back the website recording. The interviews were analyzed using the framework approach to identify usability and utility issues. Three cycles of iteration were conducted until no more major issues emerged. Thirteen patients participated: median age 65 years old, 10 men, and nine had secondary education/diploma, four were graduates/had postgraduate degree. Four usability issues were identified (navigation between pages and sections, a layout with open display, simple language, and equipment preferences). For utility, participants commented that the website influenced their decision about insulin in three ways: it had provided information about insulin, it helped them deliberate choices using the option-attribute matrix, and it allowed them to involve others in their decision making by sharing the PDA summary printout.

sandflyDST: a dynamic web-based decision support tool for the morphological identification of sandflies present in Anatolia and mainland Europe, and user study

Species identification of sandflies is mainly performed according to morphological characters using classical written identification keys. This study introduces a new web-based decision support tool (sandflyDST) for guiding the morphological identification of sandfly species present in Anatolia and mainland Europe and classified in the Phlebotomus and Sergentomyia genera (both: Diptera: Psychodidae). The current version of the tool consists of 111 questions and 36 drawings obtained from classical written keys, and 107 photographs for the quick and easy identification of 26 species of the genus Phlebotomus and four species of the genus Sergentomyia. The tool guides users through a decision tree using yes/no questions about the morphological characters of the specimen. The tool was applied by 30 individuals, who then completed study questionnaires. The results of subsequent analyses indicated that the usability (x‾SUSScore=75.4) and users' level of appreciation (86.6%) of the tool were quite high; almost all of the participants considered recommending the tool to others. The tool may also be useful in training new entomologists and maintaining their level of expertise. This is a dynamic tool and can be improved or upgraded according to feedback. The tool is now available online at http://parasitology.ege.edu.tr/sandflyDST/index.php.

Are online symptoms checkers useful for patients with inflammatory arthritis?

Background

Online symptom checkers are increasingly used by patients however there is little published evidence of their effectiveness in real patients. The aim of this study was to evaluate how patients with inflammatory arthritis and inflammatory arthralgia use the internet to look for health information and to assess the advice given and diagnoses suggested by the NHS and WebMD symptom checkers in relation to the patients’ actual diagnoses.

Methods

Thirty-four patients with inflammatory arthritis (rheumatoid arthritis (n = 13), psoriatic arthritis (n = 4), unclassified arthritis (n = 4)) and inflammatory arthralgia (n = 13) newly presenting to a secondary care based clinic were identified using a consecutive sampling approach. Consenting patients were asked questions about their internet use in relation to their presenting symptoms. They then completed the NHS and the WebMD symptom checkers and their answers and the outcomes were recorded.

Results

Sixteen patients had previously consulted the internet regarding their symptoms. Neither age nor gender significantly influenced internet usage. Actions advised via the NHS symptom checker were: call an ambulance (n = 11), attend A&E (n = 4), contact your GP straight away (n = 2), see your GP today (n = 6), or see your GP within 36 h (n = 11). The 5 most common differential diagnoses given by Web MD were gout (n = 28), rheumatoid arthritis (n = 24), psoriatic arthritis (n = 22), osteoarthritis (n = 18) and finger dislocation (n = 10). The most common first differential diagnosis was osteoarthritis (n = 12). Only 4 out of 21 patients with inflammatory arthritis were given a first diagnosis of rheumatoid arthritis or psoriatic arthritis.

Conclusions

Our data highlight that help seeking advice given online is often inappropriate and that the diagnoses suggested are frequently inaccurate. Recommendations to seek emergency advice may cause inappropriate healthcare utilization.

Electronic supplementary material

The online version of this article (doi:10.1186/s12891-016-1189-2) contains supplementary material, which is available to authorized users.

Usability Testing and Adaptation of the Pediatric Cardiovascular Risk Reduction Clinical Decision Support Tool

BACKGROUND

Cardiovascular disease (CVD) is 1 of the leading causes of death, years of life lost, and disability-adjusted years of life lost worldwide. CVD prevention for children and teens is needed, as CVD risk factors and behaviors beginning in youth contribute to CVD development. In 2012, the National Heart, Lung, and Blood Institute released their "Integrated Guidelines for Cardiovascular Health and Risk Reduction in Children and Adolescents" for clinicians, describing CVD risk factors they should address with patients at primary care preventative visits. However, uptake of new guidelines is slow. Clinical decision support (CDS) tools can improve guideline uptake. In this paper, we describe our process of testing and adapting a CDS tool to help clinicians evaluate patient risk, recommend behaviors to prevent development of risk, and complete complex calculations to determine appropriate interventions as recommended by the guidelines, using a user-centered design approach.

OBJECTIVE

The objective of the study was to assess the usability of a pediatric CVD risk factor tool by clinicians.

METHODS

The tool was tested using one-on-one in-person testing and a "think aloud" approach with 5 clinicians and by using the tool in clinical practice along with formal usability metrics with 14 pediatricians. Thematic analysis of the data from the in-person testing and clinical practice testing identified suggestions for change in 3 major areas: user experience, content refinement, and technical deployment. Descriptive statistical techniques were employed to summarize users' overall experience with the tool.

RESULTS

Data from testers showed that general reactions toward the CDS tool were positive. Clinical practice testers suggested revisions to make the application more user-friendly, especially for clinicians using the application on the iPhone, and called for refining recommendations to be more succinct and better tailored to the patient. Tester feedback was incorporated into the design when feasible, including streamlining data entry during clinical visits, reducing the volume of results displayed, and highlighting critical results.

CONCLUSIONS

This study found support for the usability of our pediatric CVD risk factor tool. Insights shared about this tool may be applicable for designing other mHealth applications and CDS tools. The usability of decision support tools in clinical practice depends critically on receiving (ie, through an accessible device) and adapting the tool to meet the needs of clinicians in the practice setting.

StatXFinder: a web-based self-directed tool that provides appropriate statistical test selection for biomedical researchers in their scientific studies

The improper use of statistical methods is common in analyzing and interpreting research data in biological and medical sciences. The objective of this study was to develop a decision support tool encompassing the commonly used statistical tests in biomedical research by combining and updating the present decision trees for appropriate statistical test selection. First, the decision trees in textbooks, published articles, and online resources were scrutinized, and a more comprehensive unified one was devised via the integration of 10 distinct decision trees. The questions also in the decision steps were revised by simplifying and enriching of the questions with examples. Then, our decision tree was implemented into the web environment and the tool titled StatXFinder was developed. Finally, usability and satisfaction questionnaires were applied to the users of the tool, and StatXFinder was reorganized in line with the feedback obtained from these questionnaires. StatXFinder provides users with decision support in the selection of 85 distinct parametric and non-parametric statistical tests by directing 44 different yes–no questions. The accuracy rate of the statistical test recommendations obtained by 36 participants, with the cases applied, were 83.3 % for “difficult” tests, and 88.9 % for “easy” tests. The mean system usability score of the tool was found 87.43 ± 10.01 (minimum: 70—maximum: 100). A statistically significant difference could not be seen between total system usability score and participants’ attributes (p value >0.05). The User Satisfaction Questionnaire showed that 97.2 % of the participants appreciated the tool, and almost all of the participants (35 of 36) thought of recommending the tool to the others. In conclusion, StatXFinder, can be utilized as an instructional and guiding tool for biomedical researchers with limited statistics knowledge. StatXFinder is freely available at http://webb.deu.edu.tr/tb/statxfinder.

Quantifying usability: an evaluation of a diabetes mHealth system on effectiveness, efficiency, and satisfaction metrics with associated user characteristics

OBJECTIVE

Mobile health (mHealth) systems are becoming more common for chronic disease management, but usability studies are still needed on patients' perspectives and mHealth interaction performance. This deficiency is addressed by our quantitative usability study of a mHealth diabetes system evaluating patients' task performance, satisfaction, and the relationship of these measures to user characteristics.

MATERIALS AND METHODS

We used metrics in the International Organization for Standardization (ISO) 9241-11 standard. After standardized training, 10 patients performed representative tasks and were assessed on individual task success, errors, efficiency (time on task), satisfaction (System Usability Scale [SUS]) and user characteristics.

RESULTS

Tasks of exporting and correcting values proved the most difficult, had the most errors, the lowest task success rates, and consumed the longest times on task. The average SUS satisfaction score was 80.5, indicating good but not excellent system usability. Data trends showed males were more successful in task completion, and younger participants had higher performance scores. Educational level did not influence performance, but a more recent diabetes diagnosis did. Patients with more experience in information technology (IT) also had higher performance rates.

DISCUSSION

Difficult task performance indicated areas for redesign. Our methods can assist others in identifying areas in need of improvement. Data about user background and IT skills also showed how user characteristics influence performance and can provide future considerations for targeted mHealth designs.

CONCLUSION

Using the ISO 9241-11 usability standard, the SUS instrument for satisfaction and measuring user characteristics provided objective measures of patients' experienced usability. These could serve as an exemplar for standardized, quantitative methods for usability studies on mHealth systems.

Developing Internet-based health interventions: a guide for public health researchers and practitioners

Background

Researchers and practitioners interested in developing online health interventions most often rely on Web-based and print resources to guide them through the process of online intervention development. Although useful for understanding many aspects of best practices for website development, missing from these resources are concrete examples of experiences in online intervention development for health apps from the perspective of those conducting online health interventions.

Objective

This study aims to serve as a series of case studies in the development of online health interventions to provide insights for researchers and practitioners who are considering technology-based interventional or programmatic approaches.

Methods

A convenience sample of six study coordinators and five principal investigators at a large, US-based land grant university were interviewed about the process of developing online interventions in the areas of alcohol policy, adolescent health, medication adherence, and human immunodeficiency virus prevention in transgender persons and in men who have sex with men. Participants were asked questions that broadly addressed each of the four phases of the User-Centered Design Process Map from the US Department of Health and Human Services' Research-Based Web Design & Usability Guidelines. Interviews were audio recorded and transcribed. Qualitative codes were developed using line-by-line open coding for all transcripts, and all transcripts were coded independently by at least 2 authors. Differences among coders were resolved with discussion.

Results

We identified the following seven themes: (1) hire a strong (or at least the right) research team, (2) take time to plan before beginning the design process, (3) recognize that vendors and researchers have differing values, objectives, and language, (4) develop a detailed contract, (5) document all decisions and development activities, (6) use a content management system, and (7) allow extra time for testing and debugging your intervention. Each of these areas is discussed in detail, with supporting quotations from principal investigators and study coordinators.

Conclusions

The values held by members of each participating organization involved in the development of the online intervention or program, as well as the objectives that are trying to be met with the website, must be considered. These defined values and objectives should prompt an open and explicit discussion about the scope of work, budget, and other needs from the perspectives of each organization. Because of the complexity of developing online interventions, researchers and practitioners should become familiar with the process and how it may differ from the development and implementation of in-person interventions or programs. To assist with this, the intervention team should consider expanding the team to include experts in computer science or learning technologies, as well as taking advantage of institutional resources that will be needed for successful completion of the project. Finally, we describe the tradeoff between funds available for online intervention or program development and the complexity of the project.

Development and Evaluation of an Educational E-Tool to Help Patients With Non-Hodgkin's Lymphoma Manage Their Personal Care Pathway

BACKGROUND

An overload of health-related information is available for patients on numerous websites, guidelines, and information leaflets. However, the increasing need for personalized health-related information is currently unmet.

OBJECTIVE

This study evaluates an educational e-tool for patients with non-Hodgkin's lymphoma (NHL) designed to meet patient needs with respect to personalized and complete health-related information provision. The e-tool aims to help NHL patients manage and understand their personal care pathway, by providing them with insight into their own care pathway, the possibility to keep a diary, and structured health-related information.

METHODS

Together with a multidisciplinary NHL expert panel, we developed an e-tool consisting of two sections: (1) a personal section for patients' own care pathway and their experiences, and (2) an informative section including information on NHL. We developed an ideal NHL care pathway based on the available (inter)national guidelines. The ideal care pathway, including date of first consultation, diagnosis, and therapy start, was used to set up the personal care pathway. The informative section was developed in collaboration with the patient association, Hematon. Regarding participants, 14 patients and 6 laymen were asked to evaluate the e-tool. The 24-item questionnaire used discussed issues concerning layout (6 questions), user convenience (3 questions), menu clarity (3 questions), information clarity (5 questions), and general impression (7 questions). In addition, the panel members were asked to give their feedback by email.

RESULTS

A comprehensive overview of diagnostics, treatments, and aftercare can be established by patients completing the questions from the personal section. The informative section consisted of NHL information regarding NHL in general, diagnostics, therapy, aftercare, and waiting times. Regarding participants, 6 patients and 6 laymen completed the questionnaire. Overall, the feedback was positive, with at least 75% satisfaction on each feedback item. Important strengths mentioned were the use of a low health-literacy level, the opportunity to document the personal care pathway and experiences, and the clear overview of the information provided. The added value of the e-tool in general was pointed out as very useful for preparing the consultation with one's doctor and for providing all information on one website, including the opportunity for a personalized care pathway and diary. The majority of the revisions concerned wording and clarity. In addition, more explicit information on immunotherapy, experimental therapy, and psychosocial support was added.

CONCLUSIONS

We have developed a personal care management e-tool for NHL patients. This tool contains a unique way to help patients manage their personal care pathway and give them insight into their NHL by providing health-related information and a personal diary. This evaluation showed that our e-tool meets patients' needs concerning personalized health-related information, which might serve as a good example for other oncologic diseases. Future research should focus on the possible impact of the e-tool on doctor-patient communication during consultations.