The use of think-aloud and instant data analysis in evaluation research: Exemplar and lessons learned

Evaluating the usability of Iran's national comprehensive health information system: a think-aloud study to uncover usability problems in the recording of childcare data

INTRODUCTION

Health information systems play a crucial role in the delivery of efficient and effective healthcare. Poor usability is one of the reasons for their lack of acceptance and low usage by users. The aim of this study was to identify the usability problems of a national comprehensive health information system using the concurrent think-aloud method in the recording of childcare data.

METHODS

A descriptive cross-sectional study was conducted in the health centers of Kashan University of Medical Sciences, Iran, in 2020. Ten healthcare providers as system's users were purposively selected to evaluate the system. To identify problems, a concurrent think-aloud evaluation was conducted. Two administrators of the system designed scenarios for ten childcare data recording tasks. By analysing the recorded files, usability problems were identified. The severity of the problems was then determined with the help of the users and problems were assigned to usability attributes based on their impact on the user.

RESULTS

A total of 68 unique problems were identified in the system, of which 47.1% were rated as catastrophic problems. The participants assigned 47 problems (69%) to the user satisfaction attribute and 45 problems (66%) to the efficiency attribute; they also did not assign any problems to the effectiveness attribute.

CONCLUSION

The problems identified in the national comprehensive health information system using the think-aloud method were rated as major and catastrophic, which indicates poor usability of this system. Therefore, resolving the system problems will help increase user satisfaction and system efficiency, allowing more time to be spent on patient care and parent's education as well as improving overall quality of care.

Evaluating the Usability and Quality of a Clinical Mobile App for Assisting Physicians in Head Computed Tomography Scan Ordering: Mixed Methods Study

BACKGROUND

Among the numerous factors contributing to health care providers' engagement with mobile apps, including user characteristics (eg, dexterity, anatomy, and attitude) and mobile features (eg, screen and button size), usability and quality of apps have been introduced as the most influential factors.

OBJECTIVE

This study aims to investigate the usability and quality of the Head Computed Tomography Scan Appropriateness Criteria (HAC) mobile app for physicians' computed tomography scan ordering.

METHODS

Our study design was primarily based on methodological triangulation by using mixed methods research involving quantitative and qualitative think-aloud usability testing, quantitative analysis of the Mobile Apps Rating Scale (MARS) for quality assessment, and debriefing across 3 phases. In total, 16 medical interns participated in quality assessment and testing usability characteristics, including efficiency, effectiveness, learnability, errors, and satisfaction with the HAC app.

RESULTS

The efficiency and effectiveness of the HAC app were deemed satisfactory, with ratings of 97.8% and 96.9%, respectively. MARS assessment scale indicated the overall favorable quality score of the HAC app (82 out of 100). Scoring 4 MARS subscales, Information (73.37 out of 100) and Engagement (73.48 out of 100) had the lowest scores, while Aesthetics had the highest score (87.86 out of 100). Analysis of the items in each MARS subscale revealed that in the Engagement subscale, the lowest score of the HAC app was "customization" (63.6 out of 100). In the Functionality subscale, the HAC app's lowest value was "performance" (67.4 out of 100). Qualitative think-aloud usability testing of the HAC app found notable usability issues grouped into 8 main categories: lack of finger-friendly touch targets, poor search capabilities, input problems, inefficient data presentation and information control, unclear control and confirmation, lack of predictive capabilities, poor assistance and support, and unclear navigation logic.

CONCLUSIONS

Evaluating the quality and usability of mobile apps using a mixed methods approach provides valuable information about their functionality and disadvantages. It is highly recommended to embrace a more holistic and mixed methods strategy when evaluating mobile apps, because results from a single method imperfectly reflect trustworthy and reliable information regarding the usability and quality of apps.

Exploring patient-centered design solutions of a telehealth app for HIV - A qualitative study

BACKGROUND

The Communication and Tracing App HIV (COMTRAC-HIV) project is developing a mobile health (mHealth) app for integrated care of HIV patients in Germany. The complexity of HIV treatment and continuous care necessitates the need for tailored mHealth solutions. This qualitative study explores design solutions and a prototype to enhance the app's functionality and effectiveness.

METHODS

A total of eight HIV patients and pre-exposure prophylaxis (PrEP) users, recruited at the HIV Center Frankfurt, participated in focus groups and thinking-aloud tests (TA test). In the focus groups, design solutions were discussed for user-interface clarity, leading to the development of an interactive prototype, the usability of which was evaluated with a TA test. Data collection involved video/audio recordings. Qualitative analysis was conducted using a deductive category system, and focused on app design and usage in focus groups, and layout, navigation, interaction, terminology, comprehension, feedback, and level of satisfaction in TA tests.

RESULTS

The app was commended for its simple, clear design, especially its medication reminders and health tracking features. Opinions on the symptom diary varied however, respondents noting it more suitable for HIV users than PrEP users. Privacy concerns suggest avoiding display of HIV-specific information. Suggested improvements include e.g. image uploads, drug interaction checks and prescription tracking. A total of 25 usability issues were identified in the TA test, with most found in the layout (n = 6), navigation (n = 5), interaction (n = 5), and terminology (n = 5) categories. Two examples are non-intuitive controls and illogical button placement. Despite these disadvantages, participants noted positive impressions (n = 5) in the satisfaction category.

CONCLUSION

The study emphasizes the need for patient-centered design in mobile HIV care solutions, highlighting to the app's user-friendliness and potential to enhance care. Further research is necessary to refine the app's functionality and to align it with clinical and patients' privacy needs.

Development of minimum data set and dashboard for monitoring adverse events in radiology departments

Background

To reduce the risk of errors, patient safety monitoring in the medical imaging department is crucial. Interventions are required and these can be provided as a framework for documenting, reporting, evaluating, and recognizing events that pose a threat to patient safety. The aim of this study was to develop minimum data set and dashboard for monitoring adverse events in radiology departments.

Material and methods

This developmental research was conducted in multiple phases, including content determination using the Delphi technique; database designing using SQL Server; user interface (UI) building using PHP; and dashboard evaluation in three aspects: the accuracy of calculating; UI requirements; and usability.

Results

This study identified 26 patient safety (PS) performance metrics and 110 PS-related significant data components organized into 14 major groupings as the system contents. The UI was built with three tabs: pre-procedure, intra-procedure, and post-procedure. The evaluation results proved the technical feasibility of the dashboard. Finally, the dashboard's usability was highly rated (76.3 out of 100).

Conclusion

The dashboard can be used to supplement datasets to obtain a more accurate picture of the PS condition and to draw attention to characteristics that professionals might otherwise overlook or undervalue.

The differences between adults and adolescents using a mobile health application for menstrual complaints: A usability and qualitative study

BACKGROUND

A "Menstruatie Educatie Kalender" application (Menstruation Education Calendar, (MEK-APP)) was developed for adults to evaluate menstrual complaints. The future aim of this app is to use it as a self-diagnostic instrument for menstrual abnormalities for both adults and adolescents. Early identification of the potential of an application for future use by both user groups would increase implementation success and adoption of the application.

OBJECTIVE

To compare differences in experienced usability by adults versus adolescents and to identify factors influencing future use for both age groups in one mHealth application (in this study the MEK-APP).

METHODS

This study consisted of three phases: (1) usability testing of the MEK-APP for iOS and Android by think-aloud method, (2) two-month daily use and (3) in-depth individual interview. During the think-aloud sessions, twelve tasks were performed in the application while they were thinking aloud. Usability problems were rated for their severity with Nielsen' Severity Scale. Both the think-aloud sessions and in-depth interviews were verbatim transcribed and thematically analyzed to determine the factors influencing future use for both groups. In addition, the System Usability Scale (SUS) and Intrinsic Motivation Inventory (IMI) questionnaires were filled out during the interviews.

RESULTS

Seven adults (>18 years) and seven adolescents (14-18 years) evaluated the MEK-APP. There were 14 usability issues and 16 bugs in both groups. There were no differences between adults and adolescents. In the thematic qualitative analysis, the following future use factors were identified: user-expectation, motivation, privacy, understandability, and user-experience. The user-expectation, motivation and privacy differed between both groups but did not influence usage. No differences were observed in SUS and IMI scores between both groups.

CONCLUSIONS

There are five factors influencing the future use of a menstrual-related mHealth application for both adults and adolescents. It is possible to serve different age groups with a single application.

Towards co-design of rehabilitation technologies: a collaborative approach to prioritize usability issues

Introduction

Early stakeholder engagement is critical to the successful development and translation of rehabilitation technologies, a pivotal step of which is usability testing with intended end-users. To this end, several methods employ end-user feedback to identify usability and implementation issues. However, the process of prioritizing identified issues seldom leverages the knowledge and expertise of the range of stakeholders who will ultimately affect the demand and supply of a device. This paper describes a novel method to prioritize end-user feedback using transdisciplinary stakeholder consultation and address it in subsequent product development. The proposed approach was demonstrated using a case study relating to the development of a novel technology for neural recovery after spinal cord injury.

Method

Feedback from five individuals with chronic spinal cord injury was collected during two-hour usability evaluation sessions with a fully functional high-fidelity system prototype. A think-aloud and semi-structured interview protocol was used with each participant to identify usability and acceptability issues relating to the system in a 3-phase approach. Phase 1 involved extracting usability issues from think-aloud and semi-structured interview data. Phase 2 involved rating the usability issues based on their significance, technical feasibility, and implementation priority by relevant internal and external stakeholders. Finally, Phase 3 involved aggregating the usability issues according to design and implementation elements to facilitate solution generation, and these solutions were then raised as action tasks for future design iterations.

Results

Sixty usability issues representing nine facets of usability were rated. Eighty percent of issues were rated to be of moderate to high significance, 83% were rated as being feasible to address, and 75% were rated as addressable using existing project resources. Fifty percent of the issues were rated to be a high priority for implementation. Evaluation of the grouped issues identified 21 tasks which were mapped to the product roadmap for integration into future design iterations.

Discussion

This paper presents a method for meaningful transdisciplinary stakeholder engagement in rehabilitation technology development that can extended to other projects. Alongside a worked example, we offer practical considerations for others seeking to co-develop rehabilitation technologies.

Development and optimisation of a mobile app (iMPAKT) for improving person-centred practice in healthcare settings: A multi-methods evaluation study

Objective

To develop and optimise an app (iMPAKT) for improving implementation and measurement of person-centred practice in healthcare settings.

Methods

Two iterative rounds of testing were carried out based on cognitive task analysis and qualitative interview methods. The System Usability Scale (SUS) was also used to evaluate the app. Quantitative data on task completion and SUS scores were evaluated descriptively, with thematic analysis performed on qualitative data. The MoSCoW prioritisation system was used to identify key modifications to improve the app.

Results

Twelve participants took part (eight health professionals and four patient and public involvement representatives). Views on design and structure of the app were positive. The majority of the 16 tasks undertaken during the cognitive task analysis were easy to complete. Mean SUS scores were 73.5/100 (SD: 7.9; range = 60-92.5), suggesting good overall usability. For one section of the app that transcribes patients speaking about their experience of care, a non-intuitive user interface and lack of transcription accuracy were identified as key issues influencing usability and acceptability.

Conclusions

Findings from the evaluation were used to inform iterative modifications to further develop and optimise the iMPAKT App. These included improved navigational flow, and implementation of an updated artificial intelligence (AI) based Speech-To-Text software; allowing for more accurate, real-time transcription. Use of such AI-based software represents an interesting area that requires further evaluation. This is particularly apparent in relation to potential for large-scale collection of data on person-centred measures using the iMPAKT App, and for assessing initiatives designed to improve patient experience.

Designing and validating novel communication measures for pediatric, adolescent, and young adult oncology care and research: The PedCOM measures

BACKGROUND

High-quality communication in pediatric and adolescent cancer is the standard of care. Yet, we lack pediatric-specific communication measures. We designed self-report and caregiver-report communication measures for use in pediatric oncology settings.

METHODS

We recruited adolescent and young adults (AYAs; 12-24 years) with cancer and parents of children and AYAs between 1 month post-diagnosis and 2 months after completing treatment. Participants completed measures including 58 questions addressing eight previously characterized communication functions. For each function, we conducted factor analysis, and assessed reliability and construct validity. Based on psychometric properties, we created final long-form (31 items) and short-form (eight items) communication measures (PedCOM) for both self- and caregiver-report.

RESULTS

Participants included 200 parents and 88 AYAs. The final 31-item and eight-item PedCOM measures for parents and AYAs demonstrated good unidimensional model fit. Each communication function (e.g., building relationships) demonstrated high reliability, coefficient alphas ranged from .83 to .93 for parents and .85 to .93 for AYAs. The overall 31-item and eight-item PedCOM measures also demonstrated high reliability (alphas >.9). For construct validity, PedCOM-Parent-31 and PedCOM-Parent-8 correlated positively with satisfaction with care, trust, emotional self-management, and decisional satisfaction, and negatively with anxiety and caregiver burden. PedCOM-AYA-31 and PedCOM-AYA-8 correlated positively with satisfaction with care, trust, emotional self-management, symptoms self-management, and decisional satisfaction, and negatively with anxiety.

DISCUSSION

We developed valid and reliable measures of communication functions for parents and AYAs with cancer. These measures can support organizations and stakeholder groups that are striving to improve the quality of cancer care.

User-Centered Design of a Gamified Mental Health App for Adolescents in Sub-Saharan Africa: Multicycle Usability Testing Study

BACKGROUND

There is an urgent need for scalable psychological treatments to address adolescent depression in low-resource settings. Digital mental health interventions have many potential advantages, but few have been specifically designed for or rigorously evaluated with adolescents in sub-Saharan Africa.

OBJECTIVE

This study had 2 main objectives. The first was to describe the user-centered development of a smartphone app that delivers behavioral activation (BA) to treat depression among adolescents in rural South Africa and Uganda. The second was to summarize the findings from multicycle usability testing.

METHODS

An iterative user-centered agile design approach was used to co-design the app to ensure that it was engaging, culturally relevant, and usable for the target populations. An array of qualitative methods, including focus group discussions, in-depth individual interviews, participatory workshops, usability testing, and extensive expert consultation, was used to iteratively refine the app throughout each phase of development.

RESULTS

A total of 160 adolescents from rural South Africa and Uganda were involved in the development process. The app was built to be consistent with the principles of BA and supported by brief weekly phone calls from peer mentors who would help users overcome barriers to engagement. Drawing on the findings of the formative work, we applied a narrative game format to develop the Kuamsha app. This approach taught the principles of BA using storytelling techniques and game design elements. The stories were developed collaboratively with adolescents from the study sites and included decision points that allowed users to shape the narrative, character personalization, in-app points, and notifications. Each story consists of 6 modules ("episodes") played in sequential order, and each covers different BA skills. Between modules, users were encouraged to work on weekly activities and report on their progress and mood as they completed these activities. The results of the multicycle usability testing showed that the Kuamsha app was acceptable in terms of usability and engagement.

CONCLUSIONS

The Kuamsha app uniquely delivered BA for adolescent depression via an interactive narrative game format tailored to the South African and Ugandan contexts. Further studies are currently underway to examine the intervention's feasibility, acceptability, and efficacy in reducing depressive symptoms.

The Healthy Hearts Project: Development and evaluation of a website for cardiovascular risk assessment and visualisation and self-management through healthy lifestyle goal-setting

Materially deprived communities in the UK have excess morbidity and mortality from cardiovascular disease (CVD) but are less likely to engage with formal care pathways. Community engagement and e-health may be more effective ways to promote risk-reducing lifestyle change. The "Healthy Hearts Project" website was designed for use by community health workers (CHWs) for cardiovascular risk assessment and lifestyle goal setting, or for independent use by community members. This paper describes the website's development and evaluation. The website was developed using interactive wire frame prototypes in a user-led approach. Qualitative evaluation of the completed website's usability and acceptability was conducted using the "Thinking Aloud" method in a purposive sample of 10 participants (one voluntary sector employee, three CHWs, two community members and four healthcare professionals). Thinking Aloud interview transcripts were thematically analysed using an inductive approach. A separate quantitative evaluation of usability and the effect of using the website on CVD knowledge and beliefs was conducted. A random sample of 134 participants, recruited using the online platform Prolific, completed the "Attitudes and Beliefs About Cardiovascular Disease" (ABCD) questionnaire before and after using the website, along with the System Usability Scale (SUS). Qualitative evaluation-Four key themes were identified: 1) Website functionality and design-participants generally found the website easy to use and understood the risk communication graphics and the feedback and goal-setting features,; 2) Inclusivity and representation-most participants considered the website inclusive of a range of users/cultures; 3) Language and comprehension-participants found the language used easy to understand but suggested reducing the amount of text; 4) Motivation and barriers to change-participants liked the personalized feedback and empowerment offered by goal-setting but commented on the need for self-motivation. Quantitative evaluation-The mean score across all domains of the ABCD questionnaire (from 2.99 to 3.11, p<0.001) and in the sub-domains relating to attitudes and beliefs around healthy eating and exercise increased after using the website. The mean(sd) score on the SUS was 77.5 (13.5). The website's usability was generally rated well by both quantitative and qualitative measures, and measures of CVD knowledge improved after use. A number of general recommendations for the design of eHealth behaviour change tools are made based on participants' suggestions to improve the website.

Gamification to Improve Medication Adherence: A Mixed-method Usability Study for MedScrab

Medication non-adherence is a prevalent healthcare problem with poor health outcomes and added healthcare costs. MedScrab, a gamification-based mHealth app, is the first attempt to deliver crucial life-saving medication information to patients and increase their medication adherence. The paper presents the development of MedScrab and a two-phase mixed-method usability evaluation of MedScrab. Phase I qualitatively evaluated MedScrab using a think-aloud protocol for its usability. With 51 participants, qualitative data analysis of Phase I revealed two themes: positive functionality of the app and four areas of improvement. The improvement recommendations were incorporated into MedScrab's design. Phase I also validated a widely used mHealth App Usability Questionnaire (MAUQ). Quantitative data analysis of Phase I reduced the original 18-item MAUQ scale to a 15-item scale with two factors: ease of use (4 items) and usefulness and satisfaction (11 items). Phase II surveyed 83 participants from Amazon's Mechanical Turk using a modified MAUQ. The modified MAUQ scale showed strong internal consistency (Cronbach alpha = 0.959) and high factor loadings (between 0.623 and 0.987). The study design of the usability evaluation can serve as a methodological guide for designing, evaluating, and improving mHealth apps.The usability study showed that MedScrab was perceived as ease of use (6.24 out of 7) with high usefulness and satisfaction (5.72 out of 7). The quantitative data analysis results support the use of the modified MAUQ as a valid instrument to measure the usability of the MedScrab. However, the instrument should be used with adaptation based on the app's characteristics.

User-Centered Mobile Applications for Stroke Survivors (MAPPS): A Mixed-Methods Study of Patient Preferences

OBJECTIVE

Investigate stroke survivors' (SS) preferences for a hypothetical mHealth app for post-stroke care and to study the influence of demographic variables on these preferences.

DESIGN

Mixed-methods, sequential, observational study.

SETTING

Focus groups (phase 1) were conducted to identify SS perceptions and knowledge of mHealth applications (apps). Using grounded theory approach, recurring themes were identified. A multiple-choice questionnaire of 5 desired app features was generated using these themes and mailed to SS (national survey, phase 2). SS' demographics and perceived usefulness (yes/no) for each feature were recorded. In-person usability testing (phase 3) was conducted to identify areas of improvement in user interfaces of existing apps. Summative telephone interviews (phase 4) were conducted for final impressions supplementary to national survey.

PARTICIPANTS

SS aged >18 years recruited from study hospital, national stroke association database, stroke support and advocacy groups. Non-English speakers and those unable to communicate were excluded.

INTERVENTIONS

None.

MAIN OUTCOME MEASURES

(1) Percentage of SS (phase 2) identifying proposed app features to be useful. (2) Influence of age, sex, race, education, and time since stroke on perceived usefulness.

RESULTS

Ninety-six SS participated in focus groups. High cost, complexity, and lack of technical support were identified as barriers to adoption of mHealth apps. In the national survey (n=1194), ability to track fitness and diet (84%) and communication (70%) were the most and least useful features, respectively. Perceived usefulness was higher among younger SS (P<.001 to .006) and SS of color (African American and Hispanic) (ORs 1.73-4.41). Simple design and accommodation for neurologic deficits were main recommendations from usability testing.

CONCLUSIONS

SS are willing to adopt mHealth apps that are free of cost and provide technical support. Apps for SS should perform multiple tasks and be of simple design. Greater interest for the app's features among SS of color may provide opportunities to address health inequities.

Towards Translation of Novel Neurorehabilitation Systems: A Practical Approach to Usability Testing

Usability testing is important for the effective translation of neurorehabilitation technologies but is often overlooked and under-reported. The aim of this paper is to present a method of collecting and analyzing usability data, using a think-aloud and semi-structured interview protocol and qualitative analysis techniques. We present a worked case study of this method with a novel neurorehabilitation system that utilizes thought-controlled robotics to partially restore lower-limb function of people with spinal cord injury (SCI). Five male participants (mean age = 32.6 years) with SCI who identified as users of related neurorehabilitation technologies completed the usability study. Video-recorded usability sessions utilized a combination of concurrent and retrospective think-aloud methods as well as semi-structured interviews. Recordings were analyzed to identify verbal and behavioral feedback from participants regarding system performance and acceptability. In total, 538 data points were logged, which were aggregated into 60 usability issues, 44 positive evaluations, and 31 strategies for improvement. The approach undertaken was novel in that we sought to not only capture usability issues but also system elements that were positively evaluated by intended users and strategies for improvement from the perspective of intended users. These observations will be used to inform the further development of the neurorehabilitation system.

Interviews with HIV Experts for Development of a Mobile Health Application in HIV Care-A Qualitative Study

The Communication and Tracing App HIV (COMTRAC-HIV) project aims to develop a mobile health application for integrated care of HIV patients due to the low availability of those apps in Germany. This study addressed organizational conditions and necessary app functionalities, especially for the care of late diagnosed individuals (late presenters) and those using pre-exposure prophylaxis. We followed a human-centered design approach and interviewed HIV experts in Germany to describe the context of use of the app. The interviews were paraphrased and analyzed with a qualitative content analysis. To define the context of use, user group profiles were defined and tasks derived, which will represent the functionalities of the app. A total of eight experts were included in the study. The results show that the app should include a symptom diary for entering symptoms, side effects, and their intensity. It offers chat/video call functionality for communication with an HIV expert, appointment organization, and sharing findings. The app should also provide medication overview and reminders for medications and appointments. This qualitative study is a first step towards the development of an app for HIV individuals in Germany. Further research includes involving patients in the initial app design and test design usability.

Improving the lives of ethnically diverse family carers and people living with dementia using digital media resources - Protocol for the Draw-Care randomised controlled trial

Objectives

Ethnically diverse family carers of people living with dementia (hereafter carers and people with dementia) experience more psychological distress than other carers. To reduce this inequality, culturally adapted, multilingual, evidence-based practical assistance is needed. This paper details the Draw-Care study protocol including a randomised control trial (RCT) to test the effectiveness of a digital intervention comprising a multilingual website, virtual assistant, animated films, and information, on the lives of carers and people with dementia in Australia.

Methods

The Draw-Care intervention will be evaluated in a 12-week active waitlist parallel design RCT with 194 carers from Arabic, Cantonese, Greek, Hindi, Italian, Mandarin, Spanish, Tamil, and Vietnamese-speaking language groups. Our intervention was based on the World Health Organization's (WHO) iSupport Lite online carer support messages and was co-designed with carers, people with dementia, service providers, and clinicians. Culturally adapted multilingual digital resources were created in nine languages and English.

Results

In Phase I (2022), six co-design workshops with stakeholders and interviews with people with dementia informed the development of the intervention which will be trialled and evaluated in Phases II and III (2023 and 2024).

Conclusions

Digital media content is a novel approach to providing cost-effective access to health care information. This study protocol details the three study phases including the RCT of a co-designed, culturally adapted, multilingual, digital intervention for carers and people with dementia to advance the evidence in dementia and digital healthcare research and help meet the needs of carers and people with dementia in Australia and globally.

A visual analytic tool, VIADS, to assist the hypothesis generation process in clinical research—A usability study using mixed methods (Preprint)

BACKGROUND

Visualization can be a powerful tool for comprehending datasets, especially when they can be represented via hierarchical structures. Enhanced comprehension can facilitate the development of scientific hypotheses. However, the inclusion of excessive data can make a visualization overwhelming.

OBJECTIVE

We developed a Visual Interactive Analytic tool for filtering and summarizing large health Data Sets (VIADS) coded with hierarchical terminologies. In this study, we evaluated the usability of VIADS for visualizing data sets of patient diagnoses and procedures coded in the International Classification of Diseases, ninth revisions, clinical modification (ICD-9-CM).

METHODS

We used mixed methods in the study. A group of 12 clinical researchers participated in the generation of data-driven hypotheses using the same datasets and time frame (a 1-hour training session and a 2-hour study session), utilizing VIADS via the think-aloud protocol. The audio and screen activities were recorded remotely. A modified version of the System Usability Scale (SUS) survey and a brief survey with open-ended questions were administered after the study to assess the usability of VIADS and verify their intense usage experience of VIADS.

RESULTS

The range of SUS scores was 37.5 - 87.5. The mean SUS score for VIADS was 71.88 (out of a possible 100, standard deviation: 14.62 ), and the median SUS was 75. The participants unanimously agreed that VIADS offers new perspectives on data sets (100%), while 75% agreed that VIADS facilitates understanding, presentation, and interpretation of underlying datasets. The comments on the utility of VIADS were positive and aligned well with the design objectives of VIADS. The answers to the open-ended questions in the modified SUS provided specific suggestions regarding potential improvements in VIADS, and identified problems in usability were used to update the tool.

CONCLUSIONS

This usability study demonstrates that VIADS is a usable tool for analyzing secondary datasets with good average usability, SUS score, and favorable utility. Currently, VIADS accepts datasets with hierarchical codes and their corresponding frequencies. Consequently, only specific types of use cases are supported by the analytical results. Participants agreed, however, that VIADS provides new perspectives on datasets and is relatively easy to use. The functionalities mostly appreciated by participants were VIADS' ability to filter, summarize, compare, and visualize data.

CLINICALTRIAL

INTERNATIONAL REGISTERED REPORT

RR2-10.2196/39414.

Digital mental health interventions for chronic serious mental illness: Findings from a qualitative study on usability and scale-up of the Life Goals app for bipolar disorder

The Life Goals (LG) application is an evidence-based self-management tool intended to help individuals with bipolar disorder (BD) by aligning symptom coping strategies with personal goals. The program has traditionally been offered in-person or via the web, but has recently been translated into an individualized, customizable mobile intervention to improve access to care and reduce provider burden. The LG app previously showed acceptability with ease of use and satisfaction with user interface, but less success in encouraging self-management. To better understand patient needs, our team conducted semi-structured interviews with 18 individuals with BD who used the LG app for 6 months. These interviews also investigated participant interest in sharing LG app data with their provider through an online dashboard. Using affinity mapping, a collaborative, qualitative data analysis technique, our team identified emerging common themes in the interviews. Through this process, team members identified 494 pieces of salient information from interviews that were mapped and translated into three main findings: (1) many participants found Mood Monitoring and LG modules helpful/interesting and stated the app overall had positive impacts on their mental health, (2) some components of the app were too rudimentary or impersonal to be beneficial, and (3) feedback was mixed regarding future implementation of an LG provider dashboard, with some participants seeing potential positive impacts and others hesitating due to perceived efficacy and privacy concerns. These findings can help researchers improve app-based interventions for individuals with BD by increasing app usage and improving care overall.

The design and development of an experience measure for a peer community moderated forum in a digital mental health service

Online digital mental health communities can contribute to users' mental health positively and negatively. Yet the measurement of experience, outcomes and impact mechanisms relating to digital mental health communities is difficult to capture. In this paper we demonstrate the development of an online experience measure for a specific children and young people's community forum inside a digital mental health service. The development of the Peer Online Community Experience Measure (POCEM) is informed by a multi-phased design: (i) item reduction through Estimate-Talk-Estimate modified Delphi methods, (ii) user testing with think-aloud protocols and (iii) a pilot study within the digital service community to explore observational data within the platform. Experts in the field were consulted to help reduce the items in the pool and to check their theoretical coherence. User testing workshops helped to inform the usability appearance, wording, and purpose of the measure. Finally, the pilot results highlight completion rates, differences in scores for age and roles and "relate to others", as the most frequent domain mechanism of support for this community. Outcomes frequently selected show the importance of certain aspects of the community, such as safety, connection, and non-judgment previously highlighted in the literature. Experience measures like this one could be used as indicators of active therapeutic engagement within the forum community and its content but further research is required to ascertain its acceptability and validity. Multi-phased approaches involving stakeholders and user-centred design activities enhances the development of digitally enabled measurement tools.

A Pilot Study Conducting Online Think Aloud Qualitative Method during Social Distancing: Benefits and Challenges

COVID-19 social distancing restrictions provided unprecedented insights into online research methodologies and approaches for both participants and researchers. Field research traditionally conducted face-to-face had to be transferred online, highlighting the great strides made in communication technologies (particularly live video streaming) over the last two decades for online qualitative research. However, dedicated research on these phenomena is tentative, including with regard to specific methods such as Think Aloud. This paper contributes to literature on online Think Aloud in qualitative research, evaluating new insights on its adoption online. It draws on findings from an online piloting study of Think Aloud tasks to explore the implications of using real-time internet video calls via SoIP applications by MS Teams. To assess the online Think Aloud process, this review called upon some of the comments made by participants during the semi-structured interview or comments made during the Think Aloud process, when they were relevant to the online process itself. It focuses on different dimensions of benefits, rapport in the session’s encounter, challenges, and ethical concerns. Overall, the findings indicate that online Think Aloud sessions cannot completely replace in-person sessions for some particular and highly in-depth research areas, but they can greatly facilitate qualitative data collection in most conventional contexts. It is necessary to carry out further studies exploring the use of this and other online approaches and instructions.

Conducting rapid qualitative interview research during the COVID-19 pandemic-Reflections on methodological choices

As the COVID-19 pandemic has shown, setting up studies in time to gather relevant, real-world data enables researchers to capture current views and experiences, focus on practicalities on the ground, and deliver actionable results. Delivering high quality rapid studies in healthcare poses several challenges even in non-emergency situations. There is an expanding literature discussing benefits and challenges of conducting rapid research, yet there are relatively few examples related to methodological dilemmas and decisions that researchers may face when conducting rapid studies. In rapidly-changing emergency contexts, some of these challenges may be more easily overcome, while others may be unique to the emergency, magnified, or emerge in different ways. In this manuscript, we discuss our reflections and lessons learnt across the research process when conducting rapid qualitative interview studies in the context of a healthcare emergency, focusing on methodological issues. By this we mean the challenging considerations and pragmatic choices we made, and their downstream impacts, that shaped our studies. We draw on our extensive combined experience of delivering several projects during the COVID-19 pandemic in both single and multi-country settings, where we implemented rapid studies, or rapidly adapted an existing study. In the context of these studies, we discuss two main considerations, with a particular focus on the complexities, multiple facets, and trade-offs involved in: (i) team-based approaches to qualitative studies; and (ii) timely and rapid data collection, analysis and dissemination. We contribute a transparent discussion of these issues, describing them, what helped us to deal with them, and which issues have been difficult to overcome. We situate our discussion of arising issues in relation to existing literature, to offer broader recommendations while also identifying gaps in current understandings of how to deal with these methodological challenges. We thus identify key considerations, lessons, and possibilities for researchers implementing rapid studies in healthcare emergencies and beyond. We aim to promote transparency in reporting, assist other researchers in making informed choices, and consequently contribute to the development of the rapid qualitative research.

Artificial intelligence-powered smartphone application, AICaries, improves at-home dental caries screening in children: Moderated and unmoderated usability test

Early Childhood Caries (ECC) is the most common childhood disease worldwide and a health disparity among underserved children. ECC is preventable and reversible if detected early. However, many children from low-income families encounter barriers to dental care. An at-home caries detection technology could potentially improve access to dental care regardless of patients' economic status and address the overwhelming prevalence of ECC. Our team has developed a smartphone application (app), AICaries, that uses artificial intelligence (AI)-powered technology to detect caries using children's teeth photos. We used mixed methods to assess the acceptance, usability, and feasibility of the AICaries app among underserved parent-child dyads. We conducted moderated usability testing (Step 1) with ten parent-child dyads using "Think-aloud" methods to assess the flow and functionality of the app and analyze the data to refine the app and procedures. Next, we conducted unmoderated field testing (Step 2) with 32 parent-child dyads to test the app within their natural environment (home) over two weeks. We administered the System Usability Scale (SUS) and conducted semi-structured individual interviews with parents and conducted thematic analyses. AICaries app received a 78.4 SUS score from the participants, indicating an excellent acceptance. Notably, the majority (78.5%) of parent-taken photos of children's teeth were satisfactory in quality for detection of caries using the AI app. Parents suggested using community health workers to provide training to parents needing assistance in taking high quality photos of their young child's teeth. Perceived benefits from using the AICaries app include convenient at-home caries screening, informative on caries risk and education, and engaging family members. Data from this study support future clinical trial that evaluates the real-world impact of using this innovative smartphone app on early detection and prevention of ECC among low-income children.

Development of a Digital Support Application With Evidence-Based Content for Sustainable Return to Work for Persons With Chronic Pain and Their Employers: User-Centered Agile Design Approach

BACKGROUND

Persons with chronic pain experience a lack of support after completing rehabilitation and the responsibility for the return-to-work (RTW) process is taken over by the employer. In addition, employers describe not knowing how to support their employees. Smartphone apps have been increasingly used for self-management, but there is a lack of available eHealth apps with evidence-based content providing digital support for persons with chronic pain and their employers when they return to work.

OBJECTIVE

This study aims to describe the development of a digital support application with evidence-based content that includes a biopsychosocial perspective on chronic pain for sustainable RTW for persons with chronic pain and their employers (SWEPPE [Sustainable Worker Digital Support for Persons With Chronic Pain and Their Employers]).

METHODS

A user-centered agile design approach was applied. The multidisciplinary project team consisted of health care researchers, a user representative, and a software team. A total of 2 reference groups of 7 persons with chronic pain and 4 employers participated in the development process and usability testing. Mixed methods were used for data collection. The design was revised using feedback from the reference groups. The content of SWEPPE was developed based on existing evidence and input from the reference groups.

RESULTS

The reference groups identified the following as important characteristics to include in SWEPPE: keeping users motivated, tracking health status and work situation, and following progress. SWEPPE was developed as a smartphone app for the persons with chronic pain and as a web application for their employers. SWEPPE consists of six modules: the action plan, daily self-rating, self-monitoring graphs, the coach, the library, and shared information with the employer. The employers found the following functions in SWEPPE to be the most useful: employees' goals related to RTW, barriers to RTW, support wanted from the employer, and the ability to follow employees' progress. The persons with chronic pain found the following functions in SWEPPE to be the most useful: setting a goal related to RTW, identifying barriers and strategies, and self-monitoring. Usability testing revealed that SWEPPE was safe, useful (ie, provided relevant information), logical, and easy to use with an appealing interface.

CONCLUSIONS

This study reports the development of a digital support application for persons with chronic pain and their employers. SWEPPE fulfilled the need of support after an interdisciplinary pain rehabilitation program with useful functions such as setting a goal related to RTW, identification of barriers and strategies for RTW, self-monitoring, and sharing information between the employee and the employer. The user-centered agile design approach contributed to creating SWEPPE as a relevant and easy-to-use eHealth intervention. Further studies are needed to examine the effectiveness of SWEPPE in a clinical setting.

Assessment of an Innovative Mobile Dentistry eHygiene Model Amid the COVID-19 Pandemic in the National Dental Practice-Based Research Network: Protocol for Design, Implementation, and Usability Testing

BACKGROUND

Amid COVID-19, and other possible future infectious disease pandemics, dentistry needs to consider modified dental examination regimens that render quality care, are cost effective, and ensure the safety of patients and dental health care personnel (DHCP). Traditional dental examinations, which number more than 300 million per year in the United States, rely on person-to-person tactile examinations, pose challenges to infection control, and consume large quantities of advanced-level personal protective equipment (PPE). Therefore, our long-term goal is to develop an innovative mobile dentistry (mDent) model that takes these issues into account. This model supplements the traditional dental practice with virtual visits, supported by mobile devices such as mobile telephones, tablets, and wireless infrastructure. The mDent model leverages the advantages of digital mobile health (mHealth) tools such as intraoral cameras to deliver virtual oral examinations, treatment planning, and interactive oral health management, on a broad population basis. Conversion of the traditional dental examinations to mDent virtual examinations builds upon (1) the reliability of teledentistry, which uses intraoral photos and live videos to make diagnostic decisions, and (2) rapid advancement in mHealth tool utilization.

OBJECTIVE

In this pilot project, we designed a 2-stage implementation study to assess 2 critical components of the mDent model: virtual hygiene examination (eHygiene) and patient self-taken intraoral photos (SELFIE). Our specific aims are to (1) assess the acceptance and barriers of mDent eHygiene among patients and DHCP, (2) assess the economic impact of mDent eHygiene, and (3) assess the patient's capability to generate intraoral photos using mHealth tools (exploratory aim, SELFIE).

METHODS

This study will access the rich resources of the National Dental Practice-Based Research Network to recruit 12 dentists, 12 hygienists, and 144 patients from 12 practices. For aims 1 and 2, we will use role-specific questionnaires to collect quantitative data on eHygiene acceptance and economic impact. The questionnaire components include participant characteristics, the System Usability Scale, a dentist-patient communication scale, practice operation cost, and patient opportunity cost. We will further conduct a series of iterative qualitative research activities using individual interviews to further elicit feedback and suggestion for changes to the mDent eHygiene model. For aim 3, we will use mixed methods (quantitative and qualitative) to assess the patient's capability of taking intraoral photos, by analyzing obtained photos and recorded videos.

RESULTS

The study is supported by the US National Institute of Dental and Craniofacial Research. This study received "single" institutional review board approval in August 2021. Data collection and analysis are expected to conclude by December 2021 and March 2022, respectively.

CONCLUSIONS

The study results will inform the logistics of conducting virtual dental examinations and empowering patients with mHealth tools, providing better safety and preserving PPE amid the COVID-19 and possible future pandemics.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/32345.

Assessing a Smartphone App (AICaries) That Uses Artificial Intelligence to Detect Dental Caries in Children and Provides Interactive Oral Health Education: Protocol for a Design and Usability Testing Study

BACKGROUND

Early childhood caries (ECC) is the most common chronic childhood disease, with nearly 1.8 billion new cases per year worldwide. ECC afflicts approximately 55% of low-income and minority US preschool children, resulting in harmful short- and long-term effects on health and quality of life. Clinical evidence shows that caries is reversible if detected and addressed in its early stages. However, many low-income US children often have poor access to pediatric dental services. In this underserved group, dental caries is often diagnosed at a late stage when extensive restorative treatment is needed. With more than 85% of lower-income Americans owning a smartphone, mobile health tools such as smartphone apps hold promise in achieving patient-driven early detection and risk control of ECC.

OBJECTIVE

This study aims to use a community-based participatory research strategy to refine and test the usability of an artificial intelligence-powered smartphone app, AICaries, to be used by children's parents/caregivers for dental caries detection in their children.

METHODS

Our previous work has led to the prototype of AICaries, which offers artificial intelligence-powered caries detection using photos of children's teeth taken by the parents' smartphones, interactive caries risk assessment, and personalized education on reducing children's ECC risk. This AICaries study will use a two-step qualitative study design to assess the feedback and usability of the app component and app flow, and whether parents can take photos of children's teeth on their own. Specifically, in step 1, we will conduct individual usability tests among 10 pairs of end users (parents with young children) to facilitate app module modification and fine-tuning using think aloud and instant data analysis strategies. In step 2, we will conduct unmoderated field testing for app feasibility and acceptability among 32 pairs of parents with their young children to assess the usability and acceptability of AICaries, including assessing the number/quality of teeth images taken by the parents for their children and parents' satisfaction.

RESULTS

The study is funded by the National Institute of Dental and Craniofacial Research, United States. This study received institutional review board approval and launched in August 2021. Data collection and analysis are expected to conclude by March 2022 and June 2022, respectively.

CONCLUSIONS

Using AICaries, parents can use their regular smartphones to take photos of their children's teeth and detect ECC aided by AICaries so that they can actively seek treatment for their children at an early and reversible stage of ECC. Using AICaries, parents can also obtain essential knowledge on reducing their children's caries risk. Data from this study will support a future clinical trial that evaluates the real-world impact of using this smartphone app on early detection and prevention of ECC among low-income children.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/32921.

A Smartphone-Based Self-management Intervention for Bipolar Disorder (LiveWell): User-Centered Development Approach

BACKGROUND

Bipolar disorder is a serious mental illness that results in significant morbidity and mortality. Pharmacotherapy is the primary treatment for bipolar disorder; however, adjunctive psychotherapy can help individuals use self-management strategies to improve outcomes. Yet access to this therapy is limited. Smartphones and other technologies have the potential to increase access to therapeutic strategies that enhance self-management while simultaneously providing real-time user feedback and provider alerts to augment care.

OBJECTIVE

This paper describes the user-centered development of LiveWell, a smartphone-based self-management intervention for bipolar disorder, to contribute to and support the ongoing improvement and dissemination of technology-based mental health interventions.

METHODS

Individuals with bipolar disorder first participated in a field trial of a simple smartphone app for self-monitoring of behavioral targets. To develop a complete technology-based intervention for bipolar disorder, this field trial was followed by design sessions, usability testing, and a pilot study of a smartphone-based self-management intervention for bipolar disorder. Throughout all phases of development, intervention revisions were made based on user feedback.

RESULTS

The core of the LiveWell intervention consists of a daily self-monitoring tool, the Daily Check-in. This self-monitoring tool underwent multiple revisions during the user-centered development process. Daily Check-in mood and thought rating scales were collapsed into a single wellness rating scale to accommodate user development of personalized scale anchors. These anchors are meant to assist users in identifying early warning signs and symptoms of impending episodes to take action based on personalized plans. When users identified personal anchors for the wellness scale, the anchors most commonly reflected behavioral signs and symptoms (40%), followed by cognitive (25%), mood (15%), physical (10%), and motivational (7%) signs and symptoms. Changes to the Daily Check-in were also made to help users distinguish between getting adequate sleep and keeping a regular routine. At the end of the pilot study, users reported that the Daily Check-in made them more aware of early warning signs and symptoms and how much they were sleeping. Users also reported that they liked personalizing their anchors and plans and felt this process was useful. Users experienced some difficulties with developing, tracking, and achieving target goals. Users also did not consistently follow up with app recommendations to contact providers when Daily Check-in data suggested they needed additional assistance. As a result, the human support roles for the technology were expanded beyond app use support to include support for self-management and clinical care communication. The development of these human support roles was aided by feedback on the technology's usability from the users and the coaches who provided the human support.

CONCLUSIONS

User input guided the development of intervention content, technology, and coaching support for LiveWell. Users valued the provision of monitoring tools and the ability to personalize plans for staying well, supporting the role of monitoring and personalization as important features of digital mental health technologies. Users also valued human support of the technology in the form of a coach, and user difficulties with aspects of self-management and care-provider communication led to an expansion of the coach's support roles. Obtaining feedback from both users and coaches played an important role in the development of both the LiveWell technology and human support. Attention to all stakeholders involved in the use of mental health technologies is essential for optimizing intervention development.

Development of a Health Information Technology Tool for Behavior Change to Address Obesity and Prevent Chronic Disease Among Adolescents: Designing for Dissemination and Sustainment Using the ORBIT Model

Health information technology (HIT) has not been broadly adopted for use in outpatient healthcare settings to effectively address obesity in youth, especially among disadvantaged populations that face greater barriers to good health. A well-designed HIT tool can deliver behavior change recommendations and provide community resources to address this gap, and the Obesity-Related Behavioral Intervention Trials (ORBIT) model can guide its development and refinement. This article reports the application of the ORBIT model to (1) describe the characteristics and design of a novel HIT tool (the PREVENT tool) using behavioral theory, (2) illustrate the use of stakeholder-centered "designing for dissemination and sustainability" principles, and (3) discuss the practical implications and directions for future research. Two types of stakeholder engagement (customer discovery and user testing) were conducted with end users (outpatient healthcare teams). Customer discovery interviews (n = 20) informed PREVENT tool components and intervention targets by identifying (1) what healthcare teams (e.g., physicians, dietitians) identified as their most important "jobs to be done" in helping adolescents who are overweight/obese adopt healthy behaviors, (2) their most critical "pains" and "gains" related to overweight/obesity treatment, and (3) how they define success compared to competing alternatives. Interviews revealed the need for a tool to help healthcare teams efficiently deliver tailored, evidence-based behavior change recommendations, motivate patients, and follow-up with patients within the constraints of clinic schedules and workflows. The PREVENT tool was developed to meet these needs. It facilitates prevention discussions, delivers tailored, evidence-based recommendations for physical activity and food intake, includes an interactive map of community resources to support behavior change, and automates patient follow-up. Based on Self-Determination Theory, the PREVENT tool engages the patient to encourage competence and autonomy to motivate behavior change. The use of this intentional, user-centered design process should increase the likelihood of the intended outcomes (e.g., behavior change, weight stabilization/loss) and ultimately increase uptake, implementation success, and long-term results. After initial tool development, user-testing interviews (n = 13) were conducted using a think-aloud protocol that provided insight into users' (i.e., healthcare teams) cognitive processes, attitudes, and challenges when using the tool. Overall, the PREVENT tool was perceived to be useful, well-organized, and visually appealing.

Augmented Reality to Enable Users to Identify Deviations for Model Reconciliation

Effective use of Building Information Modeling (BIM) during operation typically requires modeled content to accurately match the built spaces, which necessitates effective field verification techniques to ensure that BIM content matches the actual built conditions. Some contractors leverage laser scanning and other reality-capture technologies to verify modeled content prior to turnover, but these approaches can be time- and resource-intensive. Augmented reality (AR) enables users to view BIM content overlaid on their field of view of the built space. Research suggests potential for using AR for tasks related to field verification, but a study that systematically explores the specific types of deviations that can be detected with this technology is missing from the current literature. This paper tasks participants with using immersive AR to identify deviations from BIM in a ceiling plenum space that includes installed Mechanical, Electrical and Plumbing (MEP) components, which would typically be included in a coordinated BIM. The results suggest that AR can enable users to effectively identify large deviations and missing building elements. However, the results do not indicate that AR effectively enables users to identify small deviations and can potentially lead to identifying false positive observations, where accurately constructed elements are perceived as deviating from BIM. These results suggest that immersive AR can effectively be used to check whether recently built elements conform to the intended BIM in instances where speed of verification is more important than adhering to strict tolerances. For instances where accuracy (less than two inches) is critical, the results of this work suggest that AR can be used to help determine where in the building more accurate, but more resource-intensive, reality-capture technologies should be used.

Evaluation of a clinical decision support system for rare diseases: a qualitative study

BACKGROUND

Rare Diseases (RDs) are difficult to diagnose. Clinical Decision Support Systems (CDSS) could support the diagnosis for RDs. The Medical Informatics in Research and Medicine (MIRACUM) consortium developed a CDSS for RDs based on distributed clinical data from eight German university hospitals. To support the diagnosis for difficult patient cases, the CDSS uses data from the different hospitals to perform a patient similarity analysis to obtain an indication of a diagnosis. To optimize our CDSS, we conducted a qualitative study to investigate usability and functionality of our designed CDSS.

METHODS

We performed a Thinking Aloud Test (TA-Test) with RDs experts working in Rare Diseases Centers (RDCs) at MIRACUM locations which are specialized in diagnosis and treatment of RDs. An instruction sheet with tasks was prepared that the participants should perform with the CDSS during the study. The TA-Test was recorded on audio and video, whereas the resulting transcripts were analysed with a qualitative content analysis, as a ruled-guided fixed procedure to analyse text-based data. Furthermore, a questionnaire was handed out at the end of the study including the System Usability Scale (SUS).

RESULTS

A total of eight experts from eight MIRACUM locations with an established RDC were included in the study. Results indicate that more detailed information about patients, such as descriptive attributes or findings, can help the system perform better. The system was rated positively in terms of functionality, such as functions that enable the user to obtain an overview of similar patients or medical history of a patient. However, there is a lack of transparency in the results of the CDSS patient similarity analysis. The study participants often stated that the system should present the user with an overview of exact symptoms, diagnosis, and other characteristics that define two patients as similar. In the usability section, the CDSS received a score of 73.21 points, which is ranked as good usability.

CONCLUSIONS

This qualitative study investigated the usability and functionality of a CDSS of RDs. Despite positive feedback about functionality of system, the CDSS still requires some revisions and improvement in transparency of the patient similarity analysis.

Development of a web tool to increase research literacy in underserved populations through public library partnerships

OBJECTIVE

Inadequate diversity in clinical trials is widely recognized as a significant contributing factor to health disparities experienced by racial/ethnic minorities and other diverse populations in the US. To address this in a scalable way, we sought to develop a web tool that could help enhance underserved minority participation in clinical research.

METHODS

We used our research literacy support flashcard tool as the initial prototype for human-centered design and usability testing of the web tool Health for All in public library settings. After forming partnerships with leadership from Chicago Public Libraries (CPL), local medical libraries, and the Chicago Department of Public Health, we conducted seven iterative design sessions with focus groups of library patrons and library staff from six CPL branches serving underserved communities followed by two rounds of usability testing and website modification.

RESULTS

Based on the qualitative research findings from Design Sessions 1-7, we enacted the design decision of a website that was a hybrid of fact-filled and vignette (personal stories) paper prototypes divided into 4 modules (trust, diversity, healthy volunteers, pros/cons), each with their own outcome metrics. The website was thus constructed, and navigation issues identified in two rounds of usability testing by library patrons were addressed through further website modification, followed by the launch of a beta version of a hybridized single-scrolling and guided module prototype to allow further development with website analytics.

CONCLUSIONS

We report the development of Health for All, a website designed to enhance racial/ethnic minority participation in clinical trials by imparting research literacy, mitigating distrust engendered by longstanding racism and discrimination, and providing connections to clinical trials recruiting participants.

A patient-centered knowledge translation tool for treat-to-target strategy in rheumatoid arthritis: Patient and rheumatologist perspectives

AIM

Implementation of treat-to-target (T2T) for rheumatoid arthritis (RA) presents many challenges and an evidence-practice gap has emerged. This study assessed clinician and patient barriers to the implementation of an RA-T2T strategy and developed a knowledge translation (KT) tool for use in "real-life" clinical settings.

METHODS

Surveys of patients and rheumatologists measured agreement with RA-T2T recommendations and use in daily practice. Patient knowledge and perceptions were assessed as was clinician willingness to alter practice and barriers to RA-T2T using visual analog scales. An electronic KT-tool was developed and a two-phase usability trial undertaken to assess use in clinical interactions.

RESULTS

Ninety-one percent of patients had no prior knowledge of RA-T2T but agreed with the recommendations showing mean level agreement scores (8.39-9.54, SD 2.37-1.54). Ninety percent were willing to try RA-T2T, 49% felt their treatment could be improved and 28% wanted more involvement in treatment decisions. Rheumatologists agreed with RA-T2T recommendations (7.30-9.27, SD 2.59-0.91). Barriers to implementation identified by rheumatologists included time, appointment availability and perceived patient reluctance to escalate medications. Usability experiences with the KT-tool were tracked and clinicians reported it was easy to use (100%), resulted in a discussion of RA-T2T (73%) and a target being set for 63% of consults. Patients reported they read (92%) and understood (87%) the information in the KT-tool, and that a target was set in 62% of interactions.

CONCLUSIONS

RA-T2T uptake in clinical practice may be improved through understanding local clinician and patient barriers and an implementation strategy utilizing a patient-driven KT-tool.

Investigation of a Mobile Health Texting Tool for Embedding Patient-Reported Data Into Diabetes Management (i-Matter): Development and Usability Study

Background

Patient-reported outcomes (PROs) are increasingly being used in the management of type 2 diabetes (T2D) to integrate data from patients’ perspective into clinical care. To date, the majority of PRO tools have lacked patient and provider involvement in their development, thus failing to meet the unique needs of end users, and lack the technical infrastructure to be integrated into the clinic workflow.

Objective

This study aims to apply a systematic, user-centered design approach to develop i-Matter (investigating a mobile health [mHealth] texting tool for embedding patient-reported data into diabetes management), a theory-driven, mobile PRO system for patients with T2D and their primary care providers.

Methods

i-Matter combines text messaging with dynamic data visualizations that can be integrated into electronic health records (EHRs) and personalized patient reports. To build i-Matter, we conducted semistructured group and individual interviews with patients with T2D and providers, a design thinking workshop to refine initial ideas and design the prototype, and user testing sessions of prototypes using a rapid-cycle design (ie, design-test-modify-retest).

Results

Using an iterative user-centered process resulted in the identification of 6 PRO messages that were relevant to patients and providers: medication adherence, dietary behaviors, physical activity, sleep quality, quality of life, and healthy living goals. In user testing, patients recommended improvements to the wording and timing of the PRO text messages to increase clarity and response rates. Patients also recommended including motivational text messages to help sustain engagement with the program. The personalized report was regarded as a key tool for diabetes self-management by patients and providers because it aided in the identification of longitudinal patterns in the PRO data, which increased patient awareness of their need to adopt healthier behaviors. Patients recommended adding individualized tips to the journal on how they can improve their behaviors. Providers preferred having a separate tab built into the EHR that included the personalized report and highlighted key trends in patients’ PRO data over the past 3 months.

Conclusions

PRO tools that capture patients’ well-being and the behavioral aspects of T2D management are important to patients and providers. A clinical trial will test the efficacy of i-Matter in 282 patients with uncontrolled T2D.

Trial Registration

ClinicalTrials.gov NCT03652389; https://clinicaltrials.gov/ct2/show/NCT03652389

Primary Care Access to an Online Decision Support Tool is Associated With Improvements in Some Aspects of Pediatric Migraine Care

OBJECTIVE

To evaluate whether primary care provider (PCP) access to an online decision support tool is associated with a change in evidence-based primary care medical management of pediatric migraine.

METHODS

In this prospective observational study, PCPs serving a target community were educated on the availability and use of an online clinical decision support tool that was developed to inform treatment of pediatric migraine. For 9 months before and after implementation of the decision tool, the proportions of children with migraine prescribed evidence-based and contraindicated medications by PCPs in the target region were monitored using electronic medical record query and statistically compared to these same proportions for patients in surrounding (control) regions. Rates of visits to the emergency department for migraine also were tracked pre- and postimplementation as an indirect measure of impact of the decision tool. Provider usage of the decision tool was monitored and summarized using web analytics.

RESULTS

Approximately half (56%) of target region PCPs used the online tool at least once over the project period. Relative to control regions and baseline trends, the proportion of children residing in the target region who were prescribed recommended abortive and preventive medications for treating migraine was statistically significantly higher following implementation of the tool. No significant changes to frequency of emergency care visits for migraine by youth in the target region were observed.

CONCLUSIONS

Availability to PCPs of an online decision support tool for pediatric migraine is associated with a modest change in some aspects of evidence-based medical care.

A Practitioner's Guide to Performing a Holistic Evaluation of Technology-Enhanced Learning in Medical Education

Technology-enhanced learning (TEL) is now a common mode of educational delivery within medical education. Despite this upsurge, there remains a paucity in comprehensive evaluation of TEL efficacy. In order to make meaningful and evidence-informed decisions on 'how' and 'when' to utilise technology within a course, 'useful knowledge' is required to support faculty in these decision-making processes. In this monograph, a series of pragmatic and achievable approaches for conducting a holistic evaluation of a TEL resource intervention are detailed. These suggestions are based on an established TEL evaluation framework, as well as the author's own experience and that of the broader literature. The approaches cover development of an appropriate research question that is based on the availability of existing TEL resources alongside the peer-reviewed literature; the development of an appropriate team as well as recommendations for navigating ethical approval; conducting small-scale quantitative and qualitative measure; and performing a large-scale mixed methods assessment to understand the holistic impact of the TEL resource.

The Use of Technology to Support Precision Health in Nursing Science

PURPOSE

This article outlines how current nursing research can utilize technology to advance symptom and self-management science for precision health and provides a roadmap for the development and use of technologies designed for this purpose.

APPROACH

At the 2018 annual conference of the National Institute of Nursing Research (NINR) Research Centers, nursing and interdisciplinary scientists discussed the use of technology to support precision health in nursing research projects and programs of study. Key themes derived from the presentations and discussion were summarized to create a proposed roadmap for advancement of technologies to support health and well-being.

CONCLUSIONS

Technology to support precision health must be centered on the user and designed to be desirable, feasible, and viable. The proposed roadmap is composed of five iterative steps for the development, testing, and implementation of technology-based/enhanced self-management interventions. These steps are (a) contextual inquiry, focused on the relationships among humans, and the tools and equipment used in day-to-day life; (b) value specification, translating end-user values into end-user requirements; (c) design, verifying that the technology/device can be created and developing the prototype(s); (d) operationalization, testing the intervention in a real-world setting; and (e) summative evaluation, collecting and analyzing viability metrics, including process data, to evaluate whether the technology and the intervention have the desired effect.

CLINICAL RELEVANCE

Interventions using technology are increasingly popular in precision health. Use of a standard multistep process for the development and testing of technology is essential.

Evaluation of a Dental Diagnostic Terminology Subset

The objective of this study was to determine how well a subset of SNODENT, specifically designed for general dentistry, meets the needs of dental practitioners. Participants were asked to locate their written diagnosis for tooth conditions among the SNODENT terminology uploaded into an electronic dental record. Investigators found that 65% of providers’ original written diagnoses were in “agreement” with their selected SNODENT dental diagnostic subset concept(s).

Integrating Taxonomies Into Theory-Based Digital Health Interventions for Behavior Change: A Holistic Framework

Digital health interventions (DHIs) have been emerging in the last decade. Due to their interdisciplinary nature, DHIs are guided and influenced by theories (eg, behavioral theories, behavior change technologies, and persuasive technology) from different research communities. However, DHIs are always coded using various taxonomies and reported in insufficient perspectives. This inconsistency and incomprehensiveness will cause difficulty in conducting systematic reviews and sharing contributions among communities. Therefore, based on existing related work, we propose a holistic framework that embeds behavioral theories, behavior change technique taxonomy, and persuasive system design principles. Including four development steps, two toolboxes, and one workflow, our framework aims to guide DHI developers to design, evaluate, and report their work in a formative and comprehensive way.

Bridging the Gap Between Academic Research and Pragmatic Needs in Usability: A Hybrid Approach to Usability Evaluation of Health Care Information Systems

BACKGROUND

Technology is increasingly embedded into the full spectrum of health care. This movement has benefited from the application of software development practices such as usability testing and agile development processes. These practices are frequently applied in both commercial or operational and academic settings. However, the relative importance placed on rapid iteration, validity, reproducibility, generalizability, and efficiency differs between the 2 settings and the needs and objectives of academic versus pragmatic usability evaluations.

OBJECTIVE

This paper explores how usability evaluation typically varies on key dimensions in pragmatic versus academic settings that impact the rapidity, validity, and reproducibility of findings and proposes a hybrid approach aimed at satisfying both pragmatic and academic objectives.

METHODS

We outline the characteristics of pragmatic versus academically oriented usability testing in health care, describe the tensions and gaps resulting from differing contexts and goals, and present a model of this hybrid process along with 2 case studies of digital development projects in which we demonstrate this integrated approach to usability evaluation.

RESULTS

The case studies presented illustrate design choices characteristic of our hybrid approach to usability evaluation.

CONCLUSIONS

Designed to leverage the strengths of both pragmatically and academically focused usability studies, a hybrid approach allows new development projects to efficiently iterate and optimize from usability data as well as preserves the ability of these projects to produce deeper insights via thorough qualitative analysis to inform further tool development and usability research by way of academically focused dissemination.

A Human-Centered Design Methodology to Enhance the Usability, Human Factors, and User Experience of Connected Health Systems: A Three-Phase Methodology

BACKGROUND

Design processes such as human-centered design, which involve the end user throughout the product development and testing process, can be crucial in ensuring that the product meets the needs and capabilities of the user, particularly in terms of safety and user experience. The structured and iterative nature of human-centered design can often present a challenge when design teams are faced with the necessary, rapid, product development life cycles associated with the competitive connected health industry.

OBJECTIVE

We wanted to derive a structured methodology that followed the principles of human-centered design that would allow designers and developers to ensure that the needs of the user are taken into account throughout the design process, while maintaining a rapid pace of development. In this paper, we present the methodology and its rationale before outlining how it was applied to assess and enhance the usability, human factors, and user experience of a connected health system known as the Wireless Insole for Independent and Safe Elderly Living (WIISEL) system, a system designed to continuously assess fall risk by measuring gait and balance parameters associated with fall risk.

METHODS

We derived a three-phase methodology. In Phase 1 we emphasized the construction of a use case document. This document can be used to detail the context of use of the system by utilizing storyboarding, paper prototypes, and mock-ups in conjunction with user interviews to gather insightful user feedback on different proposed concepts. In Phase 2 we emphasized the use of expert usability inspections such as heuristic evaluations and cognitive walkthroughs with small multidisciplinary groups to review the prototypes born out of the Phase 1 feedback. Finally, in Phase 3 we emphasized classical user testing with target end users, using various metrics to measure the user experience and improve the final prototypes.

RESULTS

We report a successful implementation of the methodology for the design and development of a system for detecting and predicting falls in older adults. We describe in detail what testing and evaluation activities we carried out to effectively test the system and overcome usability and human factors problems.

CONCLUSIONS

We feel this methodology can be applied to a wide variety of connected health devices and systems. We consider this a methodology that can be scaled to different-sized projects accordingly.