Key methodological considerations for usability testing of electronic patient-reported outcome (ePRO) systems

APPROACH e-PROM system: a user-centered development and evaluation of an electronic patient-reported outcomes measurement system for management of coronary artery disease

BACKGROUND

Coronary artery disease (CAD) confers increased risks of premature mortality, non-fatal morbidity, and significant impairment in functional status and health-related quality of life. Routine administration of electronic patient-reported outcome measures (PROMs) and its real time delivery to care providers is known to have the potential to inform routine cardiac care and to improve quality of care and patient outcomes. This study describes a user-centered development and evaluation of the Alberta Provincial Project for Outcomes Assessment (APPROACH) electronic Patient Reported Outcomes Measurement (e-PROM) system. This e-PROM system is an electronic system for the administration of PROMs to patients with CAD and the delivery of the summarized information to their care providers to facilitate patient-physician communication and shared decision-making. This electronic platform was designed to be accessible via web-based and hand-held devices. Heuristic and user acceptance evaluation were conducted with patients and attending care providers.

RESULTS

The APPROACH e-PROM system was co-developed with patients and care providers, research investigators, informaticians and information technology experts. Five PROMs were selected for inclusion in the online platform after consultations with patient partners, care providers, and PROMs experts: the Seattle Angina Questionnaire, Patient Health Questionnaire, EuroQOL, and Medical Outcomes Study Social Support Survey, and Self-Care of Coronary Heart Disease Inventory. The heuristic evaluation was completed by four design experts who examined the usability of the prototype interfaces. User acceptance testing was completed with 13 patients and 10 cardiologists who evaluated prototype user interfaces of the e-PROM system.

CONCLUSION

Both patients and physicians found the APPROACH e-PROM system to be easy to use, understandable, and acceptable. The APPROACH e-PROM system provides a user-informed electronic platform designed to incorporate PROMs into the delivery of individualized cardiac care for persons with CAD.

Development, Feasibility, and Acceptability of the Electronic Patient Benefit Index for Psoriasis in Clinical Practice: Mixed Methods Study

BACKGROUND

Patient-reported outcomes are relevant in clinical practice showing patient benefits, supporting clinicians' decision-making, and contributing to the delivery of high standards of care. Digital monitoring of patient-reported outcomes is still rare. The Patient Benefit Index (PBI) measures benefits and goals from patients' views and may be relevant for regular documentation and shared decision-making.

OBJECTIVE

This study aimed to develop electronic versions of the PBI to examine their feasibility and acceptability in clinical practice for patients with psoriasis.

METHODS

We developed an app and a web version of the existing, valid PBI using focus groups and cognitive debriefings with patients before conducting a quantitative survey on its feasibility and acceptability. Conduction took part in an outpatient dermatology care unit in Germany. Descriptive and subgroup analyses were conducted.

RESULTS

A total of 139 patients completed the electronic PBIs (ePBIs) and took part in the survey. The ePBI was understandable (n=129-137, 92.8%-98.6%) and feasible, for example, easy to read (n=135, 97.1%) and simple to handle (n=137, 98.5%). Acceptability was also high, for example, patients can imagine using and discussing the ePBI data in practice (n=91, 65.5%) and documenting it regularly (n=88, 63.3%). They believe it could support treatment decisions (n=118, 84.9%) and improve communication with their physician (n=112, 81.3%). They can imagine filling in electronic questionnaires regularly (n=118, 84.9%), even preferring electronic over paper versions (n=113, 81.2%). Older and less educated people show less feasibility, but the latter expected the relationship with their physician to improve and would be more willing to invest time or effort.

CONCLUSIONS

The app and web version of the PBI are usable and acceptable for patients offering comprehensive documentation and patient participation in practice. An implementation strategy should consider patients' needs, barriers, and facilitators but also physicians' attitudes and requirements from the health care system.

A scoping review assessing the usability of digital health technologies targeting people with multiple sclerosis

Digital health technologies (DHTs) have become progressively more integrated into the healthcare of people with multiple sclerosis (MS). To ensure that DHTs meet end-users' needs, it is essential to assess their usability. The objective of this study was to determine how DHTs targeting people with MS incorporate usability characteristics into their design and/or evaluation. We conducted a scoping review of DHT studies in MS published from 2010 to the present using PubMed, Web of Science, OVID Medline, CINAHL, Embase, and medRxiv. Covidence was used to facilitate the review. We included articles that focused on people with MS and/or their caregivers, studied DHTs (including mhealth, telehealth, and wearables), and employed quantitative, qualitative, or mixed methods designs. Thirty-two studies that assessed usability were included, which represents a minority of studies (26%) that assessed DHTs in MS. The most common DHT was mobile applications (n = 23, 70%). Overall, studies were highly heterogeneous with respect to what usability principles were considered and how usability was assessed. These findings suggest that there is a major gap in the application of standardized usability assessments to DHTs in MS. Improvements in the standardization of usability assessments will have implications for the future of digital health care for people with MS.

Usability of an Automated System for Real-Time Monitoring of Shared Decision-Making for Surgery: Mixed Methods Evaluation

BACKGROUND

Improving shared decision-making (SDM) for patients has become a health policy priority in many countries. Achieving high-quality SDM is particularly important for approximately 313 million surgical treatment decisions patients make globally every year. Large-scale monitoring of surgical patients' experience of SDM in real time is needed to identify the failings of SDM before surgery is performed. We developed a novel approach to automating real-time data collection using an electronic measurement system to address this. Examining usability will facilitate its optimization and wider implementation to inform interventions aimed at improving SDM.

OBJECTIVE

This study examined the usability of an electronic real-time measurement system to monitor surgical patients' experience of SDM. We aimed to evaluate the metrics and indicators relevant to system effectiveness, system efficiency, and user satisfaction.

METHODS

We performed a mixed methods usability evaluation using multiple participant cohorts. The measurement system was implemented in a large UK hospital to measure patients' experience of SDM electronically before surgery using 2 validated measures (CollaboRATE and SDM-Q-9). Quantitative data (collected between April 1 and December 31, 2021) provided measurement system metrics to assess system effectiveness and efficiency. We included adult patients booked for urgent and elective surgery across 7 specialties and excluded patients without the capacity to consent for medical procedures, those without access to an internet-enabled device, and those undergoing emergency or endoscopic procedures. Additional groups of service users (group 1: public members who had not engaged with the system; group 2: a subset of patients who completed the measurement system) completed user-testing sessions and semistructured interviews to assess system effectiveness and user satisfaction. We conducted quantitative data analysis using descriptive statistics and calculated the task completion rate and survey response rate (system effectiveness) as well as the task completion time, task efficiency, and relative efficiency (system efficiency). Qualitative thematic analysis identified indicators of and barriers to good usability (user satisfaction).

RESULTS

A total of 2254 completed surveys were returned to the measurement system. A total of 25 service users (group 1: n=9; group 2: n=16) participated in user-testing sessions and interviews. The task completion rate was high (169/171, 98.8%) and the survey response rate was good (2254/5794, 38.9%). The median task completion time was 3 (IQR 2-13) minutes, suggesting good system efficiency and effectiveness. The qualitative findings emphasized good user satisfaction. The identified themes suggested that the measurement system is acceptable, easy to use, and easy to access. Service users identified potential barriers and solutions to acceptability and ease of access.

CONCLUSIONS

A mixed methods evaluation of an electronic measurement system for automated, real-time monitoring of patients' experience of SDM showed that usability among patients was high. Future pilot work will optimize the system for wider implementation to ultimately inform intervention development to improve SDM.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR2-10.1136/bmjopen-2023-079155.

Usability testing of a mobile health application to support individuals with active tuberculosis: a mixed methods study

Poor adherence to tuberculosis (TB) treatment leads to further disease transmission, worsened outcomes, and the development of drug resistance. Digital adherence technologies may facilitate a more patient-centered approach for improving TB treatment outcomes than current strategies. The objective of this study was to evaluate and explore improving usability of the TB Treatment Support Tools (TB-TST) mobile application. We used an iterative convergent mixed-method design consisting of two quantitative surveys and a qualitative think-aloud interview. Testing was conducted in three testing cycles consisting of a total of 16 interviews and 26 surveys. Results were thematically analyzed and reported to the development team during weekly team meetings. Participants rated the TB-TSTs application as having high usability and the iterative approach resulted in several refinements to the application in response to participant feedback. These refinements were well received during qualitative interviews but did not result in a statistically significant improvement in usability testing scores between cycles. Using an iterative convergent mixed-method design was an effective method for refining our mHealth application. Data collected from think-aloud interviews, the MAUQ, and the Health-ITUES identified key areas of application design that needed refinement.

Protocol for a mixed-methods study to develop and feasibility test a digital system for the capture of patient-reported outcomes (PROs) in patients receiving chimeric antigen receptor T-cell (CAR-T) therapies (the PRO-CAR-T study)

INTRODUCTION

Chimeric antigen receptor (CAR) T-cell therapies are novel, potentially curative therapies for haematological malignancies. CAR T-cell therapies are associated with severe toxicities, meaning patients require monitoring during acute and postacute treatment phases. Electronic patient-reported outcomes (ePROs), self-reports of health status provided via online questionnaires, can complement clinician observation with potential to improve patient outcomes. This study will develop and evaluate feasibility of a new ePRO system for CAR-T patients in routine care.

METHODS AND ANALYSIS

Multiphase, mixed-methods study involving multiple stakeholder groups (patients, family members, carers, clinicians, academics/researchers and policy-makers). The intervention development phase comprises a Delphi study to select PRO measures for the digital system, a codesign workshop and consensus meetings to establish thresholds for notifications to the clinical team if a patient reports severe symptoms or side effects. Usability testing will evaluate how users interact with the digital system and, lastly, we will evaluate ePRO system feasibility with 30 CAR-T patients (adults aged 18+ years) when used in addition to usual care. Feasibility study participants will use the ePRO system to submit self-reports of symptoms, treatment tolerability and quality of life at specific time points. The CAR-T clinical team will respond to system notifications triggered by patients' submitted responses with actions in line with standard clinical practice. Feasibility measures will be collected at prespecified time points following CAR T-cell infusion. A qualitative substudy involving patients and clinical team members will explore acceptability of the ePRO system.

ETHICS AND DISSEMINATION

Favourable ethical opinion was granted by the Health and Social Care Research Ethics Committee B(HSC REC B) (ref: 23/NI/0104) on 28 September 2023. Findings will be submitted for publication in high-quality, peer-reviewed journals. Summaries of results, codeveloped with the Blood and Transplant Research Unit Patient and Public Involvement and Engagement group, will be disseminated to all interested groups.

TRIAL REGISTRATION NUMBER

ISCTRN11232653.

The administration of the paper and electronic versions of the Manual Ability Measure-36 (MAM-36) and Fatigue Severity Scale (FSS) is equivalent in people with multiple sclerosis

BACKGROUND

The mobile device diffusion has increasingly highlighted the opportunity to collect patient-reported outcomes (PROs) through electronic patient-reported outcomes measurements (ePROMs) during the clinical routine. Despite the ePROMs promises and advantages, the equivalence when a PRO measure is moved from the original paper-and-pencil to the electronic version is still little investigated. This study aims at evaluating equivalence between PROMs and ePROMs self-administration in people with multiple sclerosis (PwMS); in addition, preference of self-administration type was evaluated.

METHODS

The Manual Ability Measure-36 (MAM-36) and Fatigue Severity Scale (FSS) were selected for the equivalence test. The app ABOUTCOME was developed through a user-centered design approach to administer the questionnaires on tablet. Both paper-and-pencil and electronic versions were randomly self-administered. Intrarater reliability between both versions was evaluated through the intraclass correlation coefficient (ICC, excellent for values ≥ 0.75).

RESULTS

Fifty PwMS (35 females) participated to the study (mean age: 54.7±11.0 years, disease course: 27 relapsing-remitting and 23 progressive; mean EDSS: 4.7±1.9; mean disease duration: 13.3±9.5 years). No statistically significant differences were found for the means total scores of MAM-36 (p = 0.61) and FSS (p = 0.78). The ICC value for MAM-36 and FSS was excellent (0.98 and 0.94, respectively). Most of participants preferred the tablet version (84%).

CONCLUSION

The results of the study provide evidence about the equivalence between the paper-and-pencil and electronic versions of PROs administration. In addition, PwMS prefer electronic methods rather than paper because the information can be provided more efficiently and accurately. The results could be easily extended to other MS PROs.

Using a multi-stakeholder co-design process to develop a health service organisation-wide patient reported outcome measure collection system

PURPOSE

Limited examples exist of successful Patient Reported Outcome Measure (PROM) implementation across an entire healthcare organisation. The aim of this study was to use a multi-stakeholder co-design process to develop a PROM collection system, which will inform implementation of routine collection of PROMs across an entire healthcare organisation.

METHODS

Co-design comprised semi-structured interviews with clinicians (n = 11) and workshops/surveys with consumers (n = 320). The interview guide with clinicians focused on their experience using PROMs, preferences for using PROMs, and facilitators/barriers to using PROMs. Co-design activities specific to consumers focused on: (1) how PROMs will be administered (mode), (2) when PROMs will be administered (timing), (3) who will assist with PROMs collection, and (4) how long a PROM will take to complete. Data were analysed using a manifest qualitative content analysis approach.

RESULTS

Core elements identified during the co-design process included: PROMs collection should be consumer-led and administered by someone other than a clinician; collection at discharge from the healthcare organisation and at 3-6 months post discharge would be most suitable for supporting comprehensive assessment; PROMs should be administered using a variety of modes to accommodate the diversity of consumer preferences, with electronic as the default; and the time taken to complete PROMs should be no longer than 5-10 min.

CONCLUSION

This study provides new information on the co-design of a healthcare organisation-wide PROM collection system. Implementing a clinician and patient informed strategy for PROMs collection, that meets their preferences across multiple domains, should address known barriers to routine collection.

Recommendations to address respondent burden associated with patient-reported outcome assessment

Patient-reported outcomes (PROs) are increasingly used in healthcare research to provide evidence of the benefits and risks of interventions from the patient perspective and to inform regulatory decisions and health policy. The use of PROs in clinical practice can facilitate symptom monitoring, tailor care to individual needs, aid clinical decision-making and inform value-based healthcare initiatives. Despite their benefits, there are concerns that the potential burden on respondents may reduce their willingness to complete PROs, with potential impact on the completeness and quality of the data for decision-making. We therefore conducted an initial literature review to generate a list of candidate recommendations aimed at reducing respondent burden. This was followed by a two-stage Delphi survey by an international multi-stakeholder group. A consensus meeting was held to finalize the recommendations. The final consensus statement includes 19 recommendations to address PRO respondent burden in healthcare research and clinical practice. If implemented, these recommendations may reduce PRO respondent burden.

Electronic adaptation and danish cross-cultural translation of PEmb-QoL and VEINES-QoL/Sym for patients with venous thromboembolism

PURPOSE

Most patient-reported outcome (PROs) used in thrombosis research and clinical practice are delivered using technology like online questionnaires. However, only few have undergone formal electronic adaptation from paper to digital versions, threatening the validity and reliability of the PROs. The present study aimed to perform an electronic adaption and cross-cultural translation of two PROs measuring health-related quality of life in a Danish cohort of patients with venous thrombosis (VTE), specifically the VEINES-QoL/Sym questionnaire and the PEmb-QoL questionnaire.

METHODS

The electronic adaption and cross-cultural translation processes followed the international guidelines recommended by ISPOR. The migration of the questionnaires from paper to electronic versions was conducted in the Research Electronic Data Capture (REDCap). Following approval of the electronically adapted and translated versions, a pretest of the questionnaires was performed by cognitive interviewing patients with VTE recruited from a hospital setting.

RESULTS

Nine men and ten women between the age of 19 and 73 years participated in cognitive interviews. The questionnaires were successfully adapted from paper to electronic versions, and during the migration process only a few modifications to the content and format were made. Most comments were related to technicalities, e.g. touch functions and checkboxes. The cross-cultural translation of both questionnaires was satisfactory, as only minor rephrasing was required.

CONCLUSIONS

The original and Danish version of VEINES-QoL/Sym and PEmb-QoL were successfully adapted into electronic versions and are ready to share for REDCap users. Furthermore, the Danish versions of the two questionnaires have shown satisfactory face validity.

Feasibility of an Electronic Patient-Reported Outcome Tool for Screening Distress and Supportive Care Needs of Adolescents and Young Adults with Cancer

Purpose: Although adolescent and young adult (AYA) cancer patients are digital natives and have high digital communication needs, previous studies of screening tools for AYAs have primarily used paper when measuring patient-reported outcomes (PROs). There are no reports on using an electronic PRO (ePRO) screening tool with AYAs. This study evaluated the feasibility of such a tool in clinical settings, and assessed the prevalence of AYAs' distress and supportive care needs. Methods: An ePRO tool based on the Distress Thermometer and Problem List (DTPL)-Japanese (DTPL-J) version for AYAs was implemented in a clinical setting for 3 months. To determine the prevalence of distress and need for supportive care, descriptive statistics were calculated for participant characteristics, selected items, and Distress Thermometer (DT) scores. Response rates, referral rates to an attending physician and other experts, and time required to complete PRO tools were assessed to evaluate feasibility. Results: From February to April 2022, 244 (93.8%) of 260 AYAs completed the ePRO tool based on the DTPL-J for AYAs. Based on a DT cutoff of ≥5, 65 of 244 patients (26.6%) had high distress. Worry (n = 81, 33.2%) was the most frequently selected item. Primary nurses referred 85 (32.7%) patients to an attending physician or other experts. The referral rate resulting from ePRO screening was significantly higher than that after PRO screening (χ2(1) = 17.99, p < 0.001). The average response time did not differ significantly between ePRO and PRO screening (p = 0.252). Conclusion: This study suggests the feasibility of an ePRO tool based on the DTPL-J for AYAs.

Self-reported MeasUrement of Physical and PsychosOcial Symptoms Response Tool (SUPPORT-dialysis): systematic symptom assessment and management in patients on in-centre haemodialysis - a parallel arm, non-randomised feasibility pilot study protocol

INTRODUCTION

Patients with kidney failure experience symptoms that are often under-recognised and undermanaged. These symptoms negatively impact health-related quality of life and are associated with adverse clinical outcomes. Regular symptom assessment, using electronic patient reported outcomes measure (ePROMs) linked to systematic symptom management, could improve such outcomes. Clinical implementation of ePROMs have been successful in routine oncology care, but not used for patients on dialysis. In this study, we describe a pilot study of ePROM-based systematic symptom monitoring and management intervention in patients treated with in-centre haemodialysis.

METHODS AND ANALYSIS

This is a parallel-arm, controlled pilot of adult patients receiving in-centre maintenance haemodialysis. Participants in the intervention arm will complete ePROMs once a month for 6 months. ePROMs will be scored real time and the results will be shared with participants and with the clinical team. Moderate-severe symptoms will be flagged using established cut-off scores. Referral options for those symptoms will be shared with the clinical team, and additional symptom management resources will also be provided for both participants and clinicians. Participants in the control arm will be recruited at a different dialysis unit, to prevent contamination. They will receive usual care, except that they will complete ePROMs without the presentation of results to participants of the clinical team. The primary objectives of the pilot are to assess (1) the feasibility of a larger, randomised clinical effectiveness trial and (2) the acceptability of the intervention. Interviews conducted with participants and staff will be assessed using a content analysis approach.

ETHICS AND DISSEMINATION

Ethical approval for this study was obtained from the University Health Network (REB#21-5199) and the William Osler Health System (#23-0005). All study procedures will be conducted in accordance with the standards of University Health Network research ethics board and with the 1964 Helsinki declaration and its later amendments. Results of this study will be shared with participants, patients on dialysis and other stakeholders using lay language summaries, oral presentations to patients and nephrology professionals. We will also be publishing the results in a peer-reviewed journal and at scientific meetings.

PROTOCOL VERSION

4 (16 November 2022).

TRIAL REGISTRATION NUMBER

NCT05515991.

Real-time monitoring and feedback to improve shared decision-making for surgery (the ALPACA Study): protocol for a mixed-methods study to inform co-development of an inclusive intervention

INTRODUCTION

High-quality shared decision-making (SDM) is a priority of health services, but only achieved in a minority of surgical consultations. Improving SDM for surgical patients may lead to more effective care and moderate the impact of treatment consequences. There is a need to establish effective ways to achieve sustained and large-scale improvements in SDM for all patients whatever their background. The ALPACA Study aims to develop, pilot and evaluate a decision support intervention that uses real-time feedback of patients' experience of SDM to change patients' and healthcare professionals' decision-making processes before adult elective surgery and to improve patient and health service outcomes.

METHODS AND ANALYSIS

This protocol outlines a mixed-methods study, involving diverse stakeholders (adult patients, healthcare professionals, members of the community) and three National Health Service (NHS) trusts in England. Detailed methods for the assessment of the feasibility, usability and stakeholder views of implementing a novel system to monitor the SDM process for surgery automatically and in real time are described. The study will measure the SDM process using validated instruments (CollaboRATE, SDM-Q-9, SHARED-Q10) and will conduct semi-structured interviews and focus groups to examine (1) the feasibility of automated data collection, (2) the usability of the novel system and (3) the views of diverse stakeholders to inform the use of the system to improve SDM. Future phases of this work will complete the development and evaluation of the intervention.

ETHICS AND DISSEMINATION

Ethical approval was granted by the NHS Health Research Authority North West-Liverpool Central Research Ethics Committee (reference: 21/PR/0345). Approval was also granted by North Bristol NHS Trust to undertake quality improvement work (reference: Q80008) overseen by the Consent and SDM Programme Board and reporting to an Executive Assurance Committee.

TRIAL REGISTRATION NUMBER

ISRCTN17951423; Pre-results.

Development of a telemedicine virtual clinic system for remote, rural, and underserved areas using user-centered design methods

Introduction

Virtual clinics are an emerging form of telemedicine which can positively impact healthcare accessibility in remote, rural, and underserved areas. However, a virtual clinic system for these contexts must be designed appropriately. The user-centered design method can be employed to develop an appropriate virtual clinic.

Methods

The development of the virtual clinic used the user-centered design method. First, a situational analysis was conducted to understand the context of public primary healthcare facilities in South Africa. Literature review, observations, and interviews were conducted, which then informed system requirements. A virtual clinic system was then developed and its usability was evaluated with doctors and nurses in a lab setting using healthy participants acting as patients. Doctors and nurses completed system usability scale surveys and provided interview feedback.

Results

The situational analysis revealed 10 key themes which were translated into a problem statement and 10 system requirements. A virtual clinic system was then developed based on these requirements. 5 doctors and 11 nurses were recruited to complete usability testing with the system. The system received an average of 80.6 scores (good to excellent) out of 100 on the system usability scale. Feedback from participants revealed key areas for improvement of the virtual clinic system, as well as opportunities for further implementation.

Conclusion

The developed virtual clinic system demonstrated the application of the user-centered design method to telemedicine technologies for remote, rural, and underserved areas. The positive feedback received from the participants demonstrated the importance of the user-centered design method in developing technologies for enhancing service delivery in health systems. Further work will implement this system in real-world clinical settings.

Development and Formative Evaluation of a Virtual Exercise Platform for a Community Fitness Center Serving Individuals With Physical Disabilities: Mixed Methods Study

BACKGROUND

People with disabilities experience numerous barriers to being physically active, such as transportation issues, a lack of trained exercise professionals who understand disabilities, and facility access. The use of a virtual exercise platform (VEP) may provide an alternative and limit certain barriers.

OBJECTIVE

The aim of this mixed method study was to evaluate user interaction (effectiveness, efficiency, and satisfaction), the strengths and weaknesses of the user interface, and the user experience with a VEP.

METHODS

Participants were recruited from a community fitness facility that offers programs for people with disabilities. Inclusion criteria were being older than 18 years, fluent in English, and availability of internet access. Features of the VEP included articles, prerecorded videos, live Zoom classes, web-based class registration, weekly progress tracking, incentives, and surveys. A one-on-one Zoom session was scheduled with each participant, during which they completed certain tasks: (1) create an account or login, (2) register for class, (3) join class, (4) add to calendar, and (5) complete surveys. As participants completed tasks, quantitative observations (time on task, task success, rate of task completion, and number of errors by users, which determined task difficulty), qualitative observations were made and interviews were conducted at the end of the session. The "concurrent think-aloud" method was encouraged by the moderator to gauge participants' thoughts as they worked through testing. Participants also completed the System Usability Scale (SUS) and Questionnaire for User Interface Satisfaction (QUIS).

RESULTS

A total of 5 people with disabilities (3 male, 2 female), aged 36-78 (mean 54) years, with education levels from high school to PhD, were recruited. Devices used for testing included a laptop (n=3), a Chromebook (n=1), and a desktop (n=1). All participants completed tasks #1 and #2 without errors but could not complete task #4. One participant completed task #5 with difficulty and another completed task #3 with difficulty. The average time to complete each task was: (1) 82 seconds (55-110), (2) 11 seconds (4-21), (3) 9 seconds (5-27), and (4) 921.5 seconds (840-958). The mean SUS score was 86.5/100, and the mean user QUIS score was 8.08 out of 10. Qualitative observations indicated that the system was simple, user-friendly, and accessible.

CONCLUSIONS

People with disabilities reported high usability and user satisfaction with the web-based exercise platform, and the system appears to be an efficient and effective physical activity option.

Let's talk about it: an exploration of the comparative use of three different digital platforms to gather patient-reported outcome measures

BACKGROUND

Patient-reported outcome (PRO) measures provide valuable evidence in clinical trials; however, poor compliance with PRO measures is a notable and long-standing problem, resulting in missing data that potentially impact the interpretation of trial results. Interactive, patient-centric platforms may increase participants' motivation to complete PRO measures over the course of a clinical trial. Thus, the aim of this study was to evaluate and optimize the usability of 3 popular consumer technologies-a traditional app-based interface, a chatbot interface, and a speech-operated interface-that may be used to improve user engagement and compliance with PRO measures.

METHODS

Participants aged 18-75 years from the general United States population tested the usability of 3 ePRO platforms: a traditional app-based interface using Datacubed Health Platform (Datacubed), a web-based chatbot interface using the Orbita platform, and a speech-operated Alexa interface using an Alexa Skill called "My Daily Wellness." The usability of these platforms was tested with 2 PRO measures: the EQ-5D-5 L and the SF-12v2 Health Survey (SF-12v2), Daily recall. Using a crossover design, 3 cohorts of participants tested each ePRO platform daily for 1 week. After testing, interviews were conducted regarding the participants' experience with each platform.

RESULTS

A total of 24 adults participated in the study. The mean age of participants was 45 years (range, 21-71 years), and half were female (n = 12; 50%). Overall, participants prioritized speed, ease of use, and device portability in selecting their preferred platform. The Datacubed app met these criteria and was the preferred platform among most participants (n = 20; 83%). Participants also suggested various modifications to the platforms, such as programmable notifications, adjustable speed, and additional daily reminders.

CONCLUSIONS

These data demonstrate the importance of speed, ease of use, and device portability, features that are currently incorporated in the Datacubed app, in ePRO platforms used in future clinical trials. Additionally, the usability of ePRO platforms may be optimized by adding programmable notifications, adjustable speed, and increased daily reminders. The results of this study may be used to enhance the usability and patient centricity of these platforms to improve user compliance and engagement during clinical trials.

Development and usability testing of an electronic patient-reported outcome (ePRO) solution for patients with inflammatory diseases in an Advanced Therapy Medicinal Product (ATMP) basket trial

BACKGROUND

Electronic patient-reported outcome (ePRO) systems are increasingly used in clinical trials to provide evidence of efficacy and tolerability of treatment from the patient perspective. The aim of this study is twofold: (1) to describe how we developed an electronic platform for patients to report their symptoms, and (2) to develop and undertake usability testing of an ePRO solution for use in a study of cell therapy seeking to provide early evidence of efficacy and tolerability of treatment and test the feasibility of the system for use in later phase studies.

METHODS

An ePRO system was designed to be used in a single arm, multi-centre, phase II basket trial investigating the safety and activity of the use of ORBCEL-C™ in the treatment of patients with inflammatory conditions. ORBCEL-C™ is an enriched Mesenchymal Stromal Cells product isolated from human umbilical cord tissue using CD362+ cell selection. Usability testing sessions were conducted using cognitive interviews and the 'Think Aloud' method with patient advisory group members and Research Nurses to assess the usability of the system.

RESULTS

Nine patient partners and seven research nurses took part in one usability testing session. Measures of fatigue and health-related quality of life, the PRO-CTCAE™ and FACT-GP5 global tolerability question were included in the ePRO system. Alert notifications to the clinical team were triggered by PRO-CTCAE™ and FACT-GP5 scores. Patient participants liked the simplicity and responsiveness of the patient-facing app. Two patients were unable to complete the testing session, due to technical issues. Research Nurses suggested minor modifications to improve functionality and the layout of the clinician dashboard and the training materials.

CONCLUSION

By testing the effectiveness, efficiency, and satisfaction of our novel ePRO system (PROmicsR), we learnt that most people with an inflammatory condition found it easy to report their symptoms using an app on their own device. Their experiences using the PROmicsR ePRO system within a trial environment will be further explored in our upcoming feasibility testing. Research nurses were also positive and found the clinical dashboard easy-to-use. Using ePROs in early phase trials is important in order to provide evidence of therapeutic responses and tolerability, increase the evidence based, and inform methodology development.

TRIAL REGISTRATION

ISRCTN, ISRCTN80103507. Registered 01 April 2022, https://www.isrctn.com/ISRCTN80103507.

Study within a trial of electronic versus paper-based Patient-Reported oUtcomes CollEction (SPRUCE): study protocol for a partially randomised patient preference study

INTRODUCTION

Patient-reported outcomes (PRO) are currently collected from trial participants using paper questionnaires by the Clinical Trials and Statistics Unit at The Institute of Cancer Research (ICR-CTSU). Streamlining PRO collection using electronic questionnaires (ePRO) may improve data collection and patient experience. Here, we outline our protocol for a Study within a trial of electronic versus paper-based Patient-Reported oUtcomes CollEction (SPRUCE), which investigates the acceptability of ePRO in oncology clinical trials.

METHODS AND ANALYSIS

SPRUCE was developed alongside patient and public contributors. SPRUCE runs in multiple host trials with a partially randomised patient preference design, allowing participants to be randomised or choose their preference of electronic or paper questionnaires. Questionnaires are scheduled in accordance with host trial follow-up. The primary objective will assess differences in return rates (compliance) between ePRO and paper PROs at the first timepoint post-host trial intervention in the randomised group. Paper PRO compliance is expected to be 90%. 244 randomised participants are required to exclude ≤80% compliance rates with ePRO (10% non-inferiority margin, with 80% power and one-sided alpha=0.05). SPRUCE aims to assess acceptability of ePRO in oncology clinical trials, establish whether ePRO is acceptable to ICR-CTSU trial participants and can capture complete PRO data, consistent with paper PROs.

ETHICS AND DISSEMINATION

The SPRUCE protocol (ICR-CTSU/2021/10074) was approved by the Coventry and Warwick Central Research Ethics Committee (21/WM/0223) on 21 October 2021. Results will be disseminated via presentations, publications and lay summaries. No participant identifiable data will be included.

TRIAL REGISTRATION

SWAT169.

Electronic Patient-Reported Outcome Data Collection Systems in Oncology Clinical Trials: A Survey of Clinical Research Professionals (an Alliance Study)

PURPOSE

To describe clinical research professionals (CRPs)' experiences with electronic patient-reported outcome (ePRO) data collection systems in oncology clinical trials and identify correlates of CRPs' attitude toward technology.

METHODS

An online survey was conducted among 210 CRPs from 125 National Cancer Institute-funded research sites. Measures included CRPs' demographic characteristics, working years, employment locations, and previous experiences with various types of ePROs. Their attitude toward technology was measured by the Technology Attitude Scale-Adapted. The Wilcoxon signed-rank test was used to compare two subdomains of attitude (perceived usefulness [PU] and perceived ease of use [PEU]). Multiple linear regression was used to explore correlates of (1) overall attitude, (2) PU, and (3) PEU. The significance level was 5%.

RESULTS

Participants' median age was 41 years (range, 21-67). Most were female (90%) and White (82%). More than half of the participants had previous experiences with web-based ePROs using patients' own devices (72%) or site-/sponsor-provided on-site devices (eg, kiosks or tablets; 64%). CRPs who were 60 years or older (β = -0.32, P < .05) or worked for 10-20 years (β = -0.11, P < .05) had relatively negative attitudes, controlling for other factors. Previous experiences with more ePRO types were associated with more positive attitudes (β = 0.08, P = .02). Similar correlates were found with PU but not with PEU.

CONCLUSION

This study revealed that CRPs had various experiences with ePRO systems and attitudes toward technology. Age, working years, and previous experiences with ePROs were correlates of overall attitude toward technology and PU. These findings suggest necessary targeted training to facilitate ePRO use in oncology clinical trials by improving CRPs' awareness and attitude toward technology.

An App to Support Fathers' Mental Health and Well-Being: User-Centered Development Study

BACKGROUND

Numerous studies describe the popularity and usefulness of parenting programs. In particular, parenting programs are generally viewed as effective for supporting parents' mental well-being during key transition periods. However, the evidence base for fathers is limited owing to their lack of involvement in parenting programs and scarcity of tailored support.

OBJECTIVE

This paper aimed to describe the co-design process for a universal digital intervention for fathers (fatherli) and the outline of a logic model with its expected outcomes.

METHODS

Following established guidelines for co-designing and developing complex interventions, we conducted a nonsystematic review of the available literature to gather key information, developed market surveys to assess fathers' needs and interests, consulted with key stakeholders to obtain expert opinions, and engaged in a rapid iterative prototyping process with app developers. Each step was summarized, and the information was collated and integrated to inform a logic model and the features of the resulting intervention.

RESULTS

The steps in the co-design process confirmed a need for and interest in a digital intervention for fathers. In response to this finding, fatherli was developed, consisting of 5 key features: a discussion forum for anyone to post information about various topics (the forum), a socializing platform for fathers to create and engage with others in small groups about topics or points of shared interest (dad hub), a tool for fathers to find other fathers with shared interests or within the same geographic location (dad finder), a resource for fathers to access up-to-date information about topics that interest them (dad wiki), and a portal to book sessions with coaches who specialize in different topics (dad coaching space). The evidence-based logic model proposes that if fatherli is successfully implemented, important outcomes such as increased parental efficacy and mental health help-seeking behaviors may be observed.

CONCLUSIONS

We documented the co-design and development process of fatherli, which confirmed that it is possible to use input from end users and experts, integrated with theory and research evidence, to create suitable digital well-being interventions for fathers. In general, the key findings suggest that an app that facilitates connection, communication, and psychoeducation may appeal to fathers. Further studies will now focus on acceptability, feasibility, and effectiveness. Feedback gathered during pilot-testing will inform any further developments in the app to increase its applicability to fathers and its usability.

Usability testing of an electronic patient-reported outcome system linked to an electronic chemotherapy prescribing and patient management system for patients with cancer

Background

People affected by cancer experience a wide range of symptoms which have a major impact on their functioning and health-related quality of life (HRQoL). One way to measure the impact of cancer symptoms is through the use of patient-reported outcomes.

Methods

An electronic patient-reported outcome (ePRO) application (ChemoPRO®) was designed to be used by cancer patients to report their symptoms and communicate with their clinical team. Usability testing sessions were conducted with people with lived experience of cancer to understand how real users interact with the ChemoPRO® system. One-to-one testing sessions were conducted to assess use of the system and identify areas for further refinement. User satisfaction was assessed using a brief satisfaction questionnaire previously used by Aiyegbusi et al. (date).

Results

Ten people with lived experience of cancer took part in the usability study. Symptoms and HRQoL measures, including the Euroqol EQ5D5L and the PRO-CTCAE™ were included in the ePRO system.

Participants

had a mean age of 62.3 years. Three critical errors and 21 non-critical errors were reported. All participants were enthusiastic about the app. Participants liked the simplicity and responsiveness of the patient-facing app and highlighted the potential for communicating with their clinical team. The overall usability and satisfaction score was 4.5 (sd = 0.09).

Conclusion

This usability study suggests that people with lived experience of cancer found the ChemoPRO® app acceptable and easy to use. One of the key features of this particular ePRO system that should be developed further is system functionality to facilitate communication between patients and clinicians. Future testing should include testing in a clinical setting and testing with people from ethnic minorities.

Patient-Reported Outcomes, Digital Health, and the Quest to Improve Health Equity

The theme of the 2023 American Society of Clinical Oncology Annual Meeting is Partnering With Patients: The Cornerstone of Cancer Care and Research. As we aim to partner with patients to improve their health care, digital tools have the potential to enhance patient-centered cancer care and make clinical research more accessible and generalizable. Using electronic patient-reported outcomes (ePROs) to collect patients' reports of symptoms, functioning, and well-being facilitates patient-clinician communication and improves care and outcomes. Early studies suggest that racial and ethnic minority populations, older patients, and patients with less education may benefit even more from ePRO implementation. Clinical practices looking to implement ePROs can refer to the resources of the PROTEUS Consortium (Patient-Reported Outcomes Tools: Engaging Users & Stakeholders). Beyond ePROs, in response to the COVID-19 pandemic, cancer practices have rapidly adopted other digital tools (eg, telemedicine, remote patient monitoring). As implementation grows, we must be aware of the limitations of these tools and implement them in ways to promote optimal function, access, and ease of use. Infrastructure, patient, provider, and system-level barriers need to be addressed. Partnerships across all levels can inform development and implementation of digital tools to meet the needs of diverse groups. In this article, we describe how we use ePROs and other digital health tools in cancer care, how digital tools can expand access to and generalizability of oncology care and research, and prospects for broader implementation and use.

Differences in Center for Epidemiologic Studies Depression Scale, Generalized Anxiety Disorder-7 and Kessler Screening Scale for Psychological Distress Scores between Smartphone Version versus Paper Version Administration: Evidence of Equivalence

The use of electronic patient-reported outcomes has increased recently, and smartphones offer distinct advantages over other devices. However, previous systematic reviews have not investigated the reliability of the Center for Epidemiologic Studies Depression Scale (CES-D), Generalized Anxiety Disorder-7 (GAD-7), and Kessler Screening Scale for Psychological Distress (K6) when used with smartphones, and this has not been fully explored. This study aimed to evaluate the equivalence of the paper and smartphone versions of the CES-D, GAD-7, and K6, which were compared following a randomized crossover design method in 100 adults in Gunma, Japan. Participants responded to the paper and smartphone versions at 1-week intervals. The equivalence of paper and smartphone versions was evaluated using the intraclass correlation coefficient (ICC_agreement). The mean participant age was 19.86 years (SD = 1.08, 23% male). The ICC_agreements for the paper and smartphone versions of the CES-D, GAD-7, and K6 were 0.76 (95% confidence interval [CI] 0.66-0.83), 0.68 (95% CI 0.59-0.77), and 0.83 (95% CI 0.75-0.88), respectively. Thus, the CES-D and K6 scales are appropriate for use in a smartphone version, which could be applied to clinical and research settings in which the paper or smartphone versions could be used as needed.

'Why should I fill out this questionnaire?' A qualitative study of cancer patients' perspectives on the integration of e-PROMs in routine clinical care

PURPOSE

There is increasing interest in integrating electronic patient-reported outcomes (e-PROs) into clinical routines in cancer settings. However, little is known about patients' experiences with and perceptions of e-PRO measures (e-PROMs). This study examines patients' experiences with e-PROMS, particularly their perspectives about its usefulness and its implications for the clinical encounter with their doctors.

METHOD

A total of 19 individual in-person interviews with cancer patients at a Comprehensive Cancer Center in northern Italy conducted in 2021 inform this study.

RESULTS

The findings suggested that, overall, patients had positive attitudes towards data collection using e-PROMs. On the one hand, most patients found the integration of e-PROMs into routine clinical practice as beneficial in treating patients with cancer. The main benefits of e-PROMs according to this group of patients were that: they promoted patient-centred care; could be used to tailor and improve the quality of care through a holistic approach; supported early detection of problematic symptoms; increased patient self-awareness; and contributed to clinical research. On the other hand, many patients did not fully understand the purpose of e-PROMs and some patients were also sceptical about their usefulness in routine clinical practice.

CONCLUSIONS

These findings have several practical implications for ensuring the successful implementation of e-PROMs in routine clinical practice. These include the following preconditions: patients are informed about the purposes of data collection; physicians provide feedback to patients about the e-PROMs' results; and that hospital administrators allocate sufficient time for clinical interactions to integrate e-PROMs into routine clinical practice.

Graphical user interface design to improve understanding of the patient-reported outcome symptom response

BACKGROUND

Symptom monitoring application (SMA) has clinical benefits to cancer patients but patients experience difficulties in using it. Few studies have identified which types of graphical user interface (GUI) are preferred by cancer patients for using the SMA.

METHODS

This is a cross-sectional study aimed to identify preferred GUI among cancer patients to use SMA. Total of 199 patients were asked to evaluate 8 types of GUIs combining text, icon, illustration, and colors using mixed-methods. Subgroup analyses were performed according to age and gender.

RESULTS

The mean age of the patients was 57 and 42.5% was male. The most preferred GUI was "Text + Icon + Color" (mean = 4.43), followed by "Text + Icon" (mean = 4.39). Older patients (≥ 60 years) preferred "Text + Icon" than younger patients (p for interaction < 0.01). Simple and intuitive text and icons were the most useful GUI for cancer patients to use the SMA.

CONCLUSION

Simple and intuitive text and icons were the most useful GUI for cancer patients to use the SMA. Researchers need to be careful when applying realistic face drawings to cancer symptom monitoring applications because they can recall negative images of cancer.

Feasibility of electronic patient-reported outcome monitoring and self-management program in aplastic anemia and paroxysmal nocturnal hemoglobinuria-a pilot study (ePRO-AA-PNH)

INTRODUCTION

Electronic patient-reported outcomes (ePRO) are increasingly recognized in health care, as they have been demonstrated to improve patient outcomes in cancer, but have been less studied in rare hematological diseases. The aim of this study was to develop and test the feasibility of an ePRO system specifically customized for aplastic anemia (AA) and paroxysmal nocturnal hemoglobinuria (PNH).

METHODS

After performing a user-centered design evaluation an ePRO system for AA and PNH patients could be customized and the application was tested by patients and their medical teams for 6 months. Symptom-reporting triggered self-management advice for patients and prompts them to contact clinicians in case of severe symptoms, while the medical team received alerts of severe symptoms for patient care.

RESULTS

All nine included patients showed a high adherence rate to the weekly symptom-reporting (72%) and reported high satisfaction. The system was rated high for usage, comprehensibility, and integration into daily life. Most patients (78%) would continue and all would recommend the application to other AA/PNH patients. Technical performance was rarely a barrier and healthcare providers saw ePRO-AA-PNH as a useful supplement, but the lacking integration into the hospital information system was identified as a major barrier to usage.

CONCLUSION

An ePRO system customized for AA and PNH was feasible in terms of adherence, satisfaction, and performance, showing a high potential for these rare conditions in terms of data collection and patient guidance. However, the integration into clinical workflows is crucial for further routine use.

TRIAL REGISTRATION

ClinicalTrials.gov NCT04128943.

Systematic surveillance of patient-reported symptoms of viral respiratory tract infectious Syndromes in diverse populations

BACKGROUND

Patient reported outcome measures (PROM) can improve patient care and be crucial for symptom tracking especially during disease outbreaks. FLU-PRO Plus is a validated PROM used to track viral respiratory symptoms. Our study aimed to evaluate the feasibility of using FLU-PRO© Plus, to track symptoms across three healthcare systems.

METHODS

The prospective, longitudinal study recruited adults between February-May 2021 from HealthPartners Institute (HP), Kaiser Permanente Georgia (KPGA), and Kaiser Permanente Mid-Atlantic States (KPMAS). Adult members were eligible if they had a positive lab or diagnosis for either COVID-19 or influenza-like illness (ILI) or exhibited 2 + viral respiratory symptoms. Descriptive statistics were calculated to describe the patient characteristics for participants that were eligible for FLU-PRO Plus, successfully contacted, attempted to log in to the FLU-PRO Plus website, and participants who completed FLU-PRO Plus Day 1. Bivariable and multivariable logistic regression using PROC GLIMMIXX investigated the patient characteristics associated with (1) successful contact and (2) FLU-PRO Plus Day 1 completion.

RESULTS

We identified a total of 15,650 eligible participants during the enrollment period: 9,582 from HP, 1,740 from KPGA, and 4,328 from KPMAS. Among the total of 409 eligible adults who attempted to participate in FLU-PRO Plus, 317 completed FLU-PRO Plus Day 1. Among the 317 individuals that completed FLU-PRO Plus Day 1, 205 (67.5%) were diagnosed with COVID-19; 112 adults diagnosed with COVID-19 completed FLU-PRO Plus Day 14. Among adults successfully contacted, adults aged 35-64 (OR = 1.40, 95% CI 1.05, 1.87), females (OR = 1.77, 95% CI 1.38, 2.27), and adults diagnosed with COVID-19 (OR = 1.66, 95% CI 1.27, 2.17) had higher odds of completing FLU-PRO Plus Day 1; Asian adults (OR = 0.38, 95% CI 0.19, 0.76) and Black and African American adults (OR = 0.33, 95% CI 0.19, 0.76) had lower odds compared to White adults.

CONCLUSION

Our study reports on the feasibility of patients across three integrated healthcare systems utilizing FLU-PRO Plus to monitor their respiratory symptoms. Patient reported outcome measures (PROM) can improve patient care, quality of life, and reduce the strain of limited resources on healthcare systems. Future FLU-PRO Plus studies should develop an implementation strategy to fully integrate FLU-PRO Plus within clinical care and patient management.

The Vision Correction Questionnaire: an electronic patient-reported outcome measure for refractive surgery

PURPOSE

To develop a psychometrically robust electronic patient reported outcome measure (ePROM) for refractive surgery.

SETTING

Moorfields Eye Hospital, London, United Kingdom.

DESIGN

Questionnaire development study.

METHODS

Items were derived in 6 domains (spectacle dependence, visual quality, eye comfort, functional freedom, emotional wellbeing, and satisfaction with treatment) from existing Rasch adjusted instruments, patient and surgeon feedback, and refinement in semistructured interviews before administration to a field test cohort (n = 360) of patients undergoing routine refractive surgery. Spectacle dependence and satisfaction with treatment items were used to provide descriptive statistical information only. Contemporary criteria for item reduction and Rasch modeling were applied to the remaining domains. The finalized questionnaire was then administered to a second patient cohort (n = 120) before and after surgery to assess sensitivity to change.

RESULTS

A 5-item scale derived for emotional wellbeing was unidimensional and a good fit to the Rasch model with ordered category response profiles, adequate precision (person separation 2.22 logits, reliability coefficient 0.83), and no misfitting items. Mean logit scores were 0.91 higher after treatment (effect size 1.26) suggesting a positive impact on emotional wellbeing. Functional scales could not be derived for visual quality, eye comfort, or functional freedom. Single-item ratings for visual quality and eye comfort were retained in our final 11-item questionnaire.

CONCLUSIONS

This short ePROM should integrate well with routine clinical care and clinical trials in refractive surgery. The Rasch adjusted emotional wellbeing scale may help quantify the way patients feel about refractive surgery, with the remaining items providing useful descriptive information.

Digital Future of Emergency Medical Services: Envisioning and Usability of Electronic Patient Care Report System

Despite the efforts of emerging technologies in the healthcare system, there is still a slower rate of acceleration in prehospital settings compared with the hospitals in digital transformation adaptation. The acknowledgment that digital transformation is significant to healthcare is reflected in planning for the future of digital healthcare. Thus, this study aimed to measure the usability of the electronic patient care report (ePCR) system among emergency medical services (EMS) staff who work in prehospital settings. A descriptive cross-sectional correlation study was used. Two hundred fifty EMS staff who are working in the prehospital setting at Saudi Red Crescent Authority in the Kingdom of Saudi Arabia were surveyed, and the response rate was 79.2% (198). An adapted tool of the Computer System Usability Questionnaire survey was used to collect data. The data were coded numerically and subjected to descriptive and inferential statistical analysis including Pearson’s correlation coefficient using the statistical software (SPSS 21). The majority of the participants rate their ePCR system as “useable” at a high level with a score of 3.41 (SD = 1.021). The overall mean of the ePCR system’s three subscales: system usefulness, information quality, interface quality, and overall satisfaction were 3.39 (SD = 1.152), 3.30 (SD = 1.052), 3.57 (SD = 1.064), and 3.37 (SD = 1.239), respectively. The least liked aspect of ePCR system software was information quality 81 (40.9%). Furthermore, there was a significant correlation between the age of EMS staff and the usability of the ePCR system (r = −0.150 ${\hspace{0.17em}}^{\ast }$ , $P=0.035$ ). The results suggest that healthcare institutions’ policy and decision-makers pay close attention to performing standardized training for the staff on their ePCR system before going to the field to increase efficiency and productivity. Furthermore, the users in this study identified other system features that, if included, could have enhanced usability, and improved functions and capabilities of the design to meet the EMS staff’s expectations.

Collection and Utilization of Patient-Reported Outcome Measures in a Colorectal Surgery Clinic

INTRODUCTION

The routine collection of patient-reported outcome measures (PROMs) promises to improve patient care. However, in colorectal surgery, PROMs are uncommonly collected outside of clinical research studies and rarely used in clinical care. We designed and implemented a quality improvement project with the goals of routinely collecting PROMs and increasing the frequency that PROMs are utilized by colorectal surgeons in clinical practice.

METHODS

This mixed-methods, quality improvement project was conducted in the colorectal surgery clinic of a tertiary academic medical center. Patients were administered up to five PROMs before each appointment. PROM completion rates were measured. Additionally, we performed two educational interventions to increase utilization of our electronic health record's PROM dashboard by colorectal surgeons. Utilization rates and attitudes toward the PROM dashboard were measured.

RESULTS

Overall, patients completed 3600 of 3977 (90.9%) administered PROMs during the study period. At baseline, colorectal surgeons reviewed 6.7% of completed PROMs. After two educational interventions, this increased to 39.3% (P = 0.004). Colorectal surgeons also felt that the PROM dashboard was easier to use. Barriers to greater PROM dashboard utilization included poor user interface/user experience and a perceived lack of knowledge, time, and relevance.

CONCLUSIONS

The collection of PROMs in colorectal surgery clinics is feasible and can result in high PROM completion rates. Educational interventions can improve the utilization of PROMs by colorectal surgeons in clinical practice. Our experience collecting PROMs through this quality improvement initiative can serve as a template for other colorectal surgery clinics interested in collecting and utilizing data from PROMs.

Remote Moderator and Observer Experiences and Decision-making During Usability Testing of a Web-Based Empathy Training Portal: Content Analysis

Background

COVID-19 restrictions severely curtailed empirical endeavors that involved in-person interaction, such as usability testing sessions for technology development. Researchers and developers found themselves using web-based moderation for usability testing. Skilled remote moderators and observers are fundamental in this approach. However, to date, more empirical work is needed that captures the perceptions and support needs of moderators and observers in testing situations.

Objective

The aim of this paper was to identify remote moderator and observer participant experiences and their use of certain tools to capture feedback of users as they interact with the web browser application.

Methods

This research is part of a broader study on an educational web browser application for nursing students to learn perspective taking and enhance their perceptual understanding of a dialogue partner’s thoughts and feelings. The broader study used a quantitative and think-aloud qualitative problem-discovery usability study design. This case study explored written accounts of the remote moderator and observer participants regarding their roles, experiences, and reactions to the testing protocol and their suggestions for improved techniques and strategies for conducting remote usability testing. Content analysis was used to analyze participants’ experiences in the usability testing sessions.

Results

We collected data from 1 remote moderator and 2 remote observers. Five themes were identified: dealing with personal stressors, dealing with user anxiety, maintaining social presence, ethical response to the study protocol, and communication during sessions. The participants offered recommendations for the design of future remote testing activities as well as evidence-informed training materials for usability project personnel.

Conclusions

This study’s findings contribute to a growing body of endeavors to understand human-computer interaction and its impact on remote moderator and observer roles. As technology rapidly advances, more remote usability testing will occur where the knowledge gleaned in this study can have an impact. Recommendations based on moderator and observer participant perspectives identify the need for more evidence-informed training materials for their roles that focus on web-based interpersonal communication skills, execution of user testing protocols, troubleshooting technology and test user issues, proficiency in web conferencing platforms, behavior analysis and feedback technologies, and time management.

Smart Textile Sock System for Athletes’ Self-Correction during Functional Tasks: Formative Usability Evaluation

(1) Background: The development of a lightweight, easy-to-use system that measures the foot’s plantar pressure is becoming an increasingly important area of research in physiotherapy. For further development of the smart sock system, a formative usability study was conducted, where the smart textile sock sensor system was used for self-correction during functional tasks; (2) Methods: Five athletes from the football school participated in the formative usability study. Athletes performed pre-defined functional tasks for self-correction when interacting with the smart textile sock system. Formative usability evaluation methods: effectiveness (task success rate, error rate), efficiency (time-based), satisfaction evaluated by System Usability Scale (SUS); (3) Results: Formative usability indicators: task completeness effectiveness ranged from 40% to 100% in the first- and second-stage tasks. Completed task efficiency time: Stage 1, from 4.2 s (SD 1.3) to 88.8 s (SD 19.8); Stage 2, from 7.2 s (SD 1.9) to 9.6 s (SD 2.1). Satisfaction was assessed by the SUS system user group with 76 points (SD 7.42), which indicates “good” satisfaction; (4) Conclusions: formative usability indicators showed the need for technical improvements to the smart textile sock pressure sensor system. The SUS results indicate “good” satisfaction with the smart textile sock pressure sensor system and its application.

Interprofessional Collaboration and Diabetes Management in Primary Care: A Systematic Review and Meta-Analysis of Patient-Reported Outcomes

The global spread of diabetes poses serious threats to public health requiring a patient-centered approach based both on interprofessional collaboration (IPC) given by the cooperation of several different health professionals, and patients’ perspective through the assessment of Patient-Reported Outcomes (PROs). The aim of the present study is to evaluate the impact of interprofessional collaboration interventions, for the management of type 2 diabetes in primary care settings, through PROs. A systematic review and meta-analysis was conducted querying the PubMed, Scopus and Embase databases. Out of the 1961 papers initially retrieved, 19 met the inclusion criteria. Interprofessional collaboration is significantly associated with an increase in both patient’s satisfaction (SMD 0.32 95% CI 0.05–0.59) and in the mental well-being component of the HRQoL (SMD 0.18; 95% CI 0.06–0.30), and there was also promising evidence supporting the association between an interprofessional approach and an increase in self-care and in generic and specific quality-of-life. No statistical differences were found, supporting the positive impact on IPC interventions on the physical component of the HRQoL, depression, emotional distress, and self-efficacy. In conclusion, the effect of IPC impacts positively on the few areas assessed by PROMs. Policymakers should promote the widespread adoption of a collaborative approach as well as to endorse an active engagement of patients across the whole process of care.

Usability evaluation of the Computer-Based Health Evaluation System (CHES) eDiary for patients with faecal incontinence: a pilot study

Background

Faecal incontinence (FI) is prevalent in 15–20% of elderly individuals and is frequently monitored in clinical trials and practice. Bowel diaries are the most common way to document FI, but, in clinical practice, are mainly used as paper-based versions. Electronic diaries (eDiaries) offer many potential benefits over paper-based diaries. The aim of this study was to develop and test an eDiary to document FI.

Methods

We migrated a paper FI diary to an eDiary app based on the Computer-based Health Evaluation System (CHES). To assess usability, we conducted functionality and usability tests at two time points in a sample of patients with FI. In the first assessment, the eDiary functionalities were tested, patients completed the System Usability Scale (SUS, range 0–100) and compared the paper diary with the eDiary. We set a threshold for minimum acceptable average usability at 70 points. Patients were then instructed to use the eDiary for 2 days at home and contacted to report on their usage and completed the SUS a second time.

Results

We recruited a sample of N = 14 patients to use the eDiary. All patients were able to use all functionalities of the eDiary and only a few patients with lower technological literacy or access to devices (n = 3) needed initial assistance. The mean usability rating given at the first time point was high with 88 points (SD 18, 95% CI 78.2–96.8) and most patients (n = 10) reported they would prefer the eDiary over the paper-based version. Nine patients (n = 9) participated in the follow-up assessment and the mean SUS rating at the second time point was 97 points (SD 7, 95% CI 92.8–100).

Conclusion

The eDiary showed excellent usability scores for the assessment of FI at both assessments. Generally, patients preferred the eDiary over the paper-based version. We recommend the eDiary for usage with patients who own and use a smartphone and discuss potential solutions for patients with lower technological literacy or access.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12911-022-01818-5.

Evaluation of an Electronic Medical Record Module for Nursing Documentation in Paediatric Palliative Care: Involvement of Nurses with a Think-Aloud Approach

BACKGROUND

Paediatric palliative care (PPC) is a noncurative approach to the care of children and adolescents with life-limiting and life-threatening illnesses. Electronic medical records (EMRs) play an important role in documenting such complex processes. Despite their benefits, they can introduce unintended consequences if future users are not involved in their development.

AIM

The aim of this study was to evaluate the acceptance of a novel module for nursing documentation by nurses working in the context of PPC.

METHODS

An observational study employing concurrent think-aloud and semi-structured qualitative interviews were conducted with 11 nurses working in PPC. Based on the main determinants of the unified theory of acceptance and use of technology (UTAUT), data were analysed using qualitative content analysis.

RESULTS

The main determinants of UTAUT were found to potentially influence acceptance of the novel module. Participants perceived the module to be self-explanatory and intuitive. Some adaptations, such as the reduction of fragmentation in the display, the optimization of confusing mouseover fields, and the use of familiar nursing terminology, are reasonable ways of increasing software adoption.

CONCLUSIONS

After adaptation of the modules based on the results, further evaluation with the participation of future users is required.

Upper Digestive Disease App for Electronic Patient Reported Outcomes: A Mixed Methods Pilot Study

BACKGROUND

Patient reported outcomes (PROs) can enable communication of symptoms, function, and quality of life. Mobile devices capture PROs electronically (ePRO). The purpose of this study was to determine the feasibility and acceptability of collecting ePROs in esophagectomy patients using the Upper Digestive Disease application (UDD App^TM).

METHODS

A single group, mixed methods design was used to address the study purposes. A convenience sample of esophagectomy patients were recruited after resumption of an oral diet from January 2020-December 2020. Demographic (age, gender) and clinical data (surgical procedure) were obtained after informed consent. Participants utilized the the UDD App^TM for one year, followed by scripted telephone interviews. Descriptive statistics and thematic analyses of the interviews were the primary data analyses.

RESULTS

Sixty-four patients were enrolled (80% male, 62.9 + 12 years old). Of these, 50 patients initiated 108 evaluations via the UDD App^TM with 98% completing the questionnaire on the first attempt indicating feasibility. A subset of 32 patients participated in the scripted telephone interview. Participants (74%) reported high computer literacy; all reported using the UDD App^TM was easier than or equivalent to a paper form. Interview themes revealed value of the app for identifying problems and enhanced communication with providers and caregivers, with an overall appreciation for the convenience of remote monitoring, but concerns about data privacy.

CONCLUSIONS

The UDD App^TM is feasible and acceptable for collecting ePROs in esophagectomy patients. Future work will determine if the UDD App^TM improves symptoms, function, and quality of life.

Electronic Patient-Reported Outcomes in Nephrology: Focus on Hemodialysis

The success of hemodialysis (HD) treatments has been evaluated using objective measures of analytical parameters, or machine-measured parameters, despite having available validated instruments that assess patient perspective. There is an emerging interest regarding the use and relevance of patient-related outcomes (PROs). Electronic PROs (ePROs) involve the use of electronic technology, provide rapid access to this information, and are becoming more widely used in clinical trials and studies to evaluate efficacy and safety. Despite the scarce literature, this review suggests that ePROs are useful in providing a more customized and multidimensional approach to patient management and in making better clinical decisions in relevant aspects such as vascular access, duration and frequency of dialysis sessions, treatment of anemia, mental health, fatigue, and quality of life. The purpose of this review is to raise interest in the systematic use of ePROs in HD and to promote the development of studies in this field, which can respond to the gaps in knowledge and contribute to the implementation of the use of ePROs through new technologies, helping to improve the quality of health care.

Participatory Design of a Medication Module in an Electronic Medical Record for Paediatric Palliative Care: A Think-Aloud Approach with Nurses and Physicians

BACKGROUND

Electronic medical records (EMRs) play a key role in improving documentation and quality of care in paediatric palliative care (PPC). Inadequate EMR design can cause incorrect prescription and administration of medications. Due to the fact of complex diseases and the resulting high level of medical complexity, patients in PPC are vulnerable to medication errors. Consequently, involving users in the development process is important. Therefore, the aim of this study was to evaluate the acceptance of a medication module from the perspective of potential users in PPC and to involve them in the development process.

METHODS

A qualitative observational study was conducted with 10 nurses and four physicians using a concurrent think-aloud protocol and semi-structured qualitative interviews. A qualitative content analysis was applied based on a unified theory of acceptance and use of technology.

RESULTS

Requirements from the user's perspective could be identified as possible influences on acceptance and actual use. Requirements were grouped into the categories "performance expectancies" and "effort expectancies".

CONCLUSIONS

The results serve as a basis for further development. Attention should be given to the reduction of display fragmentation, as it decreases cognitive load. Further approaches to evaluation should be taken.

An Electronic Patient-Reported Outcome Mobile App for Data Collection in Type A Hemophilia: Design and Usability Study

Background

There is currently limited evidence on the level and intensity of physical activity in individuals with hemophilia A. Mobile technologies can offer a rigorous and reliable alternative to support data collection processes but they are often associated with poor user retention. The lack of longitudinal continuity in their use can be partly attributed to the insufficient consideration of stakeholder inputs in the development process of mobile apps. Several user-centered models have been proposed to guarantee that a thorough knowledge of the end user needs is considered in the development process of mobile apps.

Objective

The aim of this study is to design and validate an electronic patient-reported outcome mobile app that requires sustained active input by individuals during POWER, an observational study that aims at evaluating the relationship between physical activity levels and bleeding in patients with hemophilia A.

Methods

We adopted a user-centered design and engaged several stakeholders in the development and usability testing of this mobile app. During the concept generation and ideation phase, we organized a need-assessment focus group (FG) with patient representatives to elicit specific design requirements for the end users. We then conducted 2 exploratory FGs to seek additional inputs for the app’s improvement and 2 confirmatory FGs to validate the app and test its usability in the field through the mobile health app usability questionnaire.

Results

The findings from the thematic analysis of the need-assessment FG revealed that there was a demand for sense making, for simplification of app functionalities, for maximizing integration, and for minimizing the feeling of external control. Participants involved in the later stages of the design refinement contributed to improving the design further by upgrading the app’s layout and making the experience with the app more efficient through functions such as chatbots and visual feedback on the number of hours a wearable device had been worn, to ensure that the observed data were actually registered. The end users rated the app highly during the quantitative assessment, with an average mobile health app usability questionnaire score of 5.32 (SD 0.66; range 4.44-6.23) and 6.20 (SD 0.43; range 5.72-6.88) out of 7 in the 2 iterative usability testing cycles.

Conclusions

The results of the usability test indicated a high, growing satisfaction with the electronic patient-reported outcome app. The adoption of a thorough user-centered design process using several types of FGs helped maximize the likelihood of sustained retention of the app’s users and made it fit for data collection of relevant outcomes in the observational POWER study. The continuous use of the app and the actual level of engagement will be evaluated during the ongoing trial.

Trial Registration

ClinicalTrials.gov NCT04165135; https://clinicaltrials.gov/ct2/show/NCT04165135

A Web-Based, Population-Based Cirrhosis Identification and Management System for Improving Cirrhosis Care: Qualitative Formative Evaluation

Background

Cirrhosis, or scarring of the liver, is a debilitating condition that affects millions of US adults. Early identification, linkage to care, and retention of care are critical for preventing severe complications and death from cirrhosis.

Objective

The purpose of this study is to conduct a preimplementation formative evaluation to identify factors that could impact implementation of the Population-Based Cirrhosis Identification and Management System (P-CIMS) in clinics serving patients with cirrhosis. P-CIMS is a web-based informatics tool designed to facilitate patient outreach and cirrhosis care management.

Methods

Semistructured interviews were conducted between January and May 2016 with frontline providers in liver disease and primary care clinics at 3 Veterans Health Administration medical centers. A total of 10 providers were interviewed, including 8 physicians and midlevel providers from liver-related specialty clinics and 2 primary care providers who managed patients with cirrhosis. The Consolidated Framework for Implementation Research guided the development of the interview guides. Inductive consensus coding and content analysis were used to analyze transcribed interviews and abstracted coded passages, elucidated themes, and insights.

Results

The following themes and subthemes emerged from the analyses: outer setting: needs and resources for patients with cirrhosis; inner setting: readiness for implementation (subthemes: lack of resources, lack of leadership support), and implementation climate (subtheme: competing priorities); characteristics of individuals: role within clinic; knowledge and beliefs about P-CIMS (subtheme: perceived and realized benefits; useful features; suggestions for improvement); and perceptions of current practices in managing cirrhosis cases (subthemes: preimplementation process for identifying and linking patients to cirrhosis care; structural and social barriers to follow-up). Overall, P-CIMS was viewed as a powerful tool for improving linkage and retention, but its integration in the clinical workflow required leadership support, time, and staffing. Providers also cited the need for more intuitive interface elements to enhance usability.

Conclusions

P-CIMS shows promise as a powerful tool for identifying, linking, and retaining care in patients living with cirrhosis. The current evaluation identified several improvements and advantages of P-CIMS over current care processes and provides lessons for others implementing similar population-based identification and management tools in populations with chronic disease.

Data Quality of Longitudinally Collected Patient-Reported Outcomes After Thoracic Surgery: Comparison of Paper- and Web-Based Assessments

Background

High-frequency patient-reported outcome (PRO) assessments are used to measure patients' symptoms after surgery for surgical research; however, the quality of those longitudinal PRO data has seldom been discussed.

Objective

The aim of this study was to determine data quality-influencing factors and to profile error trajectories of data longitudinally collected via paper-and-pencil (P&P) or web-based assessment (electronic PRO [ePRO]) after thoracic surgery.

Methods

We extracted longitudinal PRO data with 678 patients scheduled for lung surgery from an observational study (n=512) and a randomized clinical trial (n=166) on the evaluation of different perioperative care strategies. PROs were assessed by the MD Anderson Symptom Inventory Lung Cancer Module and single-item Quality of Life Scale before surgery and then daily after surgery until discharge or up to 14 days of hospitalization. Patient compliance and data error were identified and compared between P&P and ePRO. Generalized estimating equations model and 2-piecewise model were used to describe trajectories of error incidence over time and to identify the risk factors.

Results

Among 678 patients, 629 with at least 2 PRO assessments, 440 completed 3347 P&P assessments and 189 completed 1291 ePRO assessments. In total, 49.4% of patients had at least one error, including (1) missing items (64.69%, 1070/1654), (2) modifications without signatures (27.99%, 463/1654), (3) selection of multiple options (3.02%, 50/1654), (4) missing patient signatures (2.54%, 42/1654), (5) missing researcher signatures (1.45%, 24/1654), and (6) missing completion dates (0.30%, 5/1654). Patients who completed ePRO had fewer errors than those who completed P&P assessments (ePRO: 30.2% [57/189] vs. P&P: 57.7% [254/440]; P<.001). Compared with ePRO patients, those using P&P were older, less educated, and sicker. Common risk factors of having errors were a lower education level (P&P: odds ratio [OR] 1.39, 95% CI 1.20-1.62; P<.001; ePRO: OR 1.82, 95% CI 1.22-2.72; P=.003), treated in a provincial hospital (P&P: OR 3.34, 95% CI 2.10-5.33; P<.001; ePRO: OR 4.73, 95% CI 2.18-10.25; P<.001), and with severe disease (P&P: OR 1.63, 95% CI 1.33-1.99; P<.001; ePRO: OR 2.70, 95% CI 1.53-4.75; P<.001). Errors peaked on postoperative day (POD) 1 for P&P, and on POD 2 for ePRO.

Conclusions

It is possible to improve data quality of longitudinally collected PRO through ePRO, compared with P&P. However, ePRO-related sampling bias needs to be considered when designing clinical research using longitudinal PROs as major outcomes.

Usability Evaluation of an Offline Electronic Data Capture App in a Prospective Multicenter Dementia Registry (digiDEM Bayern): Mixed Method Study

BACKGROUND

Digital registries have been shown to provide an efficient way of gaining a better understanding of the clinical complexity and long-term progression of diseases. The paperless method of electronic data capture (EDC) during a patient interview saves both time and resources. In the prospective multicenter project "Digital Dementia Registry Bavaria (digiDEM Bayern)," interviews are also performed on site in rural areas with unreliable internet connectivity. It must be ensured that EDC can still be performed in such a context and that there is no need to fall back on paper-based questionnaires. In addition to a web-based data collection solution, the EDC system REDCap (Research Electronic Data Capture) offers the option to collect data offline via an app and to synchronize it afterward.

OBJECTIVE

The aim of this study was to evaluate the usability of the REDCap app as an offline EDC option for a lay user group and to examine the necessary technology acceptance of using mobile devices for data collection. The feasibility of the app-based offline data collection in the digiDEM Bayern dementia registry project was then evaluated before going live.

METHODS

An exploratory mixed method design was employed in the form of an on-site usability test with the "Thinking Aloud" method combined with an online questionnaire including the System Usability Scale (SUS). The acceptance of mobile devices for data collection was surveyed based on five categories of the technology acceptance model.

RESULTS

Using the "Thinking Aloud" method, usability issues were identified and solutions were accordingly derived. Evaluation of the REDCap app resulted in a SUS score of 74, which represents "good" usability. After evaluating the technology acceptance questionnaire, it can be concluded that the lay user group is open to mobile devices as interview tools.

CONCLUSIONS

The usability evaluation results show that a lay user group generally agree that data collecting partners in the digiDEM project can handle the REDCap app well. The usability evaluation provided statements about positive aspects and could also identify usability issues relating to the REDCap app. In addition, the current technology acceptance in the sample showed that heterogeneous groups of different ages with diverse experiences in handling mobile devices are also ready for the use of app-based EDC systems. Based on these results, it can be assumed that the offline use of an app-based EDC system on mobile devices is a viable solution for collecting data in a decentralized registry-based research project.

Who is more likely to adopt and comply with the electronic patient-reported outcome measure (ePROM) mobile application? A real-world study with cancer patients undergoing active treatment

PURPOSE

This study aims to identify factors associated with the adoption and compliance of electronic patient-reported outcome measure (ePROM) use among cancer patients in a real-world setting.

METHODS

This prospective cohort study was conducted at the Samsung Medical Center in Seoul, Korea, from September 2018 to January 2019. Cancer patients aged 18 years or older who owned smartphones and who were receiving chemotherapy or radiation therapy were eligible for this study. Patients were asked to use the app to report their symptoms every 7 days for a total of 21 days (3 weeks). Logistic regression was performed to identify the factors associated with the adoption and compliance.

RESULTS

Among 580 patients, 417 (71.9%) adopted the ePROM app and 159 (27.4%) out of 417 had good compliance. Patients who had greater expectations regarding the ease of use (adjusted odds ratio [aOR] 2.67, 95% CI: 1.28-5.57) and usefulness (aOR 1.69, 95% CI: 1.05-2.72) of the ePROM app were more likely to adopt the app than those who did not. Patients who had greater satisfaction with usefulness (aOR 1.89, 95% CI 1.10-3.25) were more likely to comply with using the app, but satisfaction with ease of use was not related to the compliance.

CONCLUSION

While expectation regarding the ease of use and usefulness of the ePROM app was associated with the adoption of the app, satisfaction with ease of use was not related to compliance with the ePROM app. Satisfaction with usefulness was associated with the compliance of ePROM app use.

A narrative review of current evidence supporting the implementation of electronic patient-reported outcome measures in the management of chronic diseases

An application of telemedicine of growing interest and relevance is the use of personal computers and mobile devices to collect patient-reported outcomes (PROs). PROs are self-reports of patients' health status without interpretation by anyone else. The tools developed to assess PROs are known as patient-reported outcomes measures (PROMs). The technological innovations that have led to an increased ownership of electronic devices have also facilitated the development of electronic PROMs (ePROMs). ePROMs are a conduit for telemedicine in the care of patients with chronic diseases. Various studies have demonstrated that the use of ePROMs in routine clinical practice is both acceptable and feasible with patients increasingly expressing a preference for an electronic mode of administration. There is increasing evidence that the use of electronic patient-reported outcome (ePROMs) could have significant impacts on outcomes valued by patients, healthcare providers and researchers. Whilst the development and implementation of these systems may be initially costly and resource-intensive, patient preferences and existing evidence to support their implementation suggests the need for continued research prioritisation in this area. This narrative review summarises and discusses evidence of the impact of ePROMs on clinical parameters and outcomes relevant to chronic diseases. We also explore recently published literature regarding issues that may influence the robust implementation of ePROMs for routine clinical practice.

Using a Commercially Available App for the Self-Management of Hypertension: Acceptance and Usability Study in Saudi Arabia

BACKGROUND

The use of smartphone apps to assist in the self-management of hypertension is becoming increasingly common, but few commercially available apps have the potential to be effective along with adequate security and privacy measures in place. In a previous study, we identified 5 apps that are potentially effective and safe, and based on the preferences of doctors and patients, one (Cora Health) was selected as the most suitable app for use in a Saudi context. However, there is currently no evidence of its usability and acceptance among potential users. Indeed, there has been little research into the usability and acceptance of hypertension apps in general, and less research considers this in the Gulf Region.

OBJECTIVE

This study aims to evaluate the acceptance and usability of the selected app in the Saudi context.

METHODS

This study used a mixed methods approach with 2 studies: a usability test involving patients in a controlled setting performing predefined tasks and a real-world usability study where patients used the app for 4 weeks. In the usability test, participants were asked to think aloud while performing the tasks, and an observer recorded the number of tasks they completed. At the end of the real-world pilot study, participants were interviewed, and the mHealth App Usability Questionnaire was completed. Descriptive statistics were used to analyze quantitative data, and thematic analysis was used to analyze qualitative data.

RESULTS

In total, 10 patients completed study 1. The study found that app usability was moderate and that participants needed some familiarization time before they could use the app proficiently. Some usability issues were revealed, related to app accessibility and navigation, and a few tasks remained uncompleted by most people. A total of 20 patients completed study 2, with a mean age of 51.6 (SD 11.7) years. Study 2 found that the app was generally acceptable and easy to use, with some similar usability issues identified. Participants stressed the importance of practice and training to use it more easily and proficiently. Participants had a good engagement level with 48% retention at the end of study 2, with most participants' engagement being classed as meaningful. The most recorded data were blood pressure, followed by stress and medication, and the most accessed feature was viewing graphs of data trends.

CONCLUSIONS

This study shows that a commercially available app can be usable and acceptable in the self-management of hypertension but also found a considerable number of possibilities for improvement, which needs to be considered in future app development. The results show that there is potential for a commercially available app to be used in large-scale studies of hypertension self-management if suggestions for improvements are addressed.

Development of a web tool to increase research literacy in underserved populations through public library partnerships

OBJECTIVE

Inadequate diversity in clinical trials is widely recognized as a significant contributing factor to health disparities experienced by racial/ethnic minorities and other diverse populations in the US. To address this in a scalable way, we sought to develop a web tool that could help enhance underserved minority participation in clinical research.

METHODS

We used our research literacy support flashcard tool as the initial prototype for human-centered design and usability testing of the web tool Health for All in public library settings. After forming partnerships with leadership from Chicago Public Libraries (CPL), local medical libraries, and the Chicago Department of Public Health, we conducted seven iterative design sessions with focus groups of library patrons and library staff from six CPL branches serving underserved communities followed by two rounds of usability testing and website modification.

RESULTS

Based on the qualitative research findings from Design Sessions 1-7, we enacted the design decision of a website that was a hybrid of fact-filled and vignette (personal stories) paper prototypes divided into 4 modules (trust, diversity, healthy volunteers, pros/cons), each with their own outcome metrics. The website was thus constructed, and navigation issues identified in two rounds of usability testing by library patrons were addressed through further website modification, followed by the launch of a beta version of a hybridized single-scrolling and guided module prototype to allow further development with website analytics.

CONCLUSIONS

We report the development of Health for All, a website designed to enhance racial/ethnic minority participation in clinical trials by imparting research literacy, mitigating distrust engendered by longstanding racism and discrimination, and providing connections to clinical trials recruiting participants.

Clinical Research in Neonates: Redesigning the Informed Consent Process in the Digital Era

Background: Currently, many initiatives are devoted to optimizing informed consent for participation in clinical research. Due to the digital transformation in health care, a shift toward electronic informed consent (eIC) has been fostered. However, empirical evidence on how to implement eIC in clinical research involving neonates is lacking. Methods: Semi-structured interviews were conducted with 31 health care professionals active in Belgium or the Netherlands. All health care professionals had experience in conducting clinical research involving neonates. Interviews were audio-recorded, transcribed and analyzed using the framework method. Results: Interviewees generally supported the use of eIC in clinical research involving neonates. For example, eIC could enable parents to receive study feedback via the eIC system. Requirements were expressed for parental involvement to decide on which feedback would be appropriate to return. Moreover, experts specialized in presenting information and designing electronic systems should be involved. Broad consensus among health care professionals indicates that the face-to-face-interaction between parents and the research team is vital to establish a relationship of trust. Therefore, it is necessary that the use of eIC runs alongside personal interactions with the parents. Concerns were raised about the accessibility of eIC to parents. For this reason, it was suggested that parents should always be given the possibility to read and sign a paper-based informed consent form or to use eIC. Conclusions: Health care professionals' views indicate that the use of eIC in clinical research with neonates may offer various opportunities. Further development and implementation will require a multi-stakeholder approach.