Usability of an Automated System for Real-Time Monitoring of Shared Decision-Making for Surgery: Mixed Methods Evaluation

BACKGROUND

Improving shared decision-making (SDM) for patients has become a health policy priority in many countries. Achieving high-quality SDM is particularly important for approximately 313 million surgical treatment decisions patients make globally every year. Large-scale monitoring of surgical patients' experience of SDM in real time is needed to identify the failings of SDM before surgery is performed. We developed a novel approach to automating real-time data collection using an electronic measurement system to address this. Examining usability will facilitate its optimization and wider implementation to inform interventions aimed at improving SDM.

OBJECTIVE

This study examined the usability of an electronic real-time measurement system to monitor surgical patients' experience of SDM. We aimed to evaluate the metrics and indicators relevant to system effectiveness, system efficiency, and user satisfaction.

METHODS

We performed a mixed methods usability evaluation using multiple participant cohorts. The measurement system was implemented in a large UK hospital to measure patients' experience of SDM electronically before surgery using 2 validated measures (CollaboRATE and SDM-Q-9). Quantitative data (collected between April 1 and December 31, 2021) provided measurement system metrics to assess system effectiveness and efficiency. We included adult patients booked for urgent and elective surgery across 7 specialties and excluded patients without the capacity to consent for medical procedures, those without access to an internet-enabled device, and those undergoing emergency or endoscopic procedures. Additional groups of service users (group 1: public members who had not engaged with the system; group 2: a subset of patients who completed the measurement system) completed user-testing sessions and semistructured interviews to assess system effectiveness and user satisfaction. We conducted quantitative data analysis using descriptive statistics and calculated the task completion rate and survey response rate (system effectiveness) as well as the task completion time, task efficiency, and relative efficiency (system efficiency). Qualitative thematic analysis identified indicators of and barriers to good usability (user satisfaction).

RESULTS

A total of 2254 completed surveys were returned to the measurement system. A total of 25 service users (group 1: n=9; group 2: n=16) participated in user-testing sessions and interviews. The task completion rate was high (169/171, 98.8%) and the survey response rate was good (2254/5794, 38.9%). The median task completion time was 3 (IQR 2-13) minutes, suggesting good system efficiency and effectiveness. The qualitative findings emphasized good user satisfaction. The identified themes suggested that the measurement system is acceptable, easy to use, and easy to access. Service users identified potential barriers and solutions to acceptability and ease of access.

CONCLUSIONS

A mixed methods evaluation of an electronic measurement system for automated, real-time monitoring of patients' experience of SDM showed that usability among patients was high. Future pilot work will optimize the system for wider implementation to ultimately inform intervention development to improve SDM.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR2-10.1136/bmjopen-2023-079155.

Biometric linkage of longitudinally collected electronic case report forms and confirmation of subject identity: an open framework for ODK and related tools

The availability of low-cost biometric hardware sensors and software makes it possible to rapidly, affordably and securely sample and store a unique and invariant biological signature (or biometric "template") for the purposes of identification. This has applications in research and trials, particularly for purposes of consent, linkage of case reporting forms collected at different times, and in the confirmation of participant identity for purposes of safety monitoring and adherence to international data laws. More broadly, these methods are applicable to the needs of the billion people who live in resource-restricted settings without identification credentials. The use of mobile electronic data collection software has recently become commonplace in clinical trials, research and actions for public good. A raft of tools based on the open-source ODK project now provide diverse options for data management that work consistently in resource-restricted settings, but none have built-in functionality for capturing biometric templates. In this study, we report the development and validation of a novel open-source app and associated method for capturing and matching biometric fingerprint templates during data collection with the popular data platforms ODK, KoBoToolbox, SurveyCTO, Ona and CommCare. Using data from more than 1,000 fingers, we show that fingerprint templates can be used to link data records with high accuracy. The accuracy of this process increases through the linkage of multiple fingerprints to each data record. By focussing on publishing open-source code and documentation, and by using an affordable (<£50) and mass-produced model of fingerprint sensor, we are able to make this platform freely available to the large global user community that utilises ODK and related data collection systems.

National Implementation of an Electronic Patient-Reported Outcome Measures Program for Joint Replacement Surgery: Pilot Study

Background

There is a global emphasis on expanding data collection for joint replacement procedures beyond implant attributes and progression to revision surgery. Patient-reported outcome measures (PROMs) are increasingly considered as an important measure of surgical outcomes from a patient’s perspective. However, a major limitation preventing wider use of PROMs data in national data collection has been the inability to systematically collect and share electronic information with relevant stakeholders in a comprehensive and financially sustainable manner.

Objective

This study reports on the development of an electronic data capture and reporting system by a national registry for the collection of PROMs and the processes used to identify and overcome barriers to implementation and uptake. The study also aims to provide a cost breakdown of establishing and maintaining a nationwide electronic PROMs program.

Methods

Between 2018 and 2020, 3 governance and advisory committees were established to develop and implement a PROMs pilot program nested within a nationwide joint replacement registry. The program involved electronic collection of preoperative and 6-month postoperative data for hip, knee, or shoulder replacement surgery from 44 Australian hospitals. Resource requirements for the program included a project manager, software developers, data manager, and statistician. An online platform was tested, refined, and implemented for electronic PROMs collection with scalability considered for future expansion to all Australian hospitals and additional data fields. Technical capabilities included different access for multiple user types, patient registration, automatic reminders via SMS text messages and email, online consent, and patient outcome real-time dashboards accessible for different user groups (surgeons, patients, hospitals, and project stakeholders).

Results

During the PROMs pilot period there were 19,699 primary procedures undertaken with 10,204 registered procedures in the electronic system. This equated to 51.80% of people who had a joint replacement at participating hospitals during this period. Patient registration and data collection were efficient (20-30 seconds and 10-12 minutes, respectively). Engagement with the reporting dashboards (as a proportion of those who viewed their dashboard) varied by user group: 197/277 (71.1%) hospital administrators, 68/129 (52.7%) project stakeholders, 177/391 (45.3%) surgeons, and 1138/8840 patients (12.9%). Cost analysis determined an overall cost per patient of Aus $7-15 (approximately US $5-12) for 2 PROMs collections per joint replacement procedure once the program was established.

Conclusions

Successful implementation of an orthopedic PROMs program with planned scalability for a broader national rollout requires significant funding and staffing resources. However, this expenditure can be considered worthwhile, given that collection and reporting of PROMs can drive health care improvement processes. Further consideration of strategies to improve stakeholder engagement with electronic reporting dashboards (particularly for patients and surgeons) will be critical to the ongoing success of a national PROMs program.

The Strategies for Quantitative and Qualitative Remote Data Collection: Lessons From the COVID-19 Pandemic

The COVID-19 pandemic has necessitated a rapid shift to web-based or blended design models for both ongoing and future clinical research activities. Research conducted virtually not only has the potential to increase the patient-centeredness of clinical research but may also further widen existing disparities in research participation among underrepresented individuals. In this viewpoint, we discuss practical strategies for quantitative and qualitative remote research data collection based on previous literature and our own ongoing clinical research to overcome challenges presented by the shift to remote data collection. We aim to contribute to and catalyze the dissemination of best practices related to remote data collection methodologies to address the opportunities presented by this shift and develop strategies for inclusive research.

Electronic Data Management for Vaccine Trials in Low Resource Settings: Upgrades, Scalability, and Impact of ODK

Background: ODK provides software and standards that are popular solutions for off-grid electronic data collection and has substantial code overlap and interoperability with a number of related software products including CommCare, Enketo, Ona, SurveyCTO, and KoBoToolbox. These tools provide open-source options for off-grid use in public health data collection, management, analysis, and reporting. During the 2018–2020 Ebola epidemic in the North Kivu and Ituri regions of Democratic Republic of Congo, we used these tools to support the DRC Ministère de la Santé RDC and World Health Organization in their efforts to administer an experimental vaccine (VSV-Zebov-GP) as part of their strategy to control the transmission of infection.

Method: New functions were developed to facilitate the use of ODK, Enketo and R in large scale data collection, aggregation, monitoring, and near-real-time analysis during clinical research in health emergencies. We present enhancements to ODK that include a built-in audit-trail, a framework and companion app for biometric registration of ISO/IEC 19794-2 fingerprint templates, enhanced performance features, better scalability for studies featuring millions of data form submissions, increased options for parallelization of research projects, and pipelines for automated management and analysis of data. We also developed novel encryption protocols for enhanced web-form security in Enketo.

Results: Against the backdrop of a complex and challenging epidemic response, our enhanced platform of open tools was used to collect and manage data from more than 280,000 eligible study participants who received VSV-Zebov-GP under informed consent. These data were used to determine whether the VSV-Zebov-GP was safe and effective and to guide daily field operations.

Conclusions: We present open-source developments that make electronic data management during clinical research and health emergencies more viable and robust. These developments will also enhance and expand the functionality of a diverse range of data collection platforms that are based on the ODK software and standards.

Usability Evaluation of an Offline Electronic Data Capture App in a Prospective Multicenter Dementia Registry (digiDEM Bayern): Mixed Method Study

BACKGROUND

Digital registries have been shown to provide an efficient way of gaining a better understanding of the clinical complexity and long-term progression of diseases. The paperless method of electronic data capture (EDC) during a patient interview saves both time and resources. In the prospective multicenter project "Digital Dementia Registry Bavaria (digiDEM Bayern)," interviews are also performed on site in rural areas with unreliable internet connectivity. It must be ensured that EDC can still be performed in such a context and that there is no need to fall back on paper-based questionnaires. In addition to a web-based data collection solution, the EDC system REDCap (Research Electronic Data Capture) offers the option to collect data offline via an app and to synchronize it afterward.

OBJECTIVE

The aim of this study was to evaluate the usability of the REDCap app as an offline EDC option for a lay user group and to examine the necessary technology acceptance of using mobile devices for data collection. The feasibility of the app-based offline data collection in the digiDEM Bayern dementia registry project was then evaluated before going live.

METHODS

An exploratory mixed method design was employed in the form of an on-site usability test with the "Thinking Aloud" method combined with an online questionnaire including the System Usability Scale (SUS). The acceptance of mobile devices for data collection was surveyed based on five categories of the technology acceptance model.

RESULTS

Using the "Thinking Aloud" method, usability issues were identified and solutions were accordingly derived. Evaluation of the REDCap app resulted in a SUS score of 74, which represents "good" usability. After evaluating the technology acceptance questionnaire, it can be concluded that the lay user group is open to mobile devices as interview tools.

CONCLUSIONS

The usability evaluation results show that a lay user group generally agree that data collecting partners in the digiDEM project can handle the REDCap app well. The usability evaluation provided statements about positive aspects and could also identify usability issues relating to the REDCap app. In addition, the current technology acceptance in the sample showed that heterogeneous groups of different ages with diverse experiences in handling mobile devices are also ready for the use of app-based EDC systems. Based on these results, it can be assumed that the offline use of an app-based EDC system on mobile devices is a viable solution for collecting data in a decentralized registry-based research project.

Smartphone assisted oral health data recording - an android based software application development

Background

Smartphone compared to the traditional pen-paper method could enhance oral health data recording procedure by reducing the cost of data collection, risk of data loss, early detection of errors and reducing data entry time. The present research developed a mobile/tablet-based software application to capture oral health data and test its adaptability and operations in oral health surveys.

Methods

A comparative cross-sectional study was conducted among the general population of Sanwer town, Indore district. The initial testing of the application was done on 120 individuals. A random sampling (lottery method) followed by a systematic sampling strategy was employed to select 120 households. A "one per household" design was implemented for the survey. The initial oral health data collection was done using mobile-assisted software application followed by a second examination scheduled after 15 days on the same participants using the conventional Pen-paper method to collect oral health data.

Results

Six Investigator Recorder (IR) teams conducted the oral health data collection. Data collection through Smartphone-based application displayed less meantime (3.57 minutes) in comparison to pen-paper method (4.87 minutes) (p≤0.001). Survey team response showed the majority of investigators having strong agreement on user satisfaction and speed of data entry using software application.

Conclusion

The initial testing of mobile-assisted recording system (MARS) efficiently captured oral health data among the general population with wide variations in oral disease level. The application facilitated minimal or no wastage of paper and had a high level of user-satisfaction, accuracy, speed of entry and low potential for any data loss.

Data Architecture to Support Real-Time Data Analytics for the Population-Based HIV Impact Assessments

BACKGROUND AND SETTING

Electronic data capture facilitates timely use of data. Population-based HIV impact assessments (PHIAs) were led by host governments, with funding from the President's Emergency Plan for AIDS Relief, technical assistance from the Centers for Disease Control, and implementation support from ICAP at Columbia University. We described data architectures, code-based processes, and resulting data volume and quality for 14 national PHIA surveys with concurrent timelines and varied country-level data governance (2015-2020).

METHODS

PHIA project data were collected through tablets, point-of-care and laboratory testing instruments, and inventory management systems, using open-source software, vendor solutions, and custom-built software. Data were securely uploaded to the PHIA data warehouse daily or weekly and then used to populate survey-monitoring dashboards and return timely laboratory-based test results on an ongoing basis. Automated data processing allowed timely reporting of survey results.

RESULTS

Fourteen data architectures were successfully established, and data from more than 450,000 participants in 30,000 files across 13 countries with completed PHIAs, and blood draws producing approximately 6000 aliquots each week per country, were securely collected, transmitted, and processed by 17 full-time equivalent staff. More than 25,600 viral load results were returned to clinics of participants' choice. Data cleaning was not needed for 98.5% of household and 99.2% of individual questionnaires.

CONCLUSION

The PHIA data architecture permitted secure, simultaneous collection and transmission of high-quality interview and biomarker data across multiple countries, quick turnaround time of laboratory-based biomarker results, and rapid dissemination of survey outcomes to guide President's Emergency Plan for AIDS Relief epidemic control.

Development and application of an electronic treatment register: a system for enumerating populations and monitoring treatment during mass drug administration

We developed an electronic treatment register for the DeWorm3 Project, a cluster-randomised, controlled trial in Benin, India, and Malawi testing the feasibility of interrupting transmission of soil-transmitted helminths through community-wide mass drug administration. The electronic treatment register was designed in xlsform, deployed via the SurveyCTO mobile data collection platform, and implemented on smartphones running the Android operating system. The versatile system enables collection of census and treatment status information, facilitates data aggregation and visualisation, and permits real-time feedback loops during implementation of mass drug administration. Here we describe the system’s design and use within the DeWorm3 Project and key features, and by sharing the register here, we hope our readers will further explore its use within their research and disease-control activities.

Predictors and Prospective Course of PANS: A Pilot Study Using Electronic Platforms for Data Collection

Objectives: Little is known about the longitudinal course of pediatric acute-onset neuropsychiatric syndrome (PANS) because existing literature is primarily cross-sectional. To begin to address this gap, two digital platforms were used to prospectively monitor neuropsychiatric symptoms in children with PANS. The aim was to identify baseline clinical characteristics that would predict the course of neuropsychiatric symptoms over 12 weeks. We compared relative compliance between two electronic data acquisition platforms and evaluated agreement between parent-child ratings of symptoms. Methods: For 12 weeks, 20 children with PANS and their parents completed weekly rating scales of neuropsychiatric symptoms on Research Electronic Data Capture (REDCap) and concurrently parents completed tri-weekly ratings on My Pediatric Autoimmune Neuropsychiatric Disorders Associated with Streptococcal Infections (PANDAS) Chart, a symptom monitoring website. Longitudinal data were analyzed by using regression analyses. Results: Greater duration of time between onset of PANS and study enrollment was associated with worsening of parent-rated neuropsychiatric symptoms over 12 weeks (p = 0.05). Higher scores on parents' Caregiver Burden Inventory at baseline predicted that children would report more severe symptoms over the 12-week period (p = 0.01). Compliance rates for parents were 86.3% for the weekly REDCap PANS Symptoms Rating Scale compared with 53.8% for the tri-weekly My PANDAS Chart ratings. There was moderate agreement between children and parents on the PANS Symptom Rating Scale (r = 0.55, p < 0.0001). Conclusion: Our study highlights the utility of electronic methods for tracking longitudinal symptoms in children with PANS and suggests that particular baseline characteristics (e.g., delay in identification and treatment of PANS, greater caregiver burden) may be indicative of a differential trajectory of PANS course, with more severe symptoms over the short term. clinicaltrials.gov NCT04382716.

Data Quality and Cost-effectiveness Analyses of Electronic and Paper-Based Interviewer-Administered Public Health Surveys: Systematic Review

Background

A population-level survey (PLS) is an essential and standard method used in public health research that supports the quantification of sociodemographic events, public health policy development, and intervention designs. Data collection mechanisms in PLS seem to be a significant determinant in avoiding mistakes. Using electronic devices such as smartphones and tablet computers improves the quality and cost-effectiveness of public health surveys. However, there is a lack of systematic evidence to show the potential impact of electronic data collection tools on data quality and cost reduction in interviewer-administered surveys compared with the standard paper-based data collection system.

Objective

This systematic review aims to evaluate the impact of the interviewer-administered electronic data collection methods on data quality and cost reduction in PLS compared with traditional methods.

Methods

We conducted a systematic search of MEDLINE, CINAHL, PsycINFO, the Web of Science, EconLit, Cochrane CENTRAL, and CDSR to identify relevant studies from 2008 to 2018. We included randomized and nonrandomized studies that examined data quality and cost reduction outcomes, as well as usability, user experience, and usage parameters. In total, 2 independent authors screened the title and abstract, and extracted data from selected papers. A third author mediated any disagreements. The review authors used EndNote for deduplication and Rayyan for screening.

Results

Our search produced 3817 papers. After deduplication, we screened 2533 papers, and 14 fulfilled the inclusion criteria. None of the studies were randomized controlled trials; most had a quasi-experimental design, for example, comparative experimental evaluation studies nested on other ongoing cross-sectional surveys. A total of 4 comparative evaluations, 2 pre-post intervention comparative evaluations, 2 retrospective comparative evaluations, and 4 one-arm noncomparative studies were included. Meta-analysis was not possible because of the heterogeneity in study designs, types, study settings, and level of outcome measurements. Individual paper synthesis showed that electronic data collection systems provided good quality data and delivered faster compared with paper-based data collection systems. Only 2 studies linked cost and data quality outcomes to describe the cost-effectiveness of electronic data collection systems. Field data collectors reported that an electronic data collection system was a feasible, acceptable, and preferable tool for their work. Onsite data error prevention, fast data submission, and easy-to-handle devices were the comparative advantages offered by electronic data collection systems. Challenges during implementation included technical difficulties, accidental data loss, device theft, security concerns, power surges, and internet connection problems.

Conclusions

Although evidence exists of the comparative advantages of electronic data collection compared with paper-based methods, the included studies were not methodologically rigorous enough to combine. More rigorous studies are needed to compare paper and electronic data collection systems in public health surveys considering data quality, work efficiency, and cost reduction.

International Registered Report Identifier (IRRID)

RR2-10.2196/10678

The Acceptability and Efficacy of Electronic Data Collection in a Hospital Neurodevelopmental Clinic: Pilot Questionnaire Study

Background

There is a growing need for cost-efficient and patient-centered approaches to support families in hospital- and community-based neurodevelopmental services. For such purposes, electronic data collection (EDC) may hold advantages over paper-based data collection. Such EDC approaches enable automated data collection for scoring and interpretation, saving time for clinicians and services and promoting more efficient service delivery.

Objective

This pilot study evaluated the efficacy of EDC for the Child Development Unit, a hospital-based diagnostic assessment clinic in the Sydney Children’s Hospital Network. Caregiver response rates and preference for EDC or paper-based methods were evaluated as well as the moderating role of demographic characteristics such as age, level of education, and ethnic background.

Methods

Families were sent either a paper-based questionnaire via post or an electronic mail link for completion before attending their first on-site clinic appointment for assessment. A total of 62 families were provided a paper version of the questionnaire, while 184 families were provided the online version of the same questionnaire.

Results

Completion rates of the questionnaire before the first appointment were significantly higher for EDC (164/184, 89.1%) in comparison to paper-based methods (24/62, 39%; P<.001). Within the EDC group, a vast majority of respondents indicated a preference for completing the questionnaire online (151/173, 87.3%), compared to paper completion (22/173, 12.7%; P<.001). Of the caregiver demographic characteristics, only the respondent’s level of education was associated with modality preference, such that those with a higher level of education reported a greater preference for EDC (P=.04).

Conclusions

These results show that EDC is feasible in hospital-based clinics and has the potential to offer substantial benefits in terms of centralized data collation, time and cost savings, efficiency of service, and resource allocation. The results of this study therefore support the continued use of electronic methods to improve family-centered care in clinical practices.

Comparison between an electronic version of the foot and ankle outcome score and the standard paper version: A randomized multicenter study

To prove the equivalence of the Korean version of the Foot and Ankle Outcome Score (FAOS) in the printed (PFAOS) vs the electronic (EFAOS) form in a multicenter randomized study.Overall, 227 patients with ages ranging from 20 to 79 years from 16 dedicated foot and ankle centers were included. Patients were randomized into either a 'paper first' group (P-F group, n = 113) or an 'electronic device (tablet computer) first' group (E-F group, n = 114). The first evaluation either by paper (P-F group) or tablet (E-F group) was followed by a second evaluation the following day. The difference between the PFAOS and EFAOS results in each group was calculated and analyzed. To evaluate the benefit of each methodology, the time consumed per evaluation was compared and patients were asked which methodology they preferred and which was the easiest to use.There were no significant differences in age or sex between the groups. An intraclass correlation coefficient (ICC) value of 0.934 (95% confidence interval [CI]: 0.912-0.950, P < .001) was confirmed in PFAOS and EFAOS, showing a significant correlation between the 2 methodologies. EFAOS was completed in a shorter amount of time than PFAOS. The majority of patients agreed that EFAOS was easier to complete than PFAOS.The paper or electronic forms of the Korean adaptation of FAOS were considered equivalent. The shorter time of completion and the preference for the electronic version over paper by patients deems the electronic FAOS a promising option to consider in future.

An acute knee injury: tracking a two-year recovery online

Current government policy aims to deliver high quality healthcare by recording accurate data at the point of giving care, storing it efficiently and displaying it in a format, which is easily interpreted by healthcare professionals and patients. Few published reports describe the benefits of collecting and reporting such data from the point of view of the patient. We report the case of a 62-year-old lorry driver who used a web-based system to track patient reported outcome measures (PROMs) following an acute knee injury to full recovery over a 24-month period. The system reports clinical outcome scores in real-time to the patient and their medical team, clearly illustrating and enhancing recovery from injury. This case reveals how local software meets the needs of the individual patient linking and integrating such local systems must be the future focus of eHealth within the NHS to release the benefits presented by the information revolution.

The Perfusion Downunder collaborative database project

The Perfusion Downunder Collaboration provides research infrastructure and support to the Australian and New Zealand perfusion community, with the objective of determining best practices and producing relevant research publications. The Perfusion Downunder Collaborative Database (PDUCD) has been created for the purpose of collecting a dataset for cardiopulmonary bypass (CPB) procedures that includes integration with commercially available CPB data collection software. Initial testing of the PDUCD involved collection of data from four Australian and New Zealand hospitals from March to July 2007. Data from 513 procedures were compared with the concurrent Australasian Society of Cardiac and Thoracic Surgeons (ASCTS) database report to assess the validity of the collected data. Demographic, preoperative, and procedural variables were comparable between databases. Perfusion variables showed a median nasopharyngeal temperature of 36.7degrees C at separation from CPB (range, 35.3-37.5 degrees C), which was similar to maximum nasopharyngeal temperature (median, 36.8 degrees C). Median arterial flow and mean arterial pressure were 4.2 L/min and 57.2 mmHg, respectively. Control charts indicate a central tendency of 12.5 minutes for mean arterial pressure < 50 mmHg and 3.5 minutes for arterial flow < 1.6 L/min/m2 (cumulative time). There was no difference in median minimum and maximum blood glucose between diabetic and nondiabetic patients during CPB with 40% of patients receiving insulin. Median minimum and maximum activated clotting time (ACT) during CPB was 581 and 692 seconds, respectively. Outcome data for isolated coronary artery bypass grafting were similar for mortality (only) (both 1.8%). Initial data collection showed concurrent validity compared with the ASCTS database. The inclusion of a large quantity of calculated CPB variables in the dataset highlights the benefits of electronic data collection as a research tool within a collaborative research network and the potential for the evaluation of the relationships between patient risk factors, perfusion practice, and patient outcomes.