Assessing the global reach and value of a provider-facing healthcare app using large-scale analytics

Designing a Novel Clinician Decision Support Tool for the Management of Acute Diarrhea in Bangladesh: Formative Qualitative Study

BACKGROUND

The availability of mobile clinical decision support (CDS) tools has grown substantially with the increased prevalence of smartphone devices and apps. Although health care providers express interest in integrating mobile health (mHealth) technologies into their clinical settings, concerns have been raised, including perceived disagreements between information provided by mobile CDS tools and standard guidelines. Despite their potential to transform health care delivery, there remains limited literature on the provider's perspective on the clinical utility of mobile CDS tools for improving patient outcomes, especially in low- and middle-income countries.

OBJECTIVE

This study aims to describe providers' perceptions about the utility of a mobile CDS tool accessed via a smartphone app for diarrhea management in Bangladesh. In addition, feedback was collected on the preliminary components of the mobile CDS tool to address clinicians' concerns and incorporate their preferences.

METHODS

From November to December 2020, qualitative data were gathered through 8 web-based focus group discussions with physicians and nurses from 3 Bangladeshi hospitals. Each discussion was conducted in the local language-Bangla-and audio recorded for transcription and translation by the local research team. Transcripts and codes were entered into NVivo (version 12; QSR International), and applied thematic analysis was used to identify themes that explore the clinical utility of an mHealth app for assessing dehydration severity in patients with acute diarrhea. Summaries of concepts and themes were generated from reviews of the aggregated coded data; thematic memos were written and used for the final analysis.

RESULTS

Of the 27 focus group participants, 14 (52%) were nurses and 13 (48%) were physicians; 15 (56%) worked at a diarrhea specialty hospital and 12 (44%) worked in government district or subdistrict hospitals. Participants' experience in their current position ranged from 2 to 14 years, with an average of 10.3 (SD 9.0) years. Key themes from the qualitative data analysis included current experience with CDS, overall perception of the app's utility and its potential role in clinical care, barriers to and facilitators of app use, considerations of overtreatment and undertreatment, and guidelines for the app's clinical recommendations. Participants felt that the tool would initially take time to use, but once learned, it could be useful during epidemic cholera. Some felt that clinical experience remains an important part of treatment that can be supplemented, but not replaced, by a CDS tool. In addition, diagnostic information, including mid-upper arm circumference and blood pressure, might not be available to directly inform programming decisions.

CONCLUSIONS

Participants were positive about the mHealth app and its potential to inform diarrhea management. They provided detailed feedback, which developers used to revise the mobile CDS tool. These formative qualitative data provided timely and relevant feedback to improve the utility of a CDS tool for diarrhea treatment in Bangladesh.

Regional anesthesia educational material utilization varies by World Bank income category: A mobile health application data study

INTRODUCTION

Regional anesthesia offers an alternative to general anesthesia and may be advantageous in low resource environments. There is a paucity of data regarding the practice of regional anesthesia in low- and middle-income countries. Using access data from a free Android app with curated regional anesthesia learning modules, we aimed to estimate global interest in regional anesthesia and potential applications to clinical practice stratified by World Bank income level.

METHODS

We retrospectively analyzed data collected from the free Android app "Anesthesiologist" from December 2015 to April 2020. The app performs basic anesthetic calculations and provides links to videos on performing 12 different nerve blocks. Users of the app were classified on the basis of whether or not they had accessed the links. Nerve blocks were also classified according to major use (surgical block, postoperative pain adjunct, rescue block).

RESULTS

Practitioners in low- and middle-income countries accessed the app more frequently than in high-income countries as measured by clicks. Users from low- and middle-income countries focused mainly on surgical blocks: ankle, axillary, infraclavicular, interscalene, and supraclavicular blocks. In high-income countries, more users viewed postoperative pain blocks: adductor canal, popliteal, femoral, and transverse abdominis plane blocks. Utilization of the app was constant over time with a general decline with the start of the COVID-19 pandemic.

CONCLUSION

The use of an in app survey and analytics can help identify gaps and opportunities for regional anesthesia techniques and practices. This is especially impactful in limited-resource areas, such as lower-income environments and can lead to targeted educational initiatives.

Impact of COVID-19 response on global surgical volumes: an ongoing observational study

Objective

To determine whether location-linked anaesthesiology calculator mobile application (app) data can serve as a qualitative proxy for global surgical case volumes and therefore monitor the impact of the coronavirus disease 2019 (COVID-19) pandemic.

Methods

We collected data provided by users of the mobile app “Anesthesiologist” during 1 October 2018–30 June 2020. We analysed these using RStudio and generated 7-day moving-average app use plots. We calculated country-level reductions in app use as a percentage of baseline. We obtained data on COVID-19 case counts from the European Centre for Disease Prevention and Control. We plotted changing app use and COVID-19 case counts for several countries and regions.

Findings

A total of 100 099 app users within 214 countries and territories provided data. We observed that app use was reduced during holidays, weekends and at night, correlating with expected fluctuations in surgical volume. We observed that the onset of the pandemic prompted substantial reductions in app use. We noted strong cross-correlation between COVID-19 case count and reductions in app use in low- and middle-income countries, but not in high-income countries. Of the 112 countries and territories with non-zero app use during baseline and during the pandemic, we calculated a median reduction in app use to 73.6% of baseline.

Conclusion

App data provide a proxy for surgical case volumes, and can therefore be used as a real-time monitor of the impact of COVID-19 on surgical capacity. We have created a dashboard for ongoing visualization of these data, allowing policy-makers to direct resources to areas of greatest need.

Real-time assessment of COVID-19 impact on global surgical case volumes

Importance

The COVID-19 pandemic has disrupted global surgical capacity. The impact of the pandemic in low and middle income countries has the potential to worsen already strained access to surgical care. Timely assessment of surgical volumes in these countries remains challenging.

Objective

To determine whether usage data from a globally used anesthesiology calculator mobile application can serve as a proxy for global surgical case volume and contribute to monitoring of the impact of the COVID-19 pandemic, particularly in World Bank low income countries where official data collection is not currently practical.

Design

Subset of data from an ongoing observational cohort study of users of the application collected from October 1, 2018 to April 18, 2020.

Setting

The mobile application is available from public sources; users download and use the application per their own clinical needs on personal mobile devices.

Participants

No user data was excluded from the study.

Exposures

Events with impacts on surgical case volumes, including weekends, holidays, and the COVID-19 pandemic.

Main Outcomes and Measures

It was previously noted that application usage was decreased on weekends and during winter holidays. We subsequently hypothesized that more detailed analysis would reveal impacts of country-specific or region-specific holidays on the volume of app use.

Results

4,300,975 data points from 92,878 unique users were analyzed. Physicians and other anesthesia providers comprised 85.8% of the study population. Application use was reduced on holidays and weekends and correlated with fluctuations in surgical volume. The COVID-19 pandemic was associated with substantial reductions in app use globally and regionally. There was strong cross correlation between COVID-19 case count and reductions in app use. By country, there was a median global reduction in app use to 58% of baseline (interquartile range, 46%-75%). Application use in low-income continues to decline but in high-income countries has stabilized.

Conclusions and Relevance

Application usage metadata provides a real-time indicator of surgical volume. This data may be used to identify impacted regions where disruptions to surgical care are disproportionate or prolonged. A dashboard for continuous visualization of these data has been deployed.

Moving anesthesiology educational resources to the point of care: experience with a pediatric anesthesia mobile app

Background

Educators in all disciplines recognize the need to update tools for the modern learner. Mobile applications (apps) may be useful, but real-time data is needed to demonstrate the patterns of utilization and engagement amongst learners.

Methods

We examined the use of an anesthesia app by two groups of learners (residents and anesthesiologist assistant students [AAs]) during a pediatric anesthesiology rotation. The app calculates age and weight-based information for clinical decision support and contains didactic materials for self-directed learning. The app transmitted detailed usage information to our research team.

Results

Over a 12-month period, 39 participants consented; 30 completed primary study procedures (18 residents, 12 AAs). AAs used the app more frequently than residents (P = 0.025) but spent less time in the app (P < 0.001). The median duration of app usage was 2.3 minutes. During the course of the rotation, usage of the app decreased over time. ‘Succinylcholine’ was the most accessed drug, while ‘orientation’ was the most accessed teaching module. Ten (33%) believed that the use of apps was perceived to be distracting by operating room staff and surgeons.

Conclusions

Real-time in-app analytics helped elucidate the actual usage of this educational resource and will guide future decisions regarding development and educational content. Further research is required to determine learners’ preferred choice of device, user experience, and content in the full range of clinical and nonclinical purposes.

Crowdsourcing sugammadex adverse event rates using an in-app survey: feasibility assessment from an observational study

Background

Mobile applications (apps) have become a ubiquitous source of clinical decision support. We sought to ascertain the feasibility of using an app platform to obtain a crowdsourced measure of adverse drug reaction reporting rates associated with sugammadex administration and compare it with traditionally-derived estimates.

Methods

Using the widely-distributed anesthesia calculator app, 'Anesthesiologist', we surveyed anesthesia providers regarding their experience with adverse drug reactions associated with sugammadex administration.

Results

Data were analyzed from 2770 participants in 119 countries responding between March 2016 and May 2017, who were estimated to have administered between 1.6-2.9 million doses (588-1040 administrations per participant). A low and high-end reporting rate of adverse events was estimated based on respondents' reported frequency and duration of sugammadex use. The estimated reporting rate of anaphylaxis due to sugammadex was 0.0055-0.098%, similar in range to previously published estimates.

Conclusions

Use of an in-app survey facilitated a global assessment of anesthesia providers and could have useful applications in monitoring adverse events and estimating their rates. Further work is needed to validate this approach for other medications and clinical domains.

Evidence for increased use of the Society of Pediatric Anesthesia Critical Events Checklist in resource-limited environments: A retrospective observational study of app data

BACKGROUND

Electronic decision support tools in anesthesiology practice have great value, including the potential for mobile applications to simplify delivery of best-practice guidelines.

AIMS

We sought to combine demographics with usage information to elucidate important patterns in the rate of use of the Society of Pediatric Anesthesia Critical Events Checklist, as measured by in-app accesses of the checklist via the freely available anesthesia calculator app anesthesiologist.

METHODS

We performed a retrospective analytic observational case-control study using analytics and survey data collected from the app. Users of the app were classified on the basis of whether or not they had accessed the checklist. This classification was used to perform logistic regression against a number of independent variables, including frequency of app use, country income level, professional role, rating of app importance, length of time in practice, group size, practice model, community served, and primary practice environment.

RESULTS

Individual app users practicing in low- and middle-income countries have a significantly higher rate of Society for Pediatric Anesthesia Critical Events Checklist utilization as compared with high-income countries. Rural practitioners had higher utilization of the checklist. Practice size did not affect the utilization of the checklist. The checklist was used for both provider learning and for just-in-time patient care.

CONCLUSION

mHealth apps are invaluable resource in everyday clinical practice. Mobile app analytics and in-app survey data reveal variable penetration and applicability of such technology worldwide. mHealth apps may be particularly impactful in limited-resource areas, such as lower-income environments and rural communities.

Factors influencing healthcare provider respondent fatigue answering a globally administered in-app survey

Background

Respondent fatigue, also known as survey fatigue, is a common problem in the collection of survey data. Factors that are known to influence respondent fatigue include survey length, survey topic, question complexity, and open-ended question type. There is a great deal of interest in understanding the drivers of physician survey responsiveness due to the value of information received from these practitioners. With the recent explosion of mobile smartphone technology, it has been possible to obtain survey data from users of mobile applications (apps) on a question-by-question basis. The author obtained basic demographic survey data as well as survey data related to an anesthesiology-specific drug called sugammadex and leveraged nonresponse rates to examine factors that influenced respondent fatigue.

Methods

Primary data were collected between December 2015 and February 2017. Surveys and in-app analytics were collected from global users of a mobile anesthesia calculator app. Key independent variables were user country, healthcare provider role, rating of importance of the app to personal practice, length of time in practice, and frequency of app use. Key dependent variable was the metric of respondent fatigue.

Results

Provider role and World Bank country income level were predictive of the rate of respondent fatigue for this in-app survey. Importance of the app to the provider and length of time in practice were moderately associated with fatigue. Frequency of app use was not associated. This study focused on a survey with a topic closely related to the subject area of the app. Respondent fatigue rates will likely change dramatically if the topic does not align closely.

Discussion

Although apps may serve as powerful platforms for data collection, responses rates to in-app surveys may differ on the basis of important respondent characteristics. Studies should be carefully designed to mitigate fatigue as well as powered with the understanding of the respondent characteristics that may have higher rates of respondent fatigue.