End-user perspectives of two mHealth decision support tools: Electronic Community Case Management in Northern Malawi

Developing a Comprehensive List of Criteria to Evaluate the Characteristics and Quality of eHealth Smartphone Apps: Systematic Review

BACKGROUND

The field of eHealth is growing rapidly and chaotically. Health care professionals need guidance on reviewing and assessing health-related smartphone apps to propose appropriate ones to their patients. However, to date, no framework or evaluation tool fulfills this purpose.

OBJECTIVE

Before developing a tool to help health care professionals assess and recommend apps to their patients, we aimed to create an overview of published criteria to describe and evaluate health apps.

METHODS

We conducted a systematic review to identify existing criteria for eHealth smartphone app evaluation. Relevant databases and trial registers were queried for articles. Articles were included that (1) described tools, guidelines, dimensions, or criteria to evaluate apps, (2) were available in full text, and (3) were written in English, French, German, Italian, Portuguese, or Spanish. We proposed a conceptual framework for app evaluation based on the dimensions reported in the selected articles. This was revised iteratively in discussion rounds with international stakeholders. The conceptual framework was used to synthesize the reported evaluation criteria. The list of criteria was discussed and refined by the research team.

RESULTS

Screening of 1258 articles yielded 128 (10.17%) that met the inclusion criteria. Of these 128 articles, 30 (23.4%) reported the use of self-developed criteria and described their development processes incompletely. Although 43 evaluation instruments were used only once, 6 were used in multiple studies. Most articles (83/128, 64.8%) did not report following theoretical guidelines; those that did noted 37 theoretical frameworks. On the basis of the selected articles, we proposed a conceptual framework to explore 6 app evaluation dimensions: context, stakeholder involvement, features and requirements, development processes, implementation, and evaluation. After standardizing the definitions, we identified 205 distinct criteria. Through consensus, the research team relabeled 12 of these and added 11 more-mainly related to ethical, legal, and social aspects-resulting in 216 evaluation criteria. No criteria had to be moved between dimensions.

CONCLUSIONS

This study provides a comprehensive overview of criteria currently used in clinical practice to describe and evaluate apps. This is necessary as no reviewed criteria sets were inclusive, and none included consistent definitions and terminology. Although the resulting overview is impractical for use in clinical practice in its current form, it confirms the need to craft it into a purpose-built, theory-driven tool. Therefore, in a subsequent step, based on our current criteria set, we plan to construct an app evaluation tool with 2 parts: a short section (including 1-3 questions/dimension) to quickly disqualify clearly unsuitable apps and a longer one to investigate more likely candidates in closer detail. We will use a Delphi consensus-building process and develop a user manual to prepare for this undertaking.

TRIAL REGISTRATION

PROSPERO International Prospective Register of Systematic Reviews CRD42021227064; https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42021227064.

Lessons Learned in Improving the Quality of a Free Reproductive Health Hotline in Benin

INTRODUCTION

Since 2006, the Ligne Verte project-a toll-free national hotline that provides counseling and provider linkages to family planning, HIV, and sexually transmitted infection services-has been implemented by the Association Béninoise pour le Marketing Social et la Communication pour la Santé (ABMS) in Benin. Annual evaluations have been conducted to monitor client satisfaction, with feedback mechanisms to learn from callers and staff, aiming to continually improve the program. We document lessons learned from these evaluations, outline the adaptation process to improve the quality of the hotline, present results from Ligne Verte caller and counselor studies, and describe how the studies fed client-driven changes to the project.

METHODS

Annual rounds of semistructured phone interviews with Ligne Verte callers and call center advisors were conducted. The sample of clients interviewed included consenting men and women aged 18 years and older who had used Ligne Verte services in the 3 months preceding the evaluation. All advisors working on the helpline were also eligible for inclusion in qualitative interviews.

RESULTS

From 2013 to 2020, most callers were satisfied with the Ligne Verte project. Users reported appreciating the accurate, timely, and confidential support they received from the helpline but noted the need to expand the service to other local languages and improve call quality. Call center advisors requested mental health support. In employing a systematic approach to program improvement, ABMS was able to add extended language services, upgrade their telecommunication hardware, and provide a psychologist for call center advisors. These program adaptations have increased caller satisfaction by 20 percentage points, and in 2020, 95% of callers reported being satisfied with their Ligne Verte experience.

CONCLUSION

Taking a consumer-centered approach, listening to user feedback, and making evidence-based recommendations to improve the Ligne Verte project has led to an overall increase in caller satisfaction.

"It reminds me and motivates me": Human-centered design and implementation of an interactive, SMS-based digital intervention to improve early retention on antiretroviral therapy: Usability and acceptability among new initiates in a high-volume, public clinic in Malawi

BACKGROUND

Early retention of people living with HIV (PLHIV) in antiretroviral therapy (ART) programs is critical to improve individual clinical outcomes and viral load suppression. Although many mobile health (mHealth) interventions aim to improve retention in care, there is still lack of evidence on mHealth success or failure, including from patient's perspectives. We describe the human-centered design (HCD) process and assess patient usability and acceptability of a two-way texting (2wT) intervention to improve early retention among new ART initiates at Lighthouse Trust clinic in Lilongwe, Malawi.

METHODS

An iterative HCD approach focused on patient and provider users' needs, incorporating feedback from multidisciplinary teams to adapt 2wT for the local, public clinic context. We present mixed-methods usability and acceptability results from 100 participants, 50 at 3-months and 50 at 6-months, post 2wT enrollment, and observations of these same patients completing core tasks of the 2wT system.

RESULTS

Among the 100 usability respondents, 95% were satisfied with visit reminders, and 88% would recommend reminders and motivational messages to friends; however, 17% were worried about confidentiality. In observation of participant task completion, 94% were able to successfully confirm visit attendance and 73% could request appointment date change. More participants in 4-6 months group completed tasks correctly compared to 1-3 months group, although not significantly different (78% vs. 66%, p = 0.181). Qualitative results were overwhelmingly positive, but patients did note confusion with transfer reporting and concern that 2wT would not reach patients without mobile phones or with lower literacy.

CONCLUSION

The 2wT app for early ART retention appears highly usable and acceptable, hopefully creating a solid foundation for lifelong engagement in care. The HCD approach put the local team central in this process, ensuring that both patients' and Lighthouse's priorities, policies, and practices were forefront in 2wT optimization, raising the likelihood of 2wT success in other routine program contexts.

Scoping review of health information technology usability methods leveraged in Africa

OBJECTIVE

The aim of this study was to explore the state of health information technology (HIT) usability evaluation in Africa.

MATERIALS AND METHODS

We searched three electronic databases: PubMed, Embase, and Association for Computing Machinery. We categorized the stage of evaluations, the type of interactions assessed, and methods applied using Stead's System Development Life Cycle (SDLC) and Bennett and Shackel's usability models.

RESULTS

Analysis of 73 of 1002 articles that met inclusion criteria reveals that HIT usability evaluations in Africa have increased in recent years and mainly focused on later SDLC stage (stages 4 and 5) evaluations in sub-Saharan Africa. Forty percent of the articles examined system-user-task-environment (type 4) interactions. Most articles used mixed methods to measure usability. Interviews and surveys were often used at each development stage, while other methods, such as quality-adjusted life year analysis, were only found at stage 5. Sixty percent of articles did not include a theoretical model or framework.

DISCUSSION

The use of multistage evaluation and mixed methods approaches to obtain a comprehensive understanding HIT usability is critical to ensure that HIT meets user needs.

CONCLUSIONS

Developing and enhancing usable HIT is critical to promoting equitable health service delivery and high-quality care in Africa. Early-stage evaluations (stages 1 and 2) and interactions (types 0 and 1) should receive special attention to ensure HIT usability prior to implementing HIT in the field.

Effectiveness of Smartphone-Based Community Case Management on the Urgent Referral, Reconsultation, and Hospitalization of Children Aged Under 5 Years in Malawi: Cluster-Randomized, Stepped-Wedge Trial

Background

Integrated community case management (CCM) has led to reductions in child mortality in Malawi resulting from illnesses such as malaria, pneumonia, and diarrhea. However, adherence to CCM guidelines is often poor, potentially leading to inappropriate clinical decisions and poor outcomes. We determined the impact of an e-CCM app on the referral, reconsultation, and hospitalization rates of children presenting to village clinics in Malawi.

Objective

We determined the impact of an electronic version of a smartphone-based CCM (e-CCM) app on the referral, reconsultation, and hospitalization rates of children presenting to village clinics in Malawi.

Methods

We used a stepped-wedge, cluster-randomized trial to compare paper-based CCM (control) with and without the use of an e-CCM app on smartphones from November 2016 to February 2017. A total of 102 village clinics from 2 districts in northern Malawi were assigned to 1 of 6 clusters, which were randomized on the sequencing of the crossover from the control phase to the intervention phase as well as the duration of exposure in each phase. Children aged ≥2 months to <5 years who presented with acute illness were enrolled consecutively by health surveillance assistants. The primary outcome of urgent referrals to higher-level facilities was evaluated by using multilevel mixed effects models. A logistic regression model with the random effects of the cluster and the fixed effects for each step was fitted. The adjustment for potential confounders included baseline factors, such as patient age, sex, and the geographical location of the village clinics. Calendar time was adjusted for in the analysis.

Results

A total of 6965 children were recruited—49.11% (3421/6965) in the control phase and 50.88% (3544/6965) in the intervention phase. After adjusting for calendar time, children in the intervention phase were more likely to be urgently referred to a higher-level health facility than children in the control phase (odds ratio [OR] 2.02, 95% CI 1.27-3.23; P=.003). Overall, children in the intervention arm had lower odds of attending a repeat health surveillance assistant consultation (OR 0.45, 95% CI 0.34-0.59; P<.001) or being admitted to a hospital (OR 0.75, 95% CI 0.62-0.90; P=.002), but after adjusting for time, these differences were not significant (P=.07 for consultation; P=.30 for hospital admission).

Conclusions

The addition of e-CCM decision support by using smartphones led to a greater proportion of children being referred to higher-level facilities, with no apparent increase in hospital admissions or repeat consultations in village clinics. Our findings provide support for the implementation of e-CCM tools in Malawi and other low- and middle-income countries with a need for ongoing assessments of effectiveness and integration with national digital health strategies.

Trial Registration

ClinicalTrials.gov NCT02763345; https://clinicaltrials.gov/ct2/show/NCT02763345