Leveraging electronic tablets for general pediatric care: a pilot study

Digital Health Technology to Enhance Adolescent and Young Adult Clinical Preventive Services: Affordances and Challenges

The lives of adolescents and young adults (AYAs) have become increasingly intertwined with technology. In this scoping review, studies about digital health tools are summarized in relation to five key affordances-social, cognitive, identity, emotional, and functional. Consideration of how a platform or tool exemplifies these affordances may help clinicians and researchers achieve the goal of using digital health technology to enhance clinical preventive services for AYAs. Across these five affordances, considerable research and development activity exists accompanied by signs of high promise, although the literature primarily reflects demonstration studies of acceptability or small sample experiments to discern impact. Digital health technology may afford an array of functions, yet its potential to enhance AYA clinical preventive services is met with three key challenges. The challenges discussed in this review are the disconnectedness between digital health tools and clinical care, threats to AYA privacy and security, and difficulty identifying high-value digital health products for AYA. The data presented are synthesized in calls to action for the use of digital health technology to enhance clinical preventive services and to ensure that the digital health ecosystem is relevant, effective, safe, and purposed for meeting the health needs of AYA.

Effect of an Alternative Newborn Naming Strategy on Wrong-Patient Errors: A Quasi-Experimental Study

OBJECTIVES

Newborns are often assigned temporary names at birth. Temporary newborn names-often a combination of the mother's last name and the newborn's gender-are vulnerable to patient misidentification due to similarities with other newborns or between a mother and her newborn. We developed and implemented an alternative distinct naming strategy, and then compared its effectiveness on reducing the number of wrong-patient orders with the standard distinct naming strategy.

METHODS

This study was conducted over a 14-month period in the newborn nursery and neonatal intensive care units of three hospitals that were part of the same health care system. We used a quasi-experimental study design using interrupted time series analysis to compare the differences in wrong-patient orders (an indicator of patient misidentification) before and after the implementation of the alternative distinct naming strategy.

RESULTS

Overall, there were 25 wrong-patient errors per 10,000 orders during entire study period (36.8 per 10,000 before and 19.6 per 10,000 after). However, there was no statistically significant change in the rate of wrong-patient ordering errors after the transition from the distinct to the alternative distinct naming strategy (β = 0.832, 95% confidence interval [CI] = -0.83 to 2.49, p = 0.326). We also found that, overall, 1.7% of the clinicians contributed to 62% of the wrong-patient errors.

CONCLUSION

Although we did not find statistically significant differences in wrong-patient errors, the alternative distinct naming approach provides pragmatic advantages over its predecessors. In addition, the localization of wrong-patient errors within a small set of clinicians highlights the potential for developing strategies for delivering training to clinicians.

Effect of a Computer-Based Decision Support Intervention on Autism Spectrum Disorder Screening in Pediatric Primary Care Clinics: A Cluster Randomized Clinical Trial

IMPORTANCE

Universal early screening for autism spectrum disorder (ASD) is recommended but not routinely performed.

OBJECTIVE

To determine whether computer-automated screening and clinical decision support can improve ASD screening rates in pediatric primary care practices.

DESIGN, SETTING, AND PARTICIPANTS

This cluster randomized clinical trial, conducted between November 16, 2010, and November 21, 2012, compared ASD screening rates among a random sample of 274 children aged 18 to 24 months in urban pediatric clinics of an inner-city county hospital system with or without an ASD screening module built into an existing decision support software system. Statistical analyses were conducted from February 6, 2017, to June 1, 2018.

INTERVENTIONS

Four clinics were matched in pairs based on patient volume and race/ethnicity, then randomized within pairs. Decision support with the Child Health Improvement Through Computer Automation system (CHICA) was integrated with workflow and with the electronic health record in intervention clinics.

MAIN OUTCOMES AND MEASURES

The main outcome was screening rates among children aged 18 to 24 months. Because the intervention was discontinued among children aged 18 months at the request of the participating clinics, only results for those aged 24 months were collected and analyzed. Rates of positive screening results, clinicians' response rates to screening results in the computer system, and new cases of ASD identified were also measured. Main results were controlled for race/ethnicity and intracluster correlation.

RESULTS

Two clinics were randomized to receive the intervention, and 2 served as controls. Records from 274 children (101 girls, 162 boys, and 11 missing information on sex; age range, 23-30 months) were reviewed (138 in the intervention clinics and 136 in the control clinics). Of 263 children, 242 (92.0%) were enrolled in Medicaid, 138 (52.5%) were African American, and 96 (36.5%) were Hispanic. Screening rates in the intervention clinics increased from 0% (95% CI, 0%-5.5%) at baseline to 68.4% (13 of 19) (95% CI, 43.4%-87.4%) in 6 months and to 100% (18 of 18) (95% CI, 81.5%-100%) in 24 months. Control clinics had no significant increase in screening rates (baseline, 7 of 64 children [10.9%]; 6-24 months after the intervention, 11 of 72 children [15.3%]; P = .46). Screening results were positive for 265 of 980 children (27.0%) screened by CHICA during the study period. Among the 265 patients with positive screening results, physicians indicated any response in CHICA in 151 (57.0%). Two children in the intervention group received a new diagnosis of ASD within the time frame of the study.

CONCLUSIONS AND RELEVANCE

The findings suggest that computer automation, when integrated with clinical workflow and the electronic health record, increases screening of children for ASD, but follow-up by physicians is still flawed. Automation of the subsequent workup is still needed.

TRIAL REGISTRATION

ClinicalTrials.gov identifier: NCT01612897.

Let Them Speak for Themselves: Improving Adolescent Self-Report Rate on Pre-Visit Screening

BACKGROUND

Adolescent pre-visit screening on patient-generated health data is a common and efficient practice to guide clinical decision making. However, proxy informants (eg, parents or caregivers) often complete these forms, which may lead to incorrect information or lack of confidentiality. Our objective was to improve the adolescent self-report rate on pre-visit screening.

METHODS

We conducted an interventional study using an interrupted time series design to compare adolescent self-report rates (percent of adolescents ages 12-18 years completing their own pre-visit screening) over 16 months in general pediatric ambulatory clinics. We collected data using a computerized clinical decision support system with waiting room electronic tablet screening. Preintervention rates were low, and we created and implemented 2 electronic workflow alerts, one each to the patient/caregiver and clinical staff, reminding them that the adolescent should answer the questions independently. We included the first encounter from each adolescent and evaluated changes in adolescent self-reporting between pre- and postintervention periods using interrupted time series analysis.

RESULTS

Patients or caregivers completed 2670 qualifying pre-visit screenings across 19 preintervention, 7 intervention, and 44 postintervention weeks. Self-reporting by younger adolescents nearly doubled, with a significant increase of 19.3 percentage points (confidence interval [CI], 9.1-29.5) from the baseline 20.5%. Among older adolescents, the stable baseline rate of 53.6% increased by 9.2 absolute percentage points (CI, -7.0 to 25.3). There were no significant pre- or postintervention secular trends.

CONCLUSIONS

Two automated alerts directing clinic personnel and families to have adolescents self-report significantly and sustainably improved younger adolescent self-reporting on electronic patient-generated health data instruments.

The Other Parent: An Exploratory Survey of Providers' Engagement of Fathers in Pediatric Primary Care

Although father engagement in pediatric care is associated with positive child health outcomes, pediatric primary care providers (PCPs) often focus on the mother-child dyad. This study sought to characterize pediatric PCPs' engagement of fathers in care. Pediatric PCPs affiliated with an academic health system were invited to complete an online survey. The primary outcome was the proportion of providers who routinely implement American Academy of Pediatrics recommendations for father engagement. There were 100 respondents. Of the 23 recommended practices for engaging fathers, 18 were routinely implemented by <50% of respondents. The least routinely implemented practices were parenting skills support (4%) and perinatal depression screening (5%). The most commonly endorsed barriers included lack of father attendance at visits (91%) and time constraints (75%). Despite the American Academy of Pediatrics recommendations, pediatric PCPs do not routinely engage fathers in care. Effective strategies are needed to reduce barriers and improve father engagement among pediatric providers.

Improving Response Rates and Representation of Hard-to-Reach Groups in Family Experience Surveys

OBJECTIVE

Most US hospitals conduct patient experience surveys by mail or telephone after discharge to assess patient/family centeredness of care. Pediatric response rates are usually very low, especially for black, Latino, and low-income respondents. We investigated whether day of discharge surveying using tablets improves response rates and respondent representativeness.

METHODS

This was a quasi-experimental study of parents of patients discharged from 4 units of a children's hospital. Parents were assigned to receive the Child Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) via an audio-enabled tablet before discharge or via mail at approximately 1 week postdischarge. Intervention and control conditions alternated by week. We compared response rates, child/respondent characteristics, and mean top-box scores between tablet and mail only arms.

RESULTS

Administering Child HCAHPS on a tablet was administratively feasible and did not interfere with the discharge process (median completion time, 12.4 minutes). The response rate was 71.1% (424 of 596) for tablet versus 16.3% (96 of 588) for mail only. Although the tablet response rate was higher in every subgroup, tablet respondents were more likely to be fathers (20.4% vs 6.4%; P = .006), more likely to have a high school education or less (17.5% vs 8.4%; P = .002), less likely to be white (56.8% vs 71.9%; P = .006), and more likely to be publicly insured (31.4% vs 19.8%; P = .02). Tablet scores were significantly higher than mail only scores for 3 of 17 measures.

CONCLUSIONS

The response rate for day of discharge tablet survey administration was >4-fold higher than with single-wave mail-only administration, with greater participation of hard-to-reach groups. These findings suggest tablet administration before discharge shows great promise for real-time feedback and QI and may transform the field of inpatient survey administration.

Prevalence of pain reports in pediatric primary care and association with demographics, body mass index, and exam findings: a cross-sectional study

BACKGROUND

Pediatric pain is associated to patient weight and demographics in specialized settings, but pain prevalence and its associated patient attributes in general pediatric outpatient care are unknown. Our objective was to determine the rate of positive pain screenings in pediatric primary care and evaluate the relationship between reported pain and obesity, demographics, and exam findings during routine pediatric encounters.

METHODS

Cross-sectional observational study of 26,180 patients ages 2 to 19 seen in five urban pediatric primary care clinics between 2009 and 2016. Data were collected from systematic screening using a computerized clinical decision support system. Multivariable logistic regressions were used to analyze the association between pain reporting and obesity (body mass index), age, sex, race, season, insurance status, clinic site, prior pain reporting, pain reporting method, and exam findings.

RESULTS

Pain was reported by the patient or caregiver in 14.9% of visits. In adjusted models, pain reporting was associated with obesity (Odds Ratio (OR) 1.23, 95% Confidence Intervals (CI) 1.11-1.35) and severe obesity (OR 1.32, CI 1.17-1.49); adolescents (OR 1.47, CI 1.33-1.61); and females (OR 1.21, CI 1.12-1.29). Pain reported at the preceding visit increased odds of pain reporting 2.67 times (CI 2.42-2.95). Abnormal abdominal, extremity, ear, nose, throat, and lymph node exams were associated with pain reporting. Pain reporting increased in minority races within clinics that predominantly saw a concordant race.

CONCLUSIONS

Pain is common in general pediatric encounters, and occurs more frequently in obese children and those who previously reported pain. Pain reporting may be influenced by seasonal variation and clinic factors. Future pediatric pain screening may be guided by associated risk factors to improve identification and targeted healthcare interventions.

Screening for Behavioral Risk Factors Is Not Enough to Improve Preventive Services Delivery

OBJECTIVE

Unhealthy behaviors are a major cause of chronic disease. Preappointment screening has been suggested as one way to improve preventive care delivery related to these behaviors by specifying risks to be addressed. We aimed to determine whether screening for health-related behaviors before the clinical encounter will lead to higher counseling rate and service delivery by clinicians.

METHODS

We used a pre/post design in one practice with a control practice to evaluate the effects of preappointment screening for 3 behavioral risk factors (tobacco smoke exposure, no recent dental care visit, and consumption of sugar-sweetened beverages). After their clinic visit, we asked English-speaking parents whose child had one or more risk factor whether they had received counseling or services from their pediatrician to address them.

RESULTS

We recruited 264 parents in the pre phase and 242 in the post phase. Among 215 parents whose child had one or more risk factors, parents in the post phase were as likely to report receiving counseling than parents in the pre phase for each of the risk factors: smoking odds ratio 6.75 (95% confidence interval, 0.51, 88.88), dental health odds ratio 1.44 (95% confidence interval, 0.47, 4.41), and sugar-sweetened beverage consumption odds ratio 0.34 (95% confidence interval, 0.23, 5.18). Service delivery and reported behavior change were also similar in both phases.

CONCLUSIONS

Counseling rates for tobacco, dental health, or sugar-sweetened beverage consumption were low in pediatric primary care, and preappointment screening did not significantly affect clinician counseling. Future efforts will require a more robust approach to effect change in counseling, provision of service, and family behavior.

A six-year repeated evaluation of computerized clinical decision support system user acceptability

OBJECTIVE

Long-term acceptability among computerized clinical decision support system (CDSS) users in pediatrics is unknown. We examine user acceptance patterns over six years of our continuous computerized CDSS integration and updates.

MATERIALS AND METHODS

Users of Child Health Improvement through Computer Automation (CHICA), a CDSS integrated into clinical workflows and used in several urban pediatric community clinics, completed annual surveys including 11 questions covering user acceptability. We compared responses across years within a single healthcare system and between two healthcare systems. We used logistic regression to assess the odds of a favorable response to each question by survey year, clinic role, part-time status, and frequency of CHICA use.

RESULTS

Data came from 380 completed surveys between 2011 and 2016. Responses were significantly more favorable for all but one measure by 2016 (OR range 2.90-12.17, all p < 0.01). Increasing system maturity was associated with improved perceived function of CHICA (OR range 4.24-7.58, p < 0.03). User familiarity was positively associated with perceived CDSS function (OR range 3.44-8.17, p < 0.05) and usability (OR range 9.71-15.89, p < 0.01) opinions.

CONCLUSION

We present a long-term, repeated follow-up of user acceptability of a CDSS. Favorable opinions of the CDSS were more likely in frequent users, physicians and advanced practitioners, and full-time workers. CHICA acceptability increased as it matured and users become more familiar with it. System quality improvement, user support, and patience are important in achieving wide-ranging, sustainable acceptance of CDSS.

Developing Electronic Health Record (EHR) Strategies Related to Health Center Patients' Social Determinants of Health

BACKGROUND

"Social determinants of heath" (SDHs) are nonclinical factors that profoundly affect health. Helping community health centers (CHCs) document patients' SDH data in electronic health records (EHRs) could yield substantial health benefits, but little has been reported about CHCs' development of EHR-based tools for SDH data collection and presentation.

METHODS

We worked with 27 diverse CHC stakeholders to develop strategies for optimizing SDH data collection and presentation in their EHR, and approaches for integrating SDH data collection and the use of those data (eg, through referrals to community resources) into CHC workflows.

RESULTS

We iteratively developed a set of EHR-based SDH data collection, summary, and referral tools for CHCs. We describe considerations that arose while developing the tools and present some preliminary lessons learned.

CONCLUSION

Standardizing SDH data collection and presentation in EHRs could lead to improved patient and population health outcomes in CHCs and other care settings. We know of no previous reports of processes used to develop similar tools. This article provides an example of 1 such process. Lessons from our process may be useful to health care organizations interested in using EHRs to collect and act on SDH data. Research is needed to empirically test the generalizability of these lessons.

Follow-up of Mothers with Suspected Postpartum Depression from Pediatrics Clinics

PURPOSE

Pediatric providers are increasingly screening for postpartum depression (PD), yet, it is unknown how often mothers comply with recommendations to seek treatment. The objectives were to describe the rate at which mothers with suspected PD seek treatment and explore factors that predict help-seeking behavior.

DESIGN AND METHODS

Mothers were recruited from four pediatric clinics after identification using the Child Health Improvement through Computer Automation (CHICA) system. Mothers with a positive screen were invited to participate in a telephone interview between January 2012 and December 2014. Mothers reported if they sought treatment or called a community resource.

RESULTS

73 of 133 eligible mothers participated (55% response rate). Fifty women recalled a recommendation to seek help. Only 43.8% (32/73) made a follow-up appointment with an adult provider and even fewer kept the appointment.

CONCLUSION

A majority of mothers suspected of having PD recalled a referral for further intervention; yet, less than half took action. Further investigation of barriers of help-seeking behavior is warranted.

Parent Health Literacy, Depression, and Risk for Pediatric Injury

BACKGROUND

Population-wide research on the impact of parent health literacy to children's health outcomes is limited. We assessed the relationship of low parent health literacy to a range of pediatric health risks within a large cohort of primary care patients.

METHODS

Data were from 17 845 English- and Spanish-speaking parents of children aged ≤7 years presenting for well-child care. We used a 3-item screener to measure health literacy. Outcomes included secondhand smoke exposure, asthma treatment nonadherence, parent depression, child-rearing practices, injury prevention, and parent first-aid knowledge. We summarized study variables with descriptive statistics and then performed multivariable logistic regression to identify associations between low parent literacy and our dependent measures.

RESULTS

Mean child age was 4.8 years (SD 3.7); 36.5% of parent respondents had low health literacy. In models adjusted for child gender, race/ethnicity, insurance, age, and parent language preference, low parent health literacy was related to a range of pediatric health risks, including parent depression (adjusted odds ratio [AOR] 1.32; 95% confidence interval 1.18-1.48), firearm access (AOR 1.68; 1.49-1.89), not having a working smoke detector (AOR 3.54; 2.74-4.58), and lack of first-aid knowledge about choking (AOR 1.67; 1.44-1.93) and burns (AOR 1.45; 1.29-1.63). Children of parents with low health literacy were also more likely to watch >2 hours of television per day (AOR 1.27; 1.17-1.36).

CONCLUSIONS

Low parent health literacy is independently and significantly related to parent depression, child television viewing, and at-risk family behaviors associated with child injury. Use of low-literacy approaches to health-behavior interventions may be essential to address common child morbidities.

Feasibility of an Electronic Survey on iPads with In-Person Data Collectors for Data Collection with Health Care Professionals and Health Care Consumers in General Emergency Departments

BACKGROUND

Translating Emergency Knowledge for Kids was established to bridge the research-practice gap in pediatric emergency care by bringing the best evidence to Canadian general emergency departments (EDs). The first step in this process was to conduct a national needs assessment to determine the information needs and preferences of health professionals and parents in this clinical setting.

OBJECTIVE

To describe the development and implementation of two electronic surveys, and determine the feasibility of collecting electronic survey data on iPads with in-person data collectors in a busy clinical environment.

METHODS

Two descriptive surveys were conducted in 32 general EDs. Specific factors were addressed in four survey development and implementation stages: survey design, survey delivery, survey completion, and survey return. Feasibility of the data collection approach was determined by evaluating participation rates, completion rates, average survey time to completion, and usability of the platform. Usability was assessed with the in-person data collectors on five key variables: interactivity, portability, innovativeness, security, and proficiency.

RESULTS

Health professional participation rates (1561/2575, 60.62%) and completion rates (1471/1561, 94.23%) were strong. Parental participation rates (974/1099, 88.63%) and completion rates (897/974, 92.09%) were excellent. Mean time to survey completion was 28.08 minutes for health professionals and 43.23 minutes for parents. Data collectors rated the platform "positively" to "very positively" on all five usability variables.

CONCLUSIONS

A number of design and implementation considerations were explored and integrated into this mixed-mode survey data collection approach. Feasibility was demonstrated by the robust survey participation and completion rates, reasonable survey completion times, and very positive usability evaluation results.

Experience with decision support system and comfort with topic predict clinicians' responses to alerts and reminders

OBJECTIVE

Clinicians at our institution typically respond to about half of the prompts they are given by the clinic's computer decision support system (CDSS). We sought to examine factors associated with clinician response to CDSS prompts as part of a larger, ongoing quality improvement effort to optimize CDSS use.

METHODS

We examined patient, prompt, and clinician characteristics associated with clinician response to decision support prompts from the Child Health Improvement through Computer Automation (CHICA) system. We asked pediatricians who were nonusers of CHICA to rate decision support topics as "easy" or "not easy" to discuss with patients and their guardians. We analyzed these ratings and data, from July 1, 2009 to January 29, 2013, utilizing a hierarchical regression model, to determine whether factors such as comfort with the prompt topic and the length of the user's experience with CHICA contribute to user response rates.

RESULTS

We examined 414 653 prompts from 22 260 patients. The length of time a clinician had been using CHICA was associated with an increase in their prompt response rate. Clinicians were more likely to respond to topics rated as "easy" to discuss. The position of the prompt on the page, clinician gender, and the patient's age, race/ethnicity, and preferred language were also predictive of prompt response rate.

CONCLUSION

This study highlights several factors associated with clinician prompt response rates that could be generalized to other health information technology applications, including the clinician's length of exposure to the CDSS, the prompt's position on the page, and the clinician's comfort with the prompt topic. Incorporating continuous quality improvement efforts when designing and implementing health information technology may ensure that its use is optimized.

Pediatric decision support using adapted Arden Syntax

BACKGROUND

Pediatric guidelines based care is often overlooked because of the constraints of a typical office visit and the sheer number of guidelines that may exist for a patient's visit. In response to this problem, in 2004 we developed a pediatric computer based clinical decision support system using Arden Syntax medical logic modules (MLM).

METHODS

The Child Health Improvement through Computer Automation system (CHICA) screens patient families in the waiting room and alerts the physician in the exam room. Here we describe adaptation of Arden Syntax to support production and consumption of patient specific tailored documents for every clinical encounter in CHICA and describe the experiments that demonstrate the effectiveness of this system.

RESULTS

As of this writing CHICA has served over 44,000 patients at 7 pediatric clinics in our healthcare system in the last decade and its MLMs have been fired 6182,700 times in "produce" and 5334,021 times in "consume" mode. It has run continuously for over 10 years and has been used by 755 physicians, residents, fellows, nurse practitioners, nurses and clinical staff. There are 429 MLMs implemented in CHICA, using the Arden Syntax standard. Studies of CHICA's effectiveness include several published randomized controlled trials.

CONCLUSIONS

Our results show that the Arden Syntax standard provided us with an effective way to represent pediatric guidelines for use in routine care. We only required minor modifications to the standard to support our clinical workflow. Additionally, Arden Syntax implementation in CHICA facilitated the study of many pediatric guidelines in real clinical environments.

Comparison of self-administered survey questionnaire responses collected using mobile apps versus other methods

BACKGROUND

Self-administered survey questionnaires are an important data collection tool in clinical practice, public health research and epidemiology. They are ideal for achieving a wide geographic coverage of the target population, dealing with sensitive topics and are less resource-intensive than other data collection methods. These survey questionnaires can be delivered electronically, which can maximise the scalability and speed of data collection while reducing cost. In recent years, the use of apps running on consumer smart devices (i.e., smartphones and tablets) for this purpose has received considerable attention. However, variation in the mode of delivering a survey questionnaire could affect the quality of the responses collected.

OBJECTIVES

To assess the impact that smartphone and tablet apps as a delivery mode have on the quality of survey questionnaire responses compared to any other alternative delivery mode: paper, laptop computer, tablet computer (manufactured before 2007), short message service (SMS) and plastic objects.

SEARCH METHODS

We searched MEDLINE, EMBASE, PsycINFO, IEEEXplore, Web of Science, CABI: CAB Abstracts, Current Contents Connect, ACM Digital, ERIC, Sociological Abstracts, Health Management Information Consortium, the Campbell Library and CENTRAL. We also searched registers of current and ongoing clinical trials such as ClinicalTrials.gov and the World Health Organization (WHO) International Clinical Trials Registry Platform. We also searched the grey literature in OpenGrey, Mobile Active and ProQuest Dissertation & Theses. Lastly, we searched Google Scholar and the reference lists of included studies and relevant systematic reviews. We performed all searches up to 12 and 13 April 2015.

SELECTION CRITERIA

We included parallel randomised controlled trials (RCTs), crossover trials and paired repeated measures studies that compared the electronic delivery of self-administered survey questionnaires via a smartphone or tablet app with any other delivery mode. We included data obtained from participants completing health-related self-administered survey questionnaire, both validated and non-validated. We also included data offered by both healthy volunteers and by those with any clinical diagnosis. We included studies that reported any of the following outcomes: data equivalence; data accuracy; data completeness; response rates; differences in the time taken to complete a survey questionnaire; differences in respondent's adherence to the original sampling protocol; and acceptability to respondents of the delivery mode. We included studies that were published in 2007 or after, as devices that became available during this time are compatible with the mobile operating system (OS) framework that focuses on apps.

DATA COLLECTION AND ANALYSIS

Two review authors independently extracted data from the included studies using a standardised form created for this systematic review in REDCap. They then compared their forms to reach consensus. Through an initial systematic mapping on the included studies, we identified two settings in which survey completion took place: controlled and uncontrolled. These settings differed in terms of (i) the location where surveys were completed, (ii) the frequency and intensity of sampling protocols, and (iii) the level of control over potential confounders (e.g., type of technology, level of help offered to respondents). We conducted a narrative synthesis of the evidence because a meta-analysis was not appropriate due to high levels of clinical and methodological diversity. We reported our findings for each outcome according to the setting in which the studies were conducted.

MAIN RESULTS

We included 14 studies (15 records) with a total of 2275 participants; although we included only 2272 participants in the final analyses as there were missing data for three participants from one included study.Regarding data equivalence, in both controlled and uncontrolled settings, the included studies found no significant differences in the mean overall scores between apps and other delivery modes, and that all correlation coefficients exceeded the recommended thresholds for data equivalence. Concerning the time taken to complete a survey questionnaire in a controlled setting, one study found that an app was faster than paper, whereas the other study did not find a significant difference between the two delivery modes. In an uncontrolled setting, one study found that an app was faster than SMS. Data completeness and adherence to sampling protocols were only reported in uncontrolled settings. Regarding the former, an app was found to result in more complete records than paper, and in significantly more data entries than an SMS-based survey questionnaire. Regarding adherence to the sampling protocol, apps may be better than paper but no different from SMS. We identified multiple definitions of acceptability to respondents, with inconclusive results: preference; ease of use; willingness to use a delivery mode; satisfaction; effectiveness of the system informativeness; perceived time taken to complete the survey questionnaire; perceived benefit of a delivery mode; perceived usefulness of a delivery mode; perceived ability to complete a survey questionnaire; maximum length of time that participants would be willing to use a delivery mode; and reactivity to the delivery mode and its successful integration into respondents' daily routine. Finally, regardless of the study setting, none of the included studies reported data accuracy or response rates.

AUTHORS' CONCLUSIONS

Our results, based on a narrative synthesis of the evidence, suggest that apps might not affect data equivalence as long as the intended clinical application of the survey questionnaire, its intended frequency of administration and the setting in which it was validated remain unchanged. There were no data on data accuracy or response rates, and findings on the time taken to complete a self-administered survey questionnaire were contradictory. Furthermore, although apps might improve data completeness, there is not enough evidence to assess their impact on adherence to sampling protocols. None of the included studies assessed how elements of user interaction design, survey questionnaire design and intervention design might influence mode effects. Those conducting research in public health and epidemiology should not assume that mode effects relevant to other delivery modes apply to apps running on consumer smart devices. Those conducting methodological research might wish to explore the issues highlighted by this systematic review.