Most response-inducing strategies do not increase participation in observational studies: a systematic review and meta-analysis

Strategies to increase survey participation: A randomized controlled study in a population of breast cancer survivors

BACKGROUND

Data collection by mailing questionnaires to the study population is one of the main research methods in epidemiologic studies. As participation rates are decreasing, easy-to-implement and cost-effective strategies to increase survey participation are needed. In this study, we tested the effect of a pragmatic combination of evidence-based interventions.

METHODS

We conducted a two-armed randomized controlled trial, nested in a cohort of breast cancer survivors (n = 1000) in the setting of a health outcomes survey. The intervention arm received a postal pre-notification, a non-monetary incentive (ballpoint with the study logo) and an alternative invitation letter in which several lay-out and textual adjustments were implemented according to behavioural science techniques. The alternative invitation letter also contained a QR-code through which an information video about the study could be accessed. The control arm was invited according to standard practice. Participants had the option to fill-out a questionnaire either on paper or online. A questionnaire with more than 50% of the questions answered classified as participation.

RESULTS

Overall participation rate was 62.9%. No significant difference in participation rate was observed between intervention and control arm (64.5% vs 61.3%, Risk Ratio (RR) 1.05, 95% CI [0.96 - 1.16]). Older age at study (>65 vs <51 years), and high socio-economic status (highest vs lowest quartile) were associated with higher participation rates (RR 1.30, 95% CI [1.07 - 1.57] and 1.24, 95% CI [1.09 - 1.42] respectively). In-situ carcinoma compared to invasive cancer and longer interval since treatment were associated with lower participation (RR 0.86, 95% CI [0.74 - 0.99] and RR 0.92, 95% CI [0.87 - 0.99] per 5 year increase, respectively).

CONCLUSION

Overall, the combination of four interventions tested in this study did not improve survey participation among breast cancer survivors. The overall participation rate was relatively high, possibly due to the study population of cancer survivors.

Implementation of a Mentoring Program for Mentee-Mentor Satisfaction: A Longitudinal Pilot Study

Introduction

Mentoring is a unique educational workplace relationship that can support both the mentee and mentor's skill, knowledge, social, and emotional needs. The primary aim of this longitudinal pilot study was to implement a formal mentoring program to assess its effect on mentee and mentor satisfaction.

Methods

Data was collected from two hospitals in New South Wales, Australia in late 2018 and early 2019. Junior doctors (mentees) and senior medical staff (mentors) were asked to complete pre-, mid-, and oost-program surveys, with questions relevant to mentee-mentor satisfaction, interactions, and participation. Mixed-effects ordinal logistic regression models were used to assess the program effect on mentee-mentor satisfaction, while Fishers' exact test was used to evaluate mentee-mentor interactions and participation.

Results

Although there was evidence of upwards trends in the proportion of mentees and mentors who reported their satisfaction in the program as excellent and rated their work satisfaction as being very influenced by the program, both trends were statistically non-significant. While our study was likely underpowered, high participation rates provide promising evidence of the program's acceptability and feasibility.

Conclusion

Though not reaching statistical significance, study results suggest that the implementation of a mentoring program has the potential to increase satisfaction levels among its participants, be they mentees or mentors. It is recommended that future studies recruit larger samples thereby having sufficient statistical power. Furthermore, causality should be explored in more detail through a multi-site randomized controlled trial design.

Engaging Adolescents in Contemporary Longitudinal Health Research: Strategies for Promoting Participation and Retention

PURPOSE

High (and nonselective) recruitment and retention rates in longitudinal studies of adolescence are essential for illuminating health trajectories and determinants during this critical period. Knowledge of optimal recruitment and retention strategies must keep pace with emerging challenges and opportunities, such as the shifts towards digitally-based data collection.

METHODS

We used a narrative review approach to synthesize research on promising recruitment and retention strategies for optimizing engagement in the next generation of longitudinal adolescent health studies.

RESULTS

We identified a small number of well-evidenced strategies, emerging challenges and opportunities for recruitment and retention in contemporary studies, and key evidence gaps. Core recommendations include the use of well-evidenced strategies (e.g., incentivizing participation, reducing barriers and burden, and investing in building positive relationships with participants) and coproducing recruitment and retention strategies with adolescents and parents of adolescents.

DISCUSSION

More research is needed into successful recruitment/retention strategies for digital/remote data collection methods, but initial evidence suggests that adopting principles and adapting well-evidenced strategies from traditional longitudinal studies is promising.

A comparison of survey incentive methods to recruit rural cancer survivors into cancer care delivery research studies

Unconditional (upfront) incentives are proposed to improve acceptance of cancer research among underrepresented, racial/ethnic minority populations, but few studies have tested incentive strategies among rural cancer survivors. Descriptive statistics summarized demographic characteristics of survey respondents, and response rates by arm were compared using Chi-square tests. We compared upfront ($2) and response-based ($10 conditional) incentives in a mailed survey of adult post-treatment rural survivors. Individuals meeting eligibility criteria from the electronic medical record (n = 2,830) were randomized into two incentive arms (n = 1,414 for the upfront arm and n = 1,416 for the contingent arm). Of the total delivered, presumed eligible participants (n = 1,304 upfront arm; n = 1,317 contingent arm), 67.8% were aged 65y+, 49.8% were female, and 95.1% were non-Hispanic white. The response rate for all participants was 18.5%. We received eligible surveys from 281 rural survivors in the first arm (response rate: 21.5%); and 205 surveys in the second arm (response rate: 15.6%). Participants who received the upfront incentive had a higher response rate than those receiving a response-based incentive, X2 (1, 2,621) = 15.53, p < 0.0001. Incentivizing survey completion with an upfront $2 bill encouraged a higher survey response rate; other supplemental strategies are needed to achieve a higher response rate for this population.

Digital culturally tailored marketing for enrolling Latino participants in a web-based registry: Baseline metrics from the Brain Health Registry

INTRODUCTION

This culturally tailored enrollment effort aims to determine the feasibility of enrolling 5000 older Latino adults from California into the Brain Health Registries (BHR) over 2.25 years.

METHODS

This paper describes (1) the development and deployment of culturally tailored BHR websites and digital ads, in collaboration with a Latino community science partnership board and a marketing company; (2) an interim feasibility analysis of the enrollment efforts and numbers, and participant characteristics (primary aim); as well as (3) an exploration of module completion and a preliminary efficacy evaluation of the culturally tailored digital efforts compared to BHR's standard non-culturally tailored efforts (secondary aim).

RESULTS

In 12.5 months, 3603 older Latino adults were enrolled (71% of the total California Latino BHR initiative enrollment goal). Completion of all BHR modules was low (6%).

DISCUSSION

Targeted ad placement, culturally tailored enrollment messaging, and culturally tailored BHR websites increased enrollment of Latino participants in BHR, but did not translate to increased module completion.

HIGHLIGHTS

Culturally tailored social marketing and website improvements were implemented. The efforts enrolled 5662 Latino individuals in 12.5 months. The number of Latino Brain Health Registry (BHR) participants increased by 122.7%. We failed to adequately enroll female Latinos and Latinos with lower education. Future work will evaluate effects of a newly released Spanish-language BHR website.

Randomized control trial of unconditional versus conditional incentives to increase study enrollment rates in participants at increased risk of lung cancer

INTRODUCTION

Understanding how incentives and their timing influence study enrollment rates is important to efficient study design and increasing the generalizability of findings. This 2-arm, parallel randomized trial evaluated how conditional vs. unconditional mailed incentives of a $20 gift card affected study enrollment in a sample of participants screened for lung cancer screening.

METHODS

Eligible participants included Black and White adults who underwent lung cancer screening with low-dose CT and had negative screening results at two North Carolina imaging facilities in 2018. We used a stratified randomization scheme, by sex and race, to assign incentive type (conditional vs. unconditional). We used the Tailored Design Method with six points of mailed contact to engage participants. We compared study enrollment rates using chi-square tests and logistic regression analyses.

RESULTS

After adjusting for sex, race, age, smoking status, participant residence, and screening site, participants who received unconditional incentives were 74% more likely to enroll than those who received conditional incentives (adjusted OR = 1.74 (95% CI: 1.01, 3.00).

CONCLUSIONS

Type of incentive can play a role in increasing study enrollment, especially mailed surveys that target individuals who currently or previously smoked. Unconditional incentives may be worth the initial cost to engage study participants.

Improving postal survey response using behavioural science: a nested randomised control trial

Background

Systematic reviews have identified effective strategies for increasing postal response rates to questionnaires; however, most studies have isolated single techniques, testing the effect of each one individually. Despite providing insight into explanatory mechanisms, this approach lacks ecological validity, given that multiple techniques are often combined in routine practice.

Methods

We used a two-armed parallel randomised controlled trial (n = 2702), nested within a cross-sectional health survey study, to evaluate whether using a pragmatic combination of behavioural science and evidenced-based techniques (e.g., personalisation, social norms messaging) in a study invitation letter increased response to the survey, when compared with a standard invitation letter. Participants and outcome assessors were blinded to group assignment. We tested this in a sample of women testing positive for human papillomavirus (HPV) at cervical cancer screening in England.

Results

Overall, 646 participants responded to the survey (response rate [RR] = 23.9%). Logistic regression revealed higher odds of response in the intervention arm (n = 357/1353, RR = 26.4%) compared with the control arm (n = 289/1349, RR = 21.4%), while adjusting for age, deprivation, clinical site, and clinical test result (aOR = 1.30, 95% CI: 1.09–1.55).

Conclusion

Applying easy-to-implement behavioural science and evidence-based methods to routine invitation letters improved postal response to a health-related survey, whilst adjusting for demographic characteristics. Our findings provide support for the pragmatic adoption of combined techniques in routine research to increase response to postal surveys.

Trial registration

ISRCTN, ISRCTN15113095. Registered 7 May 2019 – retrospectively registered.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-021-01476-7.

Does advance contact with research participants increase response to questionnaires: an updated systematic review and meta-analysis

Background

Questionnaires remain one of the most common forms of data collection in epidemiology, psychology and other human-sciences. However, results can be badly affected by non-response. One way to potentially reduce non-response is by sending potential study participants advance communication. The last systematic review to examine the effect of questionnaire pre-notification on response is 10 years old, and lacked a risk of bias assessment.

Objectives

Update the section of the Cochrane systematic review, Edwards et al. (2009), on pre-notification to include 1) recently published studies, 2) an assessment of risk of bias, 3) Explore if heterogeneity is reduced by: delay between pre-contact and questionnaire delivery, the method of pre-contact, if pre-contact and questionnaire delivery differ, if the pre-contact includes a foot-in-the-door manipulation, and study’s the risk of bias.

Methods

Inclusion criteria: population: any population, intervention: comparison of some type of pre-notification, comparison group: no pre-notification, outcome: response rates. Study design: randomised controlled trails. Exclusion criteria: NA. Data sources: Studies which cited or were included in Edwards et al. (2009); We additionally searched: CINAHL, Web of Science, PsycInfo, MEDLINE, EconLit, EMBASE, Cochrane Central, Cochrane CMR, ERIC, and Sociological Abstracts. The searches were implemented in June 2018 and May 2021. Study screening: a single reviewer screened studies, with a random 10% sample independently screened to ascertain accuracy. Data extraction: data was extracted by a single reviewer twice, with a week between each extraction. Risk of Bias: within studies bias was assessed using the Cochrane Risk of Bias tool (ROB1) by a single unblinded reviewer, across studies bias was assessed using funnel plots. Synthesis Method: study results were meta-analysed with a random effects model using the final response rate as the outcome. Evaluation of Uncertainty: Uncertainty was evaluated using the GRADE approach.

Results

One hundred seven trials were included with 211,802 participants. Over-all pre-notification increased response, OR = 1.33 (95% CI: 1.20–1.47). However, there was a large amount of heterogeneity (I² = 97.1%), which was not explained by the subgroup analyses. In addition, when studies at high or unclear risk of bias were excluded the effect was to reduced OR = 1.09 (95% CI: 0.99–1.20). Because of the large amount of heterogeneity, even after restricting to low risk of bias studies, there is still moderate uncertainty in these results.

Conclusions

Using the GRADE evaluation, this review finds moderate evidence that pre-notification may not have an effect on response rates.

Funding

Economic and Social Research Council.

Preregistration

None.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-021-01435-2.

Rams Have Heart, a Mobile App Tracking Activity and Fruit and Vegetable Consumption to Support the Cardiovascular Health of College Students: Development and Usability Study

BACKGROUND

With the increasing use of mobile devices to access the internet and as the main computing system of apps, there is a growing market for mobile health apps to provide self-care advice. Their effectiveness with regard to diet and fitness tracking, for example, needs to be examined. The majority of American adults fail to meet daily recommendations for healthy behavior. Testing user engagement with an app in a controlled environment can provide insight into what is effective and not effective in an app focused on improving diet and exercise.

OBJECTIVE

We developed Rams Have Heart, a mobile app, to support a cardiovascular disease (CVD) intervention course. The app tracks healthy behaviors, including fruit and vegetable consumption and physical activity, throughout the day. This paper aimed to present its functionality and evaluated adherence among the African American college student population.

METHODS

We developed the app using the Personal Health Informatics and Intervention Toolkit, a software framework. Rams Have Heart integrates self-reported health screening with health education, diary tracking, and user feedback modules to acquire data and assess progress. The parent study, conducted at a historically black college and university-designated institution in southeastern United States, consisted of a semester-long intervention administered as an academic course in the fall, for 3 consecutive years. Changes were made after the cohort 1 pilot study, so results only include cohorts 2 and 3, comprising a total of 115 students (n=55 intervention participants and n=54 control participants) aged from 17 to 24 years. Data collected over the study period were transferred using the secure Hypertext Transfer Protocol Secure protocol and stored in a secure Structured Query Language server database accessible only to authorized persons. SAS software was used to analyze the overall app usage and the specific results collected.

RESULTS

Of the 55 students in the intervention group, 27 (49%) students in cohort 2 and 25 (45%) in cohort 3 used the Rams Have Heart app at least once. Over the course of the fall semester, app participation dropped off gradually until exam week when most students no longer participated. The average fruit and vegetable intake increased slightly, and activity levels decreased over the study period.

CONCLUSIONS

Rams Have Heart was developed to allow daily tracking of fruit and vegetable intake and physical activity to support a CVD risk intervention for a student demographic susceptible to obesity, heart disease, and type 2 diabetes. We conducted an analysis of app usage, function, and user results. Although a mobile app provides privacy and flexibility for user participation in a research study, Rams Have Heart did not improve compliance or user outcomes. Health-oriented research studies relying on apps in support of user goals need further evaluation.

The effect of monetary incentive on survey response for vulnerable children and youths: A randomized controlled trial

AIM

In surveys non-responders may introduce bias and lower the validity of the studies. Ways to increase response rates are therefore important. The purpose of the study was to investigate if an unconditional monetary incentive can increase the response rate for vulnerable children and youths in a postal questionnaire survey.

METHODS

The study was designed as a randomized controlled trial. The study population consisted of 262 children and youth who participated in an established intervention study aimed at creating networks for different groups of vulnerable children and youths. The mean age of the participants was 16.7 years (range 11-28) and 67.9% were female. The questionnaire was adapted to three different age groups and covered different aspects of the participants' life situation, including the dimensions from the Strengths and Difficulties Questionnaire (SDQ). In the follow-up survey, participants were randomly allocated to two groups that either received a €15 voucher for a supermarket together with the questionnaire or only received the questionnaire. We used Poisson regression to estimate the differences in response rate (Rate Ratio RR) between the intervention group and the control group.

RESULTS

The response rate was 75.5% in the intervention group and 42.9% in the control group. The response rate in the intervention group was significantly higher than in the control group when adjusting for age and gender (Rate Ratio, RR 1.73; 95% CI 1.38-2.17). We did not find any significant differences in scales scores between the two groups for the five scales of the SDQ. In stratified analyses, we found the effect of the incentive to be higher for males (RR 2.81; 95% CI 1.61-4.91) than for females (1.43; 95% CI 1.12-1.84).

CONCLUSIONS

Monetary incentives can increase the response rate for vulnerable children and youths in surveys.

TRIAL REGISTRATION

The trial was retrospectively registered at ClinicalTrials.gov Identifier: NCT01741675.

How one small text change in a study document can impact recruitment rates and follow-up completions

BACKGROUND

The validity and reliability of longitudinal research is highly dependent on the recruitment and retention of representative samples. Various strategies have been developed and tested for improving recruitment and follow-up rates into health-behavioural research, but few have examined the role of linguistic choices and study document readability on participation rates. This study examined the impact of one small text change, assigning an inappropriate or grade-8 reading level password for intervention access, on participation rates and attrition in an online alcohol intervention trial.

METHODS

Participants were recruited into an online alcohol intervention study using Amazon's Mechanical Turk via a multi-step recruitment process which required participants to log into a study portal using a pre-assigned password. Passwords were qualitatively coded as grade-8 and/or inappropriate for use within a professional setting. Separate logistic regressions examined which demographic, clinical characteristics, and password categorizations were most strongly associated with recruitment rates and follow-up completions.

RESULTS

Inappropriate passwords were a barrier for recruitment among participants with post-secondary education as compared to those with less education (p = 0.044), while grade-8 passwords appeared to significantly facilitate the completion of 6-month follow-ups (p = 0.005).

CONCLUSIONS

Altogether, these findings suggest that some linguistic choices may play an important role in recruitment, while others, such as readability, may have longer-term effects on follow-up rates and attrition. Possible explanations for the findings, as well as, sample selection biases during recruitment and follow-up are discussed. Limitations of the study are stated and recommendations for researchers are provided.

TRIAL REGISTRATION

ClinicalTrials.gov NCT02977026. Registered 27 Nov 2016.

To send or not to send: weighing the costs and benefits of mailing an advance letter to participants before a telephone survey

Objective

A letter was mailed to half the participants (Letter = 137; No Letter = 138) of a 5-year follow-up survey regarding smoking cessation before attempting contact for a telephone interview. The primary outcome was the number of completed surveys per group (response rate). Secondary analyses of the number of telephone calls placed and a cost analysis were performed.

Results

No conclusive effect was found on the response rates per group (59.1% Letter, 50.0% No Letter; p = 0.147). Additionally, a logistic regression, controlling for demographics, revealed that there was no direct effect of sending the letter on response rate (p = 0.369). Non-parametric analysis showed significantly fewer calls (U = 7962.5, z = − 2.274, p < 0.05 two-tailed) and significantly lower costs (U = 11112.00, z = 2.521, p < 0.05 two-tailed) in reaching participants in the Letter group. Mailing an advance letter to participants did not appear to effect response rates between the groups, even when controlling for demographics. However, further analysis examining the number of call attempts and the costs per group revealed the letter may have had other effects. These findings suggest that additional analyses may be merited when evaluating the effectiveness of methods to increase participation, such as an advance letter, especially in cases where the literature largely supports its effectual use.

Trial registration ClinicalTrials.gov NCT03097445. Registered 31 March 2017

Electronic supplementary material

The online version of this article (10.1186/s13104-018-3920-6) contains supplementary material, which is available to authorized users.