Mode Equivalence of Health Indicators Between Data Collection Modes and Mixed-Mode Survey Designs in Population-Based Health Interview Surveys for Children and Adolescents: Methodological Study

Lessons Learned From a Sequential Mixed-Mode Survey Design to Recruit and Collect Data From Case-Control Study Participants: Formative Evaluation

BACKGROUND

Sequential mixed-mode surveys using both web-based surveys and telephone interviews are increasingly being used in observational studies and have been shown to have many benefits; however, the application of this survey design has not been evaluated in the context of epidemiological case-control studies.

OBJECTIVE

In this paper, we discuss the challenges, benefits, and limitations of using a sequential mixed-mode survey design for a case-control study assessing risk factors during the COVID-19 pandemic.

METHODS

Colorado adults testing positive for SARS-CoV-2 were randomly selected and matched to those with a negative SARS-CoV-2 test result from March to April 2021. Participants were first contacted by SMS text message to complete a self-administered web-based survey asking about community exposures and behaviors. Those who did not respond were contacted for a telephone interview. We evaluated the representativeness of survey participants to sample populations and compared sociodemographic characteristics, participant responses, and time and resource requirements by survey mode using descriptive statistics and logistic regression models.

RESULTS

Of enrolled case and control participants, most were interviewed by telephone (308/537, 57.4% and 342/648, 52.8%, respectively), with overall enrollment more than doubling after interviewers called nonresponders. Participants identifying as female or White non-Hispanic, residing in urban areas, and not working outside the home were more likely to complete the web-based survey. Telephone participants were more likely than web-based participants to be aged 18-39 years or 60 years and older and reside in areas with lower levels of education, more linguistic isolation, lower income, and more people of color. While there were statistically significant sociodemographic differences noted between web-based and telephone case and control participants and their respective sample pools, participants were more similar to sample pools when web-based and telephone responses were combined. Web-based participants were less likely to report close contact with an individual with COVID-19 (odds ratio [OR] 0.70, 95% CI 0.53-0.94) but more likely to report community exposures, including visiting a grocery store or retail shop (OR 1.55, 95% CI 1.13-2.12), restaurant or cafe or coffee shop (OR 1.52, 95% CI 1.20-1.92), attending a gathering (OR 1.69, 95% CI 1.34-2.15), or sport or sporting event (OR 1.05, 95% CI 1.05-1.88). The web-based survey required an average of 0.03 (SD 0) person-hours per enrolled participant and US $920 in resources, whereas the telephone interview required an average of 5.11 person-hours per enrolled participant and US $70,000 in interviewer wages.

CONCLUSIONS

While we still encountered control recruitment challenges noted in other observational studies, the sequential mixed-mode design was an efficient method for recruiting a more representative group of participants for a case-control study with limited impact on data quality and should be considered during public health emergencies when timely and accurate exposure information is needed to inform control measures.

Methods to increase response to postal and electronic questionnaires

BACKGROUND

Self-administered questionnaires are widely used to collect data in epidemiological research, but non-response reduces the effective sample size and can introduce bias. Finding ways to increase response to postal and electronic questionnaires would improve the quality of epidemiological research.

OBJECTIVES

To identify effective strategies to increase response to postal and electronic questionnaires.

SEARCH METHODS

We searched 14 electronic databases up to December 2021 and manually searched the reference lists of relevant trials and reviews. We contacted the authors of all trials or reviews to ask about unpublished trials; where necessary, we also contacted authors to confirm the methods of allocation used and to clarify results presented.

SELECTION CRITERIA

Randomised trials of methods to increase response to postal or electronic questionnaires. We assessed the eligibility of each trial using pre-defined criteria.

DATA COLLECTION AND ANALYSIS

We extracted data on the trial participants, the intervention, the number randomised to intervention and comparison groups and allocation concealment. For each strategy, we estimated pooled odds ratios (OR) and 95% confidence intervals (CI) in a random-effects model. We assessed evidence for selection bias using Egger's weighted regression method and Begg's rank correlation test and funnel plot. We assessed heterogeneity amongst trial odds ratios using a Chi2 test and quantified the degree of inconsistency between trial results using the I2 statistic.

MAIN RESULTS

Postal We found 670 eligible trials that evaluated over 100 different strategies of increasing response to postal questionnaires. We found substantial heterogeneity amongst trial results in half of the strategies. The odds of response almost doubled when: using monetary incentives (odds ratio (OR) 1.86; 95% confidence interval (CI) 1.73 to 1.99; heterogeneity I2 = 85%); using a telephone reminder (OR 1.96; 95% CI 1.03 to 3.74); and when clinical outcome questions were placed last (OR 2.05; 95% CI 1.00 to 4.24). The odds of response increased by about half when: using a shorter questionnaire (OR 1.58; 95% CI 1.40 to 1.78); contacting participants before sending questionnaires (OR 1.36; 95% CI 1.23 to 1.51; I2 = 87%); incentives were given with questionnaires (i.e. unconditional) rather than when given only after participants had returned their questionnaire (i.e. conditional on response) (OR 1.53; 95% CI 1.35 to 1.74); using personalised SMS reminders (OR 1.53; 95% CI 0.97 to 2.42); using a special (recorded) delivery service (OR 1.68; 95% CI 1.36 to 2.08; I2 = 87%); using electronic reminders (OR 1.60; 95% CI 1.10 to 2.33); using intensive follow-up (OR 1.69; 95% CI 0.93 to 3.06); using a more interesting/salient questionnaire (OR 1.73; 95% CI 1.12 to 2.66); and when mentioning an obligation to respond (OR 1.61; 95% CI 1.16 to 2.22). The odds of response also increased with: non-monetary incentives (OR 1.16; 95% CI 1.11 to 1.21; I2 = 80%); a larger monetary incentive (OR 1.24; 95% CI 1.15 to 1.33); a larger non-monetary incentive (OR 1.15; 95% CI 1.00 to 1.33); when a pen was included (OR 1.44; 95% CI 1.38 to 1.50); using personalised materials (OR 1.15; 95% CI 1.09 to 1.21; I2 = 57%); using a single-sided rather than a double-sided questionnaire (OR 1.13; 95% CI 1.02 to 1.25); using stamped return envelopes rather than franked return envelopes (OR 1.23; 95% CI 1.13 to 1.33; I2 = 69%), assuring confidentiality (OR 1.33; 95% CI 1.24 to 1.42); using first-class outward mailing (OR 1.11; 95% CI 1.02 to 1.21); and when questionnaires originated from a university (OR 1.32; 95% CI 1.13 to 1.54). The odds of response were reduced when the questionnaire included questions of a sensitive nature (OR 0.94; 95% CI 0.88 to 1.00). Electronic We found 88 eligible trials that evaluated over 30 different ways of increasing response to electronic questionnaires. We found substantial heterogeneity amongst trial results in half of the strategies. The odds of response tripled when: using a brief letter rather than a detailed letter (OR 3.26; 95% CI 1.79 to 5.94); and when a picture was included in an email (OR 3.05; 95% CI 1.84 to 5.06; I2 = 19%). The odds of response almost doubled when: using monetary incentives (OR 1.88; 95% CI 1.31 to 2.71; I2 = 79%); and using a more interesting topic (OR 1.85; 95% CI 1.52 to 2.26). The odds of response increased by half when: using non-monetary incentives (OR 1.60; 95% CI 1.25 to 2.05); using shorter e-questionnaires (OR 1.51; 95% CI 1.06 to 2.16; I2 = 94%); and using a more interesting e-questionnaire (OR 1.85; 95% CI 1.52 to 2.26). The odds of response increased by a third when: offering survey results as an incentive (OR 1.36; 95% CI 1.16 to 1.59); using a white background (OR 1.31; 95% CI 1.10 to 1.56); and when stressing the benefits to society of response (OR 1.38; 95% CI 1.07 to 1.78; I2 = 41%). The odds of response also increased with: personalised e-questionnaires (OR 1.24; 95% CI 1.17 to 1.32; I2 = 41%); using a simple header (OR 1.23; 95% CI 1.03 to 1.48); giving a deadline (OR 1.18; 95% CI 1.03 to 1.34); and by giving a longer time estimate for completion (OR 1.25; 95% CI 0.96 to 1.64). The odds of response were reduced when: "Survey" was mentioned in the e-mail subject (OR 0.81; 95% CI 0.67 to 0.97); when the email or the e-questionnaire was from a male investigator, or it included a male signature (OR 0.55; 95% CI 0.38 to 0.80); and by using university sponsorship (OR 0.84; 95%CI 0.69 to 1.01). The odds of response using a postal questionnaire were over twice those using an e-questionnaire (OR 2.33; 95% CI 2.25 to 2.42; I2 = 98%). Response also increased when: providing a choice of response mode (electronic or postal) rather than electronic only (OR 1.76 95% CI 1.67 to 1.85; I2 = 97%); and when administering the e-questionnaire by computer rather than by smartphone (OR 1.62 95% CI 1.36 to 1.94).

AUTHORS' CONCLUSIONS

Researchers using postal and electronic questionnaires can increase response using the strategies shown to be effective in this Cochrane review.

Methods, strategies, and incentives to increase response to mental health surveys among adolescents: a systematic review

BACKGROUND

This systematic review aimed to identify effective methods to increase adolescents' response to surveys about mental health and substance use, to improve the quality of survey information.

METHODS

We followed a protocol and searched for studies that compared different survey delivery modes to adolescents. Eligible studies reported response rates, mental health score variation per survey mode and participant variations in mental health scores. We searched CENTRAL, PsycINFO, MEDLINE and Scopus in May 2022, and conducted citation searches in June 2022. Two reviewers independently undertook study selection, data extraction, and risk of bias assessments. Following the assessment of heterogeneity, some studies were pooled using meta-analysis.

RESULTS

Fifteen studies were identified, reporting six comparisons related to survey methods and strategies. Results indicate that response rates do not differ between survey modes (e.g., web versus paper-and-pencil) delivered in classroom settings. However, web surveys may yield higher response rates outside classroom settings. The largest effects on response rates were achieved using unconditional monetary incentives and obtaining passive parental consent. Survey mode influenced mental health scores in certain comparisons.

CONCLUSIONS

Despite the mixed quality of the studies, the low volume for some comparisons and the limit to studies in high income countries, several effective methods and strategies to improve adolescents' response rates to mental health surveys were identified.

Design and Administration of Patient-Centred Outcome Measures: The Perspectives of Children and Young People with Life-Limiting or Life-Threatening Conditions and Their Family Members

BACKGROUND

Self-reported health data from children with life-limiting conditions is rarely collected. To improve acceptability and feasibility of child and family-centred outcome measures for children, they need to be designed in a way that reflects preferences, priorities and abilities.

OBJECTIVES

The aim was to identify preferences for patient-reported outcome measure design (recall period, response format, length, administration mode) to improve the feasibility, acceptability, comprehensibility and relevance of a child and family-centred outcome measure, among children with life-limiting conditions and their family members.

METHOD

A semi-structured qualitative interview study seeking the perspectives of children with life-limiting conditions, their siblings and parents on measure design was conducted. Participants were purposively sampled and recruited from nine UK sites. Verbatim transcripts were analysed using framework analysis.

RESULTS

A total of 79 participants were recruited: 39 children aged 5-17 years (26 living with a life-limiting condition; 13 healthy siblings) and 40 parents (of children aged 0-17 years). Children found a short recall period and a visually appealing measure with ten questions or fewer most acceptable. Children with life-limiting conditions were more familiar with using rating scales such as numeric and Likert than their healthy siblings. Children emphasised the importance of completing the measure alongside interactions with a healthcare professional to enable them to talk about their responses. While parents assumed that electronic completion methods would be most feasible and acceptable, a small number of children preferred paper.

CONCLUSIONS

This study demonstrates that children with life-limiting conditions can engage in communicating preferences regarding the design of a patient-centred outcome measure. Where possible, children should be given the opportunity to participate in the measure development process to enhance acceptability and uptake in clinical practice. Results of this study should be considered in future research on outcome measure development in children.

A realist review of digitally delivered child development assessment and screening tools: Psychometrics and considerations for future use

BACKGROUND

Developmental screening improves the detection of developmental concerns, yet numerous children are not screened/assessed. Remote child developmental tool administration has been utilized to increase screening and assessment accessibility.

METHOD

We conducted a realist review to: (1) identify existing multi-domain child development assessment and screening tools for children 0-5 years; (2) review psychometric data on their digital (i.e., only administered remotely) administration; and (3) explore contextual factors relevant to their digital administration. We searched APA PsycInfo, MEDLINE, CINAHL, and ERIC to identify tools and papers on their psychometrics. We reference-searched included articles and searched Google for relevant grey literature.

RESULTS

Of 33 multi-domain child development tools identified in objective one, five tools (in five studies) were delivered digitally and compared to traditional (e.g., paper) delivery (i.e., objective two). Studies evaluated within-group equivalence reliability (k = 2) and between-group equivalence (k = 3). Within-group equivalence reliability was established for the Vineland Adaptive Behavior Scales, and domains (e.g., gross motor) of the Ages and Stages Questionnaires 2nd edition (ASQ-2) and Revised Prescreening Denver Questionnaire (R-PDQ). Between group equivalence was demonstrated for Developmental Neuropsychological Assessment, 2nd Edition (NEPSY-II) subtests and Bayley Scales of Infant and Toddler Development, 3rd edition (Bayley-3) items. In another between group evaluation, web-based and paper versions of the ASQ-2 were deemed generally equivalent. Digital Bayley-3 inter-observer reliability ranged from 0.82 to 1.0. Examiner support, time, tool modifications, family resources, and comfort promotion supported digital administration.

CONCLUSION

Digitally delivered ASQ-2, R-PDQ, Vineland, and Bayley-3 and NEPSY-II components show promise for equivalence with traditional administration.

Unit Response and Costs in Web Versus Face-To-Face Data Collection: Comparison of Two Cross-sectional Health Surveys

Background

Potential is seen in web data collection for population health surveys due to its combined cost-effectiveness, implementation ease, and increased internet penetration. Nonetheless, web modes may lead to lower and more selective unit response than traditional modes, and this may increase bias in the measured indicators.

Objective

This research assesses the unit response and costs of a web study versus face-to-face (F2F) study.

Methods

Alongside the Belgian Health Interview Survey by F2F edition 2018 (BHISF2F; net sample used: 3316), a web survey (Belgian Health Interview Survey by Web [BHISWEB]; net sample used: 1010) was organized. Sociodemographic data on invited individuals was obtained from the national register and census linkages. Unit response rates considering the different sampling probabilities of both surveys were calculated. Logistic regression analyses examined the association between mode system and sociodemographic characteristics for unit nonresponse. The costs per completed web questionnaire were compared with the costs for a completed F2F questionnaire.

Results

The unit response rate is lower in BHISWEB (18.0%) versus BHISF2F (43.1%). A lower response rate was observed for the web survey among all sociodemographic groups, but the difference was higher among people aged 65 years and older (15.4% vs 45.1%), lower educated people (10.9% vs 38.0%), people with a non-Belgian European nationality (11.4% vs 40.7%), people with a non-European nationality (7.2% vs 38.0%), people living alone (12.6% vs 40.5%), and people living in the Brussels-Capital (12.2% vs 41.8%) region. The sociodemographic characteristics associated with nonresponse are not the same in the 2 studies. Having another European (OR 1.60, 95% CI 1.20-2.13) or non-European nationality (OR 2.57, 95% CI 1.79-3.70) compared to a Belgian nationality and living in the Brussels-Capital (OR 1.72, 95% CI 1.41-2.10) or Walloon (OR 1.47, 95% CI 1.15-1.87) regions compared to the Flemish region are associated with a higher nonresponse only in the BHISWEB study. In BHISF2F, younger people (OR 1.31, 95% CI 1.11-1.54) are more likely to be nonrespondents than older people, and this was not the case in BHISWEB. In both studies, lower educated people have a higher probability of being nonrespondent, but this effect is more pronounced in BHISWEB (low vs high education level: Web, OR 2.71, 95% CI 2.21-3.39 and F2F OR 1.70, 95% CI 1.48-1.95). The BHISWEB study had a considerable advantage; the cost per completed questionnaire was almost 3 times lower (€41 [US $48]) compared with F2F data collection (€111 [US $131]).

Conclusions

The F2F unit response rate was generally higher, yet for certain groups the difference between web and F2F was more limited. Web data collection has a considerable cost advantage. It is therefore worth experimenting with adaptive mixed-mode designs to optimize financial resources without increasing selection bias (eg, only inviting sociodemographic groups who are keener to participate online for web surveys while continuing to focus on increasing F2F response rates for other groups).

Co-establishing an infrastructure for routine data collection to address disparities in infant mortality: planning and implementation

Background

Efforts to address infant mortality disparities in Ohio have historically been adversely affected by the lack of consistent data collection and infrastructure across the community-based organizations performing front-line work with expectant mothers, and there is no established template for implementing such systems in the context of diverse technological capacities and varying data collection magnitude among participating organizations.

Methods

Taking into account both the needs and limitations of participating community-based organizations, we created a data collection infrastructure that was refined by feedback from sponsors and the organizations to serve as both a solution to their existing needs and a template for future efforts in other settings.

Results

By standardizing the collected data elements across participating organizations, integration on a scale large enough to detect changes in a rare outcome such as infant mortality was made possible. Datasets generated through the use of the established infrastructure were robust enough to be matched with other records, such as Medicaid and birth records, to allow more extensive analysis.

Conclusion

While a consistent data collection infrastructure across multiple organizations does require buy-in at the organizational level, especially among participants with little to no existing data collection experience, an approach that relies on an understanding of existing barriers, iterative development, and feedback from sponsors and participants can lead to better coordination and sharing of information when addressing health concerns that individual organizations may struggle to quantify alone.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-021-07393-1.

Enhancing validity, reliability and participation in self-reported health outcome measurement for children and young people: a systematic review of recall period, response scale format, and administration modality

Introduction

Self-report is the gold standard for measuring children’s health-related outcomes. Design of such measures is complex and challenging. This review aims to systematically appraise the evidence on recall period, response scale format, mode of administration and approaches needed to enable children and young people < 19 years to participate in valid and reliable self-reporting of their health outcomes.

Method

PsycInfo, Medline, CINAHL and Embase were searched from 1 January 1990 to 15 March 2020, and citation searching undertaken in Scopus. Articles were included if they were primary research or case reports of ≥ 3 participants reporting the following: recall period, response scale selection, administration modality. Quality was assessed using QualSyst, and results synthesised narratively. This review was conducted and reported according to PRISMA guidelines.

Results

81 of 13,215 retrieved articles met the inclusion criteria. Children < 5 years old cannot validly and reliably self-report health outcomes. Face scales demonstrate better psychometric properties than visual analogue or Likert scales. Computerised and paper scales generally show equivalent construct validity. Children prefer computerised measures. Children ≤ 7 years old think dichotomously so need two response options. Those > 8 years old can reliably use a 3-point scale.

Conclusion

The results of this review have both clinical and research implications. They can be used to inform appropriate choice of PROM for use with CYP in the clinical setting. We also give eight recommendations for future development of self-reported outcome measures for children and young people.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11136-021-02814-4.

Paper-and-pencil versus computerized administration mode: Comparison of data quality and risk behavior prevalence estimates in the European school Survey Project on Alcohol and other Drugs (ESPAD)

Purpose

The aim of this experimental study was to investigate whether paper-and-pencil and computerized surveys administered in the school setting yield equivalent data quality indicators and risk behavior prevalence estimates.

Methods

Data were drawn from the European School Survey Project on Alcohol and Other Drugs (ESPAD®) carried out in Italy to monitor drug, alcohol, tobacco use and other risk-behaviors among Italian high school students aged 15–19 years.

A sub-sample of schools was recruited for the study (1673 pupils). For each school, two entire randomly selected courses (from the first to the fifth grade) participated and were assigned randomly to the self-administered paper-and-pencil (N = 811) or computerized survey (N = 862). Differences in data quality were assessed using the following indicators: questionnaire completeness (missing gender and/or 50% of missing answers) and internal consistency (repetitive extreme response patterns). Separate logistic regression models were used to estimate the mode effect on the reporting of each risk behavior, controlling for gender and age. Finally, the prevalence estimates of the experimental study were compared to the results of the national ESPAD® study.

Results

The computerized administration mode produced a higher proportion of invalid questionnaires, but the prevalence estimates generated from responses to the paper-and-pencil and computerized surveys were generally equivalent. Nevertheless, comparing these results with those of the national ESPAD® study, some differences in the prevalence rates were found.

Conclusions

The findings suggest that in a proctored school setting, the computerized survey mode yields almost the same results as the paper-and-pencil mode. However, because of the reliance on existing informatics facilities until when all schools in the country will be sufficiently equipped for the computerized data collection, they should be given the opportunity to choose between paper-and-pencil and computerized survey modes, in order to avoid a possible selection bias.