Mixed-mode administration reduced bias and enhanced poststratification adjustments in a health behavior survey

Web-Based Data Collection for Older Adults Living With HIV in a Clinical Research Setting: Pilot Observational Study

Background

Longitudinal follow-up of older persons living with HIV is essential for the ascertainment of aging-related clinical and behavioral outcomes, and self-administered questionnaires are necessary for collecting behavioral information in research involving persons living with HIV. Web-based self-reported data collection results in higher data quality than paper-and-pencil questionnaires in a wide range of populations. The option of remote web-based surveys may also increase retention in long-term research studies. However, the acceptability and feasibility of web-based data collection in clinical research involving older persons living with HIV have never been studied.

Objective

This study aims to assess the acceptability and feasibility of a web-based survey to collect information on sexual, substance use, and physical activity behaviors; compare the data quality of the web-based survey with that of a paper-and-pencil questionnaire; and summarize web-based survey metrics.

Methods

This pilot study took place within the AIDS Clinical Trials Group A5322 study, a longitudinal cohort of men and women living with HIV (aged ≥40 years), followed at 32 clinical sites in the United States and Puerto Rico. A total of 4 sites participated in this study. A web-based survey was created using self-administered questionnaires typically completed in A5322 via paper and pencil. Pilot study participants completed these questionnaires via web-based survey at one research visit in lieu of paper-and-pencil administration. Two questions were added to assess feasibility, defined as participants’ perception of the ease of web-based survey completion (very hard, hard, easy, very easy), and their preferred format (computer or tablet, paper and pencil, no preference) for completing the questions in the future (acceptability). Feasibility and acceptability were summarized overall and by demographic and clinical characteristics; the proportion of evaluable data by web-based survey versus previously administered paper-and-pencil questionnaires (data quality) was compared for each question.

Results

Acceptability and feasibility were high overall: 50.0% (79/158) preferred computer or tablet, 38.0% (60/158) reported no preference, and 12.0% (19/158) preferred paper and pencil; 93.0% (147/158) reported survey completion easy or very easy. Older age was associated with lower odds of preferring computer or tablet to paper and pencil (odds ratio per 1-year increase in age: 0.91, 95% CI 0.85-0.98). Individuals who found the survey hard or very hard had a lower median neurocognitive test score than those who found it easy or very easy. Data quality with web-based survey administration was similar to or higher than that with paper-and-pencil administration for most questions.

Conclusions

Web-based survey administration was acceptable and feasible in this cohort of older adults living with HIV, and data quality was high. Web-based surveys can be a useful tool for valid data collection and can potentially improve retention in long-term follow-up studies.

New options for national population surveys: The implications of internet and smartphone coverage

Challenges to survey data collection have increased the costs of social research via face-to-face surveys so much that it may become extremely difficult for social scientists to continue using these methods. A key drawback to less expensive Internet-based alternatives is the threat of biased results from coverage errors in survey data. The rise of Internet-enabled smartphones presents an opportunity to re-examine the issue of Internet coverage for surveys and its implications for coverage bias. Two questions (on Internet access and smartphone ownership) were added to the National Survey of Family Growth (NSFG), a U.S. national probability survey of women and men age 15-44, using a continuous sample design. We examine 16 quarters (4 years) of data, from September 2012 to August 2016. Overall, we estimate that 82.9% of the target NSFG population has Internet access, and 81.6% has a smartphone. Combined, this means that about 90.7% of U.S. residents age 15-44 have Internet access, via either traditional devices or a smartphone. We find some evidence of compensatory coverage when looking at key race/ethnicity and age subgroups. For instance, while Black teens (15-18) have the lowest estimated rate of Internet access (81.9%) and the lowest rate of smartphone usage (72.6%), an estimated 88.0% of this subgroup has some form of Internet access. We also examine the socio-demographic correlates of Internet and smartphone coverage, separately and combined, as indicators of technology access in this population. In addition, we look at the effect of differential coverage on key estimates produced by the NSFG, related to fertility, family formation, and sexual activity. While this does not address nonresponse or measurement biases that may differ for alternative modes, our paper has implications for possible coverage biases that may arise when switching to a Web-based mode of data collection, either for follow-up surveys or to replace the main face-to-face data collection.

Mixing modes in a population-based interview survey: comparison of a sequential and a concurrent mixed-mode design for public health research

Background

Population-based surveys currently face the problem of decreasing response rates. Mixed-mode designs are now being implemented more often to account for this, to improve sample composition and to reduce overall costs. This study examines whether a concurrent or sequential mixed-mode design achieves better results on a number of indicators of survey quality.

Methods

Data were obtained from a population-based health interview survey of adults in Germany that was conducted as a methodological pilot study as part of the German Health Update (GEDA). Participants were randomly allocated to one of two surveys; each of the surveys had a different design. In the concurrent mixed-mode design (n = 617) two types of self-administered questionnaires (SAQ-Web and SAQ-Paper) and computer-assisted telephone interviewing were offered simultaneously to the respondents along with the invitation to participate. In the sequential mixed-mode design (n = 561), SAQ-Web was initially provided, followed by SAQ-Paper, with an option for a telephone interview being sent out together with the reminders at a later date. Finally, this study compared the response rates, sample composition, health indicators, item non-response, the scope of fieldwork and the costs of both designs.

Results

No systematic differences were identified between the two mixed-mode designs in terms of response rates, the socio-demographic characteristics of the achieved samples, or the prevalence rates of the health indicators under study. The sequential design gained a higher rate of online respondents. Very few telephone interviews were conducted for either design. With regard to data quality, the sequential design (which had more online respondents) showed less item non-response. There were minor differences between the designs in terms of their costs. Postage and printing costs were lower in the concurrent design, but labour costs were lower in the sequential design. No differences in health indicators were found between the two designs. Modelling these results for higher response rates and larger net sample sizes indicated that the sequential design was more cost and time-effective.

Conclusions

This study contributes to the research available on implementing mixed-mode designs as part of public health surveys. Our findings show that SAQ-Paper and SAQ-Web questionnaires can be combined effectively. Sequential mixed-mode designs with higher rates of online respondents may be of greater benefit to studies with larger net sample sizes than concurrent mixed-mode designs.

Nonresponse bias in survey research: lessons from a prospective study of breast reconstruction

BACKGROUND

Survey-based research is essential for evaluating the outcomes of health care in an era of patient-centered care. However, many such studies are hampered by poor response rates in completion of study questionnaires, thus limiting the generalizability of any findings. The objectives of this analysis were to identify independent variables associated with nonresponse to surveys following breast reconstruction to improve future patient-reported outcomes research.

MATERIALS AND METHODS

The Mastectomy Reconstruction Outcomes Consortium is a prospective cohort study involving 11 leading medical centers from the United States and Canada. Nonresponse rates for surveys assessing satisfaction with breast, satisfaction with care (BREAST-Q), depression (Patient Health Questionnaire-9), and anxiety (Generalized Anxiety Disorder-7) were measured at 1 y and 2 y postoperatively. Clinical complication rates were compared between responders and nonresponders, and multivariable models were used to assess predictors of nonresponse.

RESULTS

Among 2856 women in the analytic cohort, 1882 (65.9%) underwent implant-based, 817 (28.6%) received autologous, and 157 (5.5%) underwent latissimus dorsi myocutaneous flap breast reconstructions. Nonresponse rates to surveys at 1 y and 2 y were 27.8% and 34.4%, respectively. Race, ethnicity, and annual household income were associated with nonresponse to surveys. Women who underwent implant-based procedures were less likely to complete long-term surveys.

CONCLUSIONS

As survey-based research plays an increasingly prominent role in evaluating the outcomes of breast reconstruction, we found socioeconomic and procedure-related differences in survey response rates. Investigators must consider systematic differences in response rates among particular groups of women on the generalizability and validity of findings and perform rigorous nonresponse bias analyses.

Estimating the Effect of Nonresponse Bias in a Survey of Hospital Organizations

Nonresponse bias in survey research can result in misleading or inaccurate findings and assessment of nonresponse bias is advocated to determine response sample representativeness. Four methods of assessing nonresponse bias (analysis of known characteristics of a population, subsampling of nonresponders, wave analysis, and linear extrapolation) were applied to the results of a postal survey of U.K. hospital organizations. The purpose was to establish whether validated methods for assessing nonresponse bias at the individual level can be successfully applied to an organizational level survey. The aim of the initial survey was to investigate trends in the implementation of radiographer abnormality detection schemes, and a response rate of 63.7% (325/510) was achieved. This study identified conflicting trends in the outcomes of analysis of nonresponse bias between the different methods applied and we were unable to validate the continuum of resistance theory as applied to organizational survey data. Further work is required to ensure established nonresponse bias analysis approaches can be successfully applied to organizational survey data. Until then, it is suggested that a combination of methods should be used to enhance the rigor of survey analysis.

Evaluating survey quality in health services research: a decision framework for assessing nonresponse bias

OBJECTIVE

To address the issue of nonresponse as problematic and offer appropriate strategies for assessing nonresponse bias.

STUDY DESIGN

A review of current strategies used to assess the quality of survey data and the challenges associated with these strategies is provided along with appropriate post-data collection techniques that researchers should consider.

PRINCIPAL FINDINGS

Response rates are an incomplete assessment of survey data quality, and quick reactions to response rate should be avoided. Based on a five-question decision making framework, we offer potential ways to assess nonresponse bias, along with a description of the advantages and disadvantages to each.

CONCLUSIONS

It is important that the quality of survey data be considered to assess the relative contribution to the literature of a given study. Authors and funding agencies should consider the potential effects of nonresponse bias both before and after survey administration and report the results of assessments of nonresponse bias in addition to response rates.

Benefits of extensive recruitment effort persist during follow-ups and are consistent across age group and survey method. The TRAILS study

Background

Extensive recruitment effort at baseline increases representativeness of study populations by decreasing non-response and associated bias. First, it is not known to what extent increased attrition occurs during subsequent measurement waves among subjects who were hard-to-recruit at baseline and what characteristics the hard-to-recruit dropouts have compared to the hard-to-recruit retainers. Second, it is unknown whether characteristics of hard-to-recruit responders in a prospective population based cohort study are similar across age group and survey method.

Methods

First, we compared first wave (T1) easy-to-recruit with hard-to-recruit responders of the TRacking Adolescents’ Individual Lives Survey (TRAILS), a prospective population based cohort study of Dutch (pre)adolescents (at first wave: n = 2230, mean age = 11.09 (SD 0.56), 50.8% girls), with regard to response rates at subsequent measurement waves. Second, easy-to-recruit and hard-to-recruit participants at the fourth TRAILS measurement wave (n = 1881, mean age = 19.1 (SD 0.60), 52.3% girls) were compared with fourth wave non-responders and earlier stage drop-outs on family composition, socioeconomic position (SEP), intelligence (IQ), education, sociometric status, substance use, and psychopathology.

Results

First, over 60% of the hard-to-recruit responders at the first wave were retained in the sample eight years later at the fourth measurement wave. Hard-to-recruit dropouts did not differ from hard-to-recruit retainers. Second, extensive recruitment efforts for the web based survey convinced a population of nineteen year olds with similar characteristics as the hard-to-recruit eleven year olds that were persuaded to participate in a school-based survey. Some characteristics associated with being hard-to-recruit (as compared to being easy-to-recruit) were more pronounced among non-responders, resembling the baseline situation (De Winter et al.2005).

Conclusions

First, extensive recruitment effort at the first assessment wave of a prospective population based cohort study has long lasting positive effects. Second, characteristics of hard-to-recruit responders are largely consistent across age groups and survey methods.

Deployment of a mixed-mode data collection strategy does not reduce nonresponse bias in a general population health survey

OBJECTIVE

To assess nonresponse bias in a mixed-mode general population health survey.

DATA SOURCES

Secondary analysis of linked survey sample frame and administrative data, including demographic and health-related information.

STUDY DESIGN

The survey was administered by mail with telephone follow-up to nonrespondents after two mailings. To determine whether an additional mail contact or mode switch reduced nonresponse bias, we compared all respondents (N = 3,437) to respondents from each mailing and telephone respondents to the sample frame (N = 6,716).

PRINCIPAL FINDINGS

Switching modes did not minimize the under-representation of younger people, nonwhites, those with congestive heart failure, high users of office-based services, and low-utilizers of the emergency room but did reduce the over-representation of older adults.

CONCLUSIONS

Multiple contact and mixed-mode surveys may increase response rates, but they do not necessarily reduce nonresponse bias.

Are lower response rates hazardous to your health survey? An analysis of three state telephone health surveys

OBJECTIVE

To examine the impact of response rate variation on survey estimates and costs in three health telephone surveys.

DATA SOURCE

Three telephone surveys of noninstitutionalized adults in Minnesota and Oklahoma conducted from 2003 to 2005.

STUDY DESIGN

We examine differences in demographics and health measures by number of call attempts made before completion of the survey or whether the household initially refused to participate. We compare the point estimates we actually obtained with those we would have obtained with a less aggressive protocol and subsequent lower response rate. We also simulate what the effective sample sizes would have been if less aggressive protocols were followed.

PRINCIPAL FINDINGS

Unweighted bivariate analyses reveal many differences between early completers and those requiring more contacts and between those who initially refused to participate and those who did not. However, after making standard poststratification adjustments, no statistically significant differences were observed in the key health variables we examined between the early responders and the estimates derived from the full reporting sample.

CONCLUSIONS

Our findings demonstrate that for the surveys we examined, larger effective sample sizes (i.e., more statistical power) could have been achieved with the same amount of funding using less aggressive calling protocols. For some studies, money spent on aggressively pursuing high response rates could be better used to increase statistical power and/or to directly examine nonresponse bias.