Reliability and Validity of Noncognitive Ecological Momentary Assessment Survey Response Times as an Indicator of Cognitive Processing Speed in People's Natural Environment: Intensive Longitudinal Study

BACKGROUND

Various populations with chronic conditions are at risk for decreased cognitive performance, making assessment of their cognition important. Formal mobile cognitive assessments measure cognitive performance with greater ecological validity than traditional laboratory-based testing but add to participant task demands. Given that responding to a survey is considered a cognitively demanding task itself, information that is passively collected as a by-product of ecological momentary assessment (EMA) may be a means through which people's cognitive performance in their natural environment can be estimated when formal ambulatory cognitive assessment is not feasible. We specifically examined whether the item response times (RTs) to EMA questions (eg, mood) can serve as approximations of cognitive processing speed.

OBJECTIVE

This study aims to investigate whether the RTs from noncognitive EMA surveys can serve as approximate indicators of between-person (BP) differences and momentary within-person (WP) variability in cognitive processing speed.

METHODS

Data from a 2-week EMA study investigating the relationships among glucose, emotion, and functioning in adults with type 1 diabetes were analyzed. Validated mobile cognitive tests assessing processing speed (Symbol Search task) and sustained attention (Go-No Go task) were administered together with noncognitive EMA surveys 5 to 6 times per day via smartphones. Multilevel modeling was used to examine the reliability of EMA RTs, their convergent validity with the Symbol Search task, and their divergent validity with the Go-No Go task. Other tests of the validity of EMA RTs included the examination of their associations with age, depression, fatigue, and the time of day.

RESULTS

Overall, in BP analyses, evidence was found supporting the reliability and convergent validity of EMA question RTs from even a single repeatedly administered EMA item as a measure of average processing speed. BP correlations between the Symbol Search task and EMA RTs ranged from 0.43 to 0.58 (P<.001). EMA RTs had significant BP associations with age (P<.001), as expected, but not with depression (P=.20) or average fatigue (P=.18). In WP analyses, the RTs to 16 slider items and all 22 EMA items (including the 16 slider items) had acceptable (>0.70) WP reliability. After correcting for unreliability in multilevel models, EMA RTs from most combinations of items showed moderate WP correlations with the Symbol Search task (ranged from 0.29 to 0.58; P<.001) and demonstrated theoretically expected relationships with momentary fatigue and the time of day. The associations between EMA RTs and the Symbol Search task were greater than those between EMA RTs and the Go-No Go task at both the BP and WP levels, providing evidence of divergent validity.

CONCLUSIONS

Assessing the RTs to EMA items (eg, mood) may be a method of approximating people's average levels of and momentary fluctuations in processing speed without adding tasks beyond the survey questions.

Data Cleaning Process for mHealth Log Data to Inform Health Worker Performance

Log data, captured during use of mobile health (mHealth) applications by health providers, can play an important role in informing nature of user engagement with the application. The log data can also be employed in understanding health provider work patterns and performance. However, given that these logs are raw data, they require robust cleaning and curation if accurate conclusions are to be derived from analyzing them. This paper describes a systematic data cleaning process for mHealth-derived logs based on Broeck's framework, which involves iterative screening, diagnosis, and treatment of the log data. For this study, log data from the demonstrative mUzima mHealth application are used. The employed data cleaning process uncovered data inconsistencies, duplicate logs, missing data within logs that required imputation, among other issues. After the data cleaning process, only 39,229 log records out of the initial 91,432 usage logs (42.9%) could be included in the final dataset suitable for analyses of health provider work patterns. This work highlights the significance of having a systematic data cleaning approach for log data to derive useful information on health provider work patterns and performance.

Using Smartphone Sensor Paradata and Personalized Machine Learning Models to Infer Participants' Well-being: Ecological Momentary Assessment

Background

Sensors embedded in smartphones allow for the passive momentary quantification of people’s states in the context of their daily lives in real time. Such data could be useful for alleviating the burden of ecological momentary assessments and increasing utility in clinical assessments. Despite existing research on using passive sensor data to assess participants’ moment-to-moment states and activity levels, only limited research has investigated temporally linking sensor assessment and self-reported assessment to further integrate the 2 methodologies.

Objective

We investigated whether sparse movement-related sensor data can be used to train machine learning models that are able to infer states of individuals’ work-related rumination, fatigue, mood, arousal, life engagement, and sleep quality. Sensor data were only collected while the participants filled out the questionnaires on their smartphones.

Methods

We trained personalized machine learning models on data from employees (N=158) who participated in a 3-week ecological momentary assessment study.

Results

The results suggested that passive smartphone sensor data paired with personalized machine learning models can be used to infer individuals’ self-reported states at later measurement occasions. The mean R² was approximately 0.31 (SD 0.29), and more than half of the participants (119/158, 75.3%) had an R² of ≥0.18. Accuracy was only slightly attenuated compared with earlier studies and ranged from 38.41% to 51.38%.

Conclusions

Personalized machine learning models and temporally linked passive sensing data have the capability to infer a sizable proportion of variance in individuals’ daily self-reported states. Further research is needed to investigate factors that affect the accuracy and reliability of the inference.

Engagement in mHealth behavioral interventions for HIV prevention and care: making sense of the metrics

BACKGROUND

Engagement is the primary metric by which researchers can assess whether participants in a mHealth intervention used and interacted with the intervention's content as intended over a pre-specified period to result in behavior change. Paradata, defined as the process data documenting users' access, participation, and navigation through a mHealth intervention, have been associated with differential treatment outcomes in mHealth interventions. Within behavioral mHealth interventions, there has been an increase in the number of studies addressing the HIV prevention and care continuum in recent years, yet few have presented engagement metrics or examined how these data could inform design modifications, promote continued engagement, and supplement primary intervention efficacy and scale-up efforts.

METHODS

We review common paradata metrics in mHealth interventions (e.g., amount, frequency, duration and depth of use), using case studies from four technology-driven HIV interventions to illustrate their utility in evaluating mHealth behavioral interventions for HIV prevention and care. Across the four case studies, participants' ages ranged between 15 and 30 years and included a racially and ethnically diverse sample of youth. The four case studies had different approaches for engaging young men who have sex with men: a tailored brief intervention, an interactive modular program, a daily tool to monitor and self-regulate treatment adherence, and an online platform promoting social engagement and social support. Each focused on key outcomes across the HIV prevention and care continuum [e.g., safer sex behaviors, HIV testing, antiretroviral therapy (ART) adherence] and collected paradata metrics systematically.

RESULTS

Across the four interventions, paradata was utilized to identify patterns of use, create user profiles, and determine a minimum engagement threshold for future randomized trials based on initial pilot trial data. Evidence of treatment differences based on paradata analyses were also observed in between-arm and within-arm analyses, indicating that intervention exposure and dosage might influence the strength of the observed intervention effects. Paradata reflecting participants' engagement with intervention content was used to suggest modifications to intervention design and navigation, to understand what theoretically-driven content participants chose to engage with in an intervention, and to illustrate how engagement was linked to HIV-related outcomes.

CONCLUSIONS

Paradata monitoring and reporting can enhance the rigor of mHealth trials. Metrics of engagement must be systematically collected, analyzed and interpreted to meaningfully understand a mHealth intervention's efficacy. Future mHealth trials should work to identify suitable engagement metrics during intervention development, ensure their collection throughout the trial, and evaluate their impact on trial outcomes.

The contributions of paradata and features of respondents, interviewers and survey agencies to panel co-operation in the Survey of Health, Ageing and Retirement in Europe

This paper deals with panel cooperation in a cross-national, fully harmonized face-to-face survey. Our outcome of interest is panel cooperation in the fourth wave of the Survey of Health, Ageing and Retirement in Europe (SHARE). Following a multilevel approach, we focus on the contribution of paradata at three different levels: fieldwork strategies at the survey agency level, features of the (current) interviewer and paradata describing respondent interview experience from the previous wave. Our results highlight the importance of respondent's prior interview experience, and of interviewer's quality of work and experience. We also find that survey agency practice matters: daily communication between fieldwork coordinators and interviewers is positively associated with panel cooperation.

Addressing engagement in technology-based behavioural HIV interventions through paradata metrics

PURPOSE OF REVIEW

The goal of this review was to examine how often researchers report participants' online engagement using paradata (i.e. intervention usage metrics) when describing the outcomes of online behavioural HIV prevention and care interventions. We also highlight the utility of paradata collection and analysis in future technology-based trials.

RECENT FINDINGS

We focused on studies indexed on PubMed and published between 1 January 2016 and 31 March 2017 that reported the development and testing of online behavioural interventions for HIV prevention and/or care. Of the 705 extracted citations, six met study criteria.

SUMMARY

Only one study reported paradata reflecting participants' engagement with a technology-based intervention. Researchers should systematically collect and analyse paradata to strengthen the evidence base for technology-based interventions (do they work?), advance the use of behaviour change theory across modalities and platforms (how/why do they work?) and inform reach and scale-up efforts (for whom do they work?). Researchers may also rely on paradata to examine dose-response relationships due to user engagement, to identify replicable core components linked to behaviour change outcomes, to allocate resources judiciously and drive down development costs, and to pool these metrics for use in future meta-analyses.

The use of advanced web-based survey design in Delphi research

AIM

A discussion of the application of metadata, paradata and embedded data in web-based survey research, using two completed Delphi surveys as examples.

BACKGROUND

Metadata, paradata and embedded data use in web-based Delphi surveys has not been described in the literature. The rapid evolution and widespread use of online survey methods imply that paper-based Delphi methods will likely become obsolete. Commercially available web-based survey tools offer a convenient and affordable means of conducting Delphi research. Researchers and ethics committees may be unaware of the benefits and risks of using metadata in web-based surveys.

DESIGN

Discussion paper.

DATA SOURCES

Two web-based, three-round Delphi surveys were conducted sequentially between August 2014 - January 2015 and April - May 2016. Their aims were to validate the Australian nurse practitioner metaspecialties and their respective clinical practice standards. Our discussion paper is supported by researcher experience and data obtained from conducting both web-based Delphi surveys.

IMPLICATIONS FOR NURSING

Researchers and ethics committees should consider the benefits and risks of metadata use in web-based survey methods. Web-based Delphi research using paradata and embedded data may introduce efficiencies that improve individual participant survey experiences and reduce attrition across iterations. Use of embedded data allows the efficient conduct of multiple simultaneous Delphi surveys across a shorter timeframe than traditional survey methods.

CONCLUSION

The use of metadata, paradata and embedded data appears to improve response rates, identify bias and give possible explanation for apparent outlier responses, providing an efficient method of conducting web-based Delphi surveys.

Nonresponse and Underreporting Errors Increase over the Data Collection Week Based on Paradata from the National Household Food Acquisition and Purchase Survey

Background: Food acquisition diary surveys are important for studying food expenditures, factors affecting food acquisition decisions, and relations between these decisions with selected measures of health (e.g., body mass index, self-reported health). However, to our knowledge, no studies have evaluated the errors associated with these diary surveys, which can bias survey estimates and research findings. The use of paradata, which has been largely ignored in previous literature on diary surveys, could be useful for studying errors in these surveys.Objective: We used paradata to assess survey errors in the National Household Food Acquisition and Purchase Survey (FoodAPS).Methods: To evaluate the patterns of nonresponse over the diary period, we fit a multinomial logistic regression model to data from this 1-wk diary survey. We also assessed factors influencing respondents' probability of reporting food acquisition events during the diary process by using logistic regression models. Finally, with the use of an ordinal regression model, we studied factors influencing respondents' perceived ease of participation in the survey.Results: As the diary period progressed, nonresponse increased, especially for those starting the survey on Friday (where the odds of a refusal increased by 12% with each fielding day). The odds of reporting food acquisition events also decreased by 6% with each additional fielding day. Similarly, the odds of reporting ≥1 food-away-from-home event (i.e., meals, snacks, and drinks obtained outside the home) decreased significantly over the fielding period. Male respondents, larger households, households that eat together less often, and households with frequent guests reported a significantly more difficult time getting household members to participate, as did non-English-speaking households and households currently experiencing difficult financial conditions.Conclusions: Nonresponse and underreporting of food acquisition events tended to increase in the FoodAPS as data collection proceeded. This analysis of paradata available in the FoodAPS revealed these errors and suggests methodologic improvements for future food acquisition surveys.

Consequences of an Extended Recruitment on Participation in the Follow-Up of a Child Study: Results from the German IDEFICS Cohort

BACKGROUND

Declining response proportions in population-based studies are often countered by extended recruitment efforts at baseline that may, however, result in higher attrition in a subsequent follow-up. This study analysed the effect of extended recruitment efforts on attrition at the first follow-up of a child cohort.

METHODS

We used paradata (i.e. information about the process of data collection) from the German IDEFICS cohort investigating dietary- and life style-induced health effects on children to quantify recruitment effort and classify respondents as completing the recruitment early vs. late for baseline and follow-up separately. Multilevel logistic regression models were used to investigate the association between recruitment effort and attrition at follow-up (loss to follow-up) adjusted for sociodemographic and health related variables.

RESULTS

Individuals who were late respondents at baseline and early respondents at the follow-up had a higher chance of attrition (odds ratio 1.65, 95% confidence interval (CI) 1.19, 2.28) as compared to other groups. An investigation of reasons for non-participation revealed that members of this group were more likely to be not reachable by phone.

CONCLUSIONS

An extended recruitment effort at baseline of a child cohort study is not per se associated with a higher chance of attrition at follow-up. Much care should be taken to collect valid telephone numbers.

Web Health Monitoring Survey: A New Approach to Enhance the Effectiveness of Telemedicine Systems

BACKGROUND

Aging of the European population and interest in a healthy population in western countries have contributed to an increase in the number of health surveys, where the role of survey design, data collection, and data analysis methodology is clear and recognized by the whole scientific community. Survey methodology has had to couple with the challenges deriving from data collection through information and communications technology (ICT). Telemedicine systems have not used patients as a source of information, often limiting them to collecting only biometric data. A more effective telemonitoring system would be able to collect objective and subjective data (biometric parameters and symptoms reported by the patients themselves), and to control the quality of subjective data collected: this goal be achieved only by using and merging competencies from both survey methodology and health research.

OBJECTIVE

The objective of our study was to propose new metrics to control the quality of data, along with the well-known indicators of survey methodology. Web questionnaires administered daily to a group of patients for an extended length of time are a Web health monitoring survey (WHMS) in a telemedicine system.

METHODS

We calculated indicators based on paradata collected during a WHMS study involving 12 patients, who signed in to the website daily for 2 months.

RESULTS

The patients' involvement was very high: the patients' response rate ranged between 1.00 and 0.82, with an outlier of 0.65. Item nonresponse rate was very low, ranging between 0.0% and 7.4%. We propose adherence to the chosen time to connect to the website as a measure of involvement and cooperation by the patients: the difference from the median time ranged between 11 and 24 minutes, demonstrating very good cooperation and involvement from all patients. To measure habituation to the questionnaire, we also compared nonresponse rates to the items between the first and the second month of the study, and found no significant difference. We computed the time to complete the questionnaire both as a measure of possible burden for patient, and to detect the risk of automatic responses. Neither of these hypothesis was confirmed, and differences in time to completion seemed to depend on health conditions. Focus groups with patients confirmed their appreciation for this "new" active role in a telemonitoring system.

CONCLUSIONS

The main and innovative aspect of our proposal is the use of a Web questionnaire to virtually recreate a checkup visit, integrating subjective (patient's information) with objective data (biometric information). Our results, although preliminary and if need of further study, appear promising in proposing more effective telemedicine systems. Survey methodology could have an effective role in this growing field of research and applications.

Learning and satisficing: an analysis of sequence effects in health valuation

OBJECTIVE

To estimate the effect of sequence on response precision and response behavior in health valuation studies.

METHODS

Time trade-off (TTO) and paired comparison responses from six health valuation studies-four US, one Spanish, and one Dutch-were examined (22,225 respondents) to test whether task sequence influences response precision (e.g., rounding), response changes, and median response times. Each study used a computer-based instrument that randomized task sequence among a national sample of adults, age 18 years or older, from the general population.

RESULTS

For both TTO and paired comparisons, median response times decreased with sequence (i.e., learning), but tended to flatten after the first three tasks. Although the paired comparison evidence demonstrated that sequence had no effect on response precision, the frequency of rounded TTO responses (to either 1-year or 5-year units) increased with sequence.

CONCLUSIONS

Based on these results, randomizing or reducing the number of paired comparison tasks does not appear to influence response precision; however, generalizability, practicality, and precautionary considerations remain. Overall, participants learned to respond efficiently within the first three tasks and did not resort to satisficing, but may have rounded their TTO responses.

Survey Breakoffs in a Computer-Assisted Telephone Interview

Nearly 23% of all telephone interviews in the most recently completed wave of the Panel Study of Income Dynamics break off at least once, requiring multiple sessions to complete the interview. Given this high rate, a study was undertaken to better understand the causes and consequences of temporary breakoffs in a computer-assisted telephone interview setting. The majority of studies examining breakoffs have been conducted in the context of self-administered web surveys. The present study uses new paradata collected on telephone interview breakoffs to describe their prevalence, associated field effort, the instrument sections and questions on which they occur, their source - whether respondent-initiated, interviewer-initiated, or related to telephone problems - and associations with respondent and interviewer characteristics. The results provide information about the survey response process and suggest a set of recommendations for instrument design and interviewer training, as well as additional paradata that should be collected to provide more insight into the breakoff phenomenon.

Informed Consent for Web Paradata Use

Survey researchers are making increasing use of paradata - such as keystrokes, clicks, and timestamps - to evaluate and improve survey instruments but also to understand respondents and how they answer surveys. Since the introduction of paradata, researchers have been asking whether and how respondents should be informed about the capture and use of their paradata while completing a survey. In a series of three vignette-based experiments, we examine alternative ways of informing respondents about capture of paradata and seeking consent for their use. In all three experiments, any mention of paradata lowers stated willingness to participate in the hypothetical surveys. Even the condition where respondents were asked to consent to the use of paradata at the end of an actual survey resulted in a significant proportion declining. Our research shows that requiring such explicit consent may reduce survey participation without adequately informing survey respondents about what paradata are and why they are being used.