Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the Average Time to Discover relevant records

BACKGROUND

Conducting a systematic review demands a significant amount of effort in screening titles and abstracts. To accelerate this process, various tools that utilize active learning have been proposed. These tools allow the reviewer to interact with machine learning software to identify relevant publications as early as possible. The goal of this study is to gain a comprehensive understanding of active learning models for reducing the workload in systematic reviews through a simulation study.

METHODS

The simulation study mimics the process of a human reviewer screening records while interacting with an active learning model. Different active learning models were compared based on four classification techniques (naive Bayes, logistic regression, support vector machines, and random forest) and two feature extraction strategies (TF-IDF and doc2vec). The performance of the models was compared for six systematic review datasets from different research areas. The evaluation of the models was based on the Work Saved over Sampling (WSS) and recall. Additionally, this study introduces two new statistics, Time to Discovery (TD) and Average Time to Discovery (ATD).

RESULTS

The models reduce the number of publications needed to screen by 91.7 to 63.9% while still finding 95% of all relevant records (WSS@95). Recall of the models was defined as the proportion of relevant records found after screening 10% of of all records and ranges from 53.6 to 99.8%. The ATD values range from 1.4% till 11.7%, which indicate the average proportion of labeling decisions the researcher needs to make to detect a relevant record. The ATD values display a similar ranking across the simulations as the recall and WSS values.

CONCLUSIONS

Active learning models for screening prioritization demonstrate significant potential for reducing the workload in systematic reviews. The Naive Bayes + TF-IDF model yielded the best results overall. The Average Time to Discovery (ATD) measures performance of active learning models throughout the entire screening process without the need for an arbitrary cut-off point. This makes the ATD a promising metric for comparing the performance of different models across different datasets.

Why Do Employers (Fail to) Hire People with Disabilities? A Systematic Review of Capabilities, Opportunities and Motivations

PURPOSE

To increase the number of people with disabilities in employment, we need to understand what influences employers' hiring decisions. In this systematic review, we map out factors affecting employers' hiring decisions about people with disabilities.

METHODS

This study is a systematic review that applies the COM-B model to identify factors that contribute to employers (not) hiring people with disabilities. The COM-B model proposes that employers will perform hiring behavior (B) if they have the capability (C), opportunity (O) and motivation (M) to do so. We also investigate if factors have a negative, positive or no effect. We report in accordance with the PRISMA guidelines.

RESULTS

In a review of 47 studies, we find 32 factors. Most of these factors are barriers. The most frequently mentioned barriers are employers' (1) expectations that people with disabilities are unproductive, (2) expectations that people with disabilities cost a lot of money, and employers' (3) lack of knowledge about disabilities. The most researched facilitators for employers to hire people with disabilities include (1) the motivation to help others, (2) working in a large organization, and (3) expecting a competitive advantage. The effect of factors can differ depending on contextual circumstances, including the type of organization, the type of disability and different policies.

CONCLUSIONS

We conclude that hiring decisions are influenced by an array of different barriers and facilitators. The effect of these factors can differ across organizations and disability types. Our study of factors affecting hiring can be used by scholars, policy makers, and organizations to create interventions to increase the hiring of people with disabilities.

Habit formation of preventive behaviours during the COVID-19 pandemic: a longitudinal study of physical distancing and hand washing

Background

Since the outbreak of the COVID-19 pandemic, physical distancing and hand washing have been used as effective means to reduce virus transmission in the Netherlands. However, these measures pose a societal challenge as they require people to change their customary behaviours in various contexts. The science of habit formation is potentially useful for informing policy-making in public health, but the current literature largely overlooked the role of habit in predicting and explaining these preventive behaviours. Our research aimed to describe habit formation processes of physical distancing and hand washing and to estimate the influences of habit strength and intention on behavioural adherence.

Methods

A longitudinal survey was conducted between July and November 2020 on a representative Dutch sample (n = 800). Respondents reported their intentions, habit strengths, and adherence regarding six context-specific preventive behaviours on a weekly basis. Temporal developments of the measured variables were visualized, quantified, and mapped onto five distinct phases of the pandemic. Regression models were used to test the effects of intention, habit strength, and their interaction on behavioural adherence.

Results

Dutch respondents generally had strong intentions to adhere to all preventive measures and their adherence rates were between 70% and 90%. They also self-reported to experience their behaviours as more automatic over time, and this increasing trend in habit strength was more evident for physical-distancing than for hand washing behaviours. For all six behaviours, both intention and habit strength predicted subsequent adherence (all ps < 2e-16). In addition, the predictive power of intention decreased over time and was weaker for respondents with strong habits for physical distancing when visiting supermarkets (B = -0.63, p <.0001) and having guests at home (B = -0.54, p <.0001) in the later phases of the study, but not for hand washing.

Conclusions

People’s adaptations to physical-distancing and hand washing measures involve both intentional and habitual processes. For public health management, our findings highlight the importance of using contextual cues to promote habit formation, especially for maintaining physical-distancing practices. For habit theories, our study provides a unique dataset that covers multiple health behaviours in a critical real-world setting.

Spatiotemporal variations of public opinion on social distancing in the Netherlands: Comparison of Twitter and longitudinal survey data

Background

Social distancing has been implemented by many countries to curb the COVID-19 pandemic. Understanding public support for this policy calls for effective and efficient methods of monitoring public opinion on social distancing. Twitter analysis has been suggested as a cheaper and faster-responding alternative to traditional survey methods. The current empirical evidence is mixed in terms of the correspondence between the two methods.

Objective

We aim to compare the two methods in the context of monitoring the Dutch public's opinion on social distancing. For this comparison, we quantified the temporal and spatial variations in public opinion and their sensitivities to critical events using data from both Dutch Twitter users and respondents from a longitudinal survey.

Methods

A longitudinal survey on a representative Dutch sample (n = 1,200) was conducted between July and November 2020 to measure opinions on social distancing weekly. From the same period, near 100,000 Dutch tweets were categorized as supporting or rejecting social distancing based on a model trained with annotated data. Average stances for the 12 Dutch provinces and over the 20 weeks were computed from the two data sources and were compared through visualizations and statistical analyses.

Results

Both data sources suggested strong support for social distancing, but public opinion was much more varied among tweets than survey responses. Both data sources showed an increase in public support for social distancing over time, and a strong temporal correspondence between them was found for most of the provinces. In addition, the survey but not Twitter data revealed structured differences among the 12 provinces, while the two data sources did not correspond much spatially. Finally, stances estimated from tweets were more sensitive to critical events happened during the study period.

Conclusions

Our findings indicate consistencies between Twitter data analysis and survey methods in describing the overall stance on social distancing and temporal trends. The lack of spatial correspondence may imply limitations in the data collections and calls for surveys with larger regional samples. For public health management, Twitter analysis can be used to complement survey methods, especially for capturing public's reactivities to critical events amid the current pandemic.

“Keep your distance for me”: A field experiment on empathy prompts to promote distancing during the COVID ‐19 pandemic

The outbreak of COVID‐19 has turned out to be a major challenge to societies all over the globe. Curbing the pandemic requires rapid and extensive behavioural change to limit social interaction, including physical distancing. In this study, we tested the notion that inducing empathy for people vulnerable to the virus may result in actual distancing behaviour beyond the mere motivation to do so. In a large field experiment with a sequential case–control design, we found that (a) empathy prompts may increase distancing as assessed by camera recordings and (b) effectiveness of prompts depends on the dynamics of the pandemic and associated public health policies. In sum, the present study demonstrates the potential of empathy‐generating interventions to promote pro‐social behaviour and emphasizes the necessity of field experiments to assess the role of context before advising policy makers to implement measures derived from behavioural science. Please refer to Supplementary Material to find this article's Community and Social Impact Statement

Moral dilemmas and trust in leaders during a global health crisis

Trust in leaders is central to citizen compliance with public policies. One potential determinant of trust is how leaders resolve conflicts between utilitarian and non-utilitarian ethical principles in moral dilemmas. Past research suggests that utilitarian responses to dilemmas can both erode and enhance trust in leaders: sacrificing some people to save many others ('instrumental harm') reduces trust, while maximizing the welfare of everyone equally ('impartial beneficence') may increase trust. In a multi-site experiment spanning 22 countries on six continents, participants (N = 23,929) completed self-report (N = 17,591) and behavioural (N = 12,638) measures of trust in leaders who endorsed utilitarian or non-utilitarian principles in dilemmas concerning the COVID-19 pandemic. Across both the self-report and behavioural measures, endorsement of instrumental harm decreased trust, while endorsement of impartial beneficence increased trust. These results show how support for different ethical principles can impact trust in leaders, and inform effective public communication during times of global crisis. PROTOCOL REGISTRATION STATEMENT: The Stage 1 protocol for this Registered Report was accepted in principle on 13 November 2020. The protocol, as accepted by the journal, can be found at https://doi.org/10.6084/m9.figshare.13247315.v1 .

An open source machine learning framework for efficient and transparent systematic reviews

To help researchers conduct a systematic review or meta-analysis as efficiently and transparently as possible, we designed a tool to accelerate the step of screening titles and abstracts. For many tasks—including but not limited to systematic reviews and meta-analyses—the scientific literature needs to be checked systematically. Scholars and practitioners currently screen thousands of studies by hand to determine which studies to include in their review or meta-analysis. This is error prone and inefficient because of extremely imbalanced data: only a fraction of the screened studies is relevant. The future of systematic reviewing will be an interaction with machine learning algorithms to deal with the enormous increase of available text. We therefore developed an open source machine learning-aided pipeline applying active learning: ASReview. We demonstrate by means of simulation studies that active learning can yield far more efficient reviewing than manual reviewing while providing high quality. Furthermore, we describe the options of the free and open source research software and present the results from user experience tests. We invite the community to contribute to open source projects such as our own that provide measurable and reproducible improvements over current practice.

Adherence to Electronic Health Tools Among Vulnerable Groups: Systematic Literature Review and Meta-Analysis

Background

Electronic health (eHealth) tools are increasingly being applied in health care. They are expected to improve access to health care, quality of health care, and health outcomes. Although the advantages of using these tools in health care are well described, it is unknown to what extent eHealth tools are effective when used by vulnerable population groups, such as the elderly, people with low socioeconomic status, single parents, minorities, or immigrants.

Objective

This study aimed to examine whether the design and implementation characteristics of eHealth tools contribute to better use of these tools among vulnerable groups.

Methods

In this systematic review, we assessed the design and implementation characteristics of eHealth tools that are used by vulnerable groups. In the meta-analysis, we used the adherence rate as an effect size measure. The adherence rate is defined as the number of people who are repetitive users (ie, use the eHealth tool more than once). We also performed a meta-regression analysis to examine how different design and implementation characteristics influenced the adherence rate.

Results

Currently, eHealth tools are continuously used by vulnerable groups but to a small extent. eHealth tools that use multimodal content (such as videos) and have the possibility for direct communication with providers show improved adherence among vulnerable groups.

Conclusions

eHealth tools that use multimodal content and provide the possibility for direct communication with providers have a higher adherence among vulnerable groups. However, most of the eHealth tools are not embedded within the health care system. They are usually focused on specific problems, such as diabetes or obesity. Hence, they do not provide comprehensive services for patients. This limits the use of eHealth tools as a replacement for existing health care services.

The Effects of Leadership and Job Autonomy on Vitality: Survey and Experimental Evidence

Vitality refers to the experience of having energy available to one's self. Vital employees are full of positive energy when they work, and feel mentally and physically strong. Such employees often show higher job performance and lower stress than their less vital colleagues. Despite the importance of vitality, few public administration studies have studied vitality. More generally, by focusing on vitality, we aim to bring a "positive psychology" perspective into the domain of public administration. We analyze whether two important job characteristics (leader's task communication and job autonomy) affect vitality. We use a multi-method design. A large-scale survey (N = 1,502) shows that leader's task communication and job autonomy are positively related to vitality. A lab experiment (N = 102) replicated these findings, showing cause-and-effect relationships. In conclusion, public organizations can potentially increase employee vitality (a) by increased task communication from leaders and (b) by providing employees with greater job autonomy.

Barriers to Innovation in Urban Wastewater Utilities: Attitudes of Managers in California

In many regions of the world, urban water systems will need to transition into fundamentally different forms to address current stressors and meet impending challenges-faster innovation will need to be part of these transitions. To assess the innovation deficit in urban water organizations and to identify means for supporting innovation, we surveyed wastewater utility managers in California. Our results reveal insights about the attitudes towards innovation among decision makers, and how perceptions at the level of individual managers might create disincentives for experimentation. Although managers reported feeling relatively unhindered organizationally, they also spend less time on innovation than they feel they should. The most frequently reported barriers to innovation included cost and financing; risk and risk aversion; and regulatory compliance. Considering these results in the context of prior research on innovation systems, we conclude that collective action may be required to address underinvestment in innovation.

HRM and its effect on employee, organizational and financial outcomes in health care organizations

BACKGROUND

One of the main goals of Human Resource Management (HRM) is to increase the performance of organizations. However, few studies have explicitly addressed the multidimensional character of performance and linked HR practices to various outcome dimensions. This study therefore adds to the literature by relating HR practices to three outcome dimensions: financial, organizational and employee (HR) outcomes. Furthermore, we will analyze how HR practices influence these outcome dimensions, focusing on the mediating role of job satisfaction.

METHODS

This study uses a unique dataset, based on the 'ActiZ Benchmark in Healthcare', a benchmark study conducted in Dutch home care, nursing care and care homes. Data from autumn 2010 to autumn 2011 were analyzed. In total, 162 organizations participated during this period (approximately 35% of all Dutch care organizations). Employee data were collected using a questionnaire (61,061 individuals, response rate 42%). Clients were surveyed using the Client Quality Index for long-term care, via stratified sampling. Financial outcomes were collected using annual reports. SEM analyses were conducted to test the hypotheses.

RESULTS

It was found that HR practices are - directly or indirectly - linked to all three outcomes. The use of HR practices is related to improved financial outcomes (measure: net margin), organizational outcomes (measure: client satisfaction) and HR outcomes (measure: sickness absence). The impact of HR practices on HR outcomes and organizational outcomes proved substantially larger than their impact on financial outcomes. Furthermore, with respect to HR and organizational outcomes, the hypotheses concerning the full mediating effect of job satisfaction are confirmed. This is in line with the view that employee attitudes are an important element in the 'black box' between HRM and performance.

CONCLUSION

The results underscore the importance of HRM in the health care sector, especially for HR and organizational outcomes. Further analyses of HRM in the health care sector will prove to be a productive endeavor for both scholars and HR managers.

Facing off with Scylla and Charybdis: a comparison of scalar, partial, and the novel possibility of approximate measurement invariance

Measurement invariance (MI) is a pre-requisite for comparing latent variable scores across groups. The current paper introduces the concept of approximate MI building on the work of Muthén and Asparouhov and their application of Bayesian Structural Equation Modeling (BSEM) in the software Mplus. They showed that with BSEM exact zeros constraints can be replaced with approximate zeros to allow for minimal steps away from strict MI, still yielding a well-fitting model. This new opportunity enables researchers to make explicit trade-offs between the degree of MI on the one hand, and the degree of model fit on the other. Throughout the paper we discuss the topic of approximate MI, followed by an empirical illustration where the test for MI fails, but where allowing for approximate MI results in a well-fitting model. Using simulated data, we investigate in which situations approximate MI can be applied and when it leads to unbiased results. Both our empirical illustration and the simulation study show approximate MI outperforms full or partial MI In detecting/recovering the true latent mean difference when there are (many) small differences in the intercepts and factor loadings across groups. In the discussion we provide a step-by-step guide in which situation what type of MI is preferred. Our paper provides a first step in the new research area of (partial) approximate MI and shows that it can be a good alternative when strict MI leads to a badly fitting model and when partial MI cannot be applied.

Reproducibility of energy cost of locomotion in ambulatory children with spina bifida

OBJECTIVES

Many ambulatory children with Spina Bifida (SB) experience functional decline in ambulation despite stable or even improving motor exams. Improving or maintaining low energy cost of locomotion during childhood and throughout the teenage years, could be an important goal for children and adolescents with SB. Purpose of this study was to determine reproducibility of energy expenditure measures during gait in ambulatory children with SB.

DESIGN

Reproducibility study.

SETTING

Child Development and Exercise Center of the University Children's Hospital in Utrecht, the Netherlands.

PARTICIPANTS

Fourteen ambulatory children (6 boys/8 girls) with SB. Mean age was 10.8 years (+ or - 3.4).

INTERVENTIONS

Net and gross energy expenditure measures during locomotion were determined during a six-minute walking test. These measures consisted of energy consumption (ECS), expressed in J/kg/min, and energy cost (EC), expressed in J/kg/m. For reliability, the intra-class coefficient (ICC) was determined. For agreement, the smallest detectable difference (SDD) was calculated.

RESULTS

ICCs vary from 0.86 to 0.96 for both EC and ECS. The SDD ranges from 18-24% for gross measures, up to over 30% for net values.

CONCLUSION

Reproducibility of energy expenditure during ambulation in children with SB should be considered carefully when using these measures in the evaluation of gait. High reliability of energy expenditure measurements makes these measurements appropriate to use as discriminative tools in children with SB, while agreement of only gross EC seems acceptable to use as a evaluative tool in children with SB. Overall, measures of reliability and agreement seem higher in young children when compared to adolescents. Further research is recommended to determine clinically relevant changes in energy expenditure in children with SB.