"Crowdsourcing" ten years in: A review

A Perspective on Crowdsourcing and Human-in-the-Loop Workflows in Precision Health

Modern machine learning approaches have led to performant diagnostic models for a variety of health conditions. Several machine learning approaches, such as decision trees and deep neural networks, can, in principle, approximate any function. However, this power can be considered to be both a gift and a curse, as the propensity toward overfitting is magnified when the input data are heterogeneous and high dimensional and the output class is highly nonlinear. This issue can especially plague diagnostic systems that predict behavioral and psychiatric conditions that are diagnosed with subjective criteria. An emerging solution to this issue is crowdsourcing, where crowd workers are paid to annotate complex behavioral features in return for monetary compensation or a gamified experience. These labels can then be used to derive a diagnosis, either directly or by using the labels as inputs to a diagnostic machine learning model. This viewpoint describes existing work in this emerging field and discusses ongoing challenges and opportunities with crowd-powered diagnostic systems, a nascent field of study. With the correct considerations, the addition of crowdsourcing to human-in-the-loop machine learning workflows for the prediction of complex and nuanced health conditions can accelerate screening, diagnostics, and ultimately access to care.

Does the registry speak your language? A case study of the Global Angelman Syndrome Registry

Global disease registries are critical to capturing common patient related information on rare illnesses, allowing patients and their families to provide information about their condition in a safe, accessible, and engaging manner that enables researchers to undertake critical research aimed at improving outcomes. Typically, English is the default language of choice for these global digital health platforms. Unfortunately, language barriers can significantly inhibit participation from non-English speaking participants. In addition, there is potential for compromises in data quality and completeness. In contrast, multinational commercial entities provide access to their websites in the local language of the country they are operating in, and often provide multiple options reflecting ethnic diversity. This paper presents a case study of how the Global Angelman Syndrome Registry (GASR) has used a novel approach to enable multiple language translations for its website. Using a "semi-automated language translation" approach, the GASR, which was originally launched in English in September 2016, is now available in several other languages. In 2020, the GASR adopted a novel approach using crowd-sourcing and machine translation tools leading to the availability of the GASR in Spanish, Traditional Chinese, Italian, and Hindi. As a result, enrolments increased by 124% percent for Spain, 67% percent for Latin America, 46% percent for Asia, 24% for Italy, and 43% for India. We describe our approach here, which we believe presents an opportunity for cost-effective and timely translations responsive to changes to the registry and helps build and maintain engagement with global disease communities.

Assessing socio-economic profile of U-Reporters: Towards establishing a pool for equity analysis of future crowdsourced surveys

Background

Crowdsourcing was recognized as having the potential to collect information rapidly, inexpensively and accurately. U-Report is a mobile empowerment platform that connects young people all over the world to information that will change their lives and influence decisions. Previous studies of U-Report’s effectiveness highlight strengths in the timeliness, low cost and high credibility for collecting and sending information, however they also highlight areas to improve on concerning data representation. EquityTool has developed a simpler approach to assess the wealth quintiles of respondents based on fewer questions derived from large household surveys such as Multiple Indicators Cluster Surveys (MICS) and Demographic and Health Surveys (DHS).

Methods

The methodology of Equity Tool was adopted to assess the socio-economic profile of U-Reporters (ie, enrolled participants of U-Report) in Bangladesh. The RapidPro flow collected the survey responses and scored them against the DHS national wealth index using the EquityTool methodology. This helped placing each U-Reporter who completed all questions into the appropriate wealth quintile.

Results

With 19% of the respondents completing all questions, the respondents fell into all 5 wealth quintiles, with 79% in the top-two quintiles and only 21% in the lower-three resulting in an Equity Index of 53/100 where 100 is completely in line with Bangladesh equity distribution and 1 is the least in line. An equitable random sample of 1828 U-Reporters from among the regular and frequent respondents was subsequently created for future surveys and the sample has an Equity Index of 98/100.

Conclusions

U-Report in Bangladesh does reach the poorest quintiles while the initial recruitment skews to respondents towards better off families. It is possible to create an equitable random sub-sample of respondents from all five wealth quintiles and thus process information and data for future surveys. Moving forward, U-Reporters from the poorly represented quintiles may be incentivized to recruit peers to increase equity and representation. In times of COVID-19, U-Report in combination with the EquityTool has the potential to enhance the quality of crowdsourced data for statistical analysis.

Crowdsourcing and open innovation in drug discovery: recent contributions and future directions

The past decade has seen significant growth in the use of 'crowdsourcing' and open innovation approaches to engage 'citizen scientists' to perform novel scientific research. Here, we quantify and summarize the current state of adoption of open innovation by major pharmaceutical companies. We also highlight recent crowdsourcing and open innovation research contributions to the field of drug discovery, and interesting future directions.

Digital crowdsourced intervention to promote HIV testing among MSM in China: study protocol for a cluster randomized controlled trial

BACKGROUND

Men who have sex with men (MSM) are an important HIV key population in China. However, HIV testing rates among MSM remain suboptimal. Digital crowdsourced media interventions may be a useful tool to reach this marginalized population. We define digital crowdsourced media as using social media, mobile phone applications, Internet, or other digital approaches to disseminate messages developed from crowdsourcing contests. The proposed cluster randomized controlled trial (RCT) study aims to assess the effectiveness of a digital crowdsourced intervention to increase HIV testing uptake and decrease risky sexual behaviors among Chinese MSM.

METHODS

A two-arm, cluster-randomized controlled trial will be implemented in eleven cities (ten clusters) in Shandong Province, China. Targeted study participants will be 250 MSM per arm and 50 participants per cluster. MSM who are 18 years old or above, live in the study city, have not been tested for HIV in the past 3 months, are not living with HIV or have never been tested for HIV, and are willing to provide informed consent will be enrolled. Participants will be recruited through banner advertisements on Blued, the largest gay dating app in China, and in-person at community-based organizations (CBOs). The intervention includes a series of crowdsourced intervention materials (24 images and four short videos about HIV testing and safe sexual behaviors) and HIV self-test services provided by the study team. The intervention was developed through a series of participatory crowdsourcing contests before this study. The self-test kits will be sent to the participants in the intervention group at the 2nd and 3rd follow-ups. Participants will be followed up quarterly during the 12-month period. The primary outcome will be self-reported HIV testing uptake at 12 months. Secondary outcomes will include changes in condomless sex, self-test efficacy, social network engagement, HIV testing social norms, and testing stigma.

DISCUSSION

Innovative approaches to HIV testing among marginalized population are urgently needed. Through this cluster randomized controlled trial, we will evaluate the effectiveness of a digital crowdsourced intervention, improving HIV testing uptake among MSM and providing a resource in related public health fields.

TRIAL REGISTRATION

ChiCTR1900024350 . Registered on 6 July 2019.

Emerging Wireless Sensor Networks and Internet of Things Technologies-Foundations of Smart Healthcare

Future smart healthcare systems - often referred to as Internet of Medical Things (IoMT) - will combine a plethora of wireless devices and applications that use wireless communication technologies to enable the exchange of healthcare data. Smart healthcare requires sufficient bandwidth, reliable and secure communication links, energy-efficient operations, and Quality of Service (QoS) support. The integration of Internet of Things (IoT) solutions into healthcare systems can significantly increase intelligence, flexibility, and interoperability. This work provides an extensive survey on emerging IoT communication standards and technologies suitable for smart healthcare applications. A particular emphasis has been given to low-power wireless technologies as a key enabler for energy-efficient IoT-based healthcare systems. Major challenges in privacy and security are also discussed. A particular attention is devoted to crowdsourcing/crowdsensing, envisaged as tools for the rapid collection of massive quantities of medical data. Finally, open research challenges and future perspectives of IoMT are presented.

Toward Personalized Tinnitus Treatment: An Exploratory Study Based on Internet Crowdsensing

Introduction: Chronic tinnitus is a condition estimated to affect 10–15% of the population. No treatment has shown efficacy in randomized clinical trials to reliably and effectively suppress the phantom perceptions, and little is known why patients react differently to the same treatments. Tinnitus heterogeneity may play a central role in treatment response, but no study has tried to capture tinnitus heterogeneity in terms of treatment response.

Research Goals: To test if the individualized treatment response can be predicted using personal, tinnitus, and treatment characteristics.

Methods: A survey conducted by the web platform Tinnitus Hub collected data of 5017 tinnitus bearers. The participants reported which treatments they tried and the outcome of the given treatment. Demographic and tinnitus characteristics, alongside with treatment duration were used as predictors of treatment outcomes in both an univariate as well as a multivariate regression setup. First, simple linear regressions were used with each of the 13 predictors on all of 25 treatment outcomes to predict how much variance could be explained by each predictor individually. Then, all 13 predictors were added together in the elastic net regression to predict treatment outcomes.

Results: Individual predictors from the linear regression models explained on average 2% of the variance of treatment outcome. “Duration of treatment” was the predictor that explained, on average, most of the variance, 6.8%. When combining all the predictors in the elastic net, the model could explain on average 16% of the deviance of treatment outcomes.

Discussion: By demonstrating that different aspects predict response to various treatments, our results support the notion that tinnitus heterogeneity influences the observed variability in treatment response. Moreover, the data suggest the potential of personalized tinnitus treatment based on demographic and clinical characteristics.

Crowdsourcing in medical research: concepts and applications

Crowdsourcing shifts medical research from a closed environment to an open collaboration between the public and researchers. We define crowdsourcing as an approach to problem solving which involves an organization having a large group attempt to solve a problem or part of a problem, then sharing solutions. Crowdsourcing allows large groups of individuals to participate in medical research through innovation challenges, hackathons, and related activities. The purpose of this literature review is to examine the definition, concepts, and applications of crowdsourcing in medicine. This multi-disciplinary review defines crowdsourcing for medicine, identifies conceptual antecedents (collective intelligence and open source models), and explores implications of the approach. Several critiques of crowdsourcing are also examined. Although several crowdsourcing definitions exist, there are two essential elements: (1) having a large group of individuals, including those with skills and those without skills, propose potential solutions; (2) sharing solutions through implementation or open access materials. The public can be a central force in contributing to formative, pre-clinical, and clinical research. A growing evidence base suggests that crowdsourcing in medicine can result in high-quality outcomes, broad community engagement, and more open science.

Identifying potential uses of crowdsourcing in global health, conflict, and humanitarian settings: an adapted CHNRI (Child Health and Nutrition Initiative) exercise

Background

Crowdsourcing, outsourcing problems and tasks to a crowd, has grown exponentially since the term was coined a decade ago. Being a rapid and inexpensive approach, it is particularly amenable to addressing problems in global health, conflict and humanitarian settings, but its potential has not been systematically assessed. We employed the Child Health and Nutrition Research Initiative's (CHNRI) method to generate a ranked list of potential uses of crowdsourcing in global health and conflict.

Process

94 experts in global health and crowdsourcing submitted their ideas, and 239 ideas were scored. Each expert scored ideas against three of seven criteria, which were tailored specifically for the exercise. A relative ranking was calculated, along with an Average Expert Agreement (AEA).

Findings

On a scale from 0-100, the scores assigned to proposed ideas ranged from 80.39 to 42.01. Most ideas were related to problem solving (n = 112) or data generation (n = 91). Using health care workers to share information about disease outbreaks to ensure global response had the highest score and agreement. Within the top 15, four additional ideas related to containing communicable diseases, two ideas related to using crowdsourcing for vital registration and two to improve maternal and child health. The top conflict ideas related to epidemic responses and various aspects of disease spread. Wisdom of the crowds and machine learning scored low despite being promising in literature.

Interpretations

Experts were invited to generate ideas during the Ebola crisis and to score during reports of Zika, which may have affected the scoring. However, crowdsourcing's rapid, inexpensive characteristics make it suitable for addressing epidemics. Given that many ideas reflected Sustainable Development Goals (SDGs), crowdsourcing may be an innovative solution to achieving some of the SDGs.

Applications of crowdsourcing in health: an overview

Background

Crowdsourcing is a nascent phenomenon that has grown exponentially since it was coined in 2006. It involves a large group of people solving a problem or completing a task for an individual or, more commonly, for an organisation. While the field of crowdsourcing has developed more quickly in information technology, it has great promise in health applications. This review examines uses of crowdsourcing in global health and health, broadly.

Methods

Semantic searches were run in Google Scholar for “crowdsourcing,” “crowdsourcing and health,” and similar terms. 996 articles were retrieved and all abstracts were scanned. 285 articles related to health. This review provides a narrative overview of the articles identified.

Results

Eight areas where crowdsourcing has been used in health were identified: diagnosis; surveillance; nutrition; public health and environment; education; genetics; psychology; and, general medicine/other. Many studies reported crowdsourcing being used in a diagnostic or surveillance capacity. Crowdsourcing has been widely used across medical disciplines; however, it is important for future work using crowdsourcing to consider the appropriateness of the crowd being used to ensure the crowd is capable and has the adequate knowledge for the task at hand. Gamification of tasks seems to improve accuracy; other innovative methods of analysis including introducing thresholds and measures of trustworthiness should be considered.

Conclusion

Crowdsourcing is a new field that has been widely used and is innovative and adaptable. With the exception of surveillance applications that are used in emergency and disaster situations, most uses of crowdsourcing have only been used as pilots. These exceptions demonstrate that it is possible to take crowdsourcing applications to scale. Crowdsourcing has the potential to provide more accessible health care to more communities and individuals rapidly and to lower costs of care.

Setting research priorities for maternal, newborn, child health and nutrition in India by engaging experts from 256 indigenous institutions contributing over 4000 research ideas: a CHNRI exercise by ICMR and INCLEN

Background

Health research in low– and middle– income countries (LMICs) is often driven by donor priorities rather than by the needs of the countries where the research takes place. This lack of alignment of donor’s priorities with local research need may be one of the reasons why countries fail to achieve set goals for population health and nutrition. India has a high burden of morbidity and mortality in women, children and infants. In order to look forward toward the Sustainable Development Goals, the Indian Council of Medical Research (ICMR) and the INCLEN Trust International (INCLEN) employed the Child Health and Nutrition Research Initiative’s (CHNRI) research priority setting method for maternal, neonatal, child health and nutrition with the timeline of 2016–2025. The exercise was the largest to–date use of the CHNRI methodology, both in terms of participants and ideas generated and also expanded on the methodology.

Methods

CHNRI is a crowdsourcing–based exercise that involves using the collective intelligence of a group of stakeholders, usually researchers, to generate and score research options against a set of criteria. This paper reports on a large umbrella CHNRI that was divided into four theme–specific CHNRIs (maternal, newborn, child health and nutrition). A National Steering Group oversaw the exercise and four theme–specific Research Sub–Committees technically supported finalizing the scoring criteria and refinement of research ideas for the respective thematic areas. The exercise engaged participants from 256 institutions across India – 4003 research ideas were generated from 498 experts which were consolidated into 373 research options (maternal health: 122; newborn health: 56; child health: 101; nutrition: 94); 893 experts scored these against five criteria (answerability, relevance, equity, innovation and out–of–box thinking, investment on research). Relative weights to the criteria were assigned by 79 members from the Larger Reference Group. Given India’s diversity, priorities were identified at national and three regional levels: (i) the Empowered Action Group (EAG) and North–Eastern States; (ii) States and Union territories in Northern India (including West Bengal); and (iii) States and Union territories in Southern and Western parts of India.

Conclusions

The exercise leveraged the inherent flexibility of the CHNRI method in multiple ways. It expanded on the CHNRI methodology enabling analyses for identification of research priorities at national and regional levels. However, prioritization of research options are only valuable if they are put to use, and we hope that donors will take advantage of this prioritized list of research options.