"Crowdsourcing" ten years in: A review

Feasibility assessment of crowdsourcing slogans for promoting household waste segregation in India: a cross-sectional study

Introduction

Crowdsourcing is an emerging technique to engage or access a wider set of experts and multiple stakeholders through online platforms, which might effectively be employed in waste management. Therefore, we assessed the feasibility of the crowdsourcing method to provide an alternative approach that can improve household waste segregation using an "online-slogan-contest".

Methods

The contest was promoted via targeted emails to various governmental and non-governmental organizations and through social media platforms for around 4 weeks (25 days). The entries were received through a Google form. The slogans were assessed by the experts and analyzed using content analysis methods.

Results

Total 969 entries were received from different geographic regions in India. Of that, 456 were in English and 513 in Hindi. Five themes of waste segregation emerged from the received slogans: (1) Community awareness, responsibility, and support, (2) Significance of household waste segregation, (3) Use of separate dustbins, (4) Health and well-being, and (5) Environment and sustainability.

Discussion

Crowdsourcing approaches can be used by local authorities for improving waste management approaches and are recommended as these involve a wider audience within a short time frame. Moreover, this approach is flexible and integrating crowdsourcing approaches strengthens our understanding of existing waste management activities.

Implementation and evaluation of crowdsourcing in global health education

BACKGROUND

Current global health course is most set as elective course taught in traditional teacher-taught model with low credit and short term. Innovate teaching models are required. Crowdsourcing characterized by high flexibility and strong application-orientation holds its potential to enhance global health education. We applied crowdsourcing to global health teaching for undergraduates, aiming to develop and evaluate a new teaching model for global health education.

METHODS

Crowdsourcing was implemented into traditional course-based teaching via introducing five COVID-19 related global health debates. Undergraduate students majoring in preventative medicine and nursing grouped in teams of 5-8, were asked to resolve these debates in reference to main content of the course and with manner they thought most effective to deliver the messages. Students' experience and teaching effect, were evaluated by questionnaires and teachers' ratings, respectively. McNemar's test was used to compare the difference in students' experience before and after the course, and regression models were used to explore the influencing factors of the teaching effect.

RESULTS

A total of 172 undergraduates were included, of which 122 (71%) were females. Students' evaluation of the new teaching model improved after the course, but were polarized. Students' self-reported teaching effect averaged 67.53 ± 16.8 and the teachers' rating score averaged 90.84 ± 4.9. Students majoring in preventive medicine, participated in student union, spent more time on revision, and had positive feedback on the new teaching model tended to perform better.

CONCLUSION

We innovatively implemented crowdsourcing into global health teaching, and found this new teaching model was positively received by undergraduate students with improved teaching effects. More studies are needed to optimize the implementation of crowdsourcing alike new methods into global health education, to enrich global health teaching models.

Twitter-based crowdsourcing: What kind of measures can help to end the COVID-19 pandemic faster?

Background

Crowdsourcing is a low-cost, adaptable, and innovative method to collect ideas from numerous contributors with diverse backgrounds. Crowdsourcing from social media like Twitter can be used for generating ideas in a noticeably brief time based on contributions from globally distributed users. The world has been challenged by the COVID-19 pandemic in the last several years. Measures to combat the pandemic continue to evolve worldwide, and ideas and opinions on optimal counteraction strategies are of high interest.

Objective

This study aimed to validate the use of Twitter as a crowdsourcing platform in order to gain an understanding of public opinion on what measures can help to end the COVID-19 pandemic faster.

Methods

This cross-sectional study was conducted during the period from December 22, 2021, to February 4, 2022. Tweets were posted by accounts operated by the authors, asking “How to faster end the COVID-19 pandemic?” and encouraging the viewers to comment on measures that they perceive would be effective to achieve this goal. The ideas from the users' comments were collected and categorized into two major themes – personal and institutional measures. In the final stage of the campaign, a Twitter poll was conducted to get additional comments and to estimate which of the two groups of measures were perceived to be important amongst Twitter users.

Results

The crowdsourcing campaign generated seventeen suggested measures categorized into two major themes (personal and institutional) that received a total of 1,727 endorsements (supporting comments, retweets, and likes). The poll received a total of 325 votes with 58% of votes underscoring the importance of both personal and institutional measures, 20% favoring personal measures, 11% favoring institutional measures, and 11% of the votes given just out of curiosity to see the vote results.

Conclusions

Twitter was utilized successfully for crowdsourcing ideas on strategies how to end the COVID-19 pandemic faster. The results indicate that the Twitter community highly values the significance of both personal responsibility and institutional measures to counteract the pandemic. This study validates the use of Twitter as a primary tool that could be used for crowdsourcing ideas with healthcare significance.

Leveraging non-expert crowdsourcing to segment the optic cup and disc of multicolor fundus images

Multicolor scanning laser imaging (MCI) images have broad application potential in the diagnosis of fundus diseases such as glaucoma. However, the performance level of automatic aided diagnosis systems based on MCI images is limited by the lack of high-quality annotations of numerous images. Producing annotations for vast amounts of MCI images will be a prolonged process if we only employ experts. Therefore, we consider non-expert crowdsourcing, which is an alternative approach to produce useful annotations efficiently and low cost. In this work, we aim to explore the effectiveness of non-expert crowdsourcing on the segmentation of the optic cup (OC) and optic disc (OD), which is an upstream task for glaucoma diagnosis, using MCI images. To this end, desensitized MCI images are independently annotated by four non-expert annotators, constructing a crowdsourcing dataset. To profit from crowdsourcing, we propose a model consisting of coupled regularization network and segmentation network. The regularization network generates learnable pixel-wise confusion matrices (CMs) that reflects preferences of each annotator. During training, the CMs and segmentation network are simultaneously optimized to enable dynamic trade-offs for non-expert annotations and generate reliable predictions. Crowdsourcing learning using our method have an average Mean Intersection Over Union ( M ) of 91.34%, while the average M of model trained by expert annotations is 91.72%. In addition, comparative experiments show that in our segmentation task non-expert crowdsourcing can be on a par with the expert who annotates 90% of data. Our work suggests that crowdsourcing in the segmentation of OC and OD using MCI images has the potential to be a substitute to expert annotation, which will accelerate the construction of large datasets to facilitate the application of deep learning in clinical diagnosis using MCI images.

Understanding factors that promote uptake of HIV self-testing among young people in Nigeria: Framing youth narratives using the PEN-3 cultural model

It is important to understand how to frame the formats for promoting HIV self-testing to increase uptake among young people. In this study, we used a culture-centered model to understand the narratives of HIV self-testing preferences among young people in Nigeria. We conducted a crowdsourcing contest to solicit ideas surrounding HIV self-testing promotion among young people (10–24 years) in Nigeria from October to November 2018 as part of the 2018 World AIDS Day event. We received 903 submissions and employed thematic content analysis to evaluate 769 eligible youth narratives. Thematic content analysis of the statements from the youth narratives was guided by the PEN-3 cultural model to examine the positive, existential, and negative perceptions (beliefs and values), enablers (resources), and nurturers (roles of friends and family) of HIV self-testing promotion among young people in Nigeria. Several themes emerged as factors that influence the uptake of HIV self-testing among young people in Nigeria. Specifically, seven themes emerged as perceptions: HIV testing accessibility, stigma reduction, and autonomy (positive); HIV self-testing kit packaging and advertisements (existential); lack of knowledge and increased stigma (negative). Seven themes emerged as enablers: social media, school, and government promotion (positive); gamification and animation (existential); high cost and access to linkage to care services (negative); And seven themes emerged as nurturers: peer, families, and faith-based communities (positive); parents and family-centered approach (existential); and partners and family (negative). Our data suggests that increased awareness around HIV self-testing on current youth-friendly platforms, de-stigmatization of HIV and HIV self-testing, decreased prices for HIV self-testing kits, reliability of testing kits, increased linkage to care services, and promotion of self-testing among family members and the community will be beneficial for HIV self-testing scale-up measures among young people in Nigeria.

Flesch-Kincaid Measure as Proxy of Socio-Economic Status on Twitter

Social media gives researchers an invaluable opportunity to gain insight into different facets of human life. Researchers put a great emphasis on categorizing the socioeconomic status (SES) of individuals to help predict various findings of interest. Forum uses, hashtags and chatrooms are common tools of conversations grouping. Crowdsourcing involves gathering intelligence to group online user community based on common interest. This paper provides a mechanism to look at writings on social media and group them based on their academic background. We analyzed online forum posts from various geographical regions in the US and characterized the readability scores of users. Specifically, we collected 10,000 tweets from the members of US Senate and computed the Flesch-Kincaid readability score. Comparing the Senators’ tweets to the ones from average internet users, we note 1) US Senators’ readability based on their tweets rate is much higher, and 2) immense difference among average citizen’s score compared to those of US Senators is attributed to the wide spectrum of academic attainment.

Using a visual analog scale (VAS) to measure tinnitus-related distress and loudness: Investigating correlations using the Mini-TQ results of participants from the TrackYourTinnitus platform

INTRODUCTION

Tinnitus, a perception of ringing and buzzing sound in the ear, has not been completely understood yet. It is well known that tinnitus-related distress and loudness can change over time. However, proper comparability for the data collection approaches requires further focused studies. In this context, technology such as the use of mobile devices may be a promising approach. Repeated assessments of tinnitus-related distress and loudness in Ecological Momentary Assessment (EMA) studies require a short assessment, and a Visual Analogic Scale (VAS) is often used in this context. Yet, their comparability with psychometric questionnaires remains unclear and thus was the focus of this study. Research goals: The evaluation of the appropriateness of VAS in measuring tinnitus-related distress and loudness is pursued in this paper.

METHODS

The Mini Tinnitus Questionnaire (Mini-TQ) measured tinnitus-related distress once. Tinnitus-related distress and tinnitus loudness were measured repeatedly using VAS on a daily basis during 7 days in the TrackYourTinnitus (TYT) smartphone app and were summarized per day using mean and median results. Then, correlations between summarized VAS tinnitus-related distress and summarized VAS tinnitus loudness, on the one side, and Mini-TQ, on the other side, were calculated.

RESULTS

Correlations between Mini-TQ and VAS tinnitus-related distress ranged between r = 0.36 and r = 0.52, while correlations between Mini-TQ and VAS tinnitus loudness ranged between r = 0.25 and r = 0.36. The more time difference between the Mini-TQ and the VAS assessments is, the lower the correlations between them. Mean and median VAS values per day resulted in similar correlations.

CONCLUSIONS

Mobile-based VAS seems to be an appropriate approach to utilize daily measurements of tinnitus-related distress.

Assessing socio-economic profile of U-Reporters: Towards establishing a pool for equity analysis of future crowdsourced surveys

Background

Crowdsourcing was recognized as having the potential to collect information rapidly, inexpensively and accurately. U-Report is a mobile empowerment platform that connects young people all over the world to information that will change their lives and influence decisions. Previous studies of U-Report’s effectiveness highlight strengths in the timeliness, low cost and high credibility for collecting and sending information, however they also highlight areas to improve on concerning data representation. EquityTool has developed a simpler approach to assess the wealth quintiles of respondents based on fewer questions derived from large household surveys such as Multiple Indicators Cluster Surveys (MICS) and Demographic and Health Surveys (DHS).

Methods

The methodology of Equity Tool was adopted to assess the socio-economic profile of U-Reporters (ie, enrolled participants of U-Report) in Bangladesh. The RapidPro flow collected the survey responses and scored them against the DHS national wealth index using the EquityTool methodology. This helped placing each U-Reporter who completed all questions into the appropriate wealth quintile.

Results

With 19% of the respondents completing all questions, the respondents fell into all 5 wealth quintiles, with 79% in the top-two quintiles and only 21% in the lower-three resulting in an Equity Index of 53/100 where 100 is completely in line with Bangladesh equity distribution and 1 is the least in line. An equitable random sample of 1828 U-Reporters from among the regular and frequent respondents was subsequently created for future surveys and the sample has an Equity Index of 98/100.

Conclusions

U-Report in Bangladesh does reach the poorest quintiles while the initial recruitment skews to respondents towards better off families. It is possible to create an equitable random sub-sample of respondents from all five wealth quintiles and thus process information and data for future surveys. Moving forward, U-Reporters from the poorly represented quintiles may be incentivized to recruit peers to increase equity and representation. In times of COVID-19, U-Report in combination with the EquityTool has the potential to enhance the quality of crowdsourced data for statistical analysis.

Crowdsourcing and open innovation in drug discovery: recent contributions and future directions

The past decade has seen significant growth in the use of 'crowdsourcing' and open innovation approaches to engage 'citizen scientists' to perform novel scientific research. Here, we quantify and summarize the current state of adoption of open innovation by major pharmaceutical companies. We also highlight recent crowdsourcing and open innovation research contributions to the field of drug discovery, and interesting future directions.

Digital crowdsourced intervention to promote HIV testing among MSM in China: study protocol for a cluster randomized controlled trial

BACKGROUND

Men who have sex with men (MSM) are an important HIV key population in China. However, HIV testing rates among MSM remain suboptimal. Digital crowdsourced media interventions may be a useful tool to reach this marginalized population. We define digital crowdsourced media as using social media, mobile phone applications, Internet, or other digital approaches to disseminate messages developed from crowdsourcing contests. The proposed cluster randomized controlled trial (RCT) study aims to assess the effectiveness of a digital crowdsourced intervention to increase HIV testing uptake and decrease risky sexual behaviors among Chinese MSM.

METHODS

A two-arm, cluster-randomized controlled trial will be implemented in eleven cities (ten clusters) in Shandong Province, China. Targeted study participants will be 250 MSM per arm and 50 participants per cluster. MSM who are 18 years old or above, live in the study city, have not been tested for HIV in the past 3 months, are not living with HIV or have never been tested for HIV, and are willing to provide informed consent will be enrolled. Participants will be recruited through banner advertisements on Blued, the largest gay dating app in China, and in-person at community-based organizations (CBOs). The intervention includes a series of crowdsourced intervention materials (24 images and four short videos about HIV testing and safe sexual behaviors) and HIV self-test services provided by the study team. The intervention was developed through a series of participatory crowdsourcing contests before this study. The self-test kits will be sent to the participants in the intervention group at the 2nd and 3rd follow-ups. Participants will be followed up quarterly during the 12-month period. The primary outcome will be self-reported HIV testing uptake at 12 months. Secondary outcomes will include changes in condomless sex, self-test efficacy, social network engagement, HIV testing social norms, and testing stigma.

DISCUSSION

Innovative approaches to HIV testing among marginalized population are urgently needed. Through this cluster randomized controlled trial, we will evaluate the effectiveness of a digital crowdsourced intervention, improving HIV testing uptake among MSM and providing a resource in related public health fields.

TRIAL REGISTRATION

ChiCTR1900024350 . Registered on 6 July 2019.

Studying Unemployment Effects on Mental Health: Social Media versus the Traditional Approach

Social media, traditionally reserved for social exchanges on the Internet, has been increasingly used by researchers to gain insight into different facets of human life. Unemployment is an area that has gained attention by researchers in various fields. Medical practitioners especially in the area of mental health have traditionally monitored the effects of involuntary unemployment with great interest. The question we want to address is as follows: while many researchers have been using data from social media and microblogging sites in the past few years, do they provide results consistent with traditional research? Furthermore, if the data are indeed consistent, are they detailed enough to deduce possible reasons and remedies? We believe that having a concise answer to these questions is imperative for a sustainable mechanism for medical practitioners and researchers to gather and analyze data. The stigma of mental health prevents a good portion of society from seeking help, but the anonymity provided by the Internet could shatter such barriers, thus allowing people affected by conditions such as mental health and unemployment to express themselves freely. In this work, we compare the feedback gathered from social media using crowdsourcing techniques to results obtained prior to the advent of social media and microblogging. We find that the results are consistent in terms of (1) financial strain being the biggest stressor and concern, (2) the onslaught of depression being typical and (3) possible interventions, including reemployment and support from friends and family, playing a crucial role in minimizing the effects of involuntary unemployment. Lastly, we could not find enough evidence to study effects on physical health and somatization in this work.

High-dimensionality Data Analysis of Pharmacological Systems Associated with Complex Diseases

It is widely accepted that molecular reductionist views of highly complex human physiologic activity, e.g., the aging process, as well as therapeutic drug efficacy are largely oversimplifications. Currently some of the most effective appreciation of biologic disease and drug response complexity is achieved using high-dimensionality (H-D) data streams from transcriptomic, proteomic, metabolomics, or epigenomic pipelines. Multiple H-D data sets are now common and freely accessible for complex diseases such as metabolic syndrome, cardiovascular disease, and neurodegenerative conditions such as Alzheimer's disease. Over the last decade our ability to interrogate these high-dimensionality data streams has been profoundly enhanced through the development and implementation of highly effective bioinformatic platforms. Employing these computational approaches to understand the complexity of age-related diseases provides a facile mechanism to then synergize this pathologic appreciation with a similar level of understanding of therapeutic-mediated signaling. For informative pathology and drug-based analytics that are able to generate meaningful therapeutic insight across diverse data streams, novel informatics processes such as latent semantic indexing and topological data analyses will likely be important. Elucidation of H-D molecular disease signatures from diverse data streams will likely generate and refine new therapeutic strategies that will be designed with a cognizance of a realistic appreciation of the complexity of human age-related disease and drug effects. We contend that informatic platforms should be synergistic with more advanced chemical/drug and phenotypic cellular/tissue-based analytical predictive models to assist in either de novo drug prioritization or effective repurposing for the intervention of aging-related diseases. SIGNIFICANCE STATEMENT: All diseases, as well as pharmacological mechanisms, are far more complex than previously thought a decade ago. With the advent of commonplace access to technologies that produce large volumes of high-dimensionality data (e.g., transcriptomics, proteomics, metabolomics), it is now imperative that effective tools to appreciate this highly nuanced data are developed. Being able to appreciate the subtleties of high-dimensionality data will allow molecular pharmacologists to develop the most effective multidimensional therapeutics with effectively engineered efficacy profiles.

Emerging Wireless Sensor Networks and Internet of Things Technologies-Foundations of Smart Healthcare

Future smart healthcare systems - often referred to as Internet of Medical Things (IoMT) - will combine a plethora of wireless devices and applications that use wireless communication technologies to enable the exchange of healthcare data. Smart healthcare requires sufficient bandwidth, reliable and secure communication links, energy-efficient operations, and Quality of Service (QoS) support. The integration of Internet of Things (IoT) solutions into healthcare systems can significantly increase intelligence, flexibility, and interoperability. This work provides an extensive survey on emerging IoT communication standards and technologies suitable for smart healthcare applications. A particular emphasis has been given to low-power wireless technologies as a key enabler for energy-efficient IoT-based healthcare systems. Major challenges in privacy and security are also discussed. A particular attention is devoted to crowdsourcing/crowdsensing, envisaged as tools for the rapid collection of massive quantities of medical data. Finally, open research challenges and future perspectives of IoMT are presented.

Crowdsourced Assessment of Inanimate Biotissue Drills: A Valid and Cost-Effective Way to Evaluate Surgical Trainees

OBJECTIVE

Providing feedback to surgical trainees is a critical component for assessment of technical skills, yet remains costly and time consuming. We hypothesize that statistical selection can identify a homogenous group of nonexpert crowdworkers capable of accurately grading inanimate surgical video.

DESIGN

Applicants auditioned by grading 9 training videos using the Objective Structured Assessment of Technical Skills (OSATS) tool and an error-based checklist. The summed OSATS, summed errors, and OSATS summary score were tested for outliers using Cronbach's Alpha and single measure intraclass correlation. Accepted crowdworkers then submitted grades for videos in 3 different compositions: full video 1× speed, full video 2× speed, and critical section segmented video. Graders were blinded to this study and a similar statistical analysis was performed.

SETTING

The study was conducted at the University of Pittsburgh Medical Center (Pittsburgh, PA), a tertiary care academic teaching hospital.

PARTICIPANTS

Thirty-six premedical students participated as crowdworker applicants and 2 surgery experts were compared as the gold-standard.

RESULTS

The selected hire intraclass correlation was 0.717 for Total Errors and 0.794 for Total OSATS for the first hire group and 0.800 for Total OSATS and 0.654 for Total Errors for the second hire group. There was very good correlation between full videos at 1× and 2× speed with an interitem statistic of 0.817 for errors and 0.86 for OSATS. Only moderate correlation was found with critical section segments. In 1 year 275hours of inanimate video was graded costing $22.27/video or $1.03/minute.

CONCLUSIONS

Statistical selection can be used to identify a homogenous cohort of crowdworkers used for grading trainees' inanimate drills. Crowdworkers can distinguish OSATS metrics and errors in full videos at 2× speed but were less consistent with segmented videos. The program is a comparatively cost-effective way to provide feedback to surgical trainees.

Crowdsourcing in medical research: concepts and applications

Crowdsourcing shifts medical research from a closed environment to an open collaboration between the public and researchers. We define crowdsourcing as an approach to problem solving which involves an organization having a large group attempt to solve a problem or part of a problem, then sharing solutions. Crowdsourcing allows large groups of individuals to participate in medical research through innovation challenges, hackathons, and related activities. The purpose of this literature review is to examine the definition, concepts, and applications of crowdsourcing in medicine. This multi-disciplinary review defines crowdsourcing for medicine, identifies conceptual antecedents (collective intelligence and open source models), and explores implications of the approach. Several critiques of crowdsourcing are also examined. Although several crowdsourcing definitions exist, there are two essential elements: (1) having a large group of individuals, including those with skills and those without skills, propose potential solutions; (2) sharing solutions through implementation or open access materials. The public can be a central force in contributing to formative, pre-clinical, and clinical research. A growing evidence base suggests that crowdsourcing in medicine can result in high-quality outcomes, broad community engagement, and more open science.

Exploring the characteristics of crowdsourcing: An online observational study

This article examines the application of crowdsourcing in research studies. The aim of this study is to understand how crowdsourcing is being used in research by undertaking a content analysis of studies posted to an online site designed to facilitate crowdsourced research. While there are a number of websites that facilitate crowdsourcing, this study provides an analysis only of research studies posted on crowdcrafting.org . Characteristics of crowdsourcing, proposed by Estellés-Arolas and González-Ladrón-de-Guevara, served as the framework for the content analysis, and research projects were evaluated as to how they addressed each of the proposed criteria. This article concludes with recommendations for researchers undertaking the design and implementation of projects employing crowdsourcing.

Identifying potential uses of crowdsourcing in global health, conflict, and humanitarian settings: an adapted CHNRI (Child Health and Nutrition Initiative) exercise

Background

Crowdsourcing, outsourcing problems and tasks to a crowd, has grown exponentially since the term was coined a decade ago. Being a rapid and inexpensive approach, it is particularly amenable to addressing problems in global health, conflict and humanitarian settings, but its potential has not been systematically assessed. We employed the Child Health and Nutrition Research Initiative's (CHNRI) method to generate a ranked list of potential uses of crowdsourcing in global health and conflict.

Process

94 experts in global health and crowdsourcing submitted their ideas, and 239 ideas were scored. Each expert scored ideas against three of seven criteria, which were tailored specifically for the exercise. A relative ranking was calculated, along with an Average Expert Agreement (AEA).

Findings

On a scale from 0-100, the scores assigned to proposed ideas ranged from 80.39 to 42.01. Most ideas were related to problem solving (n = 112) or data generation (n = 91). Using health care workers to share information about disease outbreaks to ensure global response had the highest score and agreement. Within the top 15, four additional ideas related to containing communicable diseases, two ideas related to using crowdsourcing for vital registration and two to improve maternal and child health. The top conflict ideas related to epidemic responses and various aspects of disease spread. Wisdom of the crowds and machine learning scored low despite being promising in literature.

Interpretations

Experts were invited to generate ideas during the Ebola crisis and to score during reports of Zika, which may have affected the scoring. However, crowdsourcing's rapid, inexpensive characteristics make it suitable for addressing epidemics. Given that many ideas reflected Sustainable Development Goals (SDGs), crowdsourcing may be an innovative solution to achieving some of the SDGs.

Applications of crowdsourcing in health: an overview

Background

Crowdsourcing is a nascent phenomenon that has grown exponentially since it was coined in 2006. It involves a large group of people solving a problem or completing a task for an individual or, more commonly, for an organisation. While the field of crowdsourcing has developed more quickly in information technology, it has great promise in health applications. This review examines uses of crowdsourcing in global health and health, broadly.

Methods

Semantic searches were run in Google Scholar for “crowdsourcing,” “crowdsourcing and health,” and similar terms. 996 articles were retrieved and all abstracts were scanned. 285 articles related to health. This review provides a narrative overview of the articles identified.

Results

Eight areas where crowdsourcing has been used in health were identified: diagnosis; surveillance; nutrition; public health and environment; education; genetics; psychology; and, general medicine/other. Many studies reported crowdsourcing being used in a diagnostic or surveillance capacity. Crowdsourcing has been widely used across medical disciplines; however, it is important for future work using crowdsourcing to consider the appropriateness of the crowd being used to ensure the crowd is capable and has the adequate knowledge for the task at hand. Gamification of tasks seems to improve accuracy; other innovative methods of analysis including introducing thresholds and measures of trustworthiness should be considered.

Conclusion

Crowdsourcing is a new field that has been widely used and is innovative and adaptable. With the exception of surveillance applications that are used in emergency and disaster situations, most uses of crowdsourcing have only been used as pilots. These exceptions demonstrate that it is possible to take crowdsourcing applications to scale. Crowdsourcing has the potential to provide more accessible health care to more communities and individuals rapidly and to lower costs of care.

Setting research priorities for maternal, newborn, child health and nutrition in India by engaging experts from 256 indigenous institutions contributing over 4000 research ideas: a CHNRI exercise by ICMR and INCLEN

Background

Health research in low– and middle– income countries (LMICs) is often driven by donor priorities rather than by the needs of the countries where the research takes place. This lack of alignment of donor’s priorities with local research need may be one of the reasons why countries fail to achieve set goals for population health and nutrition. India has a high burden of morbidity and mortality in women, children and infants. In order to look forward toward the Sustainable Development Goals, the Indian Council of Medical Research (ICMR) and the INCLEN Trust International (INCLEN) employed the Child Health and Nutrition Research Initiative’s (CHNRI) research priority setting method for maternal, neonatal, child health and nutrition with the timeline of 2016–2025. The exercise was the largest to–date use of the CHNRI methodology, both in terms of participants and ideas generated and also expanded on the methodology.

Methods

CHNRI is a crowdsourcing–based exercise that involves using the collective intelligence of a group of stakeholders, usually researchers, to generate and score research options against a set of criteria. This paper reports on a large umbrella CHNRI that was divided into four theme–specific CHNRIs (maternal, newborn, child health and nutrition). A National Steering Group oversaw the exercise and four theme–specific Research Sub–Committees technically supported finalizing the scoring criteria and refinement of research ideas for the respective thematic areas. The exercise engaged participants from 256 institutions across India – 4003 research ideas were generated from 498 experts which were consolidated into 373 research options (maternal health: 122; newborn health: 56; child health: 101; nutrition: 94); 893 experts scored these against five criteria (answerability, relevance, equity, innovation and out–of–box thinking, investment on research). Relative weights to the criteria were assigned by 79 members from the Larger Reference Group. Given India’s diversity, priorities were identified at national and three regional levels: (i) the Empowered Action Group (EAG) and North–Eastern States; (ii) States and Union territories in Northern India (including West Bengal); and (iii) States and Union territories in Southern and Western parts of India.

Conclusions

The exercise leveraged the inherent flexibility of the CHNRI method in multiple ways. It expanded on the CHNRI methodology enabling analyses for identification of research priorities at national and regional levels. However, prioritization of research options are only valuable if they are put to use, and we hope that donors will take advantage of this prioritized list of research options.