Crowdsourcing Diagnosis for Patients With Undiagnosed Illnesses: An Evaluation of CrowdMed

Rising to the Challenge of Rare Diagnoses

Patients with rare conditions often experience substantial delays between presentation and diagnosis, and some remain undiagnosed. In this Perspective, we outline the many challenges in diagnosing rare conditions in the modern clinical context. We review relevant concepts of diagnostic reasoning as they relate to rare conditions. We present solutions currently available for clinicians to mitigate some of these problems, including facilitating deliberate reflection, utilizing a diagnostic management team, and optimizing diagnostic calibration. Finally, we speculate how technology, such as chatbots and decision support tools enhanced by artificial intelligence, may augment a clinician's ability to diagnose rare conditions in a timely and accurate manner without excessive resource use.

Experimental evidence for structured information-sharing networks reducing medical errors

Errors in clinical decision-making are disturbingly common. Recent studies have found that 10 to 15% of all clinical decisions regarding diagnoses and treatment are inaccurate. Here, we experimentally study the ability of structured information-sharing networks among clinicians to improve clinicians' diagnostic accuracy and treatment decisions. We use a pool of 2,941 practicing clinicians recruited from around the United States to conduct 84 independent group-level trials, ranging across seven different clinical vignettes for topics known to exhibit high rates of diagnostic or treatment error (e.g., acute cardiac events, geriatric care, low back pain, and diabetes-related cardiovascular illness prevention). We compare collective performance in structured information-sharing networks to collective performance in independent control groups, and find that networks significantly reduce clinical errors, and improve treatment recommendations, as compared to control groups of independent clinicians engaged in isolated reflection. Our results show that these improvements are not a result of simple regression to the group mean. Instead, we find that within structured information-sharing networks, the worst clinicians improved significantly while the best clinicians did not decrease in quality. These findings offer implications for the use of social network technologies to reduce errors among clinicians.

BargCrEx: A System for Bargaining Based Aggregation of Crowd and Expert Opinions in Crowdsourcing

Crowdsourcing and crowd voting systems are being increasingly used in societal, industry, and academic problems (labeling, recommendations, social choice, etc.) due to their possibility to exploit "wisdom of crowd" and obtain good quality solutions, and/or voter satisfaction, with high cost-efficiency. However, the decisions based on crowd vote aggregation do not guarantee high-quality results due to crowd voter data quality. Additionally, such decisions often do not satisfy the majority of voters due to data heterogeneity (multimodal or uniform vote distributions) and/or outliers, which cause traditional aggregation procedures (e.g., central tendency measures) to propose decisions with low voter satisfaction. In this research, we propose a system for the integration of crowd and expert knowledge in a crowdsourcing setting with limited resources. The system addresses the problem of sparse voting data by using machine learning models (matrix factorization and regression) for the estimation of crowd and expert votes/grades. The problem of vote aggregation under multimodal or uniform vote distributions is addressed by the inclusion of expert votes and aggregation of crowd and expert votes based on optimization and bargaining models (Kalai-Smorodinsky and Nash) usually used in game theory. Experimental evaluation on real world and artificial problems showed that the bargaining-based aggregation outperforms the traditional methods in terms of cumulative satisfaction of experts and crowd. Additionally, the machine learning models showed satisfactory predictive performance and enabled cost reduction in the process of vote collection.

[Digital diagnostic support in rheumatology]

Only the correct diagnosis enables an effective treatment of rheumatic diseases. Digitalization has already significantly accelerated and simplified our everyday life. An increasing number of digital options are available to patients and medical personnel in rheumatology to accelerate and improve the diagnosis. This work gives an overview of current developments and tools for patients and rheumatologists, regarding digital diagnostic support in rheumatology.

Rare disease patient matchmaking: development and outcomes of an internet case-finding strategy in the Undiagnosed Diseases Network

BACKGROUND

Although clinician, researcher, and patient resources for matchmaking exist, finding similar patients remains an obstacle for rare disease diagnosis. The goals of this study were to develop and test the effectiveness of an Internet case-finding strategy and identify factors associated with increased matching within a rare disease population.

METHODS

Public web pages were created for consented participants. Matches made, time to each inquiry and match, and outcomes were recorded and analyzed using descriptive statistics. A Poisson regression model was run to identify characteristics associated with matches.

RESULTS

385 participants were referred to the project and 158 had pages posted. 579 inquiries were received; 89.0% were from the general public and 24.7% resulted in a match. 81.6% of pages received at least one inquiry and 15.0% had at least one patient match. Primary symptom category of neurology, diagnosis, gene page, and photo were associated with increased matches (p ≤ 0.05).

CONCLUSIONS

This Internet case-finding strategy was of interest to patients, families, and clinicians, and similar patients were identified using this approach. Extending matchmaking efforts to the general public resulted in matches and suggests including this population in matchmaking activities can improve identification of similar patients.

Unity Is Intelligence: A Collective Intelligence Experiment on ECG Reading to Improve Diagnostic Performance in Cardiology

Medical errors have a huge impact on clinical practice in terms of economic and human costs. As a result, technology-based solutions, such as those grounded in artificial intelligence (AI) or collective intelligence (CI), have attracted increasing interest as a means of reducing error rates and their impacts. Previous studies have shown that a combination of individual opinions based on rules, weighting mechanisms, or other CI solutions could improve diagnostic accuracy with respect to individual doctors. We conducted a study to investigate the potential of this approach in cardiology and, more precisely, in electrocardiogram (ECG) reading. To achieve this aim, we designed and conducted an experiment involving medical students, recent graduates, and residents, who were asked to annotate a collection of 10 ECGs of various complexity and difficulty. For each ECG, we considered groups of increasing size (from three to 30 members) and applied three different CI protocols. In all cases, the results showed a statistically significant improvement (ranging from 9% to 88%) in terms of diagnostic accuracy when compared to the performance of individual readers; this difference held for not only large groups, but also smaller ones. In light of these results, we conclude that CI approaches can support the tasks mentioned above, and possibly other similar ones as well. We discuss the implications of applying CI solutions to clinical settings, such as cases of augmented ‘second opinions’ and decision-making.

Development of a Social Network for People Without a Diagnosis (RarePairs): Evaluation Study

Background

Diagnostic delay in rare disease (RD) is common, occasionally lasting up to more than 20 years. In attempting to reduce it, diagnostic support tools have been studied extensively. However, social platforms have not yet been used for systematic diagnostic support. This paper illustrates the development and prototypic application of a social network using scientifically developed questions to match individuals without a diagnosis.

Objective

The study aimed to outline, create, and evaluate a prototype tool (a social network platform named RarePairs), helping patients with undiagnosed RDs to find individuals with similar symptoms. The prototype includes a matching algorithm, bringing together individuals with similar disease burden in the lead-up to diagnosis.

Methods

We divided our project into 4 phases. In phase 1, we used known data and findings in the literature to understand and specify the context of use. In phase 2, we specified the user requirements. In phase 3, we designed a prototype based on the results of phases 1 and 2, as well as incorporating a state-of-the-art questionnaire with 53 items for recognizing an RD. Lastly, we evaluated this prototype with a data set of 973 questionnaires from individuals suffering from different RDs using 24 distance calculating methods.

Results

Based on a step-by-step construction process, the digital patient platform prototype, RarePairs, was developed. In order to match individuals with similar experiences, it uses answer patterns generated by a specifically designed questionnaire (Q53). A total of 973 questionnaires answered by patients with RDs were used to construct and test an artificial intelligence (AI) algorithm like the k-nearest neighbor search. With this, we found matches for every single one of the 973 records. The cross-validation of those matches showed that the algorithm outperforms random matching significantly. Statistically, for every data set the algorithm found at least one other record (match) with the same diagnosis.

Conclusions

Diagnostic delay is torturous for patients without a diagnosis. Shortening the delay is important for both doctors and patients. Diagnostic support using AI can be promoted differently. The prototype of the social media platform RarePairs might be a low-threshold patient platform, and proved suitable to match and connect different individuals with comparable symptoms. This exchange promoted through RarePairs might be used to speed up the diagnostic process. Further studies include its evaluation in a prospective setting and implementation of RarePairs as a mobile phone app.

Examining Peer-to-Peer and Patient-Provider Interactions on a Social Media Community Facilitating Ask the Doctor Services

Ask the Doctor (AtD) services provide patients the opportunity to seek medical advice using online platforms. While these services represent a new mode of healthcare delivery, study of these online health communities and how they are used is limited. In particular, it is unknown if these platforms replicate existing barriers and biases in traditional healthcare delivery across demographic groups. We present an analysis of AskDocs, a subreddit that functions as a public AtD platform on social media. We examine the demographics of users, the health topics discussed, if biases present in offline healthcare settings exist on this platform, and how empathy is expressed in interactions between users and physicians. Our findings suggest a number of implications to enhance and support peer-to-peer and patient-provider interactions on online platforms.

Phenotate: crowdsourcing phenotype annotations as exercises in undergraduate classes

PURPOSE

Computational documentation of genetic disorders is highly reliant on structured data for differential diagnosis, pathogenic variant identification, and patient matchmaking. However, most information on rare diseases (RDs) exists in freeform text, such as academic literature. To increase availability of structured RD data, we developed a crowdsourcing approach for collecting phenotype information using student assignments.

METHODS

We developed Phenotate, a web application for crowdsourcing disease phenotype annotations through assignments for undergraduate genetics students. Using student-collected data, we generated composite annotations for each disease through a machine learning approach. These annotations were compared with those from clinical practitioners and gold standard curated data.

RESULTS

Deploying Phenotate in five undergraduate genetics courses, we collected annotations for 22 diseases. Student-sourced annotations showed strong similarity to gold standards, with F-measures ranging from 0.584 to 0.868. Furthermore, clinicians used Phenotate annotations to identify diseases with comparable accuracy to other annotation sources and gold standards. For six disorders, no gold standards were available, allowing us to create some of the first structured annotations for them, while students demonstrated ability to research RDs.

CONCLUSION

Phenotate enables crowdsourcing RD phenotypic annotations through educational assignments. Presented as an intuitive web-based tool, it offers pedagogical benefits and augments the computable RD knowledgebase.

Sharing Is Caring-Data Sharing Initiatives in Healthcare

In recent years, more and more health data are being generated. These data come not only from professional health systems, but also from wearable devices. All these 'big data' put together can be utilized to optimize treatments for each unique patient ('precision medicine'). For this to be possible, it is necessary that hospitals, academia and industry work together to bridge the 'valley of death' of translational medicine. However, hospitals and academia often are reluctant to share their data with other parties, even though the patient is actually the owner of his/her own health data. Academic hospitals usually invest a lot of time in setting up clinical trials and collecting data, and want to be the first ones to publish papers on this data. There are some publicly available datasets, but these are usually only shared after study (and publication) completion, which means a severe delay of months or even years before others can analyse the data. One solution is to incentivize the hospitals to share their data with (other) academic institutes and the industry. Here, we show an analysis of the current literature around data sharing, and we discuss five aspects of data sharing in the medical domain: publisher requirements, data ownership, growing support for data sharing, data sharing initiatives and how the use of federated data might be a solution. We also discuss some potential future developments around data sharing, such as medical crowdsourcing and data generalists.

Differential Diagnosis Assessment in Ambulatory Care With an Automated Medical History-Taking Device: Pilot Randomized Controlled Trial

Background

Automated medical history–taking devices (AMHTDs) are emerging tools with the potential to increase the quality of medical consultations by providing physicians with an exhaustive, high-quality, standardized anamnesis and differential diagnosis.

Objective

This study aimed to assess the effectiveness of an AMHTD to obtain an accurate differential diagnosis in an outpatient service.

Methods

We conducted a pilot randomized controlled trial involving 59 patients presenting to an emergency outpatient unit and suffering from various conditions affecting the limbs, the back, and the chest wall. Resident physicians were randomized into 2 groups, one assisted by the AMHTD and one without access to the device. For each patient, physicians were asked to establish an exhaustive differential diagnosis based on the anamnesis and clinical examination. In the intervention group, residents read the AMHTD report before performing the anamnesis. In both the groups, a senior physician had to establish a differential diagnosis, considered as the gold standard, independent of the resident’s opinion and AMHTD report.

Results

A total of 29 patients were included in the intervention group and 30 in the control group. Differential diagnosis accuracy was higher in the intervention group (mean 75%, SD 26%) than in the control group (mean 59%, SD 31%; P=.01). Subgroup analysis showed a between-group difference of 3% (83% [17/21]-80% [14/17]) for low complexity cases (1-2 differential diagnoses possible) in favor of the AMHTD (P=.76), 31% (87% [13/15]-56% [18/33]) for intermediate complexity (3 differential diagnoses; P=.02), and 24% (63% [34/54]-39% [14/35]) for high complexity (4-5 differential diagnoses; P=.08). Physicians in the intervention group (mean 4.3, SD 2) had more years of clinical practice compared with the control group (mean 5.5, SD 2; P=.03). Differential diagnosis accuracy was negatively correlated to case complexity (r=0.41; P=.001) and the residents’ years of practice (r=0.04; P=.72). The AMHTD was able to determine 73% (SD 30%) of correct differential diagnoses. Patient satisfaction was good (4.3/5), and 26 of 29 patients (90%) considered that they were able to accurately describe their symptomatology. In 8 of 29 cases (28%), residents considered that the AMHTD helped to establish the differential diagnosis.

Conclusions

The AMHTD allowed physicians to make more accurate differential diagnoses, particularly in complex cases. This could be explained not only by the ability of the AMHTD to make the right diagnoses, but also by the exhaustive anamnesis provided.

[Computer-assisted diagnosis of rare diseases]

To establish a comprehensive diagnosis is by far the most challenging task in a physician's daily routine. Especially rare diseases place high demands on differential diagnosis, caused by the high number of around 8000 diseases and their clinical variability. No clinician can be aware of all the different entities and memorizing them all is impossible and inefficient. Specific diagnostic decision-supported systems provide better results than standard search engines in this context. The systems FindZebra, Phenomizer, Orphanet, and Isabel are presented here concisely with their advantages and limitations. An outlook is given to social media usage and big data technologies. Due to the high number of initial misdiagnoses and long periods of time until a confirmatory diagnosis is reached, these tools might be promising in practice to improve the diagnosis of rare diseases.

Payment Innovations To Improve Diagnostic Accuracy And Reduce Diagnostic Error

Diagnostic accuracy is essential for treatment decisions but is largely unaccounted for by payers, including in fee-for-service Medicare and proposed Alternative Payment Models (APMs). We discuss three payment-related approaches to reducing diagnostic error. First, coding changes in the Medicare Physician Fee Schedule could facilitate the more effective use of teamwork and information technology in the diagnostic process and better support the cognitive work and time commitment that physicians make in the quest for diagnostic accuracy, especially in difficult or uncertain cases. Second, new APMs could be developed to focus on improving diagnostic accuracy in challenging cases and make available support resources for diagnosis, including condition-specific centers of diagnostic expertise or general diagnostic centers of excellence that provide second (or even third) opinions. Performing quality improvement activities that promote safer diagnosis should be a part of the accountability of APM recipients. Third, the accuracy of diagnoses that trigger APM payments and establish payment amounts should be confirmed by APM recipients. Implementation of these multipronged approaches can make current payment models more accountable for addressing diagnostic error and position diagnostic performance as a critical component of quality-based payment.

Validity of Online Screening for Autism: Crowdsourcing Study Comparing Paid and Unpaid Diagnostic Tasks

BACKGROUND

Obtaining a diagnosis of neuropsychiatric disorders such as autism requires long waiting times that can exceed a year and can be prohibitively expensive. Crowdsourcing approaches may provide a scalable alternative that can accelerate general access to care and permit underserved populations to obtain an accurate diagnosis.

OBJECTIVE

We aimed to perform a series of studies to explore whether paid crowd workers on Amazon Mechanical Turk (AMT) and citizen crowd workers on a public website shared on social media can provide accurate online detection of autism, conducted via crowdsourced ratings of short home video clips.

METHODS

Three online studies were performed: (1) a paid crowdsourcing task on AMT (N=54) where crowd workers were asked to classify 10 short video clips of children as "Autism" or "Not autism," (2) a more complex paid crowdsourcing task (N=27) with only those raters who correctly rated ≥8 of the 10 videos during the first study, and (3) a public unpaid study (N=115) identical to the first study.

RESULTS

For Study 1, the mean score of the participants who completed all questions was 7.50/10 (SD 1.46). When only analyzing the workers who scored ≥8/10 (n=27/54), there was a weak negative correlation between the time spent rating the videos and the sensitivity (ρ=-0.44, P=.02). For Study 2, the mean score of the participants rating new videos was 6.76/10 (SD 0.59). The average deviation between the crowdsourced answers and gold standard ratings provided by two expert clinical research coordinators was 0.56, with an SD of 0.51 (maximum possible SD is 3). All paid crowd workers who scored 8/10 in Study 1 either expressed enjoyment in performing the task in Study 2 or provided no negative comments. For Study 3, the mean score of the participants who completed all questions was 6.67/10 (SD 1.61). There were weak correlations between age and score (r=0.22, P=.014), age and sensitivity (r=-0.19, P=.04), number of family members with autism and sensitivity (r=-0.195, P=.04), and number of family members with autism and precision (r=-0.203, P=.03). A two-tailed t test between the scores of the paid workers in Study 1 and the unpaid workers in Study 3 showed a significant difference (P<.001).

CONCLUSIONS

Many paid crowd workers on AMT enjoyed answering screening questions from videos, suggesting higher intrinsic motivation to make quality assessments. Paid crowdsourcing provides promising screening assessments of pediatric autism with an average deviation <20% from professional gold standard raters, which is potentially a clinically informative estimate for parents. Parents of children with autism likely overfit their intuition to their own affected child. This work provides preliminary demographic data on raters who may have higher ability to recognize and measure features of autism across its wide range of phenotypic manifestations.

A systematic review of natural language processing and text mining of symptoms from electronic patient-authored text data

OBJECTIVE

In this systematic review, we aim to synthesize the literature on the use of natural language processing (NLP) and text mining as they apply to symptom extraction and processing in electronic patient-authored text (ePAT).

MATERIALS AND METHODS

A comprehensive literature search of 1964 articles from PubMed and EMBASE was narrowed to 21 eligible articles. Data related to purpose, text source, number of users and/or posts, evaluation metrics, and quality indicators were recorded.

RESULTS

Pain (n = 18) and fatigue and sleep disturbance (n = 18) were the most frequently evaluated symptom clinical content categories. Studies accessed ePAT from sources such as Twitter and online community forums or patient portals focused on diseases, including diabetes, cancer, and depression. Fifteen studies used NLP as a primary methodology. Studies reported evaluation metrics including the precision, recall, and F-measure for symptom-specific research questions.

DISCUSSION

NLP and text mining have been used to extract and analyze patient-authored symptom data in a wide variety of online communities. Though there are computational challenges with accessing ePAT, the depth of information provided directly from patients offers new horizons for precision medicine, characterization of sub-clinical symptoms, and the creation of personal health libraries as outlined by the National Library of Medicine.

CONCLUSION

Future research should consider the needs of patients expressed through ePAT and its relevance to symptom science. Understanding the role that ePAT plays in health communication and real-time assessment of symptoms, through the use of NLP and text mining, is critical to a patient-centered health system.

Beyond Dr. Google: the evidence on consumer-facing digital tools for diagnosis

Over a third of adults go online to diagnose their health condition. Direct-to-consumer (DTC), interactive, diagnostic apps with information personalization capabilities beyond those of static search engines are rapidly proliferating. While these apps promise faster, more convenient and more accurate information to improve diagnosis, little is known about the state of the evidence on their performance or the methods used to evaluate them. We conducted a scoping review of the peer-reviewed and gray literature for the period January 1, 2014–June 30, 2017. We found that the largest category of evaluations involved symptom checkers that applied algorithms to user-answered questions, followed by sensor-driven apps that applied algorithms to smartphone photos, with a handful of evaluations examining crowdsourcing. The most common clinical areas evaluated were dermatology and general diagnostic and triage advice for a range of conditions. Evaluations were highly variable in methodology and conclusions, with about half describing app characteristics and half examining actual performance. Apps were found to vary widely in functionality, accuracy, safety and effectiveness, although the usefulness of this evidence was limited by a frequent failure to provide results by named individual app. Overall, the current evidence base on DTC, interactive diagnostic apps is sparse in scope, uneven in the information provided and inconclusive with respect to safety and effectiveness, with no studies of clinical risks and benefits involving real-world consumer use. Given that DTC diagnostic apps are rapidly evolving, rigorous and standardized evaluations are essential to inform decisions by clinicians, patients, policymakers and other stakeholders.

From second to hundredth opinion in medicine: A global consultation platform for physicians

Serious medical diagnostic errors lead to adverse patient outcomes and increased healthcare costs. The use of virtual online consultation platforms may lead to better-informed physicians and reduce the incidence of diagnostic errors. Our aim was to assess the usage characteristics of an online, physician-to-physician, no-cost, medical consultation platform, Medscape Consult, from November 2015 through October 2017. Physicians creating original content were noted as “presenters” and those following up as “responders”. During the study period, 37,706 physician users generated a combined 117,346 presentations and responses. The physicians had an average age of 56 years and were from 171 countries on every continent. Over 90% of all presentations received responses with the median time to first response of 1.5 h. Overall, computer- and device-based medical consultation has the capacity to rapidly reach a global medical community and may play a role in the reduction of diagnostic errors.

Legal and ethical issues surrounding the use of crowdsourcing among healthcare providers

As the pace of medical discovery widens the knowledge-to-practice gap, technologies that enable peer-to-peer crowdsourcing have become increasingly common. Crowdsourcing has the potential to help medical providers collaborate to solve patient-specific problems in real time. We recently conducted the first trial of a mobile, medical crowdsourcing application among healthcare providers in a university hospital setting. In addition to acknowledging the benefits, our participants also raised concerns regarding the potential negative consequences of this emerging technology. In this commentary, we consider the legal and ethical implications of the major findings identified in our previous trial including compliance with the Health Insurance Portability and Accountability Act, patient protections, healthcare provider liability, data collection, data retention, distracted doctoring, and multi-directional anonymous posting. We believe the commentary and recommendations raised here will provide a frame of reference for individual providers, provider groups, and institutions to explore the salient legal and ethical issues before they implement these systems into their workflow.

Balancing Between Privacy and Patient Needs for Health Information in the Age of Participatory Health and Social Media: A Scoping Review

OBJECTIVES

 With the increased use of participatory health enabling technologies, such as social media, balancing the need for health information with patient privacy and confidentiality has become a more complex and immediate concern. The purpose of this paper produced by the members of the IMIA Participatory Health and Social Media (PHSM) working group is to investigate patient needs for health information using participatory health enabling technologies, while balancing their needs for privacy and confidentiality.

METHODS

 Six domain areas including media sharing platforms, patient portals, web-based platforms, crowdsourcing websites, medical avatars, and other mobile health technologies were identified by five members of the IMIA PHSM working group as relevant to participatory health and the balance between data sharing and patient needs for privacy and confidentiality. After identifying the relevant domain areas, our scoping review began by searching several databases such as PubMed, MEDLINE, Scopus, and Google Scholar using a variety of key search terms.

RESULTS

 A total of 1,973 studies were identified, of which 68 studies met our inclusion criteria and were included in the analysis. Results showed that challenges for balancing patient needs for information and privacy and confidentiality concerns included: cross-cultural understanding, clinician and patient awareness, de-identification of data, and commercialization of patient data. Some opportunities identified were patient empowerment, connecting participatory health enabling technologies with clinical records, open data sharing agreement, and e-consent.

CONCLUSION

 Balancing between privacy and patient needs for health information in the age of participatory health and social media offers several opportunities and challenges. More people are engaging in actively managing health through participatory health enabling technologies. Such activity often includes sharing health information and with this comes a perennial tension between balancing individual needs and the desire to uphold privacy and confidentiality. We recommend that guidelines for both patients and clinicians, in terms of their use of participatory health-enabling technologies, are developed to ensure that patient privacy and confidentiality are protected, and a maximum benefit can be realized.

Mapping of Crowdsourcing in Health: Systematic Review

Background

Crowdsourcing involves obtaining ideas, needed services, or content by soliciting Web-based contributions from a crowd. The 4 types of crowdsourced tasks (problem solving, data processing, surveillance or monitoring, and surveying) can be applied in the 3 categories of health (promotion, research, and care).

Objective

This study aimed to map the different applications of crowdsourcing in health to assess the fields of health that are using crowdsourcing and the crowdsourced tasks used. We also describe the logistics of crowdsourcing and the characteristics of crowd workers.

Methods

MEDLINE, EMBASE, and ClinicalTrials.gov were searched for available reports from inception to March 30, 2016, with no restriction on language or publication status.

Results

We identified 202 relevant studies that used crowdsourcing, including 9 randomized controlled trials, of which only one had posted results at ClinicalTrials.gov. Crowdsourcing was used in health promotion (91/202, 45.0%), research (73/202, 36.1%), and care (38/202, 18.8%). The 4 most frequent areas of application were public health (67/202, 33.2%), psychiatry (32/202, 15.8%), surgery (22/202, 10.9%), and oncology (14/202, 6.9%). Half of the reports (99/202, 49.0%) referred to data processing, 34.6% (70/202) referred to surveying, 10.4% (21/202) referred to surveillance or monitoring, and 5.9% (12/202) referred to problem-solving. Labor market platforms (eg, Amazon Mechanical Turk) were used in most studies (190/202, 94%). The crowd workers’ characteristics were poorly reported, and crowdsourcing logistics were missing from two-thirds of the reports. When reported, the median size of the crowd was 424 (first and third quartiles: 167-802); crowd workers’ median age was 34 years (32-36). Crowd workers were mainly recruited nationally, particularly in the United States. For many studies (58.9%, 119/202), previous experience in crowdsourcing was required, and passing a qualification test or training was seldom needed (11.9% of studies; 24/202). For half of the studies, monetary incentives were mentioned, with mainly less than US $1 to perform the task. The time needed to perform the task was mostly less than 10 min (58.9% of studies; 119/202). Data quality validation was used in 54/202 studies (26.7%), mainly by attention check questions or by replicating the task with several crowd workers.

Conclusions

The use of crowdsourcing, which allows access to a large pool of participants as well as saving time in data collection, lowering costs, and speeding up innovations, is increasing in health promotion, research, and care. However, the description of crowdsourcing logistics and crowd workers’ characteristics is frequently missing in study reports and needs to be precisely reported to better interpret the study findings and replicate them.

Understanding Patient Questions about their Medical Records in an Online Health Forum: Opportunity for Patient Portal Design

There are many benefits of online patient access to their medical records through technologies such as patient portals. However, patients often have difficulties understanding the clinical data presented in portals. In response, increasingly, patients go online to make sense of this data. One commonly used online resource is health forums. In this pilot study, we focus on one type of clinical data, laboratory results, and one popular forum, MedHelp. We examined patient question posts that contain laboratory results to gain insights into the nature of these questions and of the answers. Our analyses revealed a typology of confusion (i.e., topics of their questions) and potential gaps in traditional healthcare supports (i.e., patients' requests and situational factors), as well as the supports patients may gain through the forum (i.e., what the community provides). These results offer preliminary evidence of opportunities to redesign patient portals, and will inform our future work.

North American Public Opinion Survey on the Acceptability of Crowdsourcing Basic Life Support for Out-of-Hospital Cardiac Arrest With the PulsePoint Mobile Phone App

Background

The PulsePoint Respond app is a novel system that can be implemented in emergency dispatch centers to crowdsource basic life support (BLS) for patients with cardiac arrest and facilitate bystander cardiopulmonary resuscitation (CPR) and automated external defibrillator use while first responders are en route.

Objective

The aim of this study was to conduct a North American survey to evaluate the public perception of the above-mentioned strategy, including acceptability and willingness to respond to alerts.

Methods

We designed a Web-based survey administered by IPSOS Reid, an established external polling vendor. Sampling was designed to ensure broad representation using recent census statistics.

Results

A total of 2415 survey responses were analyzed (1106 from Canada and 1309 from the United States). It was found that 98.37% (1088/1106) of Canadians and 96% (1259/1309) of Americans had no objections to PulsePoint being implemented in their community; 84.27% (932/1106) of Canadians and 55.61% (728/1309) of Americans said they would download the app to become a potential responder to cardiac arrest, respectively. Among Canadians, those who said they were likely to download PulsePoint were also more likely to have ever had CPR training (OR 1.7, 95% CI 1.2-2.4; P=.002); however, this was not true of American respondents (OR 1.0, 95% CI 0.79-1.3; P=.88). When asked to imagine themselves as a cardiac arrest victim, 95.39% (1055/1106) of Canadians and 92.44% (1210/1309) of Americans had no objections to receiving crowdsourced help in a public setting; 88.79% (982/1106) of Canadians and 84.87% (1111/1309) of Americans also had no objections to receiving help in a private setting, respectively. The most common concern identified with respect to PulsePoint implementation was a responder’s lack of ability, training, or access to proper equipment in a public setting.

Conclusions

The North American public finds the concept of crowdsourcing BLS for out-of-hospital cardiac arrest to be acceptable. It demonstrates willingness to respond to PulsePoint CPR notifications and to accept help from others alerted by the app if they themselves suffered a cardiac arrest.

Crowdsourcing Precision Cerebrovascular Health: Imaging and Cloud Seeding A Million Brains Initiative™

Crowdsourcing, an unorthodox approach in medicine, creates an unusual paradigm to study precision cerebrovascular health, eliminating the relative isolation and non-standardized nature of current imaging data infrastructure, while shifting emphasis to the astounding capacity of big data in the cloud. This perspective envisions the use of imaging data of the brain and vessels to orient and seed A Million Brains Initiative™ that may leapfrog incremental advances in stroke and rapidly provide useful data to the sizable population around the globe prone to the devastating effects of stroke and vascular substrates of dementia. Despite such variability in the type of data available and other limitations, the data hierarchy logically starts with imaging and can be enriched with almost endless types and amounts of other clinical and biological data. Crowdsourcing allows an individual to contribute to aggregated data on a population, while preserving their right to specific information about their own brain health. The cloud now offers endless storage, computing prowess, and neuroimaging applications for postprocessing that is searchable and scalable. Collective expertise is a windfall of the crowd in the cloud and particularly valuable in an area such as cerebrovascular health. The rise of precision medicine, rapidly evolving technological capabilities of cloud computing and the global imperative to limit the public health impact of cerebrovascular disease converge in the imaging of A Million Brains Initiative™. Crowdsourcing secure data on brain health may provide ultimate generalizability, enable focused analyses, facilitate clinical practice, and accelerate research efforts.

The Impact of an Online Crowdsourcing Diagnostic Tool on Health Care Utilization: A Case Study Using a Novel Approach to Retrospective Claims Analysis

Background

Patients with difficult medical cases often remain undiagnosed despite visiting multiple physicians. A new online platform, CrowdMed, uses crowdsourcing to quickly and efficiently reach an accurate diagnosis for these patients.

Objective

This study sought to evaluate whether CrowdMed decreased health care utilization for patients who have used the service.

Methods

Novel, electronic methods of patient recruitment and data collection were utilized. Patients who completed cases on CrowdMed’s platform between July 2014 and April 2015 were recruited for the study via email and screened via an online survey. After providing eConsent, participants provided identifying information used to access their medical claims data, which was retrieved through a third-party web application program interface (API). Utilization metrics including frequency of provider visits and medical charges were compared pre- and post-case resolution to assess the impact of resolving a case on CrowdMed.

Results

Of 45 CrowdMed users who completed the study survey, comprehensive claims data was available via API for 13 participants, who made up the final enrolled sample. There were a total of 221 health care provider visits collected for the study participants, with service dates ranging from September 2013 to July 2015. Frequency of provider visits was significantly lower after resolution of a case on CrowdMed (mean of 1.07 visits per month pre-resolution vs. 0.65 visits per month post-resolution, P=.01). Medical charges were also significantly lower after case resolution (mean of US $719.70 per month pre-resolution vs. US $516.79 per month post-resolution, P=.03). There was no significant relationship between study results and disease onset date, and there was no evidence of regression to the mean influencing results.

Conclusions

This study employed technology-enabled methods to demonstrate that patients who used CrowdMed had lower health care utilization after case resolution. However, since the final sample size was limited, results should be interpreted as a case study. Despite this limitation, the statistically significant results suggest that online crowdsourcing shows promise as an efficient method of solving difficult medical cases.