Distributed medical image analysis and diagnosis through crowd-sourced games: a malaria case study

Advances in Portable Optical Microscopy Using Cloud Technologies and Artificial Intelligence for Medical Applications

The need for faster and more accessible alternatives to laboratory microscopy is driving many innovations throughout the image and data acquisition chain in the biomedical field. Benchtop microscopes are bulky, lack communications capabilities, and require trained personnel for analysis. New technologies, such as compact 3D-printed devices integrated with the Internet of Things (IoT) for data sharing and cloud computing, as well as automated image processing using deep learning algorithms, can address these limitations and enhance the conventional imaging workflow. This review reports on recent advancements in microscope miniaturization, with a focus on emerging technologies such as photoacoustic microscopy and more established approaches like smartphone-based microscopy. The potential applications of IoT in microscopy are examined in detail. Furthermore, this review discusses the evolution of image processing in microscopy, transitioning from traditional to deep learning methods that facilitate image enhancement and data interpretation. Despite numerous advancements in the field, there is a noticeable lack of studies that holistically address the entire microscopy acquisition chain. This review aims to highlight the potential of IoT and artificial intelligence (AI) in combination with portable microscopy, emphasizing the importance of a comprehensive approach to the microscopy acquisition chain, from portability to image analysis.

Crowdsourcing image segmentation for deep learning: integrated platform for citizen science, paid microtask, and gamification

OBJECTIVES

Segmentation is crucial in medical imaging. Deep learning based on convolutional neural networks showed promising results. However, the absence of large-scale datasets and a high degree of inter- and intra-observer variations pose a bottleneck. Crowdsourcing might be an alternative, as many non-experts provide references. We aim to compare different types of crowdsourcing for medical image segmentation.

METHODS

We develop a crowdsourcing platform that integrates citizen science (incentive: participating in the research), paid microtask (incentive: financial reward), and gamification (incentive: entertainment). For evaluation, we choose the use case of sclera segmentation in fundus images as a proof-of-concept and analyze the accuracy of crowdsourced masks and the generalization of learning models trained with crowdsourced masks.

RESULTS

The developed platform is suited for the different types of crowdsourcing and offers an easy and intuitive way to implement crowdsourcing studies. Regarding the proof-of-concept study, citizen science, paid microtask, and gamification yield a median F-score of 82.2, 69.4, and 69.3 % compared to expert-labeled ground truth, respectively. Generating consensus masks improves the gamification masks (78.3 %). Despite the small training data (50 images), deep learning reaches median F-scores of 80.0, 73.5, and 76.5 % for citizen science, paid microtask, and gamification, respectively, indicating sufficient generalizability.

CONCLUSIONS

As the platform has proven useful, we aim to make it available as open-source software for other researchers.

A Perspective on Crowdsourcing and Human-in-the-Loop Workflows in Precision Health

Modern machine learning approaches have led to performant diagnostic models for a variety of health conditions. Several machine learning approaches, such as decision trees and deep neural networks, can, in principle, approximate any function. However, this power can be considered to be both a gift and a curse, as the propensity toward overfitting is magnified when the input data are heterogeneous and high dimensional and the output class is highly nonlinear. This issue can especially plague diagnostic systems that predict behavioral and psychiatric conditions that are diagnosed with subjective criteria. An emerging solution to this issue is crowdsourcing, where crowd workers are paid to annotate complex behavioral features in return for monetary compensation or a gamified experience. These labels can then be used to derive a diagnosis, either directly or by using the labels as inputs to a diagnostic machine learning model. This viewpoint describes existing work in this emerging field and discusses ongoing challenges and opportunities with crowd-powered diagnostic systems, a nascent field of study. With the correct considerations, the addition of crowdsourcing to human-in-the-loop machine learning workflows for the prediction of complex and nuanced health conditions can accelerate screening, diagnostics, and ultimately access to care.

Crowdsourced human-based computational approach for tagging peripheral blood smear sample images from Sickle Cell Disease patients using non-expert users

In this paper, we present a human-based computation approach for the analysis of peripheral blood smear (PBS) images images in patients with Sickle Cell Disease (SCD). We used the Mechanical Turk microtask market to crowdsource the labeling of PBS images. We then use the expert-tagged erythrocytesIDB dataset to assess the accuracy and reliability of our proposal. Our results showed that when a robust consensus is achieved among the Mechanical Turk workers, probability of error is very low, based on comparison with expert analysis. This suggests that our proposed approach can be used to annotate datasets of PBS images, which can then be used to train automated methods for the diagnosis of SCD. In future work, we plan to explore the potential integration of our findings with outcomes obtained through automated methodologies. This could lead to the development of more accurate and reliable methods for the diagnosis of SCD.

Labelling instructions matter in biomedical image analysis

Biomedical image analysis algorithm validation depends on high-quality annotation of reference datasets, for which labelling instructions are key. Despite their importance, their optimization remains largely unexplored. Here we present a systematic study of labelling instructions and their impact on annotation quality in the field. Through comprehensive examination of professional practice and international competitions registered at the Medical Image Computing and Computer Assisted Intervention Society, the largest international society in the biomedical imaging field, we uncovered a discrepancy between annotators’ needs for labelling instructions and their current quality and availability. On the basis of an analysis of 14,040 images annotated by 156 annotators from four professional annotation companies and 708 Amazon Mechanical Turk crowdworkers using instructions with different information density levels, we further found that including exemplary images substantially boosts annotation performance compared with text-only descriptions, while solely extending text descriptions does not. Finally, professional annotators constantly outperform Amazon Mechanical Turk crowdworkers. Our study raises awareness for the need of quality standards in biomedical image analysis labelling instructions.

Artificial Intelligence-Driven Serious Games in Health Care: Scoping Review

BACKGROUND

Artificial intelligence (AI)-driven serious games have been used in health care to offer a customizable and immersive experience. Summarizing the features of the current AI-driven serious games is very important to explore how they have been developed and used and their current state to plan on how to leverage them in the current and future health care needs.

OBJECTIVE

This study aimed to explore the features of AI-driven serious games in health care as reported by previous research.

METHODS

We conducted a scoping review to achieve the abovementioned objective. The most popular databases in the information technology and health fields (ie, MEDLINE, PsycInfo, Embase, CINAHL, IEEE Xplore, ACM Digital Library, and Google Scholar) were searched using keywords related to serious games and AI. Two reviewers independently performed the study selection process. Three reviewers independently extracted data from the included studies. A narrative approach was used for data synthesis.

RESULTS

The search process returned 1470 records. Of these 1470 records, 46 (31.29%) met all eligibility criteria. A total of 64 different serious games were found in the included studies. Motor impairment was the most common health condition targeted by these serious games. Serious games were used for rehabilitation in most of the studies. The most common genres of serious games were role-playing games, puzzle games, and platform games. Unity was the most prominent game engine used to develop serious games. PCs were the most common platform used to play serious games. The most common algorithm used in the included studies was support vector machine. The most common purposes of AI were the detection of disease and the evaluation of user performance. The size of the data set ranged from 36 to 795,600. The most common validation techniques used in the included studies were k-fold cross-validation and training-test split validation. Accuracy was the most commonly used metric for evaluating the performance of AI models.

CONCLUSIONS

The last decade witnessed an increase in the development of AI-driven serious games for health care purposes, targeting various health conditions, and leveraging multiple AI algorithms; this rising trend is expected to continue for years to come. Although the evidence uncovered in this study shows promising applications of AI-driven serious games, larger and more rigorous, diverse, and robust studies may be needed to examine the efficacy and effectiveness of AI-driven serious games in different populations with different health conditions.

Improving Crowdsourcing-Based Image Classification Through Expanded Input Elicitation and Machine Learning

This work investigates how different forms of input elicitation obtained from crowdsourcing can be utilized to improve the quality of inferred labels for image classification tasks, where an image must be labeled as either positive or negative depending on the presence/absence of a specified object. Five types of input elicitation methods are tested: binary classification (positive or negative); the (x, y)-coordinate of the position participants believe a target object is located; level of confidence in binary response (on a scale from 0 to 100%); what participants believe the majority of the other participants' binary classification is; and participant's perceived difficulty level of the task (on a discrete scale). We design two crowdsourcing studies to test the performance of a variety of input elicitation methods and utilize data from over 300 participants. Various existing voting and machine learning (ML) methods are applied to make the best use of these inputs. In an effort to assess their performance on classification tasks of varying difficulty, a systematic synthetic image generation process is developed. Each generated image combines items from the MPEG-7 Core Experiment CE-Shape-1 Test Set into a single image using multiple parameters (e.g., density, transparency, etc.) and may or may not contain a target object. The difficulty of these images is validated by the performance of an automated image classification method. Experiment results suggest that more accurate results can be achieved with smaller training datasets when both the crowdsourced binary classification labels and the average of the self-reported confidence values in these labels are used as features for the ML classifiers. Moreover, when a relatively larger properly annotated dataset is available, in some cases augmenting these ML algorithms with the results (i.e., probability of outcome) from an automated classifier can achieve even higher performance than what can be obtained by using any one of the individual classifiers. Lastly, supplementary analysis of the collected data demonstrates that other performance metrics of interest, namely reduced false-negative rates, can be prioritized through special modifications of the proposed aggregation methods.

Artificial Intelligence-Driven Serious Games in Healthcare: A Scoping Review (Preprint).. [DOI: 10.2196/preprints.39840] [Citation(s) in RCA: 0] [Impact Index Per Article: 0]

BACKGROUND

Artificial Intelligence (AI)-driven serious games have been used in healthcare to offer a customizable and immersive experience. Summarizing the features of the current AI-driven serious games is very important to explore how they have been developed and used and their current state in order to plan on how to leverage them in the current and future healthcare needs.

OBJECTIVE

The current study aimed to explore the features of AI-driven serious games in healthcare as reported by previous research.

METHODS

We carried out a scoping review to achieve the above-mentioned objective. The most popular databases in information technology and health fields (e.g., MEDLINE and IEEE Xplore) were searched using keywords related to serious games and AI. These terms were selected based on the target intervention (i.e., AI) and the target disease (i.e., COVID-19). Two reviewers independently performed the study selection process. Three reviewers independently used Microsoft Excel to extract data from the included studies. A narrative approach was used for data synthesis.

RESULTS

The search process returned 1470 records. Of these records, 46 met all eligibility criteria. 60 different serious games were found in the included studies. Motor impairment was the most common health condition targeted by these serious games. Serious games in most of the studies were used for rehabilitation. The serious games in the majority of the included studies can be played by only single player. Most serious games were played on standalone devices (offline games). The most common genres of serious games were role-playing games, puzzle games, and platformer games. Unity was the most prominent game engine used to develop serious games. Personal computers (PCs) were the most common platforms used to play serious games. The most common algorithms used in the included studies were Support Vector Machine (SVM), Convolutional Neural Network (CNN), Artificial Neural Networks (ANN), and Random Forest (RF). The most common purposes of AI were the detection of disease and the evaluation of user's performance. The dataset size ranged from 36 to 795,600, with an average of about 52,124. The most common validation techniques used in the included studies were K-fold cross-validation and training test split validation. Accuracy was the most commonly used metric to evaluate the performance of AI models.

CONCLUSIONS

The last decade witnessed an increase in the development of AI-driven serious games for healthcare purposes and targeting various health conditions and leveraging multiple AI algorithms; this rising trend is expected to continue for years to come. While the evidence uncovered in this study shows promising applications of AI-driven serious games, larger and more rigorous, diverse, and robust studies may be needed to examine the efficacy and effectiveness of AI-driven serious games in different populations with different health conditions.

Computational Methods for Automated Analysis of Malaria Parasite Using Blood Smear Images: Recent Advances

Malaria comes under one of the dangerous diseases in many countries. It is the primary reason for most of the causalities across the world. It is presently rated as a significant cause of the high mortality rate worldwide compared with other diseases that can be reduced significantly by its earlier detection. Therefore, to facilitate the early detection/diagnosis of malaria to reduce the mortality rate, an automated computational method is required with a high accuracy rate. This study is a solid starting point for researchers who want to look into automated blood smear analysis to detect malaria. In this paper, a comprehensive review of different computer-assisted techniques has been outlined as follows: (i) acquisition of image dataset, (ii) preprocessing, (iii) segmentation of RBC, and (iv) feature extraction and selection, and (v) classification for the detection of malaria parasites using blood smear images. This study will be helpful for: (i) researchers can inspect and improve the existing computational methods for early diagnosis of malaria with a high accuracy rate that may further reduce the interobserver and intra-observer variations; (ii) microbiologists to take the second opinion from the automated computational methods for effective diagnosis of malaria; and (iii) finally, several issues remain addressed, and future work has also been discussed in this work.

Privacy-aware sharing and collaborative analysis of personal wellness data: Process model, domain ontology, software system and user trial

Personal wellness data collected using wearable devices is a valuable resource, potentially containing knowledge that goes beyond what the device and its the associated software application can tell the user. However, extracting such knowledge from the data requires expertise that an average user cannot be expected to have. To overcome this problem, the data owner could collaborate with a data analysis expert; for such a collaboration to succeed, the collaborators need to be able to find one another, communicate with one another and share datasets and analysis results with one another. In this paper we presents a process model for such collaborations, a domain ontology and software system developed to support the process, and the results of a user trial demonstrating collaborative analysis of sleep data. Unlike existing collaborative data analytics tools, the process and software have been specifically designed with the non-expert data owner in mind, enabling them to control their data and protect their privacy by selecting the data to be shared on a case-by-case basis. Theoretical analysis and empirical results suggest that the process and its implementation are valid as a proof of concept.

Artificial Intelligence-Driven Prediction Modeling and Decision Making in Spine Surgery Using Hybrid Machine Learning Models

Healthcare systems worldwide generate vast amounts of data from many different sources. Although of high complexity for a human being, it is essential to determine the patterns and minor variations in the genomic, radiological, laboratory, or clinical data that reliably differentiate phenotypes or allow high predictive accuracy in health-related tasks. Convolutional neural networks (CNN) are increasingly applied to image data for various tasks. Its use for non-imaging data becomes feasible through different modern machine learning techniques, converting non-imaging data into images before inputting them into the CNN model. Considering also that healthcare providers do not solely use one data modality for their decisions, this approach opens the door for multi-input/mixed data models which use a combination of patient information, such as genomic, radiological, and clinical data, to train a hybrid deep learning model. Thus, this reflects the main characteristic of artificial intelligence: simulating natural human behavior. The present review focuses on key advances in machine and deep learning, allowing for multi-perspective pattern recognition across the entire information set of patients in spine surgery. This is the first review of artificial intelligence focusing on hybrid models for deep learning applications in spine surgery, to the best of our knowledge. This is especially interesting as future tools are unlikely to use solely one data modality. The techniques discussed could become important in establishing a new approach to decision-making in spine surgery based on three fundamental pillars: (1) patient-specific, (2) artificial intelligence-driven, (3) integrating multimodal data. The findings reveal promising research that already took place to develop multi-input mixed-data hybrid decision-supporting models. Their implementation in spine surgery may hence be only a matter of time.

Extrinsic parameter's adjustment and potential implications in Plasmodium falciparum malaria diagnosis

Malaria is a major public health concern, affecting over 3.2 billion people in 91 countries. The advent of digital microscopy and Machine learning with the aim of automating Plasmodium falciparum diagnosis extensively depends on the extracted image features. The color of the cells, plasma, and stained artifacts influence the topological, geometrical, and statistical parameters being used to extract image features. During microscopic image acquisition, custom adjustments to the condenser and color temperature controls often have an influence on the extracted statistical features. But, our human visual system sub-consciously adjusts the color and retains the originality in a different lighting environment. Despite the use of appropriate image preprocessing, findings from the literature indicate that statistical feature variations exist, allowing the risk of P. falciparum misinterpretation. In order to eliminate this pervasive variation, the current work focuses on preprocessing the extracted statistical features rather than the prepossessing of the source image. It begins with the augmentation of series images for a microscopic field by inducing illumination variations during the microscopic image acquisition stage. A set of such image series is analyzed using a Nonlinear Regression Model to generalize the relationship between microscopic images acquired with variable ambient brightness and a specific feature. The projection point of the centroid feature onto the brightness parameter is identified in the model and it is denoted as the optimum brightness factor (OBF). Using the model, the feature correction factor (CF) is calculated from the rate of change of feature values over the interval OBF, and the brightness of the test image is processed. The present work has investigated OBF for selected image textural features, namely Contrast, Homogeneity, Entropy, Energy, and Correlation individually from its co-occurrence matrices. For performance analysis, the best state-of-the-art method uses selected texture as a subset feature to evaluate the effectiveness of P. falciparum malaria classification. Then, the impact of proposed feature processing is evaluated on 274 blood smear images with and without Feature Correction (FC). As a result, the "p" value is less than .05, which leads to the result that it is highly significant and the classification accuracy and F-score of P. falciparum malaria are increased.

A crowdsourcing semi-automatic image segmentation platform for cell biology

State-of-the-art computer-vision algorithms rely on big and accurately annotated data, which are expensive, laborious and time-consuming to generate. This task is even more challenging when it comes to microbiological images, because they require specialized expertise for accurate annotation. Previous studies show that crowdsourcing and assistive-annotation tools are two potential solutions to address this challenge. In this work, we have developed a web-based platform to enable crowdsourcing annotation of image data; the platform is powered by a semi-automated assistive tool to support non-expert annotators to improve the annotation efficiency. The behavior of annotators with and without the assistive tool is analyzed, using biological images of different complexity. More specifically, non-experts have been asked to use the platform to annotate microbiological images of gut parasites, which are compared with annotations by experts. A quantitative evaluation is carried out on the results, confirming that the assistive tools can noticeably decrease the non-expert annotation's cost (time, click, interaction, etc.) while preserving or even improving the annotation's quality. The annotation quality of non-experts has been investigated using IoU (intersection over union), precision and recall; based on this analysis we propose some ideas on how to better design similar crowdsourcing and assistive platforms.

Digital crowdsourced intervention to promote HIV testing among MSM in China: study protocol for a cluster randomized controlled trial

BACKGROUND

Men who have sex with men (MSM) are an important HIV key population in China. However, HIV testing rates among MSM remain suboptimal. Digital crowdsourced media interventions may be a useful tool to reach this marginalized population. We define digital crowdsourced media as using social media, mobile phone applications, Internet, or other digital approaches to disseminate messages developed from crowdsourcing contests. The proposed cluster randomized controlled trial (RCT) study aims to assess the effectiveness of a digital crowdsourced intervention to increase HIV testing uptake and decrease risky sexual behaviors among Chinese MSM.

METHODS

A two-arm, cluster-randomized controlled trial will be implemented in eleven cities (ten clusters) in Shandong Province, China. Targeted study participants will be 250 MSM per arm and 50 participants per cluster. MSM who are 18 years old or above, live in the study city, have not been tested for HIV in the past 3 months, are not living with HIV or have never been tested for HIV, and are willing to provide informed consent will be enrolled. Participants will be recruited through banner advertisements on Blued, the largest gay dating app in China, and in-person at community-based organizations (CBOs). The intervention includes a series of crowdsourced intervention materials (24 images and four short videos about HIV testing and safe sexual behaviors) and HIV self-test services provided by the study team. The intervention was developed through a series of participatory crowdsourcing contests before this study. The self-test kits will be sent to the participants in the intervention group at the 2nd and 3rd follow-ups. Participants will be followed up quarterly during the 12-month period. The primary outcome will be self-reported HIV testing uptake at 12 months. Secondary outcomes will include changes in condomless sex, self-test efficacy, social network engagement, HIV testing social norms, and testing stigma.

DISCUSSION

Innovative approaches to HIV testing among marginalized population are urgently needed. Through this cluster randomized controlled trial, we will evaluate the effectiveness of a digital crowdsourced intervention, improving HIV testing uptake among MSM and providing a resource in related public health fields.

TRIAL REGISTRATION

ChiCTR1900024350 . Registered on 6 July 2019.

Rapid determination of erythrocyte sedimentation rate (ESR) by an electrically driven blood droplet biosensor

In healthcare practice, the sedimentation rate of red blood cells (erythrocytes) is a widely used clinical parameter for screening of several ailments such as stroke, infectious diseases, and malignancy. In a traditional pathological setting, the total time taken for evaluating this parameter varies typically from 1 to 2 h. Furthermore, the volume of human blood to be drawn for each test, following a gold standard laboratory technique (alternatively known as the Westergren method), varies from 4 to 5 ml. Circumventing the above constraints, here we propose a rapid (∼1 min) and highly energy efficient method for the simultaneous determination of hematocrit and erythrocyte sedimentation rate (ESR) on a microfluidic chip, deploying electrically driven spreading of a tiny drop of blood sample (∼8 μl). Our unique approach estimates these parameters by correlating the same with the time taken by the droplet to spread over a given radius, reproducing the results from more elaborate laboratory settings to a satisfactory extent. Our novel methodology is equally applicable for determining higher ranges of ESR such as high concentration of bilirubin and samples corresponding to patients with anemia and patients with some severe inflammation. Furthermore, the minimal fabrication steps involved in the process, along with the rapidity and inexpensiveness of the test, render the suitability of the strategy in extreme point-of-care settings.

Deep Learning Reveals Cancer Metastasis and Therapeutic Antibody Targeting in the Entire Body

Reliable detection of disseminated tumor cells and of the biodistribution of tumor-targeting therapeutic antibodies within the entire body has long been needed to better understand and treat cancer metastasis. Here, we developed an integrated pipeline for automated quantification of cancer metastases and therapeutic antibody targeting, named DeepMACT. First, we enhanced the fluorescent signal of cancer cells more than 100-fold by applying the vDISCO method to image metastasis in transparent mice. Second, we developed deep learning algorithms for automated quantification of metastases with an accuracy matching human expert manual annotation. Deep learning-based quantification in 5 different metastatic cancer models including breast, lung, and pancreatic cancer with distinct organotropisms allowed us to systematically analyze features such as size, shape, spatial distribution, and the degree to which metastases are targeted by a therapeutic monoclonal antibody in entire mice. DeepMACT can thus considerably improve the discovery of effective antibody-based therapeutics at the pre-clinical stage. VIDEO ABSTRACT.

The CrowdWater game: A playful way to improve the accuracy of crowdsourced water level class data

Data quality control is important for any data collection program, especially in citizen science projects, where it is more likely that errors occur due to the human factor. Ideally, data quality control in citizen science projects is also crowdsourced so that it can handle large amounts of data. Here we present the CrowdWater game as a gamified method to check crowdsourced water level class data that are submitted by citizen scientists through the CrowdWater app. The app uses a virtual staff gauge approach, which means that a digital scale is added to the first picture taken at a site and this scale is used for water level class observations at different times. In the game, participants classify water levels based on the comparison of the new picture with the picture containing the virtual staff gauge. By March 2019, 153 people had played the CrowdWater game and 841 pictures were classified. The average water level for the game votes for the classified pictures was compared to the water level class submitted through the app to determine whether the game can improve the quality of the data submitted through the app. For about 70% of the classified pictures, the water level class was the same for the CrowdWater app and game. For a quarter of the classified pictures, there was disagreement between the value submitted through the app and the average game vote. Expert judgement suggests that for three quarters of these cases, the game based average value was correct. The initial results indicate that the CrowdWater game helps to identify erroneous water level class observations from the CrowdWater app and provides a useful approach for crowdsourced data quality control. This study thus demonstrates the potential of gamified approaches for data quality control in citizen science projects.

Scientific Discovery Games for Biomedical Research

Over the past decade, scientific discovery games (SDGs) have emerged as a viable approach for biomedical research, engaging hundreds of thousands of volunteer players and resulting in numerous scientific publications. After describing the origins of this novel research approach, we review the scientific output of SDGs across molecular modeling, sequence alignment, neuroscience, pathology, cellular biology, genomics, and human cognition. We find compelling results and technical innovations arising in problem-oriented games such as Foldit and Eterna and in data-oriented games such as EyeWire and Project Discovery. We discuss emergent properties of player communities shared across different projects, including the diversity of communities and the extraordinary contributions of some volunteers, such as paper writing. Finally, we highlight connections to artificial intelligence, biological cloud laboratories, new game genres, science education, and open science that may drive the next generation of SDGs.

Crowdsourcing of Histological Image Labeling and Object Delineation by Medical Students

Crowdsourcing in pathology has been performed on tasks that are assumed to be manageable by nonexperts. Demand remains high for annotations of more complex elements in digital microscopic images, such as anatomical structures. Therefore, this paper investigates conditions to enable crowdsourced annotations of high-level image objects, a complex task considered to require expert knowledge. Seventy six medical students without specific domain knowledge who voluntarily participated in three experiments solved two relevant annotation tasks on histopathological images: 1) labeling of images showing tissue regions and 2) delineation of morphologically defined image objects. We focus on methods to ensure sufficient annotation quality including several tests on the required number of participants and on the correlation of participants' performance between tasks. In a set up simulating annotation of images with limited ground truth, we validated the feasibility of a confidence score using full ground truth. For this, we computed a majority vote using weighting factors based on individual assessment of contributors against scattered gold standard annotated by pathologists. In conclusion, we provide guidance for task design and quality control to enable a crowdsourced approach to obtain accurate annotations required in the era of digital pathology.

Identifying potential uses of crowdsourcing in global health, conflict, and humanitarian settings: an adapted CHNRI (Child Health and Nutrition Initiative) exercise

Background

Crowdsourcing, outsourcing problems and tasks to a crowd, has grown exponentially since the term was coined a decade ago. Being a rapid and inexpensive approach, it is particularly amenable to addressing problems in global health, conflict and humanitarian settings, but its potential has not been systematically assessed. We employed the Child Health and Nutrition Research Initiative's (CHNRI) method to generate a ranked list of potential uses of crowdsourcing in global health and conflict.

Process

94 experts in global health and crowdsourcing submitted their ideas, and 239 ideas were scored. Each expert scored ideas against three of seven criteria, which were tailored specifically for the exercise. A relative ranking was calculated, along with an Average Expert Agreement (AEA).

Findings

On a scale from 0-100, the scores assigned to proposed ideas ranged from 80.39 to 42.01. Most ideas were related to problem solving (n = 112) or data generation (n = 91). Using health care workers to share information about disease outbreaks to ensure global response had the highest score and agreement. Within the top 15, four additional ideas related to containing communicable diseases, two ideas related to using crowdsourcing for vital registration and two to improve maternal and child health. The top conflict ideas related to epidemic responses and various aspects of disease spread. Wisdom of the crowds and machine learning scored low despite being promising in literature.

Interpretations

Experts were invited to generate ideas during the Ebola crisis and to score during reports of Zika, which may have affected the scoring. However, crowdsourcing's rapid, inexpensive characteristics make it suitable for addressing epidemics. Given that many ideas reflected Sustainable Development Goals (SDGs), crowdsourcing may be an innovative solution to achieving some of the SDGs.

ResistanceSim: development and acceptability study of a serious game to improve understanding of insecticide resistance management in vector control programmes

The use of insecticides is the cornerstone of effective malaria vector control. However, the last two decades has seen the ubiquitous use of insecticides, predominantly pyrethroids, causing widespread insecticide resistance and compromising the effectiveness of vector control. Considerable efforts to develop new active ingredients and interventions are underway. However, it is essential to deploy strategies to mitigate the impact of insecticide resistance now, both to maintain the efficacy of currently available tools as well as to ensure the sustainability of new tools as they come to market. Although the World Health Organization disseminated best practice guidelines for insecticide resistance management (IRM), Rollback Malaria's Vector Control Working Group identified the lack of practical knowledge of IRM as the primary gap in the translation of evidence into policy. ResistanceSim is a capacity strengthening tool designed to address this gap. The development process involved frequent stakeholder consultation, including two separate workshops. These workshops defined the learning objectives, target audience, and the role of mathematical models in the game. Software development phases were interspersed with frequent user testing, resulting in an iterative design process. User feedback was evaluated via questionnaires with Likert-scale and open-ended questions. The game was regularly evaluated by subject-area experts through meetings of an external advisory panel. Through these processes, a series of learning domains were identified and a set of specific learning objectives for each domain were defined to be communicated to vector control programme personnel. A simple "game model" was proposed that produces realistic outputs based on player strategy and also runs in real-time. Early testing sessions revealed numerous usability issues that prevented adequate player engagement. After extensive revisions, later testing sessions indicated that the tool would be a valuable addition to IRM training.

Iterative multi-path tracking for video and volume segmentation with sparse point supervision

Recent machine learning strategies for segmentation tasks have shown great ability when trained on large pixel-wise annotated image datasets. It remains a major challenge however to aggregate such datasets, as the time and monetary cost associated with collecting extensive annotations is extremely high. This is particularly the case for generating precise pixel-wise annotations in video and volumetric image data. To this end, this work presents a novel framework to produce pixel-wise segmentations using minimal supervision. Our method relies on 2D point supervision, whereby a single 2D location within an object of interest is provided on each image of the data. Our method then estimates the object appearance in a semi-supervised fashion by learning object-image-specific features and by using these in a semi-supervised learning framework. Our object model is then used in a graph-based optimization problem that takes into account all provided locations and the image data in order to infer the complete pixel-wise segmentation. In practice, we solve this optimally as a tracking problem using a K-shortest path approach. Both the object model and segmentation are then refined iteratively to further improve the final segmentation. We show that by collecting 2D locations using a gaze tracker, our approach can provide state-of-the-art segmentations on a range of objects and image modalities (video and 3D volumes), and that these can then be used to train supervised machine learning classifiers.

Wearable and Implantable Sensors for Biomedical Applications

Mobile health technologies offer great promise for reducing healthcare costs and improving patient care. Wearable and implantable technologies are contributing to a transformation in the mobile health era in terms of improving healthcare and health outcomes and providing real-time guidance on improved health management and tracking. In this article, we review the biomedical applications of wearable and implantable medical devices and sensors, ranging from monitoring to prevention of diseases, as well as the materials used in the fabrication of these devices and the standards for wireless medical devices and mobile applications. We conclude by discussing some of the technical challenges in wearable and implantable technology and possible solutions for overcoming these difficulties.

Mapping of Crowdsourcing in Health: Systematic Review

Background

Crowdsourcing involves obtaining ideas, needed services, or content by soliciting Web-based contributions from a crowd. The 4 types of crowdsourced tasks (problem solving, data processing, surveillance or monitoring, and surveying) can be applied in the 3 categories of health (promotion, research, and care).

Objective

This study aimed to map the different applications of crowdsourcing in health to assess the fields of health that are using crowdsourcing and the crowdsourced tasks used. We also describe the logistics of crowdsourcing and the characteristics of crowd workers.

Methods

MEDLINE, EMBASE, and ClinicalTrials.gov were searched for available reports from inception to March 30, 2016, with no restriction on language or publication status.

Results

We identified 202 relevant studies that used crowdsourcing, including 9 randomized controlled trials, of which only one had posted results at ClinicalTrials.gov. Crowdsourcing was used in health promotion (91/202, 45.0%), research (73/202, 36.1%), and care (38/202, 18.8%). The 4 most frequent areas of application were public health (67/202, 33.2%), psychiatry (32/202, 15.8%), surgery (22/202, 10.9%), and oncology (14/202, 6.9%). Half of the reports (99/202, 49.0%) referred to data processing, 34.6% (70/202) referred to surveying, 10.4% (21/202) referred to surveillance or monitoring, and 5.9% (12/202) referred to problem-solving. Labor market platforms (eg, Amazon Mechanical Turk) were used in most studies (190/202, 94%). The crowd workers’ characteristics were poorly reported, and crowdsourcing logistics were missing from two-thirds of the reports. When reported, the median size of the crowd was 424 (first and third quartiles: 167-802); crowd workers’ median age was 34 years (32-36). Crowd workers were mainly recruited nationally, particularly in the United States. For many studies (58.9%, 119/202), previous experience in crowdsourcing was required, and passing a qualification test or training was seldom needed (11.9% of studies; 24/202). For half of the studies, monetary incentives were mentioned, with mainly less than US $1 to perform the task. The time needed to perform the task was mostly less than 10 min (58.9% of studies; 119/202). Data quality validation was used in 54/202 studies (26.7%), mainly by attention check questions or by replicating the task with several crowd workers.

Conclusions

The use of crowdsourcing, which allows access to a large pool of participants as well as saving time in data collection, lowering costs, and speeding up innovations, is increasing in health promotion, research, and care. However, the description of crowdsourcing logistics and crowd workers’ characteristics is frequently missing in study reports and needs to be precisely reported to better interpret the study findings and replicate them.

Crowdsourced versus expert evaluations of the vesico-urethral anastomosis in the robotic radical prostatectomy: is one superior at discriminating differences in automated performance metrics?

Crowdsourcing from the general population is an efficient, inexpensive method of surgical performance evaluation. In this study, we compared the discriminatory ability of experts and crowdsourced evaluators (the Crowd) to detect differences in robotic automated performance metrics (APMs). APMs (instrument motion tracking and events data directly from the robot system) of anterior vesico-urethral anastomoses (VUAs) of robotic radical prostatectomies were captured by the dVLogger (Intuitive Surgical). Crowdsourced evaluators and four expert surgeons evaluated video footage using the Global Evaluative Assessment of Robotic Skills (GEARS) (individual domains and total score). Cases were then stratified into performance groups (high versus low quality) for each evaluator based on GEARS. APMs from each group were compared using the Mann-Whitney U test. 25 VUAs performed by 11 surgeons were evaluated. The Crowd displayed moderate correlation with averaged expert scores for all GEARS domains (r > 0.58, p < 0.01). Bland-Altman analysis showed a narrower total GEARS score distribution by the Crowd compared to experts. APMs compared amongst performance groups for each evaluator showed that through GEARS scoring, the most common differentiated metric by evaluators was the velocity of the dominant instrument arm. The Crowd outperformed two out of four expert evaluators by discriminating differences in three APMs using total GEARS scores. The Crowd assigns a narrower range of GEARS scores compared to experts but maintains overall agreement with experts. The discriminatory ability of the Crowd at discerning differences in robotic movements (via APMs) through GEARS scoring is quite refined, rivaling that of expert evaluators.

Mobile Technologies for the Discovery, Analysis, and Engineering of the Global Microbiome

The microbiome has been heralded as a gauge of and contributor to both human health and environmental conditions. Current challenges in probing, engineering, and harnessing the microbiome stem from its microscopic and nanoscopic nature, diversity and complexity of interactions among its members and hosts, as well as the spatiotemporal sampling and in situ measurement limitations induced by the restricted capabilities and norm of existing technologies, leaving some of the constituents of the microbiome unknown. To facilitate significant progress in the microbiome field, deeper understanding of the constituents' individual behavior, interactions with others, and biodiversity are needed. Also crucial is the generation of multimodal data from a variety of subjects and environments over time. Mobile imaging and sensing technologies, particularly through smartphone-based platforms, can potentially meet some of these needs in field-portable, cost-effective, and massively scalable manners by circumventing the need for bulky, expensive instrumentation. In this Perspective, we outline how mobile sensing and imaging technologies could lead the way to unprecedented insight into the microbiome, potentially shedding light on various microbiome-related mysteries of today, including the composition and function of human, animal, plant, and environmental microbiomes. Finally, we conclude with a look at the future, propose a computational microbiome engineering and optimization framework, and discuss its potential impact and applications.

Applications of crowdsourcing in health: an overview

Background

Crowdsourcing is a nascent phenomenon that has grown exponentially since it was coined in 2006. It involves a large group of people solving a problem or completing a task for an individual or, more commonly, for an organisation. While the field of crowdsourcing has developed more quickly in information technology, it has great promise in health applications. This review examines uses of crowdsourcing in global health and health, broadly.

Methods

Semantic searches were run in Google Scholar for “crowdsourcing,” “crowdsourcing and health,” and similar terms. 996 articles were retrieved and all abstracts were scanned. 285 articles related to health. This review provides a narrative overview of the articles identified.

Results

Eight areas where crowdsourcing has been used in health were identified: diagnosis; surveillance; nutrition; public health and environment; education; genetics; psychology; and, general medicine/other. Many studies reported crowdsourcing being used in a diagnostic or surveillance capacity. Crowdsourcing has been widely used across medical disciplines; however, it is important for future work using crowdsourcing to consider the appropriateness of the crowd being used to ensure the crowd is capable and has the adequate knowledge for the task at hand. Gamification of tasks seems to improve accuracy; other innovative methods of analysis including introducing thresholds and measures of trustworthiness should be considered.

Conclusion

Crowdsourcing is a new field that has been widely used and is innovative and adaptable. With the exception of surveillance applications that are used in emergency and disaster situations, most uses of crowdsourcing have only been used as pilots. These exceptions demonstrate that it is possible to take crowdsourcing applications to scale. Crowdsourcing has the potential to provide more accessible health care to more communities and individuals rapidly and to lower costs of care.

Image analysis and machine learning for detecting malaria

Malaria remains a major burden on global health, with roughly 200 million cases worldwide and more than 400,000 deaths per year. Besides biomedical research and political efforts, modern information technology is playing a key role in many attempts at fighting the disease. One of the barriers toward a successful mortality reduction has been inadequate malaria diagnosis in particular. To improve diagnosis, image analysis software and machine learning methods have been used to quantify parasitemia in microscopic blood slides. This article gives an overview of these techniques and discusses the current developments in image analysis and machine learning for microscopic malaria diagnosis. We organize the different approaches published in the literature according to the techniques used for imaging, image preprocessing, parasite detection and cell segmentation, feature computation, and automatic cell classification. Readers will find the different techniques listed in tables, with the relevant articles cited next to them, for both thin and thick blood smear images. We also discussed the latest developments in sections devoted to deep learning and smartphone technology for future malaria diagnosis.

"Crowdsourcing" ten years in: A review

Background

First coined by Howe in 2006, the field of crowdsourcing has grown exponentially. Despite its growth and its transcendence across many fields, the definition of crowdsourcing has still not been agreed upon, and examples are poorly indexed in peer–reviewed literature. Many examples of crowdsourcing have not been scaled–up past the pilot phase. In spite of this, crowdsourcing has great potential, especially in global health where resources are lacking. This narrative review seeks to review both indexed and grey crowdsourcing literature broadly in order to explore the current state of the field.

Methods

This is a review of reviews of crowdsourcing. Semantic searches were conducted using Google Scholar rather than indexed databases due to poor indexing of the topic. 996 articles were retrieved, of which 69 were initially identified as being reviews or theoretically–based. 21 of these were found to be irrelevant and 48 articles were reviewed.

Results

This narrative review focuses on defining crowdsourcing, taxonomies of crowdsourcing, who constitutes the crowd, research that is amenable to crowdsourcing, regulatory and ethical aspects of crowdsourcing and some notable examples of crowdsourcing.

Conclusions

Crowdsourcing has the potential to be hugely promising, especially in global health, due to its ability to collect information rapidly, inexpensively and accurately. Rigorous ethical and regulatory controls are needed to ensure data are collected and analysed appropriately and crowdsourcing should be considered complementary to traditional research methods.

A Review on the Applications of Crowdsourcing in Human Pathology

The advent of the digital pathology has introduced new avenues of diagnostic medicine. Among them, crowdsourcing has attracted researchers' attention in the recent years, allowing them to engage thousands of untrained individuals in research and diagnosis. While there exist several articles in this regard, prior works have not collectively documented them. We, therefore, aim to review the applications of crowdsourcing in human pathology in a semi-systematic manner. We first, introduce a novel method to do a systematic search of the literature. Utilizing this method, we, then, collect hundreds of articles and screen them against a predefined set of criteria. Furthermore, we crowdsource part of the screening process, to examine another potential application of crowdsourcing. Finally, we review the selected articles and characterize the prior uses of crowdsourcing in pathology.

Converging Human and Malaria Vector Diagnostics with Data Management towards an Integrated Holistic One Health Approach

Monitoring malaria prevalence in humans, as well as vector populations, for the presence of Plasmodium, is an integral component of effective malaria control, and eventually, elimination. In the field of human diagnostics, a major challenge is the ability to define, precisely, the causative agent of fever, thereby differentiating among several candidate (also non-malaria) febrile diseases. This requires genetic-based pathogen identification and multiplexed analysis, which, in combination, are hardly provided by the current gold standard diagnostic tools. In the field of vectors, an essential component of control programs is the detection of Plasmodium species within its mosquito vectors, particularly in the salivary glands, where the infective sporozoites reside. In addition, the identification of species composition and insecticide resistance alleles within vector populations is a primary task in routine monitoring activities, aiming to support control efforts. In this context, the use of converging diagnostics is highly desirable for providing comprehensive information, including differential fever diagnosis in humans, and mosquito species composition, infection status, and resistance to insecticides of vectors. Nevertheless, the two fields of human diagnostics and vector control are rarely combined, both at the diagnostic and at the data management end, resulting in fragmented data and mis- or non-communication between various stakeholders. To this direction, molecular technologies, their integration in automated platforms, and the co-assessment of data from multiple diagnostic sources through information and communication technologies are possible pathways towards a unified human vector approach.

Crowdsourcing in Surgical Skills Acquisition: A Developing Technology in Surgical Education

BACKGROUND

The application of crowdsourcing to surgical education is a recent phenomenon and adds to increasing demands on surgical residency training. The efficacy, range, and scope of this technology for surgical education remains incompletely defined.

OBJECTIVE

A systematic review was performed using the PubMed database of English-language literature on crowdsourced evaluation of surgical technical tasks up to April 2017.

METHODS

Articles were reviewed, abstracted, and analyzed, and were assessed for quality using the Medical Education Research Study Quality Instrument (MERSQI). Articles were evaluated with eligibility criteria for inclusion. Study information, performance task, subjects, evaluative standards, crowdworker compensation, time to response, and correlation between crowd and expert or standard evaluations were abstracted and analyzed.

RESULTS

Of 63 unique publications initially identified, 13 with MERSQI scores ranging from 10 to 13 (mean = 11.85) were included in the review. Overall, crowd and expert evaluations demonstrated good to excellent correlation across a wide range of tasks (Pearson's coefficient 0.59-0.95, Cronbach's alpha 0.32-0.92), with 1 exception being a study involving medical students. There was a wide range of reported interrater variability among experts. Nonexpert evaluation was consistently quicker than expert evaluation (ranging from 4.8 to 150.9 times faster), and was more cost effective.

CONCLUSIONS

Crowdsourced feedback appears to be comparable to expert feedback and is cost effective and efficient. Further work is needed to increase consistency in expert evaluations, to explore sources of discrepant assessments between surgeons and crowds, and to identify optimal populations and novel applications for this technology.

"Crowdsourcing" ten years in: A review

BACKGROUND

First coined by Howe in 2006, the field of crowdsourcing has grown exponentially. Despite its growth and its transcendence across many fields, the definition of crowdsourcing has still not been agreed upon, and examples are poorly indexed in peer-reviewed literature. Many examples of crowdsourcing have not been scaled-up past the pilot phase. In spite of this, crowdsourcing has great potential, especially in global health where resources are lacking. This narrative review seeks to review both indexed and grey crowdsourcing literature broadly in order to explore the current state of the field.

METHODS

This is a review of reviews of crowdsourcing. Semantic searches were conducted using Google Scholar rather than indexed databases due to poor indexing of the topic. 996 articles were retrieved, of which 69 were initially identified as being reviews or theoretically-based. 21 of these were found to be irrelevant and 48 articles were reviewed.

RESULTS

This narrative review focuses on defining crowdsourcing, taxonomies of crowdsourcing, who constitutes the crowd, research that is amenable to crowdsourcing, regulatory and ethical aspects of crowdsourcing and some notable examples of crowdsourcing.

CONCLUSIONS

Crowdsourcing has the potential to be hugely promising, especially in global health, due to its ability to collect information rapidly, inexpensively and accurately. Rigorous ethical and regulatory controls are needed to ensure data are collected and analysed appropriately and crowdsourcing should be considered complementary to traditional research methods.

Counting malaria parasites with a two-stage EM based algorithm using crowsourced data

Malaria eradication of the worldwide is currently one of the main WHO's global goals. In this work, we focus on the use of human-machine interaction strategies for low-cost fast reliable malaria diagnostic based on a crowdsourced approach. The addressed technical problem consists in detecting spots in images even under very harsh conditions when positive objects are very similar to some artifacts. The clicks or tags delivered by several annotators labeling an image are modeled as a robust finite mixture, and techniques based on the Expectation-Maximization (EM) algorithm are proposed for accurately counting malaria parasites on thick blood smears obtained by microscopic Giemsa-stained techniques. This approach outperforms other traditional methods as it is shown through experimentation with real data.

Leveraging the crowd for annotation of retinal images

Medical data presents a number of challenges. It tends to be unstructured, noisy and protected. To train algorithms to understand medical images, doctors can label the condition associated with a particular image, but obtaining enough labels can be difficult. We propose an annotation approach which starts with a small pool of expertly annotated images and uses their expertise to rate the performance of crowd-sourced annotations. In this paper we demonstrate how to apply our approach for annotation of large-scale datasets of retinal images. We introduce a novel data validation procedure which is designed to cope with noisy ground-truth data and with non-consistent input from both experts and crowd-workers.

The Accuracy and Reliability of Crowdsource Annotations of Digital Retinal Images

Purpose

Crowdsourcing is based on outsourcing computationally intensive tasks to numerous individuals in the online community who have no formal training. Our aim was to develop a novel online tool designed to facilitate large-scale annotation of digital retinal images, and to assess the accuracy of crowdsource grading using this tool, comparing it to expert classification.

Methods

We used 100 retinal fundus photograph images with predetermined disease criteria selected by two experts from a large cohort study. The Amazon Mechanical Turk Web platform was used to drive traffic to our site so anonymous workers could perform a classification and annotation task of the fundus photographs in our dataset after a short training exercise. Three groups were assessed: masters only, nonmasters only and nonmasters with compulsory training. We calculated the sensitivity, specificity, and area under the curve (AUC) of receiver operating characteristic (ROC) plots for all classifications compared to expert grading, and used the Dice coefficient and consensus threshold to assess annotation accuracy.

Results

In total, we received 5389 annotations for 84 images (excluding 16 training images) in 2 weeks. A specificity and sensitivity of 71% (95% confidence interval [CI], 69%–74%) and 87% (95% CI, 86%–88%) was achieved for all classifications. The AUC in this study for all classifications combined was 0.93 (95% CI, 0.91–0.96). For image annotation, a maximal Dice coefficient (∼0.6) was achieved with a consensus threshold of 0.25.

Conclusions

This study supports the hypothesis that annotation of abnormalities in retinal images by ophthalmologically naive individuals is comparable to expert annotation. The highest AUC and agreement with expert annotation was achieved in the nonmasters with compulsory training group.

Translational Relevance

The use of crowdsourcing as a technique for retinal image analysis may be comparable to expert graders and has the potential to deliver timely, accurate, and cost-effective image analysis.

Phase correlation imaging of unlabeled cell dynamics

We present phase correlation imaging (PCI) as a novel approach to study cell dynamics in a spatially-resolved manner. PCI relies on quantitative phase imaging time-lapse data and, as such, functions in label-free mode, without the limitations associated with exogenous markers. The correlation time map outputted in PCI informs on the dynamics of the intracellular mass transport. Specifically, we show that PCI can extract quantitatively the diffusion coefficient map associated with live cells, as well as standard Brownian particles. Due to its high sensitivity to mass transport, PCI can be applied to studying the integrity of actin polymerization dynamics. Our results indicate that the cyto-D treatment blocking the actin polymerization has a dominant effect at the large spatial scales, in the region surrounding the cell. We found that PCI can distinguish between senescent and quiescent cells, which is extremely difficult without using specific markers currently. We anticipate that PCI will be used alongside established, fluorescence-based techniques to enable valuable new studies of cell function.

Crowdsourcing for error detection in cortical surface delineations

PURPOSE

With the recent trend toward big data analysis, neuroimaging datasets have grown substantially in the past years. While larger datasets potentially offer important insights for medical research, one major bottleneck is the requirement for resources of medical experts needed to validate automatic processing results. To address this issue, the goal of this paper was to assess whether anonymous nonexperts from an online community can perform quality control of MR-based cortical surface delineations derived by an automatic algorithm.

METHODS

So-called knowledge workers from an online crowdsourcing platform were asked to annotate errors in automatic cortical surface delineations on 100 central, coronal slices of MR images.

RESULTS

On average, annotations for 100 images were obtained in less than an hour. When using expert annotations as reference, the crowd on average achieves a sensitivity of 82 % and a precision of 42 %. Merging multiple annotations per image significantly improves the sensitivity of the crowd (up to 95 %), but leads to a decrease in precision (as low as 22 %).

CONCLUSION

Our experiments show that the detection of errors in automatic cortical surface delineations generated by anonymous untrained workers is feasible. Future work will focus on increasing the sensitivity of our method further, such that the error detection tasks can be handled exclusively by the crowd and expert resources can be focused on error correction.

Crowdsourcing for error detection in cortical surface delineations

PURPOSE

With the recent trend toward big data analysis, neuroimaging datasets have grown substantially in the past years. While larger datasets potentially offer important insights for medical research, one major bottleneck is the requirement for resources of medical experts needed to validate automatic processing results. To address this issue, the goal of this paper was to assess whether anonymous nonexperts from an online community can perform quality control of MR-based cortical surface delineations derived by an automatic algorithm.

METHODS

So-called knowledge workers from an online crowdsourcing platform were asked to annotate errors in automatic cortical surface delineations on 100 central, coronal slices of MR images.

RESULTS

On average, annotations for 100 images were obtained in less than an hour. When using expert annotations as reference, the crowd on average achieves a sensitivity of 82 % and a precision of 42 %. Merging multiple annotations per image significantly improves the sensitivity of the crowd (up to 95 %), but leads to a decrease in precision (as low as 22 %).

CONCLUSION

Our experiments show that the detection of errors in automatic cortical surface delineations generated by anonymous untrained workers is feasible. Future work will focus on increasing the sensitivity of our method further, such that the error detection tasks can be handled exclusively by the crowd and expert resources can be focused on error correction.

The Impact of an Online Crowdsourcing Diagnostic Tool on Health Care Utilization: A Case Study Using a Novel Approach to Retrospective Claims Analysis

Background

Patients with difficult medical cases often remain undiagnosed despite visiting multiple physicians. A new online platform, CrowdMed, uses crowdsourcing to quickly and efficiently reach an accurate diagnosis for these patients.

Objective

This study sought to evaluate whether CrowdMed decreased health care utilization for patients who have used the service.

Methods

Novel, electronic methods of patient recruitment and data collection were utilized. Patients who completed cases on CrowdMed’s platform between July 2014 and April 2015 were recruited for the study via email and screened via an online survey. After providing eConsent, participants provided identifying information used to access their medical claims data, which was retrieved through a third-party web application program interface (API). Utilization metrics including frequency of provider visits and medical charges were compared pre- and post-case resolution to assess the impact of resolving a case on CrowdMed.

Results

Of 45 CrowdMed users who completed the study survey, comprehensive claims data was available via API for 13 participants, who made up the final enrolled sample. There were a total of 221 health care provider visits collected for the study participants, with service dates ranging from September 2013 to July 2015. Frequency of provider visits was significantly lower after resolution of a case on CrowdMed (mean of 1.07 visits per month pre-resolution vs. 0.65 visits per month post-resolution, P=.01). Medical charges were also significantly lower after case resolution (mean of US $719.70 per month pre-resolution vs. US $516.79 per month post-resolution, P=.03). There was no significant relationship between study results and disease onset date, and there was no evidence of regression to the mean influencing results.

Conclusions

This study employed technology-enabled methods to demonstrate that patients who used CrowdMed had lower health care utilization after case resolution. However, since the final sample size was limited, results should be interpreted as a case study. Despite this limitation, the statistically significant results suggest that online crowdsourcing shows promise as an efficient method of solving difficult medical cases.

Use of Mechanical Turk as a MapReduce Framework for Macular OCT Segmentation

Purpose. To evaluate the feasibility of using Mechanical Turk as a massively parallel platform to perform manual segmentations of macular spectral domain optical coherence tomography (SD-OCT) images using a MapReduce framework. Methods. A macular SD-OCT volume of 61 slice images was map-distributed to Amazon Mechanical Turk. Each Human Intelligence Task was set to $0.01 and required the user to draw five lines to outline the sublayers of the retinal OCT image after being shown example images. Each image was submitted twice for segmentation, and interrater reliability was calculated. The interface was created using custom HTML5 and JavaScript code, and data analysis was performed using R. An automated pipeline was developed to handle the map and reduce steps of the framework. Results. More than 93,500 data points were collected using this framework for the 61 images submitted. Pearson's correlation of interrater reliability was 0.995 (p < 0.0001) and coefficient of determination was 0.991. The cost of segmenting the macular volume was $1.21. A total of 22 individual Mechanical Turk users provided segmentations, each completing an average of 5.5 HITs. Each HIT was completed in an average of 4.43 minutes. Conclusions. Amazon Mechanical Turk provides a cost-effective, scalable, high-availability infrastructure for manual segmentation of OCT images.

Tools for the Microbiome: Nano and Beyond

The microbiome presents great opportunities for understanding and improving the world around us and elucidating the interactions that compose it. The microbiome also poses tremendous challenges for mapping and manipulating the entangled networks of interactions among myriad diverse organisms. Here, we describe the opportunities, technical needs, and potential approaches to address these challenges, based on recent and upcoming advances in measurement and control at the nanoscale and beyond. These technical needs will provide the basis for advancing the largely descriptive studies of the microbiome to the theoretical and mechanistic understandings that will underpin the discipline of microbiome engineering. We anticipate that the new tools and methods developed will also be more broadly useful in environmental monitoring, medicine, forensics, and other areas.

Crowdsourcing Diagnosis for Patients With Undiagnosed Illnesses: An Evaluation of CrowdMed

Background

Despite visits to multiple physicians, many patients remain undiagnosed. A new online program, CrowdMed, aims to leverage the “wisdom of the crowd” by giving patients an opportunity to submit their cases and interact with case solvers to obtain diagnostic possibilities.

Objective

To describe CrowdMed and provide an independent assessment of its impact.

Methods

Patients submit their cases online to CrowdMed and case solvers sign up to help diagnose patients. Case solvers attempt to solve patients’ diagnostic dilemmas and often have an interactive online discussion with patients, including an exchange of additional diagnostic details. At the end, patients receive detailed reports containing diagnostic suggestions to discuss with their physicians and fill out surveys about their outcomes. We independently analyzed data collected from cases between May 2013 and April 2015 to determine patient and case solver characteristics and case outcomes.

Results

During the study period, 397 cases were completed. These patients previously visited a median of 5 physicians, incurred a median of US $10,000 in medical expenses, spent a median of 50 hours researching their illnesses online, and had symptoms for a median of 2.6 years. During this period, 357 active case solvers participated, of which 37.9% (132/348) were male and 58.3% (208/357) worked or studied in the medical industry. About half (50.9%, 202/397) of patients were likely to recommend CrowdMed to a friend, 59.6% (233/391) reported that the process gave insights that led them closer to the correct diagnoses, 57% (52/92) reported estimated decreases in medical expenses, and 38% (29/77) reported estimated improvement in school or work productivity.

Conclusions

Some patients with undiagnosed illnesses reported receiving helpful guidance from crowdsourcing their diagnoses during their difficult diagnostic journeys. However, further development and use of crowdsourcing methods to facilitate diagnosis requires long-term evaluation as well as validation to account for patients’ ultimate correct diagnoses.

Review of Telemicrobiology

CONTEXT

Microbiology laboratories are continually pursuing means to improve quality, rapidity, and efficiency of specimen analysis in the face of limited resources. One means by which to achieve these improvements is through the remote analysis of digital images. Telemicrobiology enables the remote interpretation of images of microbiology specimens. To date, the practice of clinical telemicrobiology has not been thoroughly reviewed.

OBJECTIVE

To identify the various methods that can be employed for telemicrobiology, including emerging technologies that may provide value to the clinical laboratory.

DATA SOURCES

Peer-reviewed literature, conference proceedings, meeting presentations, and expert opinions pertaining to telemicrobiology have been evaluated.

CONCLUSIONS

A number of modalities have been employed for telemicroscopy, including static capture techniques, whole slide imaging, video telemicroscopy, mobile devices, and hybrid systems. Telemicrobiology has been successfully implemented for several applications, including routine primary diagnosis, expert teleconsultation, and proficiency testing. Emerging areas of telemicrobiology include digital plate reading of bacterial cultures, mobile health applications, and computer-augmented analysis of digital images. To date, static image capture techniques have been the most widely used modality for telemicrobiology, despite newer technologies being available that may produce better quality interpretations. Telemicrobiology adds value, quality, and efficiency to the clinical microbiology laboratory, and increased adoption of telemicrobiology is anticipated.

Development of inexpensive blood imaging systems: where are we now?

Clinical applications in the developing world, such as malaria and anemia diagnosis, demand a change in the medical paradigm of expensive care given in central locations by highly trained professionals. There has been a recent explosion in optical technologies entering the consumer market through the widespread adoption of smartphones and LEDs. This technology commoditization has enabled the development of small, portable optical imaging systems at an unprecedentedly low cost. Here, we review the state-of-the-field of the application of these systems for low-cost blood imaging with an emphasis on cellular imaging systems. In addition to some promising results addressing specific clinical issues, an overview of the technology landscape is provided. We also discuss several key issues that need to be addressed before these technologies can be commercialized.

Malaria Diagnosis Using a Mobile Phone Polarized Microscope

Malaria remains a major global health burden, and new methods for low-cost, high-sensitivity, diagnosis are essential, particularly in remote areas with low-resource around the world. In this paper, a cost effective, optical cell-phone based transmission polarized light microscope system is presented for imaging the malaria pigment known as hemozoin. It can be difficult to determine the presence of the pigment from background and other artifacts, even for skilled microscopy technicians. The pigment is much easier to observe using polarized light microscopy. However, implementation of polarized light microscopy lacks widespread adoption because the existing commercial devices have complicated designs, require sophisticated maintenance, tend to be bulky, can be expensive, and would require re-training for existing microscopy technicians. To this end, a high fidelity and high optical resolution cell-phone based polarized light microscopy system is presented which is comparable to larger bench-top polarized microscopy systems but at much lower cost and complexity. The detection of malaria in fixed and stained blood smears is presented using both, a conventional polarized microscope and our cell-phone based system. The cell-phone based polarimetric microscopy design shows the potential to have both the resolution and specificity to detect malaria in a low-cost, easy-to-use, modular platform.

Democratization of Nanoscale Imaging and Sensing Tools Using Photonics

Providing means for researchers and citizen scientists in the developing world to perform advanced measurements with nanoscale precision can help to accelerate the rate of discovery and invention as well as improve higher education and the training of the next generation of scientists and engineers worldwide. Here, we review some of the recent progress toward making optical nanoscale measurement tools more cost-effective, field-portable, and accessible to a significantly larger group of researchers and educators. We divide our review into two main sections: label-based nanoscale imaging and sensing tools, which primarily involve fluorescent approaches, and label-free nanoscale measurement tools, which include light scattering sensors, interferometric methods, photonic crystal sensors, and plasmonic sensors. For each of these areas, we have primarily focused on approaches that have either demonstrated operation outside of a traditional laboratory setting, including for example integration with mobile phones, or exhibited the potential for such operation in the near future.