1
|
Washington P, Leblanc E, Dunlap K, Penev Y, Kline A, Paskov K, Sun MW, Chrisman B, Stockham N, Varma M, Voss C, Haber N, Wall DP. Precision Telemedicine through Crowdsourced Machine Learning: Testing Variability of Crowd Workers for Video-Based Autism Feature Recognition. J Pers Med 2020; 10:E86. [PMID: 32823538 PMCID: PMC7564950 DOI: 10.3390/jpm10030086] [Citation(s) in RCA: 29] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2020] [Revised: 08/09/2020] [Accepted: 08/10/2020] [Indexed: 02/06/2023] Open
Abstract
Mobilized telemedicine is becoming a key, and even necessary, facet of both precision health and precision medicine. In this study, we evaluate the capability and potential of a crowd of virtual workers-defined as vetted members of popular crowdsourcing platforms-to aid in the task of diagnosing autism. We evaluate workers when crowdsourcing the task of providing categorical ordinal behavioral ratings to unstructured public YouTube videos of children with autism and neurotypical controls. To evaluate emerging patterns that are consistent across independent crowds, we target workers from distinct geographic loci on two crowdsourcing platforms: an international group of workers on Amazon Mechanical Turk (MTurk) (N = 15) and Microworkers from Bangladesh (N = 56), Kenya (N = 23), and the Philippines (N = 25). We feed worker responses as input to a validated diagnostic machine learning classifier trained on clinician-filled electronic health records. We find that regardless of crowd platform or targeted country, workers vary in the average confidence of the correct diagnosis predicted by the classifier. The best worker responses produce a mean probability of the correct class above 80% and over one standard deviation above 50%, accuracy and variability on par with experts according to prior studies. There is a weak correlation between mean time spent on task and mean performance (r = 0.358, p = 0.005). These results demonstrate that while the crowd can produce accurate diagnoses, there are intrinsic differences in crowdworker ability to rate behavioral features. We propose a novel strategy for recruitment of crowdsourced workers to ensure high quality diagnostic evaluations of autism, and potentially many other pediatric behavioral health conditions. Our approach represents a viable step in the direction of crowd-based approaches for more scalable and affordable precision medicine.
Collapse
Affiliation(s)
- Peter Washington
- Department of Bioengineering, Stanford University, 443 Via Ortega, Stanford, CA 94305, USA; (P.W.); (B.C.)
| | - Emilie Leblanc
- Department of Pediatrics (Systems Medicine), Stanford University, 1265 Welch Rd., Stanford, CA 94305, USA; (E.L.); (K.D.); (Y.P.); (A.K.)
| | - Kaitlyn Dunlap
- Department of Pediatrics (Systems Medicine), Stanford University, 1265 Welch Rd., Stanford, CA 94305, USA; (E.L.); (K.D.); (Y.P.); (A.K.)
| | - Yordan Penev
- Department of Pediatrics (Systems Medicine), Stanford University, 1265 Welch Rd., Stanford, CA 94305, USA; (E.L.); (K.D.); (Y.P.); (A.K.)
| | - Aaron Kline
- Department of Pediatrics (Systems Medicine), Stanford University, 1265 Welch Rd., Stanford, CA 94305, USA; (E.L.); (K.D.); (Y.P.); (A.K.)
| | - Kelley Paskov
- Department of Biomedical Data Science, Stanford University, 1265 Welch Rd., Stanford, CA 94305, USA; (K.P.); (M.W.S.)
| | - Min Woo Sun
- Department of Biomedical Data Science, Stanford University, 1265 Welch Rd., Stanford, CA 94305, USA; (K.P.); (M.W.S.)
| | - Brianna Chrisman
- Department of Bioengineering, Stanford University, 443 Via Ortega, Stanford, CA 94305, USA; (P.W.); (B.C.)
| | - Nathaniel Stockham
- Department of Neuroscience, Stanford University, 213 Quarry Rd., Stanford, CA 94305, USA;
| | - Maya Varma
- Department of Computer Science, Stanford University, 353 Jane Stanford Way, Stanford, CA 94305, USA; (M.V.); (C.V.)
| | - Catalin Voss
- Department of Computer Science, Stanford University, 353 Jane Stanford Way, Stanford, CA 94305, USA; (M.V.); (C.V.)
| | - Nick Haber
- School of Education, Stanford University, 485 Lasuen Mall, Stanford, CA 94305, USA;
| | - Dennis P. Wall
- Department of Pediatrics (Systems Medicine), Stanford University, 1265 Welch Rd., Stanford, CA 94305, USA; (E.L.); (K.D.); (Y.P.); (A.K.)
- Department of Biomedical Data Science, Stanford University, 1265 Welch Rd., Stanford, CA 94305, USA; (K.P.); (M.W.S.)
| |
Collapse
|
2
|
Aikens EO, Monteith KL, Merkle JA, Dwinnell SPH, Fralick GL, Kauffman MJ. Drought reshuffles plant phenology and reduces the foraging benefit of green-wave surfing for a migratory ungulate. Glob Chang Biol 2020; 26:4215-4225. [PMID: 32524724 DOI: 10.1111/gcb.15169] [Citation(s) in RCA: 16] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/09/2019] [Accepted: 04/21/2020] [Indexed: 06/11/2023]
Abstract
To increase resource gain, many herbivores pace their migration with the flush of nutritious plant green-up that progresses across the landscape (termed "green-wave surfing"). Despite concerns about the effects of climate change on migratory species and the critical role of plant phenology in mediating the ability of ungulates to surf, little is known about how drought shapes the green wave and influences the foraging benefits of migration. With a 19 year dataset on drought and plant phenology across 99 unique migratory routes of mule deer (Odocoileus hemionus) in western Wyoming, United States, we show that drought shortened the duration of spring green-up by approximately twofold (2.5 weeks) and resulted in less sequential green-up along migratory routes. We investigated the possibility that some routes were buffered from the effects of drought (i.e., routes that maintained long green-up duration irrespective of drought intensity). We found no evidence of drought-buffered routes. Instead, routes with the longest green-up in non-drought years also were the most affected by drought. Despite phenological changes along the migratory route, mule deer closely followed drought-altered green waves during migration. Migrating deer did not experience a trophic mismatch with the green wave during drought. Instead, the shorter window of green-up caused by drought reduced the opportunity to accumulate forage resources during rapid spring migrations. Our work highlights the synchronization of phenological events as an important mechanism by which climate change can negatively affect migratory species by reducing the temporal availability of key food resources. For migratory herbivores, climate change poses a new and growing threat by altering resource phenology and diminishing the foraging benefit of migration.
Collapse
Affiliation(s)
- Ellen O Aikens
- Wyoming Cooperative Fish and Wildlife Research Unit, Department of Zoology and Physiology, University of Wyoming, Laramie, WY, USA
- Program in Ecology, University of Wyoming, Laramie, WY, USA
| | - Kevin L Monteith
- Wyoming Cooperative Fish and Wildlife Research Unit, Department of Zoology and Physiology, University of Wyoming, Laramie, WY, USA
- Haub School of Environment and Natural Resources, University of Wyoming, Laramie, WY, USA
| | - Jerod A Merkle
- Department of Zoology and Physiology, University of Wyoming, Laramie, WY, USA
| | - Samantha P H Dwinnell
- Haub School of Environment and Natural Resources, University of Wyoming, Laramie, WY, USA
| | | | - Matthew J Kauffman
- U.S. Geological Survey, Wyoming Cooperative Fish and Wildlife Research Unit, Department of Zoology and Physiology, University of Wyoming, Laramie, WY, USA
| |
Collapse
|