1
|
Zhao Z, Teoh HK, Carpenter J, Nemon F, Kardon B, Cohen I, Goldberg JH. Anterior forebrain pathway in parrots is necessary for producing learned vocalizations with individual signatures. Curr Biol 2023; 33:5415-5426.e4. [PMID: 38070505 PMCID: PMC10799565 DOI: 10.1016/j.cub.2023.11.014] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/01/2023] [Revised: 08/30/2023] [Accepted: 11/08/2023] [Indexed: 12/21/2023]
Abstract
Parrots have enormous vocal imitation capacities and produce individually unique vocal signatures. Like songbirds, parrots have a nucleated neural song system with distinct anterior (AFP) and posterior forebrain pathways (PFP). To test if song systems of parrots and songbirds, which diverged over 50 million years ago, have a similar functional organization, we first established a neuroscience-compatible call-and-response behavioral paradigm to elicit learned contact calls in budgerigars (Melopsittacus undulatus). Using variational autoencoder-based machine learning methods, we show that contact calls within affiliated groups converge but that individuals maintain unique acoustic features, or vocal signatures, even after call convergence. Next, we transiently inactivated the outputs of AFP to test if learned vocalizations can be produced by the PFP alone. As in songbirds, AFP inactivation had an immediate effect on vocalizations, consistent with a premotor role. But in contrast to songbirds, where the isolated PFP is sufficient to produce stereotyped and acoustically normal vocalizations, isolation of the budgerigar PFP caused a degradation of call acoustic structure, stereotypy, and individual uniqueness. Thus, the contribution of AFP and the capacity of isolated PFP to produce learned vocalizations have diverged substantially between songbirds and parrots, likely driven by their distinct behavioral ecology and neural connectivity.
Collapse
Affiliation(s)
- Zhilei Zhao
- Department of Neurobiology and Behavior, Cornell University, Ithaca, NY 14853, USA
| | - Han Kheng Teoh
- Department of Physics, Cornell University, Ithaca, NY 14853, USA
| | - Julie Carpenter
- Department of Neurobiology and Behavior, Cornell University, Ithaca, NY 14853, USA
| | - Frieda Nemon
- Department of Neurobiology and Behavior, Cornell University, Ithaca, NY 14853, USA
| | - Brian Kardon
- Department of Neurobiology and Behavior, Cornell University, Ithaca, NY 14853, USA
| | - Itai Cohen
- Department of Physics, Cornell University, Ithaca, NY 14853, USA
| | - Jesse H Goldberg
- Department of Neurobiology and Behavior, Cornell University, Ithaca, NY 14853, USA.
| |
Collapse
|
2
|
Fujimoto H, Hasegawa T. Reversible inhibition of the basal ganglia prolongs repetitive vocalization but only weakly affects sequencing at branch points in songbirds. Cereb Cortex Commun 2023; 4:tgad016. [PMID: 37675437 PMCID: PMC10477706 DOI: 10.1093/texcom/tgad016] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2023] [Revised: 08/04/2023] [Accepted: 08/07/2023] [Indexed: 09/08/2023] Open
Abstract
Although vocal signals, including languages and songbird syllables, are composed of a finite number of acoustic elements, diverse vocal sequences are composed of a combination of these elements, which are linked together by syntactic rules. However, the neural basis of syntactic vocalization generation remains poorly understood. Here, we report that inhibition using tetrodotoxin (TTX) and manipulations of gamma-aminobutyric acid (GABA) receptors within the basal ganglia Area X or lateral magnocellular nucleus of the anterior neostriatum (LMAN) alter and prolong repetitive vocalization in Bengalese finches (Lonchura striata var. domestica). These results suggest that repetitive vocalizations are modulated by the basal ganglia and not solely by higher motor cortical neurons. These data highlight the importance of neural circuits, including the basal ganglia, in the production of stereotyped repetitive vocalizations and demonstrate that dynamic disturbances within the basal ganglia circuitry can differentially affect the repetitive temporal features of songs.
Collapse
Affiliation(s)
- Hisataka Fujimoto
- Department of Anatomy, Kawasaki Medical School, 577 Matsushima, Kurashiki, Okayama 701-0192, Japan
| | - Taku Hasegawa
- Laboratory for Imagination and Executive functions, RIKEN Center for Brain Science, 2-1 Hirosawa, Wakoshi, Saitama 351-0198, Japan
| |
Collapse
|
3
|
Colquitt BM, Li K, Green F, Veline R, Brainard MS. Neural circuit-wide analysis of changes to gene expression during deafening-induced birdsong destabilization. eLife 2023; 12:e85970. [PMID: 37284822 PMCID: PMC10259477 DOI: 10.7554/elife.85970] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2023] [Accepted: 04/17/2023] [Indexed: 06/08/2023] Open
Abstract
Sensory feedback is required for the stable execution of learned motor skills, and its loss can severely disrupt motor performance. The neural mechanisms that mediate sensorimotor stability have been extensively studied at systems and physiological levels, yet relatively little is known about how disruptions to sensory input alter the molecular properties of associated motor systems. Songbird courtship song, a model for skilled behavior, is a learned and highly structured vocalization that is destabilized following deafening. Here, we sought to determine how the loss of auditory feedback modifies gene expression and its coordination across the birdsong sensorimotor circuit. To facilitate this system-wide analysis of transcriptional responses, we developed a gene expression profiling approach that enables the construction of hundreds of spatially-defined RNA-sequencing libraries. Using this method, we found that deafening preferentially alters gene expression across birdsong neural circuitry relative to surrounding areas, particularly in premotor and striatal regions. Genes with altered expression are associated with synaptic transmission, neuronal spines, and neuromodulation and show a bias toward expression in glutamatergic neurons and Pvalb/Sst-class GABAergic interneurons. We also found that connected song regions exhibit correlations in gene expression that were reduced in deafened birds relative to hearing birds, suggesting that song destabilization alters the inter-region coordination of transcriptional states. Finally, lesioning LMAN, a forebrain afferent of RA required for deafening-induced song plasticity, had the largest effect on groups of genes that were also most affected by deafening. Combined, this integrated transcriptomics analysis demonstrates that the loss of peripheral sensory input drives a distributed gene expression response throughout associated sensorimotor neural circuitry and identifies specific candidate molecular and cellular mechanisms that support the stability and plasticity of learned motor skills.
Collapse
Affiliation(s)
- Bradley M Colquitt
- Howard Hughes Medical InstituteChevy ChaseUnited States
- Department of Physiology, University of California, San FranciscoSan FranciscoUnited States
| | - Kelly Li
- Howard Hughes Medical InstituteChevy ChaseUnited States
- Department of Physiology, University of California, San FranciscoSan FranciscoUnited States
| | - Foad Green
- Howard Hughes Medical InstituteChevy ChaseUnited States
- Department of Physiology, University of California, San FranciscoSan FranciscoUnited States
| | - Robert Veline
- Howard Hughes Medical InstituteChevy ChaseUnited States
- Department of Physiology, University of California, San FranciscoSan FranciscoUnited States
| | - Michael S Brainard
- Howard Hughes Medical InstituteChevy ChaseUnited States
- Department of Physiology, University of California, San FranciscoSan FranciscoUnited States
| |
Collapse
|
4
|
Rivera M, Edwards JA, Hauber ME, Woolley SMN. Machine learning and statistical classification of birdsong link vocal acoustic features with phylogeny. Sci Rep 2023; 13:7076. [PMID: 37127781 PMCID: PMC10151348 DOI: 10.1038/s41598-023-33825-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/20/2023] [Accepted: 04/19/2023] [Indexed: 05/03/2023] Open
Abstract
Birdsong is a longstanding model system for studying evolution and biodiversity. Here, we collected and analyzed high quality song recordings from seven species in the family Estrildidae. We measured the acoustic features of syllables and then used dimensionality reduction and machine learning classifiers to identify features that accurately assigned syllables to species. Species differences were captured by the first 3 principal components, corresponding to basic frequency, power distribution, and spectrotemporal features. We then identified the measured features underlying classification accuracy. We found that fundamental frequency, mean frequency, spectral flatness, and syllable duration were the most informative features for species identification. Next, we tested whether specific acoustic features of species' songs predicted phylogenetic distance. We found significant phylogenetic signal in syllable frequency features, but not in power distribution or spectrotemporal features. Results suggest that frequency features are more constrained by species' genetics than are other features, and are the best signal features for identifying species from song recordings. The absence of phylogenetic signal in power distribution and spectrotemporal features suggests that these song features are labile, reflecting learning processes and individual recognition.
Collapse
Affiliation(s)
- Moises Rivera
- Department of Psychology, Hunter College and the Graduate Center, City University of New York, New York, NY, 10065, USA
- Mortimer B. Zuckerman Mind, Brain, and Behavior Institute, Columbia University, New York, NY, 10027, USA
| | - Jacob A Edwards
- Mortimer B. Zuckerman Mind, Brain, and Behavior Institute, Columbia University, New York, NY, 10027, USA
- Department of Psychology, Columbia University, New York, NY, 10027, USA
| | - Mark E Hauber
- Department of Evolution, Ecology, and Behavior, School of Biological Sciences, University of Illinois at Urbana-Champaign, Urbana, IL, 61801, USA
| | - Sarah M N Woolley
- Mortimer B. Zuckerman Mind, Brain, and Behavior Institute, Columbia University, New York, NY, 10027, USA.
- Department of Psychology, Columbia University, New York, NY, 10027, USA.
- Zuckerman Institute at Columbia University, Jerome L. Greene Science Center, 3227 Broadway, L3.028, New York, NY, 10027, USA.
| |
Collapse
|
5
|
Lorenz C, Hao X, Tomka T, Rüttimann L, Hahnloser RH. Interactive extraction of diverse vocal units from a planar embedding without the need for prior sound segmentation. FRONTIERS IN BIOINFORMATICS 2023; 2:966066. [PMID: 36710910 PMCID: PMC9880044 DOI: 10.3389/fbinf.2022.966066] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/20/2022] [Accepted: 11/14/2022] [Indexed: 01/15/2023] Open
Abstract
Annotating and proofreading data sets of complex natural behaviors such as vocalizations are tedious tasks because instances of a given behavior need to be correctly segmented from background noise and must be classified with minimal false positive error rate. Low-dimensional embeddings have proven very useful for this task because they can provide a visual overview of a data set in which distinct behaviors appear in different clusters. However, low-dimensional embeddings introduce errors because they fail to preserve distances; and embeddings represent only objects of fixed dimensionality, which conflicts with vocalizations that have variable dimensions stemming from their variable durations. To mitigate these issues, we introduce a semi-supervised, analytical method for simultaneous segmentation and clustering of vocalizations. We define a given vocalization type by specifying pairs of high-density regions in the embedding plane of sound spectrograms, one region associated with vocalization onsets and the other with offsets. We demonstrate our two-neighborhood (2N) extraction method on the task of clustering adult zebra finch vocalizations embedded with UMAP. We show that 2N extraction allows the identification of short and long vocal renditions from continuous data streams without initially committing to a particular segmentation of the data. Also, 2N extraction achieves much lower false positive error rate than comparable approaches based on a single defining region. Along with our method, we present a graphical user interface (GUI) for visualizing and annotating data.
Collapse
Affiliation(s)
- Corinna Lorenz
- Institute of Neuroinformatics and Neuroscience Center Zurich, University of Zurich and ETH Zurich, Zurich, Switzerland,Université Paris-Saclay, CNRS, Institut des Neurosciences Paris-Saclay, Saclay, France
| | - Xinyu Hao
- Institute of Neuroinformatics and Neuroscience Center Zurich, University of Zurich and ETH Zurich, Zurich, Switzerland,Tianjin University, School of Electrical and Information Engineering, Tianjin, China
| | - Tomas Tomka
- Institute of Neuroinformatics and Neuroscience Center Zurich, University of Zurich and ETH Zurich, Zurich, Switzerland
| | - Linus Rüttimann
- Institute of Neuroinformatics and Neuroscience Center Zurich, University of Zurich and ETH Zurich, Zurich, Switzerland
| | - Richard H.R. Hahnloser
- Institute of Neuroinformatics and Neuroscience Center Zurich, University of Zurich and ETH Zurich, Zurich, Switzerland,*Correspondence: Richard H.R. Hahnloser,
| |
Collapse
|
6
|
Provost KL, Yang J, Carstens BC. The impacts of fine-tuning, phylogenetic distance, and sample size on big-data bioacoustics. PLoS One 2022; 17:e0278522. [PMID: 36477744 PMCID: PMC9728902 DOI: 10.1371/journal.pone.0278522] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/11/2022] [Accepted: 11/17/2022] [Indexed: 12/12/2022] Open
Abstract
Vocalizations in animals, particularly birds, are critically important behaviors that influence their reproductive fitness. While recordings of bioacoustic data have been captured and stored in collections for decades, the automated extraction of data from these recordings has only recently been facilitated by artificial intelligence methods. These have yet to be evaluated with respect to accuracy of different automation strategies and features. Here, we use a recently published machine learning framework to extract syllables from ten bird species ranging in their phylogenetic relatedness from 1 to 85 million years, to compare how phylogenetic relatedness influences accuracy. We also evaluate the utility of applying trained models to novel species. Our results indicate that model performance is best on conspecifics, with accuracy progressively decreasing as phylogenetic distance increases between taxa. However, we also find that the application of models trained on multiple distantly related species can improve the overall accuracy to levels near that of training and analyzing a model on the same species. When planning big-data bioacoustics studies, care must be taken in sample design to maximize sample size and minimize human labor without sacrificing accuracy.
Collapse
Affiliation(s)
- Kaiya L. Provost
- Department of Evolution, Ecology and Organismal Biology, The Ohio State University, Columbus, Ohio, United States of America
| | - Jiaying Yang
- Department of Evolution, Ecology and Organismal Biology, The Ohio State University, Columbus, Ohio, United States of America
| | - Bryan C. Carstens
- Department of Evolution, Ecology and Organismal Biology, The Ohio State University, Columbus, Ohio, United States of America
| |
Collapse
|
7
|
Brain Dp140 alters glutamatergic transmission and social behaviour in the mdx52 mouse model of Duchenne muscular dystrophy. Prog Neurobiol 2022; 216:102288. [PMID: 35654209 DOI: 10.1016/j.pneurobio.2022.102288] [Citation(s) in RCA: 11] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2021] [Revised: 04/06/2022] [Accepted: 05/25/2022] [Indexed: 12/23/2022]
Abstract
Duchenne muscular dystrophy (DMD) is a muscle disorder caused by DMD mutations and is characterized by neurobehavioural comorbidities due to dystrophin deficiency in the brain. The lack of Dp140, a dystrophin short isoform, is clinically associated with intellectual disability and autism spectrum disorders (ASDs), but its postnatal functional role is not well understood. To investigate synaptic function in the presence or absence of brain Dp140, we utilized two DMD mouse models, mdx23 and mdx52 mice, in which Dp140 is preserved or lacking, respectively. ASD-like behaviours were observed in pups and 8-week-old mdx52 mice lacking Dp140. Paired-pulse ratio of excitatory postsynaptic currents, glutamatergic vesicle number in basolateral amygdala neurons, and glutamatergic transmission in medial prefrontal cortex-basolateral amygdala projections were significantly reduced in mdx52 mice compared to those in wild-type and mdx23 mice. ASD-like behaviour and electrophysiological findings in mdx52 mice were ameliorated by restoration of Dp140 following intra-cerebroventricular injection of antisense oligonucleotide drug-induced exon 53 skipping or intra-basolateral amygdala administration of Dp140 mRNA-based drug. Our results implicate Dp140 in ASD-like behaviour via altered glutamatergic transmission in the basolateral amygdala of mdx52 mice.
Collapse
|
8
|
Sahu PK, Campbell KA, Oprea A, Phillmore LS, Sturdy CB. Comparing methodologies for classification of zebra finch distance calls. THE JOURNAL OF THE ACOUSTICAL SOCIETY OF AMERICA 2022; 151:3305. [PMID: 35649952 DOI: 10.1121/10.0011401] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/31/2022] [Accepted: 05/04/2022] [Indexed: 06/15/2023]
Abstract
Bioacoustic analysis has been used for a variety of purposes including classifying vocalizations for biodiversity monitoring and understanding mechanisms of cognitive processes. A wide range of statistical methods, including various automated methods, have been used to successfully classify vocalizations based on species, sex, geography, and individual. A comprehensive approach focusing on identifying acoustic features putatively involved in classification is required for the prediction of features necessary for discrimination in the real world. Here, we used several classification techniques, namely discriminant function analyses (DFAs), support vector machines (SVMs), and artificial neural networks (ANNs), for sex-based classification of zebra finch (Taeniopygia guttata) distance calls using acoustic features measured from spectrograms. We found that all three methods (DFAs, SVMs, and ANNs) correctly classified the calls to respective sex-based categories with high accuracy between 92 and 96%. Frequency modulation of ascending frequency, total duration, and end frequency of the distance call were the most predictive features underlying this classification in all of our models. Our results corroborate evidence of the importance of total call duration and frequency modulation in the classification of male and female distance calls. Moreover, we provide a methodological approach for bioacoustic classification problems using multiple statistical analyses.
Collapse
Affiliation(s)
- Prateek K Sahu
- Department of Psychology, University of Alberta, Edmonton, Alberta T6G 2R3, Canada
| | - Kimberley A Campbell
- Department of Psychology, University of Alberta, Edmonton, Alberta T6G 2R3, Canada
| | - Alexandra Oprea
- Department of Psychology and Neuroscience, Dalhousie University, Halifax, Nova Scotia B3H 4R2, Canada
| | - Leslie S Phillmore
- Department of Psychology and Neuroscience, Dalhousie University, Halifax, Nova Scotia B3H 4R2, Canada
| | - Christopher B Sturdy
- Department of Psychology, University of Alberta, Edmonton, Alberta T6G 2R3, Canada
| |
Collapse
|
9
|
Cohen Y, Nicholson DA, Sanchioni A, Mallaber EK, Skidanova V, Gardner TJ. Automated annotation of birdsong with a neural network that segments spectrograms. eLife 2022; 11:63853. [PMID: 35050849 PMCID: PMC8860439 DOI: 10.7554/elife.63853] [Citation(s) in RCA: 11] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/09/2020] [Accepted: 01/19/2022] [Indexed: 11/13/2022] Open
Abstract
Songbirds provide a powerful model system for studying sensory-motor learning. However, many analyses of birdsong require time-consuming, manual annotation of its elements, called syllables. Automated methods for annotation have been proposed, but these methods assume that audio can be cleanly segmented into syllables, or they require carefully tuning multiple statistical models. Here we present TweetyNet: a single neural network model that learns how to segment spectrograms of birdsong into annotated syllables. We show that TweetyNet mitigates limitations of methods that rely on segmented audio. We also show that TweetyNet performs well across multiple individuals from two species of songbirds, Bengalese finches and canaries. Lastly, we demonstrate that using TweetyNet we can accurately annotate very large datasets containing multiple days of song, and that these predicted annotations replicate key findings from behavioral studies. In addition, we provide open-source software to assist other researchers, and a large dataset of annotated canary song that can serve as a benchmark. We conclude that TweetyNet makes it possible to address a wide range of new questions about birdsong.
Collapse
Affiliation(s)
- Yarden Cohen
- Department of Brain Sciences, Weizmann Institute of Science, Rehovot, Israel
| | | | - Alexa Sanchioni
- Department of Biology, Boston University, Boston, United States
| | | | | | - Timothy J Gardner
- Phil and Penny Knight Campus for Accelerating Scientific Impact, University of Oregon, Eugene, United States
| |
Collapse
|
10
|
Hildebrand JA, Frasier KE, Helble TA, Roch MA. Performance metrics for marine mammal signal detection and classification. THE JOURNAL OF THE ACOUSTICAL SOCIETY OF AMERICA 2022; 151:414. [PMID: 35105012 DOI: 10.1121/10.0009270] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/16/2021] [Accepted: 12/15/2021] [Indexed: 06/14/2023]
Abstract
Automatic algorithms for the detection and classification of sound are essential to the analysis of acoustic datasets with long duration. Metrics are needed to assess the performance characteristics of these algorithms. Four metrics for performance evaluation are discussed here: receiver-operating-characteristic (ROC) curves, detection-error-trade-off (DET) curves, precision-recall (PR) curves, and cost curves. These metrics were applied to the generalized power law detector for blue whale D calls [Helble, Ierley, D'Spain, Roch, and Hildebrand (2012). J. Acoust. Soc. Am. 131(4), 2682-2699] and the click-clustering neural-net algorithm for Cuvier's beaked whale echolocation click detection [Frasier, Roch, Soldevilla, Wiggins, Garrison, and Hildebrand (2017). PLoS Comp. Biol. 13(12), e1005823] using data prepared for the 2015 Detection, Classification, Localization and Density Estimation Workshop. Detection class imbalance, particularly the situation of rare occurrence, is common for long-term passive acoustic monitoring datasets and is a factor in the performance of ROC and DET curves with regard to the impact of false positive detections. PR curves overcome this shortcoming when calculated for individual detections and do not rely on the reporting of true negatives. Cost curves provide additional insight on the effective operating range for the detector based on the a priori probability of occurrence. Use of more than a single metric is helpful in understanding the performance of a detection algorithm.
Collapse
Affiliation(s)
- John A Hildebrand
- Scripps Institution of Oceanography, University of California San Diego, La Jolla, California 92093, USA
| | - Kaitlin E Frasier
- Scripps Institution of Oceanography, University of California San Diego, La Jolla, California 92093, USA
| | - Tyler A Helble
- Naval Information Warfare Center Pacific, San Diego, California 92152, USA
| | - Marie A Roch
- Department of Computer Science, San Diego State University, San Diego, California 92182, USA
| |
Collapse
|
11
|
Measuring context dependency in birdsong using artificial neural networks. PLoS Comput Biol 2021; 17:e1009707. [PMID: 34962915 PMCID: PMC8746767 DOI: 10.1371/journal.pcbi.1009707] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/29/2021] [Revised: 01/10/2022] [Accepted: 12/01/2021] [Indexed: 11/19/2022] Open
Abstract
Context dependency is a key feature in sequential structures of human language, which requires reference between words far apart in the produced sequence. Assessing how long the past context has an effect on the current status provides crucial information to understand the mechanism for complex sequential behaviors. Birdsongs serve as a representative model for studying the context dependency in sequential signals produced by non-human animals, while previous reports were upper-bounded by methodological limitations. Here, we newly estimated the context dependency in birdsongs in a more scalable way using a modern neural-network-based language model whose accessible context length is sufficiently long. The detected context dependency was beyond the order of traditional Markovian models of birdsong, but was consistent with previous experimental investigations. We also studied the relation between the assumed/auto-detected vocabulary size of birdsong (i.e., fine- vs. coarse-grained syllable classifications) and the context dependency. It turned out that the larger vocabulary (or the more fine-grained classification) is assumed, the shorter context dependency is detected.
Collapse
|
12
|
Sainburg T, Gentner TQ. Toward a Computational Neuroethology of Vocal Communication: From Bioacoustics to Neurophysiology, Emerging Tools and Future Directions. Front Behav Neurosci 2021; 15:811737. [PMID: 34987365 PMCID: PMC8721140 DOI: 10.3389/fnbeh.2021.811737] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/09/2021] [Accepted: 11/29/2021] [Indexed: 11/23/2022] Open
Abstract
Recently developed methods in computational neuroethology have enabled increasingly detailed and comprehensive quantification of animal movements and behavioral kinematics. Vocal communication behavior is well poised for application of similar large-scale quantification methods in the service of physiological and ethological studies. This review describes emerging techniques that can be applied to acoustic and vocal communication signals with the goal of enabling study beyond a small number of model species. We review a range of modern computational methods for bioacoustics, signal processing, and brain-behavior mapping. Along with a discussion of recent advances and techniques, we include challenges and broader goals in establishing a framework for the computational neuroethology of vocal communication.
Collapse
Affiliation(s)
- Tim Sainburg
- Department of Psychology, University of California, San Diego, La Jolla, CA, United States
- Center for Academic Research & Training in Anthropogeny, University of California, San Diego, La Jolla, CA, United States
| | - Timothy Q. Gentner
- Department of Psychology, University of California, San Diego, La Jolla, CA, United States
- Neurosciences Graduate Program, University of California, San Diego, La Jolla, CA, United States
- Neurobiology Section, Division of Biological Sciences, University of California, San Diego, La Jolla, CA, United States
- Kavli Institute for Brain and Mind, University of California, San Diego, La Jolla, CA, United States
| |
Collapse
|
13
|
Comparing recurrent convolutional neural networks for large scale bird species classification. Sci Rep 2021; 11:17085. [PMID: 34429468 PMCID: PMC8385065 DOI: 10.1038/s41598-021-96446-w] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/25/2021] [Accepted: 08/10/2021] [Indexed: 11/08/2022] Open
Abstract
We present a deep learning approach towards the large-scale prediction and analysis of bird acoustics from 100 different bird species. We use spectrograms constructed on bird audio recordings from the Cornell Bird Challenge (CBC)2020 dataset, which includes recordings of multiple and potentially overlapping bird vocalizations with background noise. Our experiments show that a hybrid modeling approach that involves a Convolutional Neural Network (CNN) for learning the representation for a slice of the spectrogram, and a Recurrent Neural Network (RNN) for the temporal component to combine across time-points leads to the most accurate model on this dataset. We show results on a spectrum of models ranging from stand-alone CNNs to hybrid models of various types obtained by combining CNNs with other CNNs or RNNs of the following types: Long Short-Term Memory (LSTM) networks, Gated Recurrent Units (GRU), and Legendre Memory Units (LMU). The best performing model achieves an average accuracy of 67% over the 100 different bird species, with the highest accuracy of 90% for the bird species, Red crossbill. We further analyze the learned representations visually and find them to be intuitive, where we find that related bird species are clustered close together. We present a novel way to empirically interpret the representations learned by the LMU-based hybrid model which shows how memory channel patterns change over time with the changes seen in the spectrograms.
Collapse
|
14
|
Inda M, Hotta K, Oka K. High responsiveness of auditory neurons to specific combination of acoustic features in female songbirds. Eur J Neurosci 2020; 53:1412-1427. [PMID: 33205482 DOI: 10.1111/ejn.15047] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/11/2020] [Revised: 11/11/2020] [Accepted: 11/11/2020] [Indexed: 11/26/2022]
Abstract
Zebra finch (Taeniopygia guttata) is a songbird species in which males sing their unique songs to attract females who then select their preferred male. Acoustic features in the songs of individual males are important features for female auditory perception. While the male of this species is a classic model of vocal production, it has been little known about auditory processing in female. In the higher auditory brain regions, the caudomedial mesopallium (CMM) and nidopallium (NCM) contribute to female's sound recognition, we, therefore, extracted acoustic features that induce neural activities with high detection power on both regions in female finches. A multiple linear regression analysis revealed that neurons were sensitive to mean frequency and Wiener entropy. In addition, we performed an experiment with modified artificial songs and harmonic songs to directly investigate neural responsiveness for deriving further evidence for the contribution of these two acoustic features. Finally, we illustrated a specific ratio combining these two acoustic features that showed highest sensitivity to neural responsiveness, and we found that properties of sensitivity are different between CMM and NCM. Our results indicate that the mixture of the two acoustic features with the specific ratio is important in the higher auditory regions of female songbirds, and these two regions have differences in encoding for sensitivity to these acoustic features.
Collapse
Affiliation(s)
- Masahiro Inda
- Department of Biosciences and Informatics, Faculty of Science and Technology, Keio University, Yokohama, Japan
| | - Kohji Hotta
- Department of Biosciences and Informatics, Faculty of Science and Technology, Keio University, Yokohama, Japan
| | - Kotaro Oka
- Department of Biosciences and Informatics, Faculty of Science and Technology, Keio University, Yokohama, Japan.,Graduate Institute of Medicine, College of Medicine, Kaohsiung Medical University, Kaohsiung, Taiwan.,Waseda Research Institute for Science and Engineering, Waseda University, Shinjuku, Tokyo, Japan
| |
Collapse
|
15
|
Tachibana RO, Kanno K, Okabe S, Kobayasi KI, Okanoya K. USVSEG: A robust method for segmentation of ultrasonic vocalizations in rodents. PLoS One 2020; 15:e0228907. [PMID: 32040540 PMCID: PMC7010259 DOI: 10.1371/journal.pone.0228907] [Citation(s) in RCA: 29] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/11/2019] [Accepted: 01/27/2020] [Indexed: 11/18/2022] Open
Abstract
Rodents' ultrasonic vocalizations (USVs) provide useful information for assessing their social behaviors. Despite previous efforts in classifying subcategories of time-frequency patterns of USV syllables to study their functional relevance, methods for detecting vocal elements from continuously recorded data have remained sub-optimal. Here, we propose a novel procedure for detecting USV segments in continuous sound data containing background noise recorded during the observation of social behavior. The proposed procedure utilizes a stable version of the sound spectrogram and additional signal processing for better separation of vocal signals by reducing the variation of the background noise. Our procedure also provides precise time tracking of spectral peaks within each syllable. We demonstrated that this procedure can be applied to a variety of USVs obtained from several rodent species. Performance tests showed this method had greater accuracy in detecting USV syllables than conventional detection methods.
Collapse
Affiliation(s)
- Ryosuke O. Tachibana
- Department of Life Sciences, Graduate School of Arts & Sciences, The University of Tokyo, Tokyo, Japan
- * E-mail:
| | - Kouta Kanno
- Laboratory of Neuroscience, Course of Psychology, Department of Humanities, Faculty of Law, Economics and the Humanities, Kagoshima University, Kagoshima, Japan
| | - Shota Okabe
- Division of Brain and Neurophysiology, Department of Physiology, Jichi Medical University, Tochigi, Japan
| | - Kohta I. Kobayasi
- Graduate School of Life and Medical Sciences, Doshisha University, Kyoto, Japan
| | - Kazuo Okanoya
- Department of Life Sciences, Graduate School of Arts & Sciences, The University of Tokyo, Tokyo, Japan
| |
Collapse
|
16
|
Pereira TD, Aldarondo DE, Willmore L, Kislin M, Wang SSH, Murthy M, Shaevitz JW. Fast animal pose estimation using deep neural networks. Nat Methods 2018; 16:117-125. [PMID: 30573820 DOI: 10.1038/s41592-018-0234-5] [Citation(s) in RCA: 272] [Impact Index Per Article: 45.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/25/2018] [Accepted: 10/31/2018] [Indexed: 02/06/2023]
Abstract
The need for automated and efficient systems for tracking full animal pose has increased with the complexity of behavioral data and analyses. Here we introduce LEAP (LEAP estimates animal pose), a deep-learning-based method for predicting the positions of animal body parts. This framework consists of a graphical interface for labeling of body parts and training the network. LEAP offers fast prediction on new data, and training with as few as 100 frames results in 95% of peak performance. We validated LEAP using videos of freely behaving fruit flies and tracked 32 distinct points to describe the pose of the head, body, wings and legs, with an error rate of <3% of body length. We recapitulated reported findings on insect gait dynamics and demonstrated LEAP's applicability for unsupervised behavioral classification. Finally, we extended the method to more challenging imaging situations and videos of freely moving mice.
Collapse
Affiliation(s)
- Talmo D Pereira
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA
| | - Diego E Aldarondo
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA.,Program in Neuroscience, Harvard University, Cambridge, MA, USA
| | - Lindsay Willmore
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA
| | - Mikhail Kislin
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA
| | - Samuel S-H Wang
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA.,Department of Molecular Biology, Princeton University, Princeton, NJ, USA
| | - Mala Murthy
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA. .,Department of Molecular Biology, Princeton University, Princeton, NJ, USA.
| | - Joshua W Shaevitz
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA. .,Lewis-Sigler Institute for Integrative Genomics, Princeton University, Princeton, NJ, USA. .,Department of Physics, Princeton University, Princeton, NJ, USA.
| |
Collapse
|
17
|
Wilson RP, Holton MD, Virgilio A, Williams H, Shepard ELC, Lambertucci S, Quintana F, Sala JE, Balaji B, Lee ES, Srivastava M, Scantlebury DM, Duarte CM. Give the machine a hand: A Boolean time‐based decision‐tree template for rapidly finding animal behaviours in multisensor data. Methods Ecol Evol 2018. [DOI: 10.1111/2041-210x.13069] [Citation(s) in RCA: 20] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/01/2022]
Affiliation(s)
- Rory P. Wilson
- Department of BiosciencesCollege of ScienceSwansea University Swansea UK
| | - Mark D. Holton
- Department of Computing ScienceCollege of ScienceSwansea University Swansea UK
| | - Agustina Virgilio
- Grupo de Biología de la ConservaciónLaboratorio EcotonoINIBIOMA (CONICET‐Universidad Nacional del Comahue) Bariloche Argentina
- Grupo de Ecología CuantitativaINIBIOMA (CONICET‐Universidad Nacional del Comahue) Bariloche Argentina
| | - Hannah Williams
- Department of BiosciencesCollege of ScienceSwansea University Swansea UK
| | | | - Sergio Lambertucci
- Grupo de Biología de la ConservaciónLaboratorio EcotonoINIBIOMA (CONICET‐Universidad Nacional del Comahue) Bariloche Argentina
| | - Flavio Quintana
- Instituto de Biologia de Organismos Marinos IBIOMAR‐CONICET (9120) Puerto Madryn Chubut Argentina
| | - Juan E. Sala
- Instituto de Biologia de Organismos Marinos IBIOMAR‐CONICET (9120) Puerto Madryn Chubut Argentina
| | - Bharathan Balaji
- Department of Electrical and Computer EngineeringUniversity of California, Los Angeles Los Angeles California
| | - Eun Sun Lee
- Department of Electrical and Computer EngineeringUniversity of California, Los Angeles Los Angeles California
| | - Mani Srivastava
- Department of Electrical and Computer EngineeringUniversity of California, Los Angeles Los Angeles California
| | - D. Michael Scantlebury
- School of Biological SciencesInstitute for Global Food SecurityQueen's University Belfast Belfast UK
| | - Carlos M. Duarte
- Red Sea Research CentreKing Abdullah University of Science and Technology Thuwal Saudi Arabia
| |
Collapse
|
18
|
Knight EC, Bayne EM. Classification threshold and training data affect the quality and utility of focal species data processed with automated audio-recognition software. BIOACOUSTICS 2018. [DOI: 10.1080/09524622.2018.1503971] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/28/2022]
Affiliation(s)
- Elly C. Knight
- Department of Biological Science, Bioacoustic Unit, Edmonton, Alberta, Canada
| | - Erin M. Bayne
- Department of Biological Science, Bioacoustic Unit, Edmonton, Alberta, Canada
| |
Collapse
|
19
|
Qian K, Zhang Z, Baird A, Schuller B. Active learning for bird sound classification via a kernel-based extreme learning machine. THE JOURNAL OF THE ACOUSTICAL SOCIETY OF AMERICA 2017; 142:1796. [PMID: 29092546 DOI: 10.1121/1.5004570] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/07/2023]
Abstract
In recent years, research fields, including ecology, bioacoustics, signal processing, and machine learning, have made bird sound recognition a part of their focus. This has led to significant advancements within the field of ornithology, such as improved understanding of evolution, local biodiversity, mating rituals, and even the implications and realities associated to climate change. The volume of unlabeled bird sound data is now overwhelming, and comparatively little exploration is being made into methods for how best to handle them. In this study, two active learning (AL) methods are proposed, sparse-instance-based active learning (SI-AL), and least-confidence-score-based active learning (LCS-AL), both effectively reducing the need for expert human annotation. To both of these AL paradigms, a kernel-based extreme learning machine (KELM) is then integrated, and a comparison is made to the conventional support vector machine (SVM). Experimental results demonstrate that, when the classifier capacity is improved from an unweighted average recall of 60%-80%, KELM can outperform SVM even when a limited proportion of human annotations are used from the pool of data in both cases of SI-AL (minimum 34.5% vs minimum 59.0%) and LCS-AL (minimum 17.3% vs minimum 28.4%).
Collapse
Affiliation(s)
- Kun Qian
- Machine Intelligence and Signal Processing Group, Chair of Human-Machine Communication, Technische Universität München, Arcisstr. 21, Munich 80333, Germany
| | - Zixing Zhang
- Chair of Complex and Intelligent Systems, University of Passau, Innstr. 43, Passau 94032, Germany
| | - Alice Baird
- Chair of Complex and Intelligent Systems, University of Passau, Innstr. 43, Passau 94032, Germany
| | - Björn Schuller
- GLAM-Group on Language, Audio and Music, Department of Computing, Imperial College London, 180 Queens' Gate, Huxley Building, London SW7 2AZ, United Kingdom
| |
Collapse
|
20
|
Ludeña-Choez J, Quispe-Soncco R, Gallardo-Antolín A. Bird sound spectrogram decomposition through Non-Negative Matrix Factorization for the acoustic classification of bird species. PLoS One 2017. [PMID: 28628630 PMCID: PMC5476267 DOI: 10.1371/journal.pone.0179403] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022] Open
Abstract
Feature extraction for Acoustic Bird Species Classification (ABSC) tasks has traditionally been based on parametric representations that were specifically developed for speech signals, such as Mel Frequency Cepstral Coefficients (MFCC). However, the discrimination capabilities of these features for ABSC could be enhanced by accounting for the vocal production mechanisms of birds, and, in particular, the spectro-temporal structure of bird sounds. In this paper, a new front-end for ABSC is proposed that incorporates this specific information through the non-negative decomposition of bird sound spectrograms. It consists of the following two different stages: short-time feature extraction and temporal feature integration. In the first stage, which aims at providing a better spectral representation of bird sounds on a frame-by-frame basis, two methods are evaluated. In the first method, cepstral-like features (NMF_CC) are extracted by using a filter bank that is automatically learned by means of the application of Non-Negative Matrix Factorization (NMF) on bird audio spectrograms. In the second method, the features are directly derived from the activation coefficients of the spectrogram decomposition as performed through NMF (H_CC). The second stage summarizes the most relevant information contained in the short-time features by computing several statistical measures over long segments. The experiments show that the use of NMF_CC and H_CC in conjunction with temporal integration significantly improves the performance of a Support Vector Machine (SVM)-based ABSC system with respect to conventional MFCC.
Collapse
Affiliation(s)
- Jimmy Ludeña-Choez
- Facultad de Ingeniería y Computación, Centro de Investigación en Electrónica y Telecomunicaciones (CIET), Grupo de Investigación en Ciencia y Tecnología de Materiales (CITEM), Universidad Católica San Pablo, Arequipa, Perú
- Department of Signal Theory and Communications, Universidad Carlos III de Madrid, Madrid, Spain
- * E-mail:
| | - Raisa Quispe-Soncco
- Facultad de Ingeniería y Computación, Centro de Investigación en Electrónica y Telecomunicaciones (CIET), Grupo de Investigación en Ciencia y Tecnología de Materiales (CITEM), Universidad Católica San Pablo, Arequipa, Perú
| | | |
Collapse
|
21
|
Matsubayashi S, Suzuki R, Saito F, Murate T, Masuda T, Yamamoto K, Kojima R, Nakadai K, Okuno HG. Acoustic Monitoring of the Great Reed Warbler Using Multiple Microphone Arrays and Robot Audition. JOURNAL OF ROBOTICS AND MECHATRONICS 2017. [DOI: 10.20965/jrm.2017.p0224] [Citation(s) in RCA: 16] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
[abstFig src='/00290001/21.jpg' width='300' text='Spatial distribution pattern of the observed birds and localized sounds' ] This paper reports the results of our field test of HARKBird, a portable system that consists of robot audition, a laptop PC, and omnidirectional microphone arrays. We assessed its localization accuracy to monitor songs of the great reed warbler (Acrocephalus arundinaceus) in time and two-dimensional space by comparing locational and temporal data collected by human observers and HARKBird. Our analysis revealed that stationarity of the singing individual affected the spatial accuracy. Temporally, HARKBird successfully captured the exact song duration in seconds, which cannot be easily achieved by human observers. The data derived from HARKBird suggest that one of the warbler males dominated the sound space. Given the assumption that the cost of the singing activity is represented by song duration in relation to the total recording session, this particular male paid a higher cost of singing, possibly to win the territory of best quality. Overall, this study demonstrated the high potential of HARKBird as an effective alternative to the point count method to survey bird songs in the field.
Collapse
|
22
|
Tachibana RO, Takahasi M, Hessler NA, Okanoya K. Maturation-dependent control of vocal temporal plasticity in a songbird. Dev Neurobiol 2017; 77:995-1006. [PMID: 28188699 DOI: 10.1002/dneu.22487] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/03/2016] [Revised: 02/07/2017] [Accepted: 02/07/2017] [Indexed: 11/09/2022]
Abstract
Birdsong is a unique model to address learning mechanisms of the timing control of sequential behaviors, with characteristic temporal structures consisting of serial sequences of brief vocal elements (syllables) and silent intervals (gaps). Understanding the neural mechanisms for plasticity of such sequential behavior should be aided by characterization of its developmental changes. Here, we assessed the level of acute vocal plasticity between young and adult Bengalese finches, and also quantified developmental change in variability of temporal structure. Acute plasticity was tested by delivering aversive noise bursts contingent on duration of a target gap, such that birds could avoid the noise by modifying their song. We found that temporal variability of song features decreased with birds' maturation. Noise-avoidance experiments demonstrated that maximal changes of gap durations were larger in young that in adult birds. After these young birds matured, the maximal change decreased to a similar level as adults. The variability of these target gaps also decreased as the birds matured. Such parallel changes suggest that the level of acute temporal plasticity could be predicted from ongoing temporal variability. Further, we found that young birds gradually began to stop their song at the target gap and restart from the introductory part of song, whereas adults did not. According to a synaptic chain model for timing sequence generation in premotor nuclei, adult learning would be interpreted as adaptive changes in conduction delays between chain-to-chain connections, whereas the learning of young birds could mainly depend on changes of the connections. © 2017 Wiley Periodicals, Inc. Develop Neurobiol 77: 995-1006, 2017.
Collapse
Affiliation(s)
- Ryosuke O Tachibana
- Department of Life Sciences, Graduate School of Arts and Sciences, The University of Tokyo, Tokyo, Japan
| | - Miki Takahasi
- Department of Life Sciences, Graduate School of Arts and Sciences, The University of Tokyo, Tokyo, Japan
| | - Neal A Hessler
- Department of Life Sciences, Graduate School of Arts and Sciences, The University of Tokyo, Tokyo, Japan
| | - Kazuo Okanoya
- Department of Life Sciences, Graduate School of Arts and Sciences, The University of Tokyo, Tokyo, Japan.,Cognition and Behavior Joint Laboratory, RIKEN Brain Science Institute, Saitama, Japan
| |
Collapse
|
23
|
Shiba S, Okanoya K, Tachibana RO. Effects of background noise on acoustic characteristics of Bengalese finch songs. THE JOURNAL OF THE ACOUSTICAL SOCIETY OF AMERICA 2016; 140:4039. [PMID: 28040000 DOI: 10.1121/1.4968577] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/06/2023]
Abstract
Online regulation of vocalization in response to auditory feedback is one of the essential issues for vocal communication. One such audio-vocal interaction is the Lombard effect, an involuntary increase in vocal amplitude in response to the presence of background noise. Along with vocal amplitude, other acoustic characteristics, including fundamental frequency (F0), also change in some species. Bengalese finches (Lonchura striata var. domestica) are a suitable model for comparative, ethological, and neuroscientific studies on audio-vocal interaction because they require real-time auditory feedback of their own songs to maintain normal singing. Here, the changes in amplitude and F0 with a focus on the distinct song elements (i.e., notes) of Bengalese finches under noise presentation are demonstrated. To accurately analyze these acoustic characteristics, two different bandpass-filtered noises at two levels of sound intensity were used. The results confirmed that the Lombard effect occurs at the note level of Bengalese finch song. Further, individually specific modes of changes in F0 are shown. These behavioral changes suggested the vocal control mechanisms on which the auditory feedback is based have a predictable effect on amplitude, but complex spectral effects on individual note production.
Collapse
Affiliation(s)
- Shintaro Shiba
- Department of Life Sciences, Graduate School of Arts and Sciences, University of Tokyo, 3-8-1, Komaba, Meguro-ku, Tokyo, 153-8902, Japan
| | - Kazuo Okanoya
- Department of Life Sciences, Graduate School of Arts and Sciences, University of Tokyo, 3-8-1, Komaba, Meguro-ku, Tokyo, 153-8902, Japan
| | - Ryosuke O Tachibana
- Department of Life Sciences, Graduate School of Arts and Sciences, University of Tokyo, 3-8-1, Komaba, Meguro-ku, Tokyo, 153-8902, Japan
| |
Collapse
|
24
|
Koumura T, Okanoya K. Automatic Recognition of Element Classes and Boundaries in the Birdsong with Variable Sequences. PLoS One 2016; 11:e0159188. [PMID: 27442240 PMCID: PMC4956110 DOI: 10.1371/journal.pone.0159188] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/29/2015] [Accepted: 06/28/2016] [Indexed: 11/18/2022] Open
Abstract
Researches on sequential vocalization often require analysis of vocalizations in long continuous sounds. In such studies as developmental ones or studies across generations in which days or months of vocalizations must be analyzed, methods for automatic recognition would be strongly desired. Although methods for automatic speech recognition for application purposes have been intensively studied, blindly applying them for biological purposes may not be an optimal solution. This is because, unlike human speech recognition, analysis of sequential vocalizations often requires accurate extraction of timing information. In the present study we propose automated systems suitable for recognizing birdsong, one of the most intensively investigated sequential vocalizations, focusing on the three properties of the birdsong. First, a song is a sequence of vocal elements, called notes, which can be grouped into categories. Second, temporal structure of birdsong is precisely controlled, meaning that temporal information is important in song analysis. Finally, notes are produced according to certain probabilistic rules, which may facilitate the accurate song recognition. We divided the procedure of song recognition into three sub-steps: local classification, boundary detection, and global sequencing, each of which corresponds to each of the three properties of birdsong. We compared the performances of several different ways to arrange these three steps. As results, we demonstrated a hybrid model of a deep convolutional neural network and a hidden Markov model was effective. We propose suitable arrangements of methods according to whether accurate boundary detection is needed. Also we designed the new measure to jointly evaluate the accuracy of note classification and boundary detection. Our methods should be applicable, with small modification and tuning, to the songs in other species that hold the three properties of the sequential vocalization.
Collapse
Affiliation(s)
- Takuya Koumura
- Department of Life Sciences, Graduate School of Arts and Sciences, The University of Tokyo, Tokyo, Japan
- Research Fellow of Japan Society for the Promotion of Science, Tokyo, Japan
| | - Kazuo Okanoya
- Department of Life Sciences, Graduate School of Arts and Sciences, The University of Tokyo, Tokyo, Japan
- Cognition and Behavior Joint Laboratory, RIKEN Brain Science Institute, Saitama, Japan
| |
Collapse
|
25
|
Fukushima M, Doyle AM, Mullarkey MP, Mishkin M, Averbeck BB. Distributed acoustic cues for caller identity in macaque vocalization. ROYAL SOCIETY OPEN SCIENCE 2015; 2:150432. [PMID: 27019727 PMCID: PMC4806230 DOI: 10.1098/rsos.150432] [Citation(s) in RCA: 8] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 08/25/2015] [Accepted: 11/23/2015] [Indexed: 06/05/2023]
Abstract
Individual primates can be identified by the sound of their voice. Macaques have demonstrated an ability to discern conspecific identity from a harmonically structured 'coo' call. Voice recognition presumably requires the integrated perception of multiple acoustic features. However, it is unclear how this is achieved, given considerable variability across utterances. Specifically, the extent to which information about caller identity is distributed across multiple features remains elusive. We examined these issues by recording and analysing a large sample of calls from eight macaques. Single acoustic features, including fundamental frequency, duration and Weiner entropy, were informative but unreliable for the statistical classification of caller identity. A combination of multiple features, however, allowed for highly accurate caller identification. A regularized classifier that learned to identify callers from the modulation power spectrum of calls found that specific regions of spectral-temporal modulation were informative for caller identification. These ranges are related to acoustic features such as the call's fundamental frequency and FM sweep direction. We further found that the low-frequency spectrotemporal modulation component contained an indexical cue of the caller body size. Thus, cues for caller identity are distributed across identifiable spectrotemporal components corresponding to laryngeal and supralaryngeal components of vocalizations, and the integration of those cues can enable highly reliable caller identification. Our results demonstrate a clear acoustic basis by which individual macaque vocalizations can be recognized.
Collapse
|
26
|
Tachibana RO, Koumura T, Okanoya K. Variability in the temporal parameters in the song of the Bengalese finch (Lonchura striata var. domestica). J Comp Physiol A Neuroethol Sens Neural Behav Physiol 2015; 201:1157-68. [PMID: 26512015 DOI: 10.1007/s00359-015-1046-z] [Citation(s) in RCA: 14] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/13/2015] [Revised: 10/12/2015] [Accepted: 10/14/2015] [Indexed: 11/27/2022]
Abstract
Birdsong provides a unique model for studying the control mechanisms of complex sequential behaviors. The present study aimed to demonstrate that multiple factors affect temporal control in the song production. We analyzed the song of Bengalese finches in various time ranges to address factors that affected the duration of acoustic elements (notes) and silent intervals (gaps). The gaps showed more jitter across song renditions than did notes. Gaps had longer duration in branching points of song sequence than in stereotypic transitions, and the duration of a gap was correlated with the duration of the note that preceded the gap. When looking at the variation among song renditions, we found notable factors in three time ranges: within-day drift, within-bout changes, and local jitter. Note durations shortened over time from morning to evening. Within each song bout note durations lengthened as singing progressed, while gap durations lengthened only during the late part of song bout. Further analysis after removing these drift factors confirmed that the jitter remained in local song sequences. These results suggest distinct sources of temporal variability exist at multiple levels on the basis of this note-gap relationship, and that song comprised a mixture of these sources.
Collapse
Affiliation(s)
- Ryosuke O Tachibana
- Department of Life Sciences, Graduate School of Arts and Sciences, The University of Tokyo, 3-8-1 Komaba, Meguro-ku, Tokyo, 153-8902, Japan.,Research Fellow of Japan Society for the Promotion of Science, Tokyo, Japan
| | - Takuya Koumura
- Department of Life Sciences, Graduate School of Arts and Sciences, The University of Tokyo, 3-8-1 Komaba, Meguro-ku, Tokyo, 153-8902, Japan.,Research Fellow of Japan Society for the Promotion of Science, Tokyo, Japan
| | - Kazuo Okanoya
- Department of Life Sciences, Graduate School of Arts and Sciences, The University of Tokyo, 3-8-1 Komaba, Meguro-ku, Tokyo, 153-8902, Japan. .,Cognition and Behavior Joint Laboratory, RIKEN Brain Science Institute, Saitama, Japan.
| |
Collapse
|
27
|
Burkett ZD, Day NF, Peñagarikano O, Geschwind DH, White SA. VoICE: A semi-automated pipeline for standardizing vocal analysis across models. Sci Rep 2015; 5:10237. [PMID: 26018425 PMCID: PMC4446892 DOI: 10.1038/srep10237] [Citation(s) in RCA: 54] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/10/2014] [Accepted: 04/07/2015] [Indexed: 11/09/2022] Open
Abstract
The study of vocal communication in animal models provides key insight to the neurogenetic basis for speech and communication disorders. Current methods for vocal analysis suffer from a lack of standardization, creating ambiguity in cross-laboratory and cross-species comparisons. Here, we present VoICE (Vocal Inventory Clustering Engine), an approach to grouping vocal elements by creating a high dimensionality dataset through scoring spectral similarity between all vocalizations within a recording session. This dataset is then subjected to hierarchical clustering, generating a dendrogram that is pruned into meaningful vocalization “types” by an automated algorithm. When applied to birdsong, a key model for vocal learning, VoICE captures the known deterioration in acoustic properties that follows deafening, including altered sequencing. In a mammalian neurodevelopmental model, we uncover a reduced vocal repertoire of mice lacking the autism susceptibility gene, Cntnap2. VoICE will be useful to the scientific community as it can standardize vocalization analyses across species and laboratories.
Collapse
Affiliation(s)
- Zachary D Burkett
- 1] Department of Integrative Biology &Physiology, University of California, Los Angeles, California 90095 [2] Interdepartmental Program in Molecular, Cellular, &Integrative Physiology, University of California, Los Angeles, California 90095
| | - Nancy F Day
- Department of Integrative Biology &Physiology, University of California, Los Angeles, California 90095
| | - Olga Peñagarikano
- 1] Program in Neurogenetics, Department of Neurology, David Geffen School of Medicine, University of California, Los Angeles, California 90095 [2] Center for Autism Research &Treatment, Semel Institute for Neuroscience &Human Behavior, University of California, Los Angeles, California 90095
| | - Daniel H Geschwind
- 1] Program in Neurogenetics, Department of Neurology, David Geffen School of Medicine, University of California, Los Angeles, California 90095 [2] Center for Autism Research &Treatment, Semel Institute for Neuroscience &Human Behavior, University of California, Los Angeles, California 90095 [3] Center for Neurobehavioral Genetics, Semel Institute for Neuroscience &Human Behavior, University of California, Los Angeles, California 90095
| | - Stephanie A White
- 1] Department of Integrative Biology &Physiology, University of California, Los Angeles, California 90095 [2] Interdepartmental Program in Molecular, Cellular, &Integrative Physiology, University of California, Los Angeles, California 90095
| |
Collapse
|
28
|
Ho TW, Huang CW, Lin CM, Lai F, Ding JJ, Ho YL, Hung CS. A telesurveillance system with automatic electrocardiogram interpretation based on support vector machine and rule-based processing. JMIR Med Inform 2015; 3:e21. [PMID: 25953306 PMCID: PMC4440896 DOI: 10.2196/medinform.4397] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2015] [Accepted: 04/03/2015] [Indexed: 01/19/2023] Open
Abstract
Background Telehealth care is a global trend affecting clinical practice around the world. To mitigate the workload of health professionals and provide ubiquitous health care, a comprehensive surveillance system with value-added services based on information technologies must be established. Objective We conducted this study to describe our proposed telesurveillance system designed for monitoring and classifying electrocardiogram (ECG) signals and to evaluate the performance of ECG classification. Methods We established a telesurveillance system with an automatic ECG interpretation mechanism. The system included: (1) automatic ECG signal transmission via telecommunication, (2) ECG signal processing, including noise elimination, peak estimation, and feature extraction, (3) automatic ECG interpretation based on the support vector machine (SVM) classifier and rule-based processing, and (4) display of ECG signals and their analyzed results. We analyzed 213,420 ECG signals that were diagnosed by cardiologists as the gold standard to verify the classification performance. Results In the clinical ECG database from the Telehealth Center of the National Taiwan University Hospital (NTUH), the experimental results showed that the ECG classifier yielded a specificity value of 96.66% for normal rhythm detection, a sensitivity value of 98.50% for disease recognition, and an accuracy value of 81.17% for noise detection. For the detection performance of specific diseases, the recognition model mainly generated sensitivity values of 92.70% for atrial fibrillation, 89.10% for pacemaker rhythm, 88.60% for atrial premature contraction, 72.98% for T-wave inversion, 62.21% for atrial flutter, and 62.57% for first-degree atrioventricular block. Conclusions Through connected telehealth care devices, the telesurveillance system, and the automatic ECG interpretation system, this mechanism was intentionally designed for continuous decision-making support and is reliable enough to reduce the need for face-to-face diagnosis. With this value-added service, the system could widely assist physicians and other health professionals with decision making in clinical practice. The system will be very helpful for the patient who suffers from cardiac disease, but for whom it is inconvenient to go to the hospital very often.
Collapse
Affiliation(s)
- Te-Wei Ho
- National Taiwan University, Graduate Institute of Biomedical Electronics and Bioinformatics, Taipei, Taiwan
| | | | | | | | | | | | | |
Collapse
|
29
|
Tan LN, Alwan A, Kossan G, Cody ML, Taylor CE. Dynamic time warping and sparse representation classification for birdsong phrase classification using limited training data. THE JOURNAL OF THE ACOUSTICAL SOCIETY OF AMERICA 2015; 137:1069-1080. [PMID: 25786922 DOI: 10.1121/1.4906168] [Citation(s) in RCA: 16] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/04/2023]
Abstract
Annotation of phrases in birdsongs can be helpful to behavioral and population studies. To reduce the need for manual annotation, an automated birdsong phrase classification algorithm for limited data is developed. Limited data occur because of limited recordings or the existence of rare phrases. In this paper, classification of up to 81 phrase classes of Cassin's Vireo is performed using one to five training samples per class. The algorithm involves dynamic time warping (DTW) and two passes of sparse representation (SR) classification. DTW improves the similarity between training and test phrases from the same class in the presence of individual bird differences and phrase segmentation inconsistencies. The SR classifier works by finding a sparse linear combination of training feature vectors from all classes that best approximates the test feature vector. When the class decisions from DTW and the first pass SR classification are different, SR classification is repeated using training samples from these two conflicting classes. Compared to DTW, support vector machines, and an SR classifier without DTW, the proposed classifier achieves the highest classification accuracies of 94% and 89% on manually segmented and automatically segmented phrases, respectively, from unseen Cassin's Vireo individuals, using five training samples per class.
Collapse
Affiliation(s)
- Lee N Tan
- Department of Electrical Engineering, University of California, Los Angeles 56-125B Engineering IV Building, Box 951594, Los Angeles, California 90095
| | - Abeer Alwan
- Department of Electrical Engineering, University of California, Los Angeles 56-125B Engineering IV Building, Box 951594, Los Angeles, California 90095
| | - George Kossan
- Department of Ecology and Evolutionary Biology, University of California, Los Angeles 621 Charles Young Drive South, Los Angeles, California 90095
| | - Martin L Cody
- Department of Ecology and Evolutionary Biology, University of California, Los Angeles 621 Charles Young Drive South, Los Angeles, California 90095
| | - Charles E Taylor
- Department of Ecology and Evolutionary Biology, University of California, Los Angeles 621 Charles Young Drive South, Los Angeles, California 90095
| |
Collapse
|
30
|
Wang X, Zhang M, Ma J, Zhang Y, Hong G, Sun F, Lin G, Hu L. Metabolic Changes in Paraquat Poisoned Patients and Support Vector Machine Model of Discrimination. Biol Pharm Bull 2015; 38:470-5. [DOI: 10.1248/bpb.b14-00781] [Citation(s) in RCA: 29] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/22/2022]
Affiliation(s)
- Xianqin Wang
- Analytical and Testing Center, Wenzhou Medical University
| | - Meiling Zhang
- Analytical and Testing Center, Wenzhou Medical University
| | - Jianshe Ma
- Analytical and Testing Center, Wenzhou Medical University
| | - Yuan Zhang
- Analytical and Testing Center, Wenzhou Medical University
| | - Guangliang Hong
- Department of emergency, The First Affiliated Hospital of Wenzhou Medical University
| | - Fa Sun
- Analytical and Testing Center, Wenzhou Medical University
| | - Guanyang Lin
- Department of Pharmacy, The First Affliated Hospital of Wenzhou Medical University
| | - Lufeng Hu
- Department of Pharmacy, The First Affliated Hospital of Wenzhou Medical University
| |
Collapse
|