Deep Machine Learning Techniques for the Detection and Classification of Sperm Whale Bioacoustics

An integrated passive acoustic monitoring and deep learning pipeline for black-and-white ruffed lemurs (Varecia variegata) in Ranomafana National Park, Madagascar

The urgent need for effective wildlife monitoring solutions in the face of global biodiversity loss has resulted in the emergence of conservation technologies such as passive acoustic monitoring (PAM). While PAM has been extensively used for marine mammals, birds, and bats, its application to primates is limited. Black-and-white ruffed lemurs (Varecia variegata) are a promising species to test PAM with due to their distinctive and loud roar-shrieks. Furthermore, these lemurs are challenging to monitor via traditional methods due to their fragmented and often unpredictable distribution in Madagascar's dense eastern rainforests. Our goal in this study was to develop a machine learning pipeline for automated call detection from PAM data, compare the effectiveness of PAM versus in-person observations, and investigate diel patterns in lemur vocal behavior. We did this study at Mangevo, Ranomafana National Park by concurrently conducting focal follows and deploying autonomous recorders in May-July 2019. We used transfer learning to build a convolutional neural network (optimized for recall) that automated the detection of lemur calls (57-h runtime; recall = 0.94, F1 = 0.70). We found that PAM outperformed in-person observations, saving time, money, and labor while also providing re-analyzable data. Using PAM yielded novel insights into V. variegata diel vocal patterns; we present the first published evidence of nocturnal calling. We developed a graphic user interface and open-sourced data and code, to serve as a resource for primatologists interested in implementing PAM and machine learning. By leveraging the potential of this pipeline, we can address the urgent need for effective primate population surveys to inform conservation strategies.

Development of a machine learning detector for North Atlantic humpback whale song

The study of humpback whale song using passive acoustic monitoring devices requires bioacousticians to manually review hours of audio recordings to annotate the signals. To vastly reduce the time of manual annotation through automation, a machine learning model was developed. Convolutional neural networks have made major advances in the previous decade, leading to a wide range of applications, including the detection of frequency modulated vocalizations by cetaceans. A large dataset of over 60 000 audio segments of 4 s length is collected from the North Atlantic and used to fine-tune an existing model for humpback whale song detection in the North Pacific (see Allen, Harvey, Harrell, Jansen, Merkens, Wall, Cattiau, and Oleson (2021). Front. Mar. Sci. 8, 607321). Furthermore, different data augmentation techniques (time-shift, noise augmentation, and masking) are used to artificially increase the variability within the training set. Retraining and augmentation yield F-score values of 0.88 on context window basis and 0.89 on hourly basis with false positive rates of 0.05 on context window basis and 0.01 on hourly basis. If necessary, usage and retraining of the existing model is made convenient by a framework (AcoDet, acoustic detector) built during this project. Combining the tools provided by this framework could save researchers hours of manual annotation time and, thus, accelerate their research.

Computing Sickle Erythrocyte Health Index on quantitative phase imaging and machine learning

Sickle cell disease (SCD) is a genetic disorder characterized by abnormal hemoglobin and deformation of red blood cells (RBCs), leading to complications and reduced life expectancy. This study developed an in vitro assessment, the Sickle Erythrocyte Health Index, using quantitative phase imaging (QPI) and machine learning to model the health of RBCs in people with SCD. The health index combines assessment of cell deformation, sickle-shaped classification, and membrane flexibility to evaluate erythrocyte health. Using QPI and image processing, the percentage of sickle-shaped cells and membrane flexibility were quantified. Statistically significant differences were observed between individuals with and without SCD, indicating the impact of underlying pathophysiology on erythrocyte health. Additionally, sodium metabisulfite led to an increase in sickle-shaped cells and a decrease in flexibility in the sickle cell blood samples. Based on these findings, two approaches were used to calculate the Sickle Erythrocyte Health Index: one using hand-crafted features and one using learned features from deep learning models. Both indices showed significant differences between non-SCD and SCD groups and sensitivity to changes induced by sodium metabisulfite. The Sickle Erythrocyte Health Index has important clinical implications for SCD management and could be used by providers when making treatment decisions. Further research is warranted to evaluate the clinical utility and applicability of the Sickle Erythrocyte Health Index in diverse patient populations.

Elephants and algorithms: a review of the current and future role of AI in elephant monitoring

Artificial intelligence (AI) and machine learning (ML) present revolutionary opportunities to enhance our understanding of animal behaviour and conservation strategies. Using elephants, a crucial species in Africa and Asia's protected areas, as our focal point, we delve into the role of AI and ML in their conservation. Given the increasing amounts of data gathered from a variety of sensors like cameras, microphones, geophones, drones and satellites, the challenge lies in managing and interpreting this vast data. New AI and ML techniques offer solutions to streamline this process, helping us extract vital information that might otherwise be overlooked. This paper focuses on the different AI-driven monitoring methods and their potential for improving elephant conservation. Collaborative efforts between AI experts and ecological researchers are essential in leveraging these innovative technologies for enhanced wildlife conservation, setting a precedent for numerous other species.

Towards automated long-term acoustic monitoring of endangered river dolphins: a case study in the Brazilian Amazon floodplains

Using passive acoustic monitoring (PAM) and convolutional neural networks (CNN), we monitored the movements of the two endangered Amazon River dolphin species, the boto (Inia geoffrensis) and the tucuxi (Sotalia fluviatilis) from main rivers to floodplain habitats (várzea) in the Mamirauá Reserve (Amazonas, Brazil). We detected dolphin presence in four main areas based on the classification of their echolocation clicks. Using the same method, we automatically detected boat passages to estimate a possible interaction between boat and dolphin presence. Performance of the CNN classifier was high with an average precision of 0.95 and 0.92 for echolocation clicks and boats, respectively. Peaks of acoustic activity were detected synchronously at the river entrance and channel, corresponding to dolphins seasonally entering the várzea. Additionally, the river dolphins were regularly detected inside the flooded forest, suggesting a wide dispersion of their populations inside this large area, traditionally understudied and particularly important for boto females and calves. Boats overlapped with dolphin presence 9% of the time. PAM and recent advances in classification methods bring a new insight of the river dolphins' use of várzea habitats, which will contribute to conservation strategies of these species.

ORCA-SPY enables killer whale sound source simulation, detection, classification and localization using an integrated deep learning-based segmentation

Acoustic identification of vocalizing individuals opens up new and deeper insights into animal communications, such as individual-/group-specific dialects, turn-taking events, and dialogs. However, establishing an association between an individual animal and its emitted signal is usually non-trivial, especially for animals underwater. Consequently, a collection of marine species-, array-, and position-specific ground truth localization data is extremely challenging, which strongly limits possibilities to evaluate localization methods beforehand or at all. This study presents ORCA-SPY, a fully-automated sound source simulation, classification and localization framework for passive killer whale (Orcinus orca) acoustic monitoring that is embedded into PAMGuard, a widely used bioacoustic software toolkit. ORCA-SPY enables array- and position-specific multichannel audio stream generation to simulate real-world ground truth killer whale localization data and provides a hybrid sound source identification approach integrating ANIMAL-SPOT, a state-of-the-art deep learning-based orca detection network, followed by downstream Time-Difference-Of-Arrival localization. ORCA-SPY was evaluated on simulated multichannel underwater audio streams including various killer whale vocalization events within a large-scale experimental setup benefiting from previous real-world fieldwork experience. Across all 58,320 embedded vocalizing killer whale events, subject to various hydrophone array geometries, call types, distances, and noise conditions responsible for a signal-to-noise ratio varying from [Formula: see text] dB to 3 dB, a detection rate of 94.0 % was achieved with an average localization error of 7.01[Formula: see text]. ORCA-SPY was field-tested on Lake Stechlin in Brandenburg Germany under laboratory conditions with a focus on localization. During the field test, 3889 localization events were observed with an average error of 29.19[Formula: see text] and a median error of 17.54[Formula: see text]. ORCA-SPY was deployed successfully during the DeepAL fieldwork 2022 expedition (DLFW22) in Northern British Columbia, with a mean average error of 20.01[Formula: see text] and a median error of 11.01[Formula: see text] across 503 localization events. ORCA-SPY is an open-source and publicly available software framework, which can be adapted to various recording conditions as well as animal species.

V, Vijayan NT, Subramanian M, Gauns MU. Applications of Machine Learning in Chemical and Biological Oceanography

Machine learning (ML) refers to computer algorithms that predict a meaningful output or categorize complex systems based on a large amount of data. ML is applied in various areas including natural science, engineering, space exploration, and even gaming development. This review focuses on the use of machine learning in the field of chemical and biological oceanography. In the prediction of global fixed nitrogen levels, partial carbon dioxide pressure, and other chemical properties, the application of ML is a promising tool. Machine learning is also utilized in the field of biological oceanography to detect planktonic forms from various images (i.e., microscopy, FlowCAM, and video recorders), spectrometers, and other signal processing techniques. Moreover, ML successfully classified the mammals using their acoustics, detecting endangered mammalian and fish species in a specific environment. Most importantly, using environmental data, the ML proved to be an effective method for predicting hypoxic conditions and harmful algal bloom events, an essential measurement in terms of environmental monitoring. Furthermore, machine learning was used to construct a number of databases for various species that will be useful to other researchers, and the creation of new algorithms will help the marine research community better comprehend the chemistry and biology of the ocean.

Machine-learning-based simultaneous detection and ranging of impulsive baleen whale vocalizations using a single hydrophone

The low-frequency impulsive gunshot vocalizations of baleen whales exhibit dispersive propagation in shallow-water channels which is well-modeled by normal mode theory. Typically, underwater acoustic source range estimation requires multiple time-synchronized hydrophone arrays which can be difficult and expensive to achieve. However, single-hydrophone modal dispersion has been used to range baleen whale vocalizations and estimate shallow-water geoacoustic properties. Although convenient when compared to sensor arrays, these algorithms require preliminary signal detection and human labor to estimate the modal dispersion. In this paper, we apply a temporal convolutional network (TCN) to spectrograms from single-hydrophone acoustic data for simultaneous gunshot detection and ranging. The TCN learns ranging and detection jointly using gunshots simulated across multiple environments and ranges along with experimental noise. The synthetic data are informed by only the water column depth, sound speed, and density of the experimental environment, while other parameters span empirically observed bounds. The method is experimentally verified on North Pacific right whale gunshot data collected in the Bering Sea. To do so, 50 dispersive gunshots were manually ranged using the state-of-the-art time-warping inversion method. The TCN detected these gunshots among 50 noise-only examples with high precision and estimated ranges which closely matched those of the physics-based approach.

There You Are! Automated Detection of Indris' Songs on Features Extracted from Passive Acoustic Recordings

The growing concern for the ongoing biodiversity loss drives researchers towards practical and large-scale automated systems to monitor wild animal populations. Primates, with most species threatened by extinction, face substantial risks. We focused on the vocal activity of the indri (Indri indri) recorded in Maromizaha Forest (Madagascar) from 2019 to 2021 via passive acoustics, a method increasingly used for monitoring activities in different environments. We first used indris’ songs, loud distinctive vocal sequences, to detect the species’ presence. We processed the raw data (66,443 10-min recordings) and extracted acoustic features based on the third-octave band system. We then analysed the features extracted from three datasets, divided according to sampling year, site, and recorder type, with a convolutional neural network that was able to generalise to recording sites and previously unsampled periods via data augmentation and transfer learning. For the three datasets, our network detected the song presence with high accuracy (>90%) and recall (>80%) values. Once provided the model with the time and day of recording, the high-performance values ensured that the classification process could accurately depict both daily and annual habits of indris‘ singing pattern, critical information to optimise field data collection. Overall, using this easy-to-implement species-specific detection workflow as a preprocessing method allows researchers to reduce the time dedicated to manual classification.

ANIMAL-SPOT enables animal-independent signal detection and classification using deep learning

Bioacoustic research spans a wide range of biological questions and applications, relying on identification of target species or smaller acoustic units, such as distinct call types. However, manually identifying the signal of interest is time-intensive, error-prone, and becomes unfeasible with large data volumes. Therefore, machine-driven algorithms are increasingly applied to various bioacoustic signal identification challenges. Nevertheless, biologists still have major difficulties trying to transfer existing animal- and/or scenario-related machine learning approaches to their specific animal datasets and scientific questions. This study presents an animal-independent, open-source deep learning framework, along with a detailed user guide. Three signal identification tasks, commonly encountered in bioacoustics research, were investigated: (1) target signal vs. background noise detection, (2) species classification, and (3) call type categorization. ANIMAL-SPOT successfully segmented human-annotated target signals in data volumes representing 10 distinct animal species and 1 additional genus, resulting in a mean test accuracy of 97.9%, together with an average area under the ROC curve (AUC) of 95.9%, when predicting on unseen recordings. Moreover, an average segmentation accuracy and F1-score of 95.4% was achieved on the publicly available BirdVox-Full-Night data corpus. In addition, multi-class species and call type classification resulted in 96.6% and 92.7% accuracy on unseen test data, as well as 95.2% and 88.4% regarding previous animal-specific machine-based detection excerpts. Furthermore, an Unweighted Average Recall (UAR) of 89.3% outperformed the multi-species classification baseline system of the ComParE 2021 Primate Sub-Challenge. Besides animal independence, ANIMAL-SPOT does not rely on expert knowledge or special computing resources, thereby making deep-learning-based bioacoustic signal identification accessible to a broad audience.

Silbido profundo: An open source package for the use of deep learning to detect odontocete whistles

This work presents an open-source matlab software package for exploiting recent advances in extracting tonal signals from large acoustic data sets. A whistle extraction algorithm published by Li, Liu, Palmer, Fleishman, Gillespie, Nosal, Shiu, Klinck, Cholewiak, Helble, and Roch [(2020). Proceedings of the International Joint Conference on Neural Networks, July 19-24, Glasgow, Scotland, p. 10] is incorporated into silbido, an established software package for extraction of cetacean tonal calls. The precision and recall of the new system were over 96% and nearly 80%, respectively, when applied to a whistle extraction task on a challenging two-species subset of a conference-benchmark data set. A second data set was examined to assess whether the algorithm generalized to data that were collected across different recording devices and locations. These data included 487 h of weakly labeled, towed array data collected in the Pacific Ocean on two National Oceanographic and Atmospheric Administration (NOAA) cruises. Labels for these data consisted of regions of toothed whale presence for at least 15 species that were based on visual and acoustic observations and not limited to whistles. Although the lack of per whistle-level annotations prevented measurement of precision and recall, there was strong concurrence of automatic detections and the NOAA annotations, suggesting that the algorithm generalizes well to new data.

A Review of Automated Bioacoustics and General Acoustics Classification Research

Automated bioacoustics classification has received increasing attention from the research community in recent years due its cross-disciplinary nature and its diverse application. Applications in bioacoustics classification range from smart acoustic sensor networks that investigate the effects of acoustic vocalizations on species to context-aware edge devices that anticipate changes in their environment adapt their sensing and processing accordingly. The research described here is an in-depth survey of the current state of bioacoustics classification and monitoring. The survey examines bioacoustics classification alongside general acoustics to provide a representative picture of the research landscape. The survey reviewed 124 studies spanning eight years of research. The survey identifies the key application areas in bioacoustics research and the techniques used in audio transformation and feature extraction. The survey also examines the classification algorithms used in bioacoustics systems. Lastly, the survey examines current challenges, possible opportunities, and future directions in bioacoustics.

Toward understanding the communication in sperm whales

Machine learning has been advancing dramatically over the past decade. Most strides are human-based applications due to the availability of large-scale datasets; however, opportunities are ripe to apply this technology to more deeply understand non-human communication. We detail a scientific roadmap for advancing the understanding of communication of whales that can be built further upon as a template to decipher other forms of animal and non-human communication. Sperm whales, with their highly developed neuroanatomical features, cognitive abilities, social structures, and discrete click-based encoding make for an excellent model for advanced tools that can be applied to other animals in the future. We outline the key elements required for the collection and processing of massive datasets, detecting basic communication units and language-like higher-level structures, and validating models through interactive playback experiments. The technological capabilities developed by such an undertaking hold potential for cross-applications in broader communities investigating non-human communication and behavioral research.

Parameterizing animal sounds and motion with animal-attached tags to study acoustic communication

Abstract

Stemming from the traditional use of field observers to score states and events, the study of animal behaviour often relies on analyses of discrete behavioural categories. Many studies of acoustic communication record sequences of animal sounds, classify vocalizations, and then examine how call categories are used relative to behavioural states and events. However, acoustic parameters can also convey information independent of call type, offering complementary study approaches to call classifications. Animal-attached tags can continuously sample high-resolution behavioural data on sounds and movements, which enables testing how acoustic parameters of signals relate to parameters of animal motion. Here, we present this approach through case studies on wild common bottlenose dolphins (Tursiops truncatus). Using data from sound-and-movement recording tags deployed in Sarasota (FL), we parameterized dolphin vocalizations and motion to investigate how senders and receivers modified movement parameters (including vectorial dynamic body acceleration, “VeDBA”, a proxy for activity intensity) as a function of signal parameters. We show that (1) VeDBA of one female during consortships had a negative relationship with centroid frequency of male calls, matching predictions about agonistic interactions based on motivation-structural rules; (2) VeDBA of four males had a positive relationship with modulation rate of their pulsed vocalizations, confirming predictions that click-repetition rate of these calls increases with agonism intensity. Tags offer opportunities to study animal behaviour through analyses of continuously sampled quantitative parameters, which can complement traditional methods and facilitate research replication. Our case studies illustrate the value of this approach to investigate communicative roles of acoustic parameter changes.

Significance statement

Studies of animal behaviour have traditionally relied on classification of behavioural patterns and analyses of discrete behavioural categories. Today, technologies such as animal-attached tags enable novel approaches, facilitating the use of quantitative metrics to characterize behaviour. In the field of acoustic communication, researchers typically classify vocalizations and examine usage of call categories. Through case studies of bottlenose dolphin social interactions, we present here a novel tag-based complementary approach. We used high-resolution tag data to parameterize dolphin sounds and motion, and we applied continuously sampled parameters to examine how individual dolphins responded to conspecifics’ signals and moved while producing sounds. Activity intensity of senders and receivers changed with specific call parameters, matching our predictions and illustrating the value of our approach to test communicative roles of acoustic parameter changes. Parametric approaches can complement traditional methods for animal behaviour and facilitate research replication.

Convolutional Neural Networks for the Identification of African Lions from Individual Vocalizations

The classification of vocal individuality for passive acoustic monitoring (PAM) and census of animals is becoming an increasingly popular area of research. Nearly all studies in this field of inquiry have relied on classic audio representations and classifiers, such as Support Vector Machines (SVMs) trained on spectrograms or Mel-Frequency Cepstral Coefficients (MFCCs). In contrast, most current bioacoustic species classification exploits the power of deep learners and more cutting-edge audio representations. A significant reason for avoiding deep learning in vocal identity classification is the tiny sample size in the collections of labeled individual vocalizations. As is well known, deep learners require large datasets to avoid overfitting. One way to handle small datasets with deep learning methods is to use transfer learning. In this work, we evaluate the performance of three pretrained CNNs (VGG16, ResNet50, and AlexNet) on a small, publicly available lion roar dataset containing approximately 150 samples taken from five male lions. Each of these networks is retrained on eight representations of the samples: MFCCs, spectrogram, and Mel spectrogram, along with several new ones, such as VGGish and stockwell, and those based on the recently proposed LM spectrogram. The performance of these networks, both individually and in ensembles, is analyzed and corroborated using the Equal Error Rate and shown to surpass previous classification attempts on this dataset; the best single network achieved over 95% accuracy and the best ensembles over 98% accuracy. The contributions this study makes to the field of individual vocal classification include demonstrating that it is valuable and possible, with caution, to use transfer learning with single pretrained CNNs on the small datasets available for this problem domain. We also make a contribution to bioacoustics generally by offering a comparison of the performance of many state-of-the-art audio representations, including for the first time the LM spectrogram and stockwell representations. All source code for this study is available on GitHub.

Sounding the Call for a Global Library of Underwater Biological Sounds

Aquatic environments encompass the world’s most extensive habitats, rich with sounds produced by a diversity of animals. Passive acoustic monitoring (PAM) is an increasingly accessible remote sensing technology that uses hydrophones to listen to the underwater world and represents an unprecedented, non-invasive method to monitor underwater environments. This information can assist in the delineation of biologically important areas via detection of sound-producing species or characterization of ecosystem type and condition, inferred from the acoustic properties of the local soundscape. At a time when worldwide biodiversity is in significant decline and underwater soundscapes are being altered as a result of anthropogenic impacts, there is a need to document, quantify, and understand biotic sound sources–potentially before they disappear. A significant step toward these goals is the development of a web-based, open-access platform that provides: (1) a reference library of known and unknown biological sound sources (by integrating and expanding existing libraries around the world); (2) a data repository portal for annotated and unannotated audio recordings of single sources and of soundscapes; (3) a training platform for artificial intelligence algorithms for signal detection and classification; and (4) a citizen science-based application for public users. Although individually, these resources are often met on regional and taxa-specific scales, many are not sustained and, collectively, an enduring global database with an integrated platform has not been realized. We discuss the benefits such a program can provide, previous calls for global data-sharing and reference libraries, and the challenges that need to be overcome to bring together bio- and ecoacousticians, bioinformaticians, propagation experts, web engineers, and signal processing specialists (e.g., artificial intelligence) with the necessary support and funding to build a sustainable and scalable platform that could address the needs of all contributors and stakeholders into the future.

Automatic detection and classification of bearded seal vocalizations in the northeastern Chukchi Sea using convolutional neural networks

Bearded seals vocalizations are often analyzed manually or by using automatic detections that are manually validated. In this work, an automatic detection and classification system (DCS) based on convolutional neural networks (CNNs) is proposed. Bearded seal sounds were year-round recorded by four spatially separated receivers on the Chukchi Continental Slope in Alaska in 2016-2017. The DCS is divided in two sections. First, regions of interest (ROI) containing possible bearded seal vocalizations are found by using the two-dimensional normalized cross correlation of the measured spectrogram and a representative template of two main calls of interest. Second, CNNs are used to validate and classify the ROIs among several possible classes. The CNNs are trained on 80% of the ROIs manually labeled from one of the four spatially separated recorders. When validating on the remaining 20%, the CNNs show an accuracy above 95.5%. To assess the generalization performance of the networks, the CNNs are tested on the remaining recorders, located at different positions, with a precision above 89.2% for the main class of the two types of calls. The proposed technique reduces the laborious task of manual inspection prone to inconstant bias and possible errors in detections.

BioCPPNet: automatic bioacoustic source separation with deep neural networks

We introduce the Bioacoustic Cocktail Party Problem Network (BioCPPNet), a lightweight, modular, and robust U-Net-based machine learning architecture optimized for bioacoustic source separation across diverse biological taxa. Employing learnable or handcrafted encoders, BioCPPNet operates directly on the raw acoustic mixture waveform containing overlapping vocalizations and separates the input waveform into estimates corresponding to the sources in the mixture. Predictions are compared to the reference ground truth waveforms by searching over the space of (output, target) source order permutations, and we train using an objective function motivated by perceptual audio quality. We apply BioCPPNet to several species with unique vocal behavior, including macaques, bottlenose dolphins, and Egyptian fruit bats, and we evaluate reconstruction quality of separated waveforms using the scale-invariant signal-to-distortion ratio (SI-SDR) and downstream identity classification accuracy. We consider mixtures with two or three concurrent conspecific vocalizers, and we examine separation performance in open and closed speaker scenarios. To our knowledge, this paper redefines the state-of-the-art in end-to-end single-channel bioacoustic source separation in a permutation-invariant regime across a heterogeneous set of non-human species. This study serves as a major step toward the deployment of bioacoustic source separation systems for processing substantial volumes of previously unusable data containing overlapping bioacoustic signals.

Forest Vertical Structure Mapping Using Two-Seasonal Optic Images and LiDAR DSM Acquired from UAV Platform through Random Forest, XGBoost, and Support Vector Machine Approaches

Research on the forest structure classification is essential, as it plays an important role in assessing the vitality and diversity of vegetation. However, classifying forest structure involves in situ surveying, which requires considerable time and money, and cannot be conducted directly in some instances; also, the update cycle of the classification data is very late. To overcome these drawbacks, feasibility studies on mapping the forest vertical structure from aerial images using machine learning techniques were conducted. In this study, we investigated (1) the performance improvement of the forest structure classification, using a high-resolution LiDAR-derived digital surface model (DSM) acquired from an unmanned aerial vehicle (UAV) platform and (2) the performance comparison of results obtained from the single-seasonal and two-seasonal data, using random forest (RF), extreme gradient boosting (XGBoost), and support vector machine (SVM). For the performance comparison, the UAV optic and LiDAR data were divided into three cases: (1) only used autumn data, (2) only used winter data, and (3) used both autumn and winter data. From the results, the best model was XGBoost, and the F1 scores achieved using this method were approximately 0.92 in the autumn and winter cases. A remarkable improvement was achieved when both two-seasonal images were used. The F1 score improved by 35.3% from 0.68 to 0.92. This implies that (1) the seasonal variation in the forest vertical structure can be more important than the spatial resolution, and (2) the classification performance achieved from the two-seasonal UAV optic images and LiDAR-derived DSMs can reach 0.9 with the application of an optimal machine learning approach.

Comparing recurrent convolutional neural networks for large scale bird species classification

We present a deep learning approach towards the large-scale prediction and analysis of bird acoustics from 100 different bird species. We use spectrograms constructed on bird audio recordings from the Cornell Bird Challenge (CBC)2020 dataset, which includes recordings of multiple and potentially overlapping bird vocalizations with background noise. Our experiments show that a hybrid modeling approach that involves a Convolutional Neural Network (CNN) for learning the representation for a slice of the spectrogram, and a Recurrent Neural Network (RNN) for the temporal component to combine across time-points leads to the most accurate model on this dataset. We show results on a spectrum of models ranging from stand-alone CNNs to hybrid models of various types obtained by combining CNNs with other CNNs or RNNs of the following types: Long Short-Term Memory (LSTM) networks, Gated Recurrent Units (GRU), and Legendre Memory Units (LMU). The best performing model achieves an average accuracy of 67% over the 100 different bird species, with the highest accuracy of 90% for the bird species, Red crossbill. We further analyze the learned representations visually and find them to be intuitive, where we find that related bird species are clustered close together. We present a novel way to empirically interpret the representations learned by the LMU-based hybrid model which shows how memory channel patterns change over time with the changes seen in the spectrograms.

Bioacoustic classification of avian calls from raw sound waveforms with an open-source deep learning architecture

The use of autonomous recordings of animal sounds to detect species is a popular conservation tool, constantly improving in fidelity as audio hardware and software evolves. Current classification algorithms utilise sound features extracted from the recording rather than the sound itself, with varying degrees of success. Neural networks that learn directly from the raw sound waveforms have been implemented in human speech recognition but the requirements of detailed labelled data have limited their use in bioacoustics. Here we test SincNet, an efficient neural network architecture that learns from the raw waveform using sinc-based filters. Results using an off-the-shelf implementation of SincNet on a publicly available bird sound dataset (NIPS4Bplus) show that the neural network rapidly converged reaching accuracies of over 65% with limited data. Their performance is comparable with traditional methods after hyperparameter tuning but they are more efficient. Learning directly from the raw waveform allows the algorithm to select automatically those elements of the sound that are best suited for the task, bypassing the onerous task of selecting feature extraction techniques and reducing possible biases. We use publicly released code and datasets to encourage others to replicate our results and to apply SincNet to their own datasets; and we review possible enhancements in the hope that algorithms that learn from the raw waveform will become useful bioacoustic tools.

Using context to train time-domain echolocation click detectors

This work demonstrates the effectiveness of using humans in the loop processes for constructing large training sets for machine learning tasks. A corpus of over 57 000 toothed whale echolocation clicks was developed by using a permissive energy-based echolocation detector followed by a machine-assisted quality control process that exploits contextual cues. Subsets of these data were used to train feed forward neural networks that detected over 850 000 echolocation clicks that were validated using the same quality control process. It is shown that this network architecture performs well in a variety of contexts and is evaluated against a withheld data set that was collected nearly five years apart from the development data at a location over 600 km distant. The system was capable of finding echolocation bouts that were missed by human analysts, and the patterns of error in the classifier consist primarily of anthropogenic sources that were not included as counter-training examples. In the absence of such events, typical false positive rates are under ten events per hour even at low thresholds.

Automatic detection and classification of baleen whale social calls using convolutional neural networks

Passive acoustic monitoring has proven to be an indispensable tool for many aspects of baleen whale research. Manual detection of whale calls on these large data sets demands extensive manual labor. Automated whale call detectors offer a more efficient approach and have been developed for many species and call types. However, calls with a large level of variability such as fin whale (Balaenoptera physalus) 40 Hz call and blue whale (B. musculus) D call have been challenging to detect automatically and hence no practical automated detector exists for these two call types. Using a modular approach consisting of faster region-based convolutional neural network followed by a convolutional neural network, we have created automated detectors for 40 Hz calls and D calls. Both detectors were tested on recordings with high- and low density of calls and, when selecting for detections with high classification scores, they were shown to have precision ranging from 54% to 57% with recall ranging from 72% to 78% for 40 Hz and precision ranging from 62% to 64% with recall ranging from 70 to 73% for D calls. As these two call types are produced by both sexes, using them in long-term studies would remove sex-bias in estimates of temporal presence and movement patterns.

Multi-Omics Approaches and Radiation on Lipid Metabolism in Toothed Whales

Lipid synthesis pathways of toothed whales have evolved since their movement from the terrestrial to marine environment. The synthesis and function of these endogenous lipids and affecting factors are still little understood. In this review, we focused on different omics approaches and techniques to investigate lipid metabolism and radiation impacts on lipids in toothed whales. The selected literature was screened, and capacities, possibilities, and future approaches for identifying unusual lipid synthesis pathways by omics were evaluated. Omics approaches were categorized into the four major disciplines: lipidomics, transcriptomics, genomics, and proteomics. Genomics and transcriptomics can together identify genes related to unique lipid synthesis. As lipids interact with proteins in the animal body, lipidomics, and proteomics can correlate by creating lipid-binding proteome maps to elucidate metabolism pathways. In lipidomics studies, recent mass spectroscopic methods can address lipid profiles; however, the determination of structures of lipids are challenging. As an environmental stress, the acoustic radiation has a significant effect on the alteration of lipid profiles. Radiation studies in different omics approaches revealed the necessity of multi-omics applications. This review concluded that a combination of many of the omics areas may elucidate the metabolism of lipids and possible hazards on lipids in toothed whales by radiation.

Deep embedded clustering of coral reef bioacoustics

Deep clustering was applied to unlabeled, automatically detected signals in a coral reef soundscape to distinguish fish pulse calls from segments of whale song. Deep embedded clustering (DEC) learned latent features and formed classification clusters using fixed-length power spectrograms of the signals. Handpicked spectral and temporal features were also extracted and clustered with Gaussian mixture models (GMM) and conventional clustering. DEC, GMM, and conventional clustering were tested on simulated datasets of fish pulse calls (fish) and whale song units (whale) with randomized bandwidth, duration, and SNR. Both GMM and DEC achieved high accuracy and identified clusters with fish, whale, and overlapping fish and whale signals. Conventional clustering methods had low accuracy in scenarios with unequal-sized clusters or overlapping signals. Fish and whale signals recorded near Hawaii in February-March 2020 were clustered with DEC, GMM, and conventional clustering. DEC features demonstrated the highest accuracy of 77.5% on a small, manually labeled dataset for classifying signals into fish and whale clusters.

Estimating ecoacoustic activity in the Amazon rainforest through Information Theory quantifiers

Automatic monitoring of biodiversity by acoustic sensors has become an indispensable tool to assess environmental stress at an early stage. Due to the difficulty in recognizing the Amazon's high acoustic diversity and the large amounts of raw audio data recorded by the sensors, the labeling and manual inspection of this data is not feasible. Therefore, we propose an ecoacoustic index that allows us to quantify the complexity of an audio segment and correlate this measure with the biodiversity of the soundscape. The approach uses unsupervised methods to avoid the problem of labeling each species individually. The proposed index, named the Ecoacoustic Global Complexity Index (EGCI), makes use of Entropy, Divergence and Statistical Complexity. A distinguishing feature of this index is the mapping of each audio segment, including those of varied lengths, as a single point in a 2D-plane, supporting us in understanding the ecoacoustic dynamics of the rainforest. The main results show a regularity in the ecoacoustic richness of a floodplain, considering different temporal granularities, be it between hours of the day or between consecutive days of the monitoring program. We observed that this regularity does a good job of characterizing the soundscape of the environmental protection area of Mamirauá, in the Amazon, differentiating between species richness and environmental phenomena.

Bioacoustic Classification of Antillean Manatee Vocalization Spectrograms Using Deep Convolutional Neural Networks

We evaluated the potential of using convolutional neural networks in classifying spectrograms of Antillean manatee (Trichechus manatus manatus) vocalizations. Spectrograms using binary, linear and logarithmic amplitude formats were considered. Two deep convolutional neural networks (DCNN) architectures were tested: linear (fixed filter size) and pyramidal (incremental filter size). Six experiments were devised for testing the accuracy obtained for each spectrogram representation and architecture combination. Results show that binary spectrograms with both linear and pyramidal architectures with dropout provide a classification rate of 94–99% on the training and 92–98% on the testing set, respectively. The pyramidal network presents a shorter training and inference time. Results from the convolutional neural networks (CNN) are substantially better when compared with a signal processing fast Fourier transform (FFT)-based harmonic search approach in terms of accuracy and F1 Score. Taken together, these results prove the validity of using spectrograms and using DCNNs for manatee vocalization classification. These results can be used to improve future software and hardware implementations for the estimation of the manatee population in Panama.

Performance of a deep neural network at detecting North Atlantic right whale upcalls

Passive acoustics provides a powerful tool for monitoring the endangered North Atlantic right whale (Eubalaena glacialis), but robust detection algorithms are needed to handle diverse and variable acoustic conditions and differences in recording techniques and equipment. This paper investigates the potential of deep neural networks (DNNs) for addressing this need. ResNet, an architecture commonly used for image recognition, was trained to recognize the time-frequency representation of the characteristic North Atlantic right whale upcall. The network was trained on several thousand examples recorded at various locations in the Gulf of St. Lawrence in 2018 and 2019, using different equipment and deployment techniques. Used as a detection algorithm on fifty 30-min recordings from the years 2015-2017 containing over one thousand upcalls, the network achieved recalls up to 80% while maintaining a precision of 90%. Importantly, the performance of the network improved as more variance was introduced into the training dataset, whereas the opposite trend was observed using a conventional linear discriminant analysis approach. This study demonstrates that DNNs can be trained to identify North Atlantic right whale upcalls under diverse and variable conditions with a performance that compares favorably to that of existing algorithms.

Beluga whale acoustic signal classification using deep learning neural network models

Over a decade after the Cook Inlet beluga (Delphinapterus leucas) was listed as endangered in 2008, the population has shown no sign of recovery. Lack of ecological knowledge limits the understanding of, and ability to manage, potential threats impeding recovery of this declining population. National Oceanic and Atmospheric Administration Fisheries, in partnership with the Alaska Department of Fish and Game, initiated a passive acoustics monitoring program in 2017 to investigate beluga seasonal occurrence by deploying a series of passive acoustic moorings. Data have been processed with semi-automated tonal detectors followed by time intensive manual validation. To reduce this labor intensive and time-consuming process, in addition to increasing the accuracy of classification results, the authors constructed an ensembled deep learning convolutional neural network model to classify beluga detections as true or false. Using a 0.5 threshold, the final model achieves 96.57% precision and 92.26% recall on testing dataset. This methodology proves to be successful at classifying beluga signals, and the framework can be easily generalized to other acoustic classification problems.

Machine learning in acoustics: Theory and applications

Acoustic data provide scientific and engineering insights in fields ranging from biology and communications to ocean and Earth science. We survey the recent advances and transformative potential of machine learning (ML), including deep learning, in the field of acoustics. ML is a broad family of techniques, which are often based in statistics, for automatically detecting and utilizing patterns in data. Relative to conventional acoustics and signal processing, ML is data-driven. Given sufficient training data, ML can discover complex relationships between features and desired labels or actions, or between features themselves. With large volumes of training data, ML can discover models describing complex acoustic phenomena such as human speech and reverberation. ML in acoustics is rapidly developing with compelling results and significant future promise. We first introduce ML, then highlight ML developments in four acoustics research areas: source localization in speech processing, source localization in ocean acoustics, bioacoustics, and environmental sounds in everyday scenes.