Classification of large acoustic datasets using machine learning and crowdsourcing: application to whale calls

Underwater soundscape in Seaview Bay, Antarctica, and triple ascending trill of the leopard seal (Hydrurga leptonyx) underwater vocalizations

The underwater soundscape was recorded in Seaview Bay off Inexpressible Island, Ross Sea region Marine protected area, for 3 days in December 2021. Leopard seal Hydrurga leptonyx vocalizations were a prominent sound source that led to variations in ambient sound pressure levels in a frequency range of approximately 150-4500 Hz. Among the 14 call types previously identified, except ultrasound vocalizations, six types of broadcast calls were classified, and their acoustic characteristics were analyzed. We focused on the acoustic characteristics of four low-frequency calls, clustered in a relatively narrow bandwidth, which have been relatively less studied. We identified a new call type of a triple ascending trill consisting of three trill parts, expanding upon the findings of previous studies. The audio data extracted from leopard seal vocalization videos, recorded by a monitoring camera on sea ice, enhanced the reliability of identifications of the underwater triple ascending trill. We present the unique results of underwater passive acoustic monitoring conducted at Seaview Bay, designated as Antarctic Specially Protected Area No 178. Our results could contribute to the development of detection and localization algorithms for leopard seal vocalizations and can be used as fundamental data for studies related to the vocalization and behavior of this species.

Interpretable deep dictionary learning for sound speed profiles with uncertainties

Uncertainties abound in sound speed profiles (SSPs) measured/estimated by modern ocean observing systems, which impede the knowledge acquisition and downstream underwater applications. To reduce the SSP uncertainties and draw insights into specific ocean processes, an interpretable deep dictionary learning model is proposed to cater for uncertain SSP processing. In particular, two kinds of SSP uncertainties are considered: measurement errors, which generally exist in the form of Gaussian noises; and the disturbances/anomalies caused by potential ocean dynamics, which occur at some specific depths and durations. To learn the generative patterns of these uncertainties while maintaining the interpretability of the resulting deep model, the adopted scheme first unrolls the classical K-singular value decomposition algorithm into a neural network, and trains this neural network in a supervised learning manner. The training data and model initializations are judiciously designed to incorporate the environmental properties of ocean SSPs. Experimental results demonstrate the superior performance of the proposed method over the classical baseline in mitigating noise corruptions, detecting, and localizing SSP disturbances/anomalies.

A Review of Automated Bioacoustics and General Acoustics Classification Research

Automated bioacoustics classification has received increasing attention from the research community in recent years due its cross-disciplinary nature and its diverse application. Applications in bioacoustics classification range from smart acoustic sensor networks that investigate the effects of acoustic vocalizations on species to context-aware edge devices that anticipate changes in their environment adapt their sensing and processing accordingly. The research described here is an in-depth survey of the current state of bioacoustics classification and monitoring. The survey examines bioacoustics classification alongside general acoustics to provide a representative picture of the research landscape. The survey reviewed 124 studies spanning eight years of research. The survey identifies the key application areas in bioacoustics research and the techniques used in audio transformation and feature extraction. The survey also examines the classification algorithms used in bioacoustics systems. Lastly, the survey examines current challenges, possible opportunities, and future directions in bioacoustics.

Pose-gait analysis for cetacean biologging tag data

Biologging tags are a key enabling tool for investigating cetacean behavior and locomotion in their natural habitat. Identifying and then parameterizing gait from movement sensor data is critical for these investigations, but how best to characterize gait from tag data remains an open question. Further, the location and orientation of a tag on an animal in the field are variable and can change multiple times during a deployment. As a result, the relative orientation of the tag with respect to (wrt) the animal must be determined for analysis. Currently, custom scripts that involve species-specific heuristics tend to be used in the literature. These methods require a level of knowledge and experience that can affect the reliability and repeatability of the analysis. Swimming gait is composed of a sequence of body poses that have a specific spatial pattern, and tag-based measurements of this pattern can be utilized to determine the relative orientation of the tag. This work presents an automated data processing pipeline (and software) that takes advantage of these patterns to 1) Identify relative motion between the tag and animal; 2) Estimate the relative orientation of the tag wrt the animal using a data-driven approach; and 3) Calculate gait parameters that are stable and invariant to animal pose. Validation results from bottlenose dolphin tag data show that the average relative orientation error (tag wrt the body) after processing was within 11 degrees in roll, pitch, and yaw directions. The average precision and recall for detecting instances of relative motion in the dolphin data were 0.87 and 0.89, respectively. Tag data from humpback and beluga whales were then used to demonstrate how the gait analysis can be used to enhance tag-based investigations of movement and behavior. The MATLAB source code and data presented in the paper are publicly available (https://github.com/ding-z/cetacean-pose-gait-analysis.git), along with suggested best practices.

COSMOS: Computational Shaping and Modeling of Musical Structures

This position paper makes the case for an innovative, multi-disciplinary methodological approach to advance knowledge on the nature and work of music performance, driven by a novel experiential perspective, that also benefits analysis of electrocardiographic sequences. Music performance is considered by many to be one of the most breathtaking feats of human intelligence. It is well accepted that music performance is a creative act, but the nature of its work remains elusive. Taking the view of performance as an act of creative problem solving, ideas in citizen science and data science, optimization, and computational thinking provide means through which to deconstruct the process of music performance in scalable ways. The method tackles music expression's lack of notation-based data by leveraging listeners' perception and experience of the structures elicited by the performer, with implications for data collection and processing. The tools offer ways to parse a musical sequence into coherent structures, to design a performance, and to explore the space of possible interpretations of the musical sequence. These ideas and tools can be applied to other music-like sequences such as electrocardiographic recordings of arrhythmias (abnormal heart rhythms). Leveraging musical thinking and computational approaches to performance analysis, variations in expressions of cardiac arrhythmias can be more finely characterized, with implications for tailoring therapies and stratifying heart rhythm disorders.

Sounding the Call for a Global Library of Underwater Biological Sounds

Aquatic environments encompass the world’s most extensive habitats, rich with sounds produced by a diversity of animals. Passive acoustic monitoring (PAM) is an increasingly accessible remote sensing technology that uses hydrophones to listen to the underwater world and represents an unprecedented, non-invasive method to monitor underwater environments. This information can assist in the delineation of biologically important areas via detection of sound-producing species or characterization of ecosystem type and condition, inferred from the acoustic properties of the local soundscape. At a time when worldwide biodiversity is in significant decline and underwater soundscapes are being altered as a result of anthropogenic impacts, there is a need to document, quantify, and understand biotic sound sources–potentially before they disappear. A significant step toward these goals is the development of a web-based, open-access platform that provides: (1) a reference library of known and unknown biological sound sources (by integrating and expanding existing libraries around the world); (2) a data repository portal for annotated and unannotated audio recordings of single sources and of soundscapes; (3) a training platform for artificial intelligence algorithms for signal detection and classification; and (4) a citizen science-based application for public users. Although individually, these resources are often met on regional and taxa-specific scales, many are not sustained and, collectively, an enduring global database with an integrated platform has not been realized. We discuss the benefits such a program can provide, previous calls for global data-sharing and reference libraries, and the challenges that need to be overcome to bring together bio- and ecoacousticians, bioinformaticians, propagation experts, web engineers, and signal processing specialists (e.g., artificial intelligence) with the necessary support and funding to build a sustainable and scalable platform that could address the needs of all contributors and stakeholders into the future.

Underwater Target Signal Classification Using the Hybrid Routing Neural Network

In signal analysis and processing, underwater target recognition (UTR) is one of the most important technologies. Simply and quickly identify target types using conventional methods in underwater acoustic conditions is quite a challenging task. The problem can be conveniently handled by a deep learning network (DLN), which yields better classification results than conventional methods. In this paper, a novel deep learning method with a hybrid routing network is considered, which can abstract the features of time-domain signals. The used network comprises multiple routing structures and several options for the auxiliary branch, which promotes impressive effects as a result of exchanging the learned features of different branches. The experiment shows that the used network possesses more advantages in the underwater signal classification task.

Multi-Omics Approaches and Radiation on Lipid Metabolism in Toothed Whales

Lipid synthesis pathways of toothed whales have evolved since their movement from the terrestrial to marine environment. The synthesis and function of these endogenous lipids and affecting factors are still little understood. In this review, we focused on different omics approaches and techniques to investigate lipid metabolism and radiation impacts on lipids in toothed whales. The selected literature was screened, and capacities, possibilities, and future approaches for identifying unusual lipid synthesis pathways by omics were evaluated. Omics approaches were categorized into the four major disciplines: lipidomics, transcriptomics, genomics, and proteomics. Genomics and transcriptomics can together identify genes related to unique lipid synthesis. As lipids interact with proteins in the animal body, lipidomics, and proteomics can correlate by creating lipid-binding proteome maps to elucidate metabolism pathways. In lipidomics studies, recent mass spectroscopic methods can address lipid profiles; however, the determination of structures of lipids are challenging. As an environmental stress, the acoustic radiation has a significant effect on the alteration of lipid profiles. Radiation studies in different omics approaches revealed the necessity of multi-omics applications. This review concluded that a combination of many of the omics areas may elucidate the metabolism of lipids and possible hazards on lipids in toothed whales by radiation.

Biomimicking Covert Communication by Time-Frequency Shift Modulation for Increasing Mimicking and BER Performances

Underwater acoustic (UWA) biomimicking communications have been developed for covert communications. For the UWA covert communications, it is difficult to achieve the bit error rate (BER) and the degree of mimic (DoM) performances at the same time. This paper proposes a biomimicking covert communication method to increase both BER and DoM (degree of mimic) performances based on the Time Frequency Shift Keying (TFSK). To increase DoM and BER performances, the orthogonality requirements of the time- and frequency-shifting units of the TFSK are theoretically derived, and the whistles are multiplied by the sequence with a large correlation. Two-step DoM assessments are also developed for the long-term whistle signals. Computer simulations and practical lake and ocean experiments demonstrate that the proposed method increases the DoM by 35% and attains a zero BER at −6 dB of Signal to Noise Ratio (SNR).

Complex data labeling with deep learning methods: Lessons from fisheries acoustics

Quantitative and qualitative analysis of acoustic backscattered signals from the seabed bottom to the sea surface is used worldwide for fish stocks assessment and marine ecosystem monitoring. Huge amounts of raw data are collected yet require tedious expert labeling. This paper focuses on a case study where the ground truth labels are non-obvious: echograms labeling, which is time-consuming and critical for the quality of fisheries and ecological analysis. We investigate how these tasks can benefit from supervised learning algorithms and demonstrate that convolutional neural networks trained with non-stationary datasets can be used to stress parts of a new dataset needing human expert correction. Further development of this approach paves the way toward a standardization of the labeling process in fisheries acoustics and is a good case study for non-obvious data labeling processes.

Machine Learning Based Biomimetic Underwater Covert Acoustic Communication Method Using Dolphin Whistle Contours

For underwater acoustic covert communications, biomimetic covert communications have been developed using dolphin whistles. The conventional biomimetic covert communication methods transmit slightly different signal patterns from real dolphin whistles, which results in a low degree of mimic (DoM). In this paper, we propose a novel biomimetic communication method that preserves the large DoM with a low bit error rate (BER). For the transmission, the proposed method utilizes the various contours of real dolphin whistles with the link information among consecutive whistles, and the proposed receiver uses machine learning based whistle detectors with the aid of the link information. Computer simulations and practical ocean experiments were executed to demonstrate the better BER performance of the proposed method. Ocean experiments demonstrate that the BER of the proposed method was 0.002, while the BER of the conventional Deep Neural Network (DNN) based detector showed 0.36.

Classification of Hydroacoustic Signals Based on Harmonic Wavelets and a Deep Learning Artificial Intelligence System

This paper considers two approaches to hydroacoustic signal classification, taking the sounds made by whales as an example: a method based on harmonic wavelets and a technique involving deep learning neural networks. The study deals with the classification of hydroacoustic signals using coefficients of the harmonic wavelet transform (fast computation), short-time Fourier transform (spectrogram) and Fourier transform using a kNN-algorithm. Classification quality metrics (precision, recall and accuracy) are given for different signal-to-noise ratios. ROC curves were also obtained. The use of the deep neural network for classification of whales’ sounds is considered. The effectiveness of using harmonic wavelets for the classification of complex non-stationary signals is proved. A technique to reduce the feature space dimension using a ‘modulo N reduction’ method is proposed. A classification of 26 individual whales from the Whale FM Project dataset is presented. It is shown that the deep-learning-based approach provides the best result for the Whale FM Project dataset both for whale types and individuals.

Using Virtual Reality to Improve Performance and User Experience in Manual Correction of MRI Segmentation Errors by Non-experts

Segmentation of MRI scans is a critical part of the workflow process before we can further analyze neuroimaging data. Although there are several automatic tools for segmentation, no segmentation software is perfectly accurate, and manual correction by visually inspecting the segmentation errors is required. The process of correcting these errors is tedious and time-consuming, so we present a novel method of performing this task in a head-mounted virtual reality interactive system with a new software, Virtual Brain Segmenter (VBS). We provide the results of user testing on 30 volunteers to show the benefits of our tool as a more efficient, intuitive, and engaging alternative compared with the current method of correcting segmentation errors.

Comparing Performances of Five Distinct Automatic Classifiers for Fin Whale Vocalizations in Beamformed Spectrograms of Coherent Hydrophone Array

A large variety of sound sources in the ocean, including biological, geophysical, and man-made, can be simultaneously monitored over instantaneous continental-shelf scale regions via the passive ocean acoustic waveguide remote sensing (POAWRS) technique by employing a large-aperture densely-populated coherent hydrophone array system. Millions of acoustic signals received on the POAWRS system per day can make it challenging to identify individual sound sources. An automated classification system is necessary to enable sound sources to be recognized. Here, the objectives are to (i) gather a large training and test data set of fin whale vocalization and other acoustic signal detections; (ii) build multiple fin whale vocalization classifiers, including a logistic regression, support vector machine (SVM), decision tree, convolutional neural network (CNN), and long short-term memory (LSTM) network; (iii) evaluate and compare performance of these classifiers using multiple metrics including accuracy, precision, recall and F1-score; and (iv) integrate one of the classifiers into the existing POAWRS array and signal processing software. The findings presented here will (1) provide an automatic classifier for near real-time fin whale vocalization detection and recognition, useful in marine mammal monitoring applications; and (2) lay the foundation for building an automatic classifier applied for near real-time detection and recognition of a wide variety of biological, geophysical, and man-made sound sources typically detected by the POAWRS system in the ocean.

Comparing context-dependent call sequences employing machine learning methods: an indication of syntactic structure of greater horseshoe bats

For analysis of vocal syntax, accurate classification of call sequence structures in different behavioural contexts is essential. However, an effective, intelligent program for classifying call sequences from numerous recorded sound files is still lacking. Here, we employed three machine learning algorithms (logistic regression, support vector machine and decision trees) to classify call sequences of social vocalizations of greater horseshoe bats (Rhinolophus ferrumequinum) in aggressive and distress contexts. The three machine learning algorithms obtained highly accurate classification rates (logistic regression 98%, support vector machine 97% and decision trees 96%). The algorithms also extracted three of the most important features for the classification: the transition between two adjacent syllables, the probability of occurrences of syllables in each position of a sequence, and the characteristics of a sequence. The results of statistical analysis also supported the classification of the algorithms. The study provides the first efficient method for data mining of call sequences and the possibility of linguistic parameters in animal communication. It suggests the presence of song-like syntax in the social vocalizations emitted within a non-breeding context in a bat species.

Large-Scale Whale-Call Classification by Transfer Learning on Multi-Scale Waveforms and Time-Frequency Features

Whale vocal calls contain valuable information and abundant characteristics that are important for classification of whale sub-populations and related biological research. In this study, an effective data-driven approach based on pre-trained Convolutional Neural Networks (CNN) using multi-scale waveforms and time-frequency feature representations is developed in order to perform the classification of whale calls from a large open-source dataset recorded by sensors carried by whales. Specifically, the classification is carried out through a transfer learning approach by using pre-trained state-of-the-art CNN models in the field of computer vision. 1D raw waveforms and 2D log-mel features of the whale-call data are respectively used as the input of CNN models. For raw waveform input, windows are applied to capture multiple sketches of a whale-call clip at different time scales and stack the features from different sketches for classification. When using the log-mel features, the delta and delta-delta features are also calculated to produce a 3-channel feature representation for analysis. In the training, a 4-fold cross-validation technique is employed to reduce the overfitting effect, while the Mix-up technique is also applied to implement data augmentation in order to further improve the system performance. The results show that the proposed method can improve the accuracies by more than 20% in percentage for the classification into 16 whale pods compared with the baseline method using groups of 2D shape descriptors of spectrograms and the Fisher discriminant scores on the same dataset. Moreover, it is shown that classifications based on log-mel features have higher accuracies than those based directly on raw waveforms. The phylogeny graph is also produced to significantly illustrate the relationships among the whale sub-populations.

Crowdsourcing scoring of immunohistochemistry images: Evaluating Performance of the Crowd and an Automated Computational Method

The assessment of protein expression in immunohistochemistry (IHC) images provides important diagnostic, prognostic and predictive information for guiding cancer diagnosis and therapy. Manual scoring of IHC images represents a logistical challenge, as the process is labor intensive and time consuming. Since the last decade, computational methods have been developed to enable the application of quantitative methods for the analysis and interpretation of protein expression in IHC images. These methods have not yet replaced manual scoring for the assessment of IHC in the majority of diagnostic laboratories and in many large-scale research studies. An alternative approach is crowdsourcing the quantification of IHC images to an undefined crowd. The aim of this study is to quantify IHC images for labeling of ER status with two different crowdsourcing approaches, image-labeling and nuclei-labeling, and compare their performance with automated methods. Crowdsourcing- derived scores obtained greater concordance with the pathologist interpretations for both image-labeling and nuclei-labeling tasks (83% and 87%), as compared to the pathologist concordance achieved by the automated method (81%) on 5,338 TMA images from 1,853 breast cancer patients. This analysis shows that crowdsourcing the scoring of protein expression in IHC images is a promising new approach for large scale cancer molecular pathology studies.

Physical constraints of cultural evolution of dialects in killer whales

Odontocete sounds are produced by two pairs of phonic lips situated in soft nares below the blowhole; the right pair is larger and is more likely to produce clicks, while the left pair is more likely to produce whistles. This has important implications for the cultural evolution of delphinid sounds: the greater the physical constraints, the greater the probability of random convergence. In this paper the authors examine the call structure of eight killer whale populations to identify structural constraints and to determine if they are consistent among all populations. Constraints were especially pronounced in two-voiced calls. In the calls of all eight populations, the lower component of two-voiced (biphonic) calls was typically centered below 4 kHz, while the upper component was typically above that value. The lower component of two-voiced calls had a narrower frequency range than single-voiced calls in all populations. This may be because some single-voiced calls are homologous to the lower component, while others are homologous to the higher component of two-voiced calls. Physical constraints on the call structure reduce the possible variation and increase the probability of random convergence, producing similar calls in different populations.

Quantitative analysis of the acoustic repertoire of southern right whales in New Zealand

Quantitatively describing the acoustic repertoire of a species is important for establishing effective passive acoustic monitoring programs and developing automated call detectors. This process is particularly important when the study site is remote and visual surveys are not cost effective. Little is known about the vocal behavior of southern right whales (Eubalaena australis) in New Zealand. The aim of this study was to describe and quantify their entire vocal repertoire on calving grounds in the sub-Antarctic Auckland Islands. Over three austral winters (2010-2012), 4349 calls were recorded, measured, and classified into 10 call types. The most frequently observed types were pulsive, upcall, and tonal low vocalizations. A long tonal low call (≤15.5 s duration) and a very high call (peak frequency ∼750 Hz) were described for the first time. Random Forest multivariate analysis of 28 measured variables was used to classify calls with a high degree of accuracy (82%). The most important variables for classification were maximum ceiling frequency, number of inflection points, duration, and the difference between the start and end frequency. This classification system proved to be a repeatable, fast, and objective method for categorising right whale calls and shows promise for other vocal taxa.

New Approaches to Marine Conservation Through the Scaling Up of Ecological Data

In an era of rapid global change, conservation managers urgently need improved tools to track and counter declining ecosystem conditions. This need is particularly acute in the marine realm, where threats are out of sight, inadequately mapped, cumulative, and often poorly understood, thereby generating impacts that are inefficiently managed. Recent advances in macroecology, statistical analysis, and the compilation of global data will play a central role in improving conservation outcomes, provided that global, regional, and local data streams can be integrated to produce locally relevant and interpretable outputs. Progress will be assisted by (a) expanded rollout of systematic surveys that quantify species patterns, including some carried out with help from citizen scientists; (b) coordinated experimental research networks that utilize large-scale manipulations to identify mechanisms underlying these patterns;

Cultural evolution of killer whale calls: background, mechanisms and consequences

Cultural evolution is a powerful process shaping behavioural phenotypes of many species including our own. Killer whales are one of the species with relatively well-studied vocal culture. Pods have distinct dialects comprising a mix of unique and shared call types; calves adopt the call repertoire of their matriline through social learning. We review different aspects of killer whale acoustic communication to provide insights into the cultural transmission and gene-culture co-evolution processes that produce the extreme diversity of group and population repertoires. We argue that the cultural evolution of killer whale calls is not a random process driven by steady error accumulation alone: temporal change occurs at different speeds in different components of killer whale repertoires, and constraints in call structure and horizontal transmission often degrade the phylogenetic signal. We discuss the implications from bird song and human linguistic studies, and propose several hypotheses of killer whale dialect evolution.