FathomNet: A global image database for enabling artificial intelligence in the ocean

An interactive AI-driven platform for fish age reading

Fish age is an important biological variable required as part of routine stock assessment and analysis of fish population dynamics. Age estimates are traditionally obtained by human experts from the count of ring-like patterns along calcified structures such as otoliths. To automate the process and minimize human bias, modern methods have been designed utilizing the advances in the field of artificial intelligence (AI). While many AI-based methods have been shown to attain satisfactory accuracy, there are concerns regarding the lack of explainability of some early implementations. Consequently, new explainable AI-based approaches based on U-Net and Mask R-CNN have been recently published having direct compatibility with traditional ring counting procedures. Here we further extend this endeavor by creating an interactive website housing these explainable AI methods allowing age readers to be directly involved in the AI training and development. An important aspect of the platform presented in this article is that it allows the additional use of different advanced concepts of Machine Learning (ML) such as transfer learning, ensemble learning and continual learning, which are all shown to be effective in this study.

Artificial intelligence for geoscience: Progress, challenges, and perspectives

This paper explores the evolution of geoscientific inquiry, tracing the progression from traditional physics-based models to modern data-driven approaches facilitated by significant advancements in artificial intelligence (AI) and data collection techniques. Traditional models, which are grounded in physical and numerical frameworks, provide robust explanations by explicitly reconstructing underlying physical processes. However, their limitations in comprehensively capturing Earth's complexities and uncertainties pose challenges in optimization and real-world applicability. In contrast, contemporary data-driven models, particularly those utilizing machine learning (ML) and deep learning (DL), leverage extensive geoscience data to glean insights without requiring exhaustive theoretical knowledge. ML techniques have shown promise in addressing Earth science-related questions. Nevertheless, challenges such as data scarcity, computational demands, data privacy concerns, and the "black-box" nature of AI models hinder their seamless integration into geoscience. The integration of physics-based and data-driven methodologies into hybrid models presents an alternative paradigm. These models, which incorporate domain knowledge to guide AI methodologies, demonstrate enhanced efficiency and performance with reduced training data requirements. This review provides a comprehensive overview of geoscientific research paradigms, emphasizing untapped opportunities at the intersection of advanced AI techniques and geoscience. It examines major methodologies, showcases advances in large-scale models, and discusses the challenges and prospects that will shape the future landscape of AI in geoscience. The paper outlines a dynamic field ripe with possibilities, poised to unlock new understandings of Earth's complexities and further advance geoscience exploration.

The best of two worlds: reprojecting 2D image annotations onto 3D models

Imagery has become one of the main data sources for investigating seascape spatial patterns. This is particularly true in deep-sea environments, which are only accessible with underwater vehicles. On the one hand, using collaborative web-based tools and machine learning algorithms, biological and geological features can now be massively annotated on 2D images with the support of experts. On the other hand, geomorphometrics such as slope or rugosity derived from 3D models built with structure from motion (sfm) methodology can then be used to answer spatial distribution questions. However, precise georeferencing of 2D annotations on 3D models has proven challenging for deep-sea images, due to a large mismatch between navigation obtained from underwater vehicles and the reprojected navigation computed in the process of building 3D models. In addition, although 3D models can be directly annotated, the process becomes challenging due to the low resolution of textures and the large size of the models. In this article, we propose a streamlined, open-access processing pipeline to reproject 2D image annotations onto 3D models using ray tracing. Using four underwater image datasets, we assessed the accuracy of annotation reprojection on 3D models and achieved successful georeferencing to centimetric accuracy. The combination of photogrammetric 3D models and accurate 2D annotations would allow the construction of a 3D representation of the landscape and could provide new insights into understanding species microdistribution and biotic interactions.

An in situ digital synthesis strategy for the discovery and description of ocean life

Revolutionary advancements in underwater imaging, robotics, and genomic sequencing have reshaped marine exploration. We present and demonstrate an interdisciplinary approach that uses emerging quantitative imaging technologies, an innovative robotic encapsulation system with in situ RNA preservation and next-generation genomic sequencing to gain comprehensive biological, biophysical, and genomic data from deep-sea organisms. The synthesis of these data provides rich morphological and genetic information for species description, surpassing traditional passive observation methods and preserved specimens, particularly for gelatinous zooplankton. Our approach enhances our ability to study delicate mid-water animals, improving research in the world's oceans.

A synergistic future for AI and ecology

Research in both ecology and AI strives for predictive understanding of complex systems, where nonlinearities arise from multidimensional interactions and feedbacks across multiple scales. After a century of independent, asynchronous advances in computational and ecological research, we foresee a critical need for intentional synergy to meet current societal challenges against the backdrop of global change. These challenges include understanding the unpredictability of systems-level phenomena and resilience dynamics on a rapidly changing planet. Here, we spotlight both the promise and the urgency of a convergence research paradigm between ecology and AI. Ecological systems are a challenge to fully and holistically model, even using the most prominent AI technique today: deep neural networks. Moreover, ecological systems have emergent and resilient behaviors that may inspire new, robust AI architectures and methodologies. We share examples of how challenges in ecological systems modeling would benefit from advances in AI techniques that are themselves inspired by the systems they seek to model. Both fields have inspired each other, albeit indirectly, in an evolution toward this convergence. We emphasize the need for more purposeful synergy to accelerate the understanding of ecological resilience whilst building the resilience currently lacking in modern AI systems, which have been shown to fail at times because of poor generalization in different contexts. Persistent epistemic barriers would benefit from attention in both disciplines. The implications of a successful convergence go beyond advancing ecological disciplines or achieving an artificial general intelligence-they are critical for both persisting and thriving in an uncertain future.

Semi-supervised Visual Tracking of Marine Animals Using Autonomous Underwater Vehicles

In-situ visual observations of marine organisms is crucial to developing behavioural understandings and their relations to their surrounding ecosystem. Typically, these observations are collected via divers, tags, and remotely-operated or human-piloted vehicles. Recently, however, autonomous underwater vehicles equipped with cameras and embedded computers with GPU capabilities are being developed for a variety of applications, and in particular, can be used to supplement these existing data collection mechanisms where human operation or tags are more difficult. Existing approaches have focused on using fully-supervised tracking methods, but labelled data for many underwater species are severely lacking. Semi-supervised trackers may offer alternative tracking solutions because they require less data than fully-supervised counterparts. However, because there are not existing realistic underwater tracking datasets, the performance of semi-supervised tracking algorithms in the marine domain is not well understood. To better evaluate their performance and utility, in this paper we provide (1) a novel dataset specific to marine animals located at http://warp.whoi.edu/vmat/, (2) an evaluation of state-of-the-art semi-supervised algorithms in the context of underwater animal tracking, and (3) an evaluation of real-world performance through demonstrations using a semi-supervised algorithm on-board an autonomous underwater vehicle to track marine animals in the wild.

Context-Driven Detection of Invertebrate Species in Deep-Sea Video

Each year, underwater remotely operated vehicles (ROVs) collect thousands of hours of video of unexplored ocean habitats revealing a plethora of information regarding biodiversity on Earth. However, fully utilizing this information remains a challenge as proper annotations and analysis require trained scientists’ time, which is both limited and costly. To this end, we present a Dataset for Underwater Substrate and Invertebrate Analysis (DUSIA), a benchmark suite and growing large-scale dataset to train, validate, and test methods for temporally localizing four underwater substrates as well as temporally and spatially localizing 59 underwater invertebrate species. DUSIA currently includes over ten hours of footage across 25 videos captured in 1080p at 30 fps by an ROV following pre-planned transects across the ocean floor near the Channel Islands of California. Each video includes annotations indicating the start and end times of substrates across the video in addition to counts of species of interest. Some frames are annotated with precise bounding box locations for invertebrate species of interest, as seen in Fig. 1. To our knowledge, DUSIA is the first dataset of its kind for deep sea exploration, with video from a moving camera, that includes substrate annotations and invertebrate species that are present at significant depths where sunlight does not penetrate. Additionally, we present the novel context-driven object detector (CDD) where we use explicit substrate classification to influence an object detection network to simultaneously predict a substrate and species class influenced by that substrate. We also present a method for improving training on partially annotated bounding box frames. Finally, we offer a baseline method for automating the counting of invertebrate species of interest.

Image dataset for benchmarking automated fish detection and classification algorithms

Multiparametric video-cabled marine observatories are becoming strategic to monitor remotely and in real-time the marine ecosystem. Those platforms can achieve continuous, high-frequency and long-lasting image data sets that require automation in order to extract biological time series. The OBSEA, located at 4 km from Vilanova i la Geltrú at 20 m depth, was used to produce coastal fish time series continuously over the 24-h during 2013-2014. The image content of the photos was extracted via tagging, resulting in 69917 fish tags of 30 taxa identified. We also provided a meteorological and oceanographic dataset filtered by a quality control procedure to define real-world conditions affecting image quality. The tagged fish dataset can be of great importance to develop Artificial Intelligence routines for the automated identification and classification of fishes in extensive time-lapse image sets.