A game theoretic approach to multimodal communication

Sensory collectives in natural systems

Groups of animals inhabit vastly different sensory worlds, or umwelten, which shape fundamental aspects of their behaviour. Yet the sensory ecology of species is rarely incorporated into the emerging field of collective behaviour, which studies the movements, population-level behaviours, and emergent properties of animal groups. Here, we review the contributions of sensory ecology and collective behaviour to understanding how animals move and interact within the context of their social and physical environments. Our goal is to advance and bridge these two areas of inquiry and highlight the potential for their creative integration. To achieve this goal, we organise our review around the following themes: (1) identifying the promise of integrating collective behaviour and sensory ecology; (2) defining and exploring the concept of a 'sensory collective'; (3) considering the potential for sensory collectives to shape the evolution of sensory systems; (4) exploring examples from diverse taxa to illustrate neural circuits involved in sensing and collective behaviour; and (5) suggesting the need for creative conceptual and methodological advances to quantify 'sensescapes'. In the final section, (6) applications to biological conservation, we argue that these topics are timely, given the ongoing anthropogenic changes to sensory stimuli (e.g. via light, sound, and chemical pollution) which are anticipated to impact animal collectives and group-level behaviour and, in turn, ecosystem composition and function. Our synthesis seeks to provide a forward-looking perspective on how sensory ecologists and collective behaviourists can both learn from and inspire one another to advance our understanding of animal behaviour, ecology, adaptation, and evolution.

Honey bees respond to multimodal stimuli following the Principle of Inverse Effectiveness

Multisensory integration is assumed to entail benefits for receivers across multiple ecological contexts. However, signal integration effectiveness is constrained by features of the spatiotemporal and intensity domains. How sensory modalities are integrated during tasks facilitated by learning and memory, such as pollination, remains unsolved. Honey bees use olfactory and visual cues during foraging, making them a good model to study the use of multimodal signals. Here, we examined the effect of stimulus intensity on both learning and memory performance of bees trained using unimodal or bimodal stimuli. We measured the performance and the latency response across planned discrete levels of stimulus intensity. We employed the conditioning of the proboscis extension response protocol in honey bees using an electromechanical setup allowing us to control simultaneously and precisely olfactory and visual stimuli at different intensities. Our results show that the bimodal enhancement during learning and memory was higher as the intensity decreased when the separate individual components were least effective. Still, this effect was not detectable for the latency of response. Remarkably, these results support the principle of inverse effectiveness, traditionally studied in vertebrates, predicting that multisensory stimuli are more effectively integrated when the best unisensory response is relatively weak. Thus, we argue that the performance of the bees while using a bimodal stimulus depends on the interaction and intensity of its individual components. We further hold that the inclusion of findings across all levels of analysis enriches the traditional understanding of the mechanics and reliance of complex signals in honey bees.

Summary: Bimodal enhancement during learning and memory tasks in africanized honey bees increases as the stimulus intensity of its unimodal components decreases; this indicates that learning performance depends on the interaction between the intensity of its components and the nature of the sensory modalities involved, supporting the principle of inverse effectiveness.

Androgen Receptor Modulates Multimodal Displays in the Bornean Rock Frog (Staurois parvus)

Multimodal communication is common in the animal kingdom. It occurs when animals display by stimulating two or more receiver sensory systems, and often arises when selection favors multiple ways to send messages to conspecifics. Mechanisms of multimodal display behavior are poorly understood, particularly with respect to how animals coordinate the production of different signals. One important question is whether all components in a multimodal display share an underlying physiological basis, or whether different components are regulated independently. We investigated the influence of androgen receptors (ARs) on the production of both visual and vocal signal components in the multimodal display repertoire of the Bornean rock frog (Staurois parvus). To assess the role of AR in signal production, we treated reproductively active adult males with the antiandrogen flutamide (FLUT) and measured the performance of each component signal in the multimodal display. Our results show that blocking AR inhibited the production of multiple visual signals, including a conspicuous visual signal known as the "foot flag," which is produced by rotating the hind limb above the body. However, FLUT treatment caused no measurable change in vocal signaling behavior, or in the frequency or fine temporal properties of males' calls. Our study, therefore, suggests that activation of AR is not a physiological prerequisite to the coordination of multiple signals, in that it either does not regulate all signaling behaviors in a male's display repertoire or it does so only in a context-dependent manner.

Volatile fatty acid and aldehyde abundances evolve with behavior and habitat temperature in Sceloporus lizards

Animal signals evolve by striking a balance between the need to convey information through particular habitats and the limitations of what types of signals can most easily be produced and perceived. Here, we present new results from field measures of undisturbed behavior and biochemical analyses of scent marks from 12 species of Sceloporus lizards to explore whether evolutionary changes in chemical composition are better predicted by measures of species behavior, particularly those associated with visual displays, chemoreception, and locomotion, or by measures of habitat climate (precipitation and temperature). We found that more active lizard species used fewer compounds in their volatile scent marks, perhaps conveying less specific information about individual and species identity. Scent marks from more active lizard species also had higher proportions of saturated fatty acids, and the evolution of these compounds has been tracking the phylogeny closely as we would expect for a metabolic byproduct. In contrast, the proportions of unsaturated fatty acids were better explained by evolutionary shifts in habitat temperature (and not precipitation), with species in warmer climates using almost no volatile unsaturated fatty acids. The proportion of aldehydes was explained by both behavior and environment, decreasing with behavioral activity and increasing with habitat temperature. Our results highlight the evolutionary flexibility of complex chemical signals, with different chemical compounds responding to different elements of the selective landscape over evolutionary time.

Multimodal pair-bond maintenance: A review of signaling across modalities in pair-bonded nonhuman primates

Only a handful of primate species exhibit the social relationship of pair-bonding. Efficient communication is critical for behavioral coordination within pair-bonds to maintain proximity and respond appropriately to extra-pair individuals, and possibly coordinate infant care. The use of complex signaling across modalities may help individuals improve communicative outcomes. We review many ways that pair-bonded species use signals to communicate and maintain bonds, though little previous research has taken a truly multimodal approach within a single species. We make a call for further investigation into pair-bonded communication using a multimodal approach to better understand how these species use all their senses to build, maintain, and advertise their bonds.

Evolution and function of multimodal courtship displays

Courtship displays are behaviours aimed to facilitate attraction and mating with the opposite sex and are very common across the animal kingdom. Most courtship displays are multimodal, meaning that they are composed of concomitant signals occurring in different sensory modalities. Although courtship often strongly influences reproductive success, the question of why and how males use multimodal courtship to increase their fitness has not yet received much attention. Very little is known about the role of different components of male courtship and their relative importance for females. Indeed, most of the work on courtship displays have focused on effects on female choice, often neglecting other possible roles. Additionally, a number of scientists have recently stressed the importance of considering the complexity of a display and the interactions between its different components in order to grasp all the information contained in those multimodal signals. Unfortunately, these methods have not yet been extensively adapted in courtship studies. The aim of this study was to review what is currently known about the functional significance of courtship displays, particularly about the role of multimodality in the courtship communication context. Emphasis is placed on those cases where a complete picture of the communication system can only be assessed by taking complexity and interaction between different modalities into account.

The genetics of phenotypic plasticity. XVI. Interactions among traits and the flow of information

Although the environment varies, adaptive trait plasticity is uncommon, which can be due to either costs or limitations. Currently there is little evidence for costs of plasticity; limitations are a more promising explanation, including information reliability. A possible cause for a decrease in information reliability is the channeling of environmental information through one trait that then affects the phenotype of a second trait, the information path. Using an individual-based simulation model, I explored the ways in which configurations of trait interactions and patterns of environmental variation in space and time affect the evolution of phenotypic plasticity. I found that genotypes and phenotypes evolved to shorten and simplify the information path from the environment to fitness. A shortened path was characterized by a decrease in the amount of plasticity for traits that had a less direct connection between the environment of development and fitness. A simplified path was characterized by a decrease in the amount of plasticity for traits that had multiple paths between the environment and their phenotype. These results suggest that an eighth proposition be added to the theory of the evolution of phenotypic plasticity: Trait plasticity will evolve to minimize the information path between the environment and fitness.

Multimodal cues improve prey localization under complex environmental conditions

Predators often eavesdrop on sexual displays of their prey. These displays can provide multimodal cues that aid predators, but the benefits in attending to them should depend on the environmental sensory conditions under which they forage. We assessed whether bats hunting for frogs use multimodal cues to locate their prey and whether their use varies with ambient conditions. We used a robotic set-up mimicking the sexual display of a male túngara frog (Physalaemus pustulosus) to test prey assessment by fringe-lipped bats (Trachops cirrhosus). These predatory bats primarily use sound of the frog's call to find their prey, but the bats also use echolocation cues returning from the frog's dynamically moving vocal sac. In the first experiment, we show that multimodal cues affect attack behaviour: bats made narrower flank attack angles on multimodal trials compared with unimodal trials during which they could only rely on the sound of the frog. In the second experiment, we explored the bat's use of prey cues in an acoustically more complex environment. Túngara frogs often form mixed-species choruses with other frogs, including the hourglass frog (Dendropsophus ebraccatus). Using a multi-speaker set-up, we tested bat approaches and attacks on the robofrog under three different levels of acoustic complexity: no calling D. ebraccatus males, two calling D. ebraccatus males and five D. ebraccatus males. We found that bats are more directional in their approach to the robofrog when more D. ebraccatus males were calling. Thus, bats seemed to benefit more from multimodal cues when confronted with increased levels of acoustic complexity in their foraging environments. Our data have important consequences for our understanding of the evolution of multimodal sexual displays as they reveal how environmental conditions can alter the natural selection pressures acting on them.

Wind- and Rain-Induced Vibrations Impose Different Selection Pressures on Multimodal Signaling

The world is a noisy place, and animals have evolved a myriad of strategies to communicate in it. Animal communication signals are, however, often multimodal; their components can be processed by multiple sensory systems, and noise can thus affect signal components across different modalities. We studied the effect of environmental noise on multimodal communication in the túngara frog (Physalaemus pustulosus). Males communicate with rivals using airborne sounds combined with call-induced water ripples. We tested males under control as well as noisy conditions in which we mimicked rain- and wind-induced vibrations on the water surface. Males responded more strongly to a multimodal playback in which sound and ripples were combined, compared to a unimodal sound-only playback, but only in the absence of rain and wind. Under windy conditions, males decreased their response to the multimodal playback, suggesting that wind noise interferes with the detection of rival ripples. Under rainy conditions, males increased their response, irrespective of signal playback, suggesting that different noise sources can have different impacts on communication. Our findings show that noise in an additional sensory channel can affect multimodal signal perception and thereby drive signal evolution, but not always in the expected direction.

Signal Diversity, Sexual Selection, and Speciation

Communication is ubiquitous. Developing a framework for the diversity of signals has important consequences for understanding alternative models of sexual selection and the processes contributing to speciation. In this article we review how models of neutral evolution in the perceptual space of signal perceivers provide a first step toward constructing a framework for signal diversity. We discuss how the distinction between additive and multiplicative effects of multimodal signaling represents a second step. We then assess how signal efficiency, reliability, and the aesthetics of perceivers provide distinct mechanisms for signals to be effective, thereby partly explaining signal diversity. Understanding the relative contribution of each of these mechanisms to the effectiveness of mate choice signals unravels the relative importance of alternative models of sexual selection. It can also help to distinguish whether divergence of communication is a driver or a consequence of speciation. Throughout the review we emphasize the importance of verification and learning in repeated interactions for understanding variation in signals.

Multimodal signalling in the North American barn swallow: a phenotype network approach

Complex signals, involving multiple components within and across modalities, are common in animal communication. However, decomposing complex signals into traits and their interactions remains a fundamental challenge for studies of phenotype evolution. We apply a novel phenotype network approach for studying complex signal evolution in the North American barn swallow (Hirundo rustica erythrogaster). We integrate model testing with correlation-based phenotype networks to infer the contributions of female mate choice and male-male competition to the evolution of barn swallow communication. Overall, the best predictors of mate choice were distinct from those for competition, while moderate functional overlap suggests males and females use some of the same traits to assess potential mates and rivals. We interpret model results in the context of a network of traits, and suggest this approach allows researchers a more nuanced view of trait clustering patterns that informs new hypotheses about the evolution of communication systems.

The problem of multimodal concurrent serial order in behavior

The "problem of serial order in behavior," as formulated and discussed by Lashley (1951), is arguably more pervasive and more profound both than originally stated and than currently appreciated. We spell out two complementary aspects of what we term the generalized problem of behavior: (i) multimodality, stemming from the disparate nature of the sensorimotor variables and processes that underlie behavior, and (ii) concurrency, which reflects the parallel unfolding in time of these processes and of their asynchronous interactions. We illustrate these on a number of examples, with a special focus on language, briefly survey the computational approaches to multimodal concurrency, offer some hypotheses regarding the manner in which brains address it, and discuss some of the broader implications of these as yet unresolved issues for cognitive science.

Evolving from static to dynamic signals: evolutionary compensation between two communicative signals

Signals that convey related information may impose selection on each other, creating evolutionary links between different components of the communicative repertoire. Here, we ask about the consequences of the evolutionary loss of one signal (a colour patch) on another (a motion display) in Sceloporus lizards. We present data on male lizards of four species: two pairs of sister taxa representing two independent evolutionary losses of the static colour patch (Sceloporus cozumelae and Sceloporus parvus; Sceloporus siniferus and Sceloporus merriami). Males of the two species that have undergone an evolutionary loss of blue-belly patches (S. cozumelae, S. siniferus) were less active than their blue-bellied sister taxa (S. parvus, S. merriami), consistent with the suggestion that the belly patches were lost to reduce conspicuousness of species with high predation pressure. In contrast, the headbob display appears to have become more, rather than less, conspicuous over evolutionary time. The colour patch is exhibited primarily during aggressive encounters, whereas headbob displays are multifunction signals used in several different contexts, including aggressive encounters. Males of species that have lost the colour patch produced more motion displays, and the structure of those motion displays were more similar to those produced during combat. In both evolutionary episodes, a static colour signal appears to have been replaced by dynamic motion displays that can be turned off in the presence of predators and other unwanted receivers. The predominant pattern is one of evolutionary compensation and interactions between multiple signals that convey related information.

Risks of multimodal signaling: bat predators attend to dynamic motion in frog sexual displays

Many sexual displays contain multiple components that are received through a variety of sensory modalities. Primary and secondary signal components can interact to induce novel receiver responses and become targets of sexual selection as complex signals. However, predators can also use these complex signals for prey assessment, which may limit the evolution of elaborate sexual signals. We tested whether a multimodal sexual display of the male túngara frog (Physalaemus pustulosus) increases predation risk from the fringe-lipped bat (Trachops cirrhosus) when compared with a unimodal display. We gave bats a choice to attack one of two frog models: a model with a vocal sac moving in synchrony with a mating call (multisensory cue), or a control model with the call but no vocal sac movement (unimodal cue). Bats preferred to attack the model associated with the multimodal display. Furthermore, we determined that bats perceive the vocal sac using echolocation rather than visual cues. Our data illustrate the costs associated with multimodal signaling and that sexual and natural selection pressures on the same trait are not always mediated through the same sensory modalities. These data are important when considering the role of environmental fluctuations on signal evolution as different sensory modalities will be differentially affected.

Multimodality: a way to cope with road traffic noise? The case of European treefrog (Hyla arborea)

In the last decades, traffic noise has become a new challenge for efficient animal communication and several studies suggest that it is involved in population declines. Although poorly investigated in a traffic noise context, communication is generally multimodal, which can be viewed as a way to improve communication efficiency by allowing shift from a sensory modality to another when one modality suffers from noise. In the present study, we investigated multimodal shift in the European treefrog (Hyla arborea), a species using both acoustic and visual cues during male quality assessment task performed by females. Females were used in a discrimination task in two environmental conditions: with or without traffic noise. In traffic noise conditions, we showed that females' reliance on acoustic signal embedded in noise pollution did not decrease in favour of visual signals, therefore showing that females do not shift between modalities in response to traffic noise. Although, we did not evidence multimodal shift in our study, many species facing traffic noise present multimodal communication, and should be investigated to evaluate the importance of multimodal shift as a way to cope with traffic noise pollution.

Ten unanswered questions in multimodal communication

The study of multimodal communication has become an active and vibrant field. This special issue of Behavioral Ecology and Sociobiology brings together new developments in this rapidly expanding area. In this final contribution to the special issue, I look to the future and discuss ten questions in need of further work, touching on issues ranging from theoretical modeling and the evolution of behavior to molecular mechanisms and the development of behavior. In particular, I emphasize that the use of multimodal communication allows animals to switch between sensory channels when one channel becomes too noisy, and suggest that a better understanding of this process may help us both to understand the evolution of multisensory signaling and to predict the success of species facing environmental changes that affect signaling channels, such as urbanization and climate change. An expanded section is included on the effects of climate change on animal communication across sensory channels, urging researchers to pursue this topic due to the rapidity with which the environment is currently transforming.