Looking at paintings in the Vincent Van Gogh Museum: Eye movement patterns of children and adults

Effectiveness of labels in digital art experience: psychophysiological and behavioral evidence

Introduction

Nowadays museums make large use of digital materials (e.g., virtual tours) to attract visitors. Therefore, it is worthwhile investigating which variables affect the engagement with art outside the museum, and whether digital reproductions of artworks are as effective as museum originals in producing a satisfying aesthetic experience.

Methods

Here we tested the effectiveness of introducing additional informative materials on the artistic enjoyment of contemporary paintings presented on a computer screen. Naïve observers were exposed to essential and descriptive labels before viewing artworks. We flanked traditional measurement methods - viewing times and questionnaires, with biometric parameters - pupil responses, eye movements, heart rate, and electrodermal activity. The results were then compared to our previous museum study that adopted the same experimental paradigm.

Results

Our behavioral and psychophysiological data lead to a complex pattern of results. As found in the museum setting, providing detailed descriptions decreases complexity, evokes more positive sensations, and induces pupil dilation but does not enhance aesthetic appreciation. These results suggested that informative labels improve understanding and emotions but have a limited impact on the hedonic evaluation of artworks in both contexts. However, other results do not mirror those found in the museum; in the laboratory setting, participants spend a similar amount of time, have a comparable gaze behavior, and their electrodermal activity and heart rate do not change when viewing artworks with different types of labels. The main difference between the lab and museum settings is the shorter time spent viewing digital reproductions vs. real paintings, although subjective ratings (e.g., liking, interest) are comparable.

Discussion

Overall, this study indicates that the environmental context does impact the aesthetic experience; although, some beneficial effects of introducing additional relevant content in labels accompanying artworks can also be acquainted through digital media outside of the museum.

Noise estimation for head-mounted 3D binocular eye tracking using Pupil Core eye-tracking goggles

Head-mounted, video-based eye tracking is becoming increasingly common and has promise in a range of applications. Here, we provide a practical and systematic assessment of the sources of measurement uncertainty for one such device - the Pupil Core - in three eye-tracking domains: (1) the 2D scene camera image; (2) the physical rotation of the eye relative to the scene camera 3D space; and (3) the external projection of the estimated gaze point location onto the target plane or in relation to world coordinates. We also assess eye camera motion during active tasks relative to the eye and the scene camera, an important consideration as the rigid arrangement of eye and scene camera is essential for proper alignment of the detected gaze. We find that eye camera motion, improper gaze point depth estimation, and erroneous eye models can all lead to added noise that must be considered in the experimental design. Further, while calibration accuracy and precision estimates can help assess data quality in the scene camera image, they may not be reflective of errors and variability in gaze point estimation. These findings support the importance of eye model constancy for comparisons across experimental conditions and suggest additional assessments of data reliability may be warranted for experiments that require the gaze point or measure eye movements relative to the external world.

How do people distribute their attention while observing The Night Watch?

This study explored how people look at The Night Watch (1642), Rembrandt's masterpiece. Twenty-one participants each stood in front of the painting for 5 min, while their eyes were recorded with a mobile eye-tracker and their thoughts were verbalized with a think-aloud method. We computed a heatmap of the participants’ attentional distribution using a novel markerless mapping method. The results showed that the participants’ attention was mainly directed at the faces of the two central figures, the bright mascot girl in the painting, and detailed elements such as the apparel of the key figures. The eye-movement analysis and think-aloud data also showed that participants’ attention shifted from the faces of the key figures to other elements of the scene over the course of the 5 min. Our analyses are consistent with the theory that Rembrandt used light and texture to capture the viewer's attention. Finally, the robustness of the eye-tracking method was demonstrated by replicating the study on a smaller replica.

Relations between children's exploration in a children's museum and their reflections about their exploration

We examined 6- to 9-year-olds' (N = 60, 35 girls, 34% White, 23% Hispanic, 2% Black/African American, 2% Asian/Asian American, 22% Mixed Ethnicity/Race, 17% Unavailable, collected April-September 2019 in Providence, RI, USA) first-person perspectives on their exploration of museum exhibits. We coded goal setting, goal completion, and behaviors that reflected changes to how goals were accomplished. Whether children played collaboratively related to how often they revised behaviors to accomplish goals (OR = 2.14). When asked to reflect on their play, older children related talk about goals with behavioral revisions, demonstrating that children develop the ability to reflect on their goals when they watch their behaviors change (OR = 1.23). We discuss how these results inform the development of metacognitive reflection on learning through exploration.

The influence of visitor-based social contextual information on visitors’ museum experience

Visitor-centered approaches have been widely discussed in the museum experience research field. One notable approach was suggested by Falk and Dierking, who defined museum visitor experience as having a physical, personal, and social context. Many studies have been conducted based on this approach, yet the interactions between personal and social contexts have not been fully researched. Since previous studies related to these interactions have focused on the face-to-face conversation of visitor groups, attempts to provide the social information contributed by visitors have not progressed. To fill this gap, we examined such interactions in collaboration with the Lee-Ungno Art Museum in South Korea. Specifically, we investigated the influence of individual visitors’ social contextual information about their art museum experience. This data, which we call “visitor-based social contextual information” (VSCI), is the social information individuals provide—feedback, reactions, or behavioral data—that can be applied to facilitate interactions in a social context. The study included three stages: In Stage 1, we conducted an online survey for a preliminary investigation of visitors’ requirements for VSCI. In Stage 2, we designed a mobile application prototype. Finally, in Stage 3, we used the prototype in an experiment to investigate the influence of VSCI on museum experience based on visitors’ behaviors and reactions. Our results indicate that VSCI positively impacts visitors’ museum experiences. Using VSCI enables visitors to compare their thoughts with others and gain insights about art appreciation, thus allowing them to experience the exhibition from new perspectives. The results of this novel examination of a VSCI application suggest that it may be used to guide strategies for enhancing the experience of museum visitors.

Titles and Semantic Violations Affect Eye Movements When Viewing Contemporary Paintings

The role of titles in perception of visual art is a topic of interesting discussions that brings together artists, curators, and researchers. Titles provide contextual cues and guide perception. They can be particularly useful when paintings include semantic violations that make them challenging for viewers, especially viewers lacking expert knowledge. The aim of this study is to investigate the effects of titles and semantic violations on eye movements. A total of 127 participants without expertise in visual art viewed 40 paintings with and without semantic violations (20 each) in one of three conditions: untitled, consistent titles and inconsistent titles. After each painting was viewed participants also rated liking and understanding. Our results suggest that titles affect the way paintings are viewed: both titled conditions were associated with shorter first fixation duration, longer saccade durations, and amplitudes and higher dynamic entropy than the untitled conditions. Titles were fixated on more frequently (but only in the time window between 1,200 and 2,800 ms) when presented alongside paintings with semantic violations than paintings without violations, and the percentage of fixations to titles was particularly high in the case of paintings with double inconsistencies (inconsistent titles and semantic violations). Also, we found that semantic violations attracted attention early on (300–900 ms), whereas titles received attention later (average first fixation on title was at 936.28 ms) and inconsistencies in titles were processed even later (after 4,000 ms). Finally, semantic violations were associated with higher dynamic entropy than paintings without violations. Our results demonstrate the importance of titles for processing of artworks, especially artworks that present a challenge for the viewers.

Exploring Visitors' Visual Behavior Using Eye-Tracking: The Case of the "Studiolo Del Duca"

Although the understanding of cognitive disciplines has progressed, we know relatively little about how the human brain perceives art. Thanks to the growing interest in visual perception, eye-tracking technology has been increasingly used for studying the interaction between individuals and artworks. In this study, eye-tracking was used to provide insights into non-expert visitors’ visual behaviour as they move freely in the historical room of the “Studiolo del Duca” of the Ducal Palace in Urbino, Italy. Visitors looked for an average of almost two minutes. This study revealed which parts of the artefact captured visitors’ attention and also gives interesting information about the main patterns of fruition.

Similarity of gaze patterns across physical and virtual versions of an installation artwork

An experiment was conducted to compare museum visitors' gaze patterns using mobile eye-trackers, whilst they were engaging with a physical and a virtual reality (VR) installation of Piet Mondrian's Neo-plasticist room design. Visitors' eye movements produced approximately 25,000 fixations and were analysed using linear mixed-effects models. Absolute and area-normalized dwell time analyses yielded mostly non-significant main effects of the environment, indicating similarity of visual exploration patterns between physical and VR settings. One major difference observed was the decrease of average fixation duration in VR, where visitors tended to more rapidly switch focus in this environment with shorter bursts of attentional focus. The experiment demonstrated the ability to compare gaze data between physical and virtual environments as a proxy to measure the similarity of aesthetic experience. Similarity of viewing patterns along with questionnaire results suggested that virtual galleries can be treated as ecologically valid environments that are parallel to physical art galleries.

Spontaneous perception: a framework for task-free, self-paced perception

Flipping through social media feeds, viewing exhibitions in a museum, or walking through the botanical gardens, people consistently choose to engage with and disengage from visual content. Yet, in most laboratory settings, the visual stimuli, their presentation duration, and the task at hand are all controlled by the researcher. Such settings largely overlook the spontaneous nature of human visual experience, in which perception takes place independently from specific task constraints and its time course is determined by the observer as a self-governing agent. Currently, much remains unknown about how spontaneous perceptual experiences unfold in the brain. Are all perceptual categories extracted during spontaneous perception? Does spontaneous perception inherently involve volition? Is spontaneous perception segmented into discrete episodes? How do different neural networks interact over time during spontaneous perception? These questions are imperative to understand our conscious visual experience in daily life. In this article we propose a framework for spontaneous perception. We first define spontaneous perception as a task-free and self-paced experience. We propose that spontaneous perception is guided by four organizing principles that grant it temporal and spatial structures. These principles include coarse-to-fine processing, continuity and segmentation, agency and volition, and associative processing. We provide key suggestions illustrating how these principles may interact with one another in guiding the multifaceted experience of spontaneous perception. We point to testable predictions derived from this framework, including (but not limited to) the roles of the default-mode network and slow cortical potentials in underlying spontaneous perception. We conclude by suggesting several outstanding questions for future research, extending the relevance of this framework to consciousness and spontaneous brain activity. In conclusion, the spontaneous perception framework proposed herein integrates components in human perception and cognition, which have been traditionally studied in isolation, and opens the door to understand how visual perception unfolds in its most natural context.

The aesthetic experience of critical art: The effects of the context of an art gallery and the way of providing curatorial information

The aim of our research was to investigate the influence of the situational context of presenting contemporary critical artworks (in an art gallery vs in a laboratory setting) and the way in which one is acquainted with contextual information, i.e. a curatorial description (reading it on one's own vs listening to it vs a lack of curatorial information), on the reception of critical art. All experimental stimuli were exemplars of contemporary art which raise current controversial social and political issues. Non-experts in the field of art were asked to rate their emotional reactions on non-verbal scales and estimate their liking and understanding of the artworks. As predicted, the art gallery context increased both the experience of aesthetic emotions-in terms of valence, arousal, subjective significance, and dominance and aesthetic judgements-in terms of liking. Thus, for critical art (i.e. current artworks which critically address serious, up-to-date issues) the situational context of the gallery increased the aesthetic experience-which is in line with previous studies on the gallery (or museum) effect. Curatorial information increased understanding, so non-experts seem to need interpretative guidance in the reception of critical art. Subjective significance was higher in the reading of curatorial information condition than the listening to curatorial information condition or the control condition (a lack of curatorial information). It seems, therefore, that art non-experts have a better understanding of critical art after being exposed to the curatorial description, but this does not result in an increase in liking and aesthetic emotions. Probably this is because the curatorial description allows one to grasp the difficult, often unpleasant issue addressed by critical art.

Compare the performance of the models in art classification

Because large numbers of artworks are preserved in museums and galleries, much work must be done to classify these works into genres, styles and artists. Recent technological advancements have enabled an increasing number of artworks to be digitized. Thus, it is necessary to teach computers to analyze (e.g., classify and annotate) art to assist people in performing such tasks. In this study, we tested 7 different models on 3 different datasets under the same experimental setup to compare their art classification performances when either using or not using transfer learning. The models were compared based on their abilities for classifying genres, styles and artists. Comparing the result with previous work shows that the model performance can be effectively improved by optimizing the model structure, and our results achieve state-of-the-art performance in all classification tasks with three datasets. In addition, we visualized the process of style and genre classification to help us understand the difficulties that computers have when tasked with classifying art. Finally, we used the trained models described above to perform similarity searches and obtained performance improvements.

Interaction between image and text during the process of biblical art reception

In our exploratory study, we ask how naive observers, without a distinct religious background, approach biblical art that combines image and text. For this purpose, we choose the book ‘New biblical figures of the Old and New Testament’ published in 1569 as source of the stimuli. This book belongs to the genre of illustrated Bibles, which were very popular during the Reformation. Since there is no empirical knowledge regarding the interaction between image and text during the process of such biblical art reception, we selected four relevant images from the book and measured the eye movements of participants in order to characterize and quantify their scanning behavior related to such stimuli in terms of i) looking at text (text usage), ii) text vs. image interaction measures (semantic or contextual relevance of text), and iii) narration. We show that texts capture attention early in the process of inspection and that text and image interact. Moreover, semantics of texts are used to guide eye movements later through the image, supporting the formation of the narrative.

Task-Dependent Eye-Movement Patterns in Viewing Art

In art schools and classes for art history students are trained to pay attention to different aspects of an artwork, such as art movement characteristics and painting techniques. Experts are better at processing style and visual features of an artwork than nonprofessionals. Here we tested the hypothesis that experts in art use different, task-dependent viewing strategies than nonprofessionals when analyzing a piece of art. We compared a group of art history students with a group of students with no art education background, while viewing 36 paintings under three discrimination tasks. Participants were asked to determine the art movement, the date and the medium of the paintings. We analyzed behavioral and eye-movement data of 27 participants. Our observers adjusted their viewing strategies according to the task, resulting in longer fixation durations and shorter saccade amplitudes for the medium detection task. We found higher task accuracy and subjective confidence, less congruence and higher dispersion in fixation locations in experts. Expertise also influenced saccade metrics, biasing it towards larger saccade amplitudes, advocating a more holistic scanning strategy of experts in all three tasks.

The role that composition plays in determining how a viewer looks at landscape art

Viewing artworks may be subject to the same processes as everyday scene selection in respect of gaze behaviour. However, artists may employ carefully constructed composition in their paintings to lead the eyes of viewers along a predetermined path. This paper investigates whether composition is successful through comparison of expected scanpaths (constructed using the known intention of the artist) and actual scanpaths (as captured using an eye-tracker) based on a loci and sequence similarity index. The findings suggest that composition is successful in leading the eye, although the order of fixations can vary. It could thus be concluded that composition is largely successful in terms of salient elements, but less so for guiding elements. Furthermore, using Cognitive Linguistics theories and applying it to the paintings with reference to the statistical results, the Art Creation Continuum that captures the role of composition on a spectrum is proposed.

Testing a Calibration - Free Eye Tracker Prototype at the Kunsthistorisches Museum in Vienna

Eye tracking research in art viewership is often conducted in a laboratory setting where reproductions must be used in place of original art works and the viewing environment is less natural than in a museum. Recent technological developments have made museum studies possible but head-mounted eye tracking gear and interruptions by researchers still influence the experience of the viewer. In order to find a more ecologically valid way of recording eye movements while viewing artworks, we employed a prototype of a calibrationfree remote eye tracker hidden below selected paintings at the Kunsthistorisches Museum in Vienna. Museum visitors were unaware of the study and informed post hoc that we had registered their viewing behavior and asked to give consent for the use of their data. This article presents the study design as well as results from over 800 participants. While the data quality from the eye tracker prototype was not sufficient to conduct the intended analysis on within-painting gaze movements, this study might serve as a step towards an unobtrusive examination of the art viewing experience. It was possible to analyze time spent viewing paintings and those results show that certain paintings consistently drew significantly more prolonged attention from viewers.

Can we accurately predict where we look at paintings?

The objective of this study is to investigate and to simulate the gaze deployment of observers on paintings. For that purpose, we built a large eye tracking dataset composed of 150 paintings belonging to 5 art movements. We observed that the gaze deployment over the proposed paintings was very similar to the gaze deployment over natural scenes. Therefore, we evaluate existing saliency models and propose a new one which significantly outperforms the most recent deep-based saliency models. Thanks to this new saliency model, we can predict very accurately what are the salient areas of a painting. This opens new avenues for many image-based applications such as animation of paintings or transformation of a still painting into a video clip.

Exploring the awe-some: Mobile eye-tracking insights into awe in a science museum

Informal learning environments provide the opportunity to study guests' experiences as they engage with exhibits specifically designed to invoke the emotional experience of awe. The current paper presents insight gained by using both traditional survey measures and innovative mobile eye-tracking technology to examine guests' experiences of awe in a science museum. We present results for guests' visual attention in two exhibit spaces, one chosen for its potential to evoke positive awe and one for negative awe, and examine associations between visual attention and survey responses with regard to different facets of awe. In this exploratory study, we find relationships between how guests attend to features within an exhibit space (e.g., signage) and their feelings of awe. We discuss implications of using both methods concurrently to shed new light on exhibit design, and more generally for working in transdisciplinary multimethod teams to move scientific knowledge and application forward.

The Display Makes a Difference: A Mobile Eye Tracking Study on the Perception of Art Before and After a Museum's Rearrangement

There is increasing awareness that the perception of art is affected by the way it is presented. In 2018, the Austrian Gallery Belvedere redisplayed its permanent collection. Our multidisciplinary team seized this opportunity to investigate the viewing behavior of specific artworks both before and after the museum's rearrangement. In contrast to previous mobile eye tracking (MET) studies in museums, this study benefits from the comparison of two realistic display conditions (without any research interference), an unconstrained study design (working with regular museum visitors), and a large data sample (comprising 259 participants). We employed a mixed-method approach that combined mobile eye tracking, subjective mapping (a drawing task in conjunction with an open interview), and a questionnaire in order to relate gaze patterns to processes of meaning-making. Our results show that the new display made a difference in that it 1) generally increased the viewing times of the artworks; 2) clearly extended the reading times of labels; and 3) deepened visitors' engagement with the artworks in their exhibition reflections. In contrast, interest in specific artworks and art form preferences proved to be robust and independent of presentation modes.

Viewing Art in Different Contexts

While aesthetic experiences are not limited to any particular context, their sensorial, cognitive and behavioral properties can be profoundly affected by the circumstances in which they occur. Given the ubiquitous nature of contextual effects in nearly all aspects of behavior, investigations aimed at delineating the context-dependent and context-independent aspects of aesthetic experience and engagement with aesthetic objects in a diverse range of settings are important in empirical aesthetics. Here, we analyze the viewing behavior of visitors (N = 19) freely viewing 15 paintings in the 20th-century Australian collection room at the Art Gallery of New South Wales. In particular, we focus on how aspects of viewing behavior including viewing distance in the gallery condition and eye gaze measures such as fixation count, total fixation duration and average fixation duration are affected by the artworks’ physical characteristics including size and image statistics properties such as Fourier amplitude spectrum, fractal dimension and entropy. In addition, the same artworks were viewed in the laboratory, either scaled to fit most of the screen (N = 22) or to preserve their relative size as in the museum condition (N = 17) to assess the robustness of these relationships across different presentation contexts. We find that the effects of presentation context are modulated by the artworks’ physical characteristics.

The Dynamics of Attention Shifts Among Concurrent Speech in a Naturalistic Multi-speaker Virtual Environment

Focusing attention on one speaker on the background of other irrelevant speech can be a challenging feat. A longstanding question in attention research is whether and how frequently individuals shift their attention towards task-irrelevant speech, arguably leading to occasional detection of words in a so-called unattended message. However, this has been difficult to gauge empirically, particularly when participants attend to continuous natural speech, due to the lack of appropriate metrics for detecting shifts in internal attention. Here we introduce a new experimental platform for studying the dynamic deployment of attention among concurrent speakers, utilizing a unique combination of Virtual Reality (VR) and Eye-Tracking technology. We created a Virtual Café in which participants sit across from and attend to the narrative of a target speaker. We manipulated the number and location of distractor speakers by placing additional characters throughout the Virtual Café. By monitoring participant's eye-gaze dynamics, we studied the patterns of overt attention-shifts among concurrent speakers as well as the consequences of these shifts on speech comprehension. Our results reveal important individual differences in the gaze-pattern displayed during selective attention to speech. While some participants stayed fixated on a target speaker throughout the entire experiment, approximately 30% of participants frequently shifted their gaze toward distractor speakers or other locations in the environment, regardless of the severity of audiovisual distraction. Critically, preforming frequent gaze-shifts negatively impacted the comprehension of target speech, and participants made more mistakes when looking away from the target speaker. We also found that gaze-shifts occurred primarily during gaps in the acoustic input, suggesting that momentary reductions in acoustic masking prompt attention-shifts between competing speakers, in line with "glimpsing" theories of processing speech in noise. These results open a new window into understanding the dynamics of attention as they wax and wane over time, and the different listening patterns employed for dealing with the influx of sensory input in multisensory environments. Moreover, the novel approach developed here for tracking the locus of momentary attention in a naturalistic virtual-reality environment holds high promise for extending the study of human behavior and cognition and bridging the gap between the laboratory and real-life.

Capturing Aesthetic Experiences With Installation Art: An Empirical Assessment of Emotion, Evaluations, and Mobile Eye Tracking in Olafur Eliasson's "Baroque, Baroque!"

Installation art is one of the most important and provocative developments in the visual arts during the last half century and has become a key focus of artists and of contemporary museums. It is also seen as particularly challenging or even disliked by many viewers, and—due to its unique in situ, immersive setting—is equally regarded as difficult or even beyond the grasp of present methods in empirical aesthetic psychology. In this paper, we introduce an exploratory study with installation art, utilizing a collection of techniques to capture the eclectic, the embodied, and often the emotionally-charged viewing experience. We present results from an investigation of two pieces, both part of Olafur Eliasson’s exhibition “Baroque, Baroque” held at the Belvedere museum in Vienna. These were assessed by pre- and post-viewing questionnaires focusing on emotion, meaning-making, and appraisals, in tandem with mobile eye tracking to consider viewers’ attention to both installed artworks and/or to the museum environment. The data showed differences in participants’ emotional states, appraisals, and visual exploration, which together paint a picture of the aesthetic reactions to the works. These differences also showed how viewers’ appraisal strategies, meaning making, and physical actions facilitated relatively more or less deep engagement with, and enjoyment of, the art. The results are discussed in terms of their implications for museum studies, art education, and theory in empirical aesthetics.

User-Centered Predictive Model for Improving Cultural Heritage Augmented Reality Applications: An HMM-Based Approach for Eye-Tracking Data

Today, museum visits are perceived as an opportunity for individuals to explore and make up their own minds. The increasing technical capabilities of Augmented Reality (AR) technology have raised audience expectations, advancing the use of mobile AR in cultural heritage (CH) settings. Hence, there is the need to define a criteria, based on users’ preference, able to drive developers and insiders toward a more conscious development of AR-based applications. Starting from previous research (performed to define a protocol for understanding the visual behaviour of subjects looking at paintings), this paper introduces a truly predictive model of the museum visitor’s visual behaviour, measured by an eye tracker. A Hidden Markov Model (HMM) approach is presented, able to predict users’ attention in front of a painting. Furthermore, this research compares users’ behaviour between adults and children, expanding the results to different kind of users, thus providing a reliable approach to eye trajectories. Tests have been conducted defining areas of interest (AOI) and observing the most visited ones, attempting the prediction of subsequent transitions between AOIs. The results demonstrate the effectiveness and suitability of our approach, with performance evaluation values that exceed 90%.