The Efficiency of Question-Asking Strategies in a Real-World Visual Search Task

In recent years, a multitude of datasets of human-human conversations has been released for the main purpose of training conversational agents based on data-hungry artificial neural networks. In this paper, we argue that datasets of this sort represent a useful and underexplored source to validate, complement, and enhance cognitive studies on human behavior and language use. We present a method that leverages the recent development of powerful computational models to obtain the fine-grained annotation required to apply metrics and techniques from Cognitive Science to large datasets. Previous work in Cognitive Science has investigated the question-asking strategies of human participants by employing different variants of the so-called 20-question-game setting and proposing several evaluation methods. In our work, we focus on GuessWhat, a task proposed within the Computer Vision and Natural Language Processing communities that is similar in structure to the 20-question-game setting. Crucially, the GuessWhat dataset contains tens of thousands of dialogues based on real-world images, making it a suitable setting to investigate the question-asking strategies of human players on a large scale and in a natural setting. Our results demonstrate the effectiveness of computational tools to automatically code how the hypothesis space changes throughout the dialogue in complex visual scenes. On the one hand, we confirm findings from previous work on smaller and more controlled settings. On the other hand, our analyses allow us to highlight the presence of "uninformative" questions (in terms of Expected Information Gain) at specific rounds of the dialogue. We hypothesize that these questions fulfill pragmatic constraints that are exploited by human players to solve visual tasks in complex scenes successfully. Our work illustrates a method that brings together efforts and findings from different disciplines to gain a better understanding of human question-asking strategies on large-scale datasets, while at the same time posing new questions about the development of conversational systems.

Artificial Intelligence Models Do Not Ground Negation, Humans Do. GuessWhat?! Dialogues as a Case Study

Negation is widely present in human communication, yet it is largely neglected in the research on conversational agents based on neural network architectures. Cognitive studies show that a supportive visual context makes the processing of negation easier. We take GuessWhat?!, a referential visually grounded guessing game, as test-bed and evaluate to which extent guessers based on pre-trained language models profit from negatively answered polar questions. Moreover, to get a better grasp of models' results, we select a controlled sample of games and run a crowdsourcing experiment with subjects. We evaluate models and humans against the same settings and use the comparison to better interpret the models' results. We show that while humans profit from negatively answered questions to solve the task, models struggle in grounding negation, and some of them barely use it; however, when the language signal is poorly informative, visual features help encoding the negative information. Finally, the experiments with human subjects put us in the position of comparing humans and models' predictions and get a grasp about which models make errors that are more human-like and as such more plausible.

Revising the European Society of Gastrointestinal Endoscopy (ESGE) research priorities: a research progress update

BACKGROUND

One of the aims of the European Society of Gastrointestinal Endoscopy (ESGE) is to encourage high quality endoscopic research at a European level. In 2016, the ESGE research committee published a set of research priorities. As endoscopic research is flourishing, we aimed to review the literature and determine whether endoscopic research over the last 4 years had managed to address any of our previously published priorities.

METHODS

As the previously published priorities were grouped under seven different domains, a working party with at least two European experts was created for each domain to review all the priorities under that domain. A structured review form was developed to standardize the review process. The group conducted an extensive literature search relevant to each of the priorities and then graded the priorities into three categories: (1) no longer a priority (well-designed trial, incorporated in national/international guidelines or adopted in routine clinical practice); (2) remains a priority (i. e. the above criterion was not met); (3) redefine the existing priority (i. e. the priority was too vague with the research question not clearly defined).

RESULTS

The previous ESGE research priorities document published in 2016 had 26 research priorities under seven domains. Our review of these priorities has resulted in seven priorities being removed from the list, one priority being partially removed, another seven being redefined to make them more precise, with eleven priorities remaining unchanged. This is a reflection of a rapid surge in endoscopic research, resulting in 27 % of research questions having already been answered and another 27 % requiring redefinition.

CONCLUSIONS

Our extensive review process has led to the removal of seven research priorities from the previous (2016) list, leaving 19 research priorities that have been redefined to make them more precise and relevant for researchers and funding bodies to target.