Hemispheric differences in the activation of perceptual information during sentence comprehension

Visual simulations in the two cerebral hemispheres: A bilingual perspective

The ability of each hemisphere to construct visual simulations during first language (L1) and second language (L2) sentence reading was investigated. Late bilinguals read L1 and L2 sentences and decided after each sentence whether a pictured object was mentioned in the sentence. Target pictures were presented laterally in the left/right visual field (LVF/RVF) to the right/left hemisphere (RH/LH), respectively. 'Yes' responses were faster when the pictured object's shape matched, rather than mismatched, the sentence-implied shape, irrespective of the language involved. Critically, this visual shape effect was significant only under LVF/RH presentation, indicating that visual simulations are more likely to occur in the RH than in the LH. The fact that a similar experiment with central picture presentation has produced a significant shape effect only in the L1 (Norman & Peleg, 2022), suggests that under normal (central) reading conditions, the RH may be less involved in L2 than in L1 reading.

Hemispheric processing of mental representations during text comprehension: evidence for inhibition of inconsistent shape information

To successfully understand a text, readers often mentally represent the shape of an object described in a text (e.g., creating a mental image of a sliced tomato when reading about a tomato on a pizza). However, it is currently unclear how the cerebral hemispheres contribute to these mental images during reading. In the current study, participants were presented with sentences consistent with the shape of an object (i.e., the match condition), sentences inconsistent with the shape of an object (i.e., the mismatch condition), or sentences that did not specify the shape of an object (i.e., the neutral condition). Participants read each sentence and then viewed an image of an object that was quickly presented to either the right visual field-left hemisphere (rvf-LH) or the left visual field-right hemisphere (lvf-RH). Results indicate that when the shape of an object was implicitly described in the text (in Experiment 1), response times for images presented to the rvf-LH were longer in the mismatch condition than in the neutral or match conditions. However, no response time differences were evident in the lvf-RH. When the shape of an object was explicitly described in the text (in Experiment 2), response times were longer in the mismatch condition than in the neutral and match conditions in both hemispheres. Thus, hemispheric involvement in mental representation depends on how explicit information is described in a text. Furthermore, these findings suggest that readers inhibit information that does not match an object׳s shape described in a text.

Revisiting mental simulation in language comprehension: six replication attempts

The notion of language comprehension as mental simulation has become popular in cognitive science. We revisit some of the original empirical evidence for this. Specifically, we attempted to replicate the findings from earlier studies that examined the mental simulation of object orientation, shape, and color, respectively, in sentence-picture verification. For each of these sets of findings, we conducted two web-based replication attempts using Amazon's Mechanical Turk. Our results are mixed. Participants responded faster to pictures that matched the orientation or shape implied by the sentence, replicating the original findings. The effect was larger and stronger for shape than orientation. Participants also responded faster to pictures that matched the color implied by the sentence, whereas the original studies obtained mismatch advantages. We argue that these results support mental simulation theory, show the importance of replication studies, and show the viability of web-based data collection.

Hemispheric differences in the organization of memory for text ideas

The goal of this study was to examine hemispheric asymmetries in episodic memory for discourse. Access to previously comprehended information is essential for mapping incoming information to representations of "who did what to whom" in memory. An item-priming-in-recognition paradigm was used to examine differences in how the hemispheres represent discourse. Both hemispheres retained accurate information about concepts from short passages, but the information was organized differently. The left hemisphere was sensitive to the structural relations among concepts in a text, whereas the right hemisphere differentiated information that appeared in one passage from information that appeared in another. Moreover, the right hemisphere, but not the left hemisphere, retained information about the spatial/temporal proximity among concepts in a passage. Implications of these results for the roles of the right and left hemispheres in comprehending connected discourse are discussed.

Incidental picture exposure affects later reading: evidence from the N400

Language comprehenders form a mental representation of the implied shape of objects mentioned in the text. In the present study, the influence of prior visual experience on subsequent reading was assessed. In two separate phases, participants saw a picture of an object and read a text about the object, suggesting the same or a different shape. When the shapes in the two phases mismatched, ERPs during reading showed a larger N400 amplitude than when the shapes matched, suggesting that a picture presented incidentally 15 min earlier affected reading. These results further strengthen the case for the interaction of language and visual experience during language comprehension.

Out of mind, out of sight: language affects perceptual vividness in memory

We examined whether language affects the strength of a visual representation in memory. Participants studied a picture, read a story about the depicted object, and then selected out of two pictures the one whose transparency level most resembled that of the previously presented picture. The stories contained two linguistic manipulations that have been demonstrated to affect concept availability in memory, i.e., object presence and goal-relevance. The results show that described absence of an object caused people to select the most transparent picture more often than described presence of the object. This effect was not moderated by goal-relevance, suggesting that our paradigm tapped into the perceptual quality of representations rather than, for example, their linguistic availability. We discuss the implications of these findings within a framework of grounded cognition.

A logarithmic speed-difficulty trade-off in speech production

We explored the relations between task difficulty and speech time in picture description tasks. Six native speakers of Mandarin Chinese (CH group) and six native speakers or Indo-European languages (IE group) produced quick and accurate verbal descriptions of pictures in a self-paced manner. The pictures always involved two objects, a plate and one of the three objects (a stick, a fork, or a knife) located and oriented differently with respect to the plate in different trials. An index of difficulty was assigned to each picture. CH group showed lower reaction time and much lower speech time. Speech time scaled linearly with the log-transformed index of difficulty in all subjects. The results suggest generality of Fitts' law for movement and speech tasks, and possibly for other cognitive tasks as well. The differences between the CH and IE groups may be due to specific task features, differences in the grammatical rules of CH and IE languages, and possible use of tone for information transmission.

How long does it take to describe what one sees? The first step using picture description tasks

The study explored the dependences between quantifiable features of a picture and the time it takes to describe it. Six native English speakers and six bilinguals watched pictures presented on the monitor and described them "as quickly and accurately as possible". The bilingual participants performed the test twice, in English and in their native language. The pictures could contain one to six objects. There were four series of trials that differed in the number of characteristics of the objects the participants were instructed to describe. Reaction time showed a modest, close to linear scaling with the number of objects. Both reaction time and speech time were significantly longer for the bilingual participants performing in English as compared to their performance in the native language and to the English speaking participants. The difference in reaction time did not depend on the number of objects. Speech time showed a close to linear scaling with the number of objects within each of the four series. The linear regression coefficient in this relationship increased linearly with the number of characteristics of the objects across all series. The results are discussed in relation to speed-accuracy trade-off and different strategies of picture description.

Integration of perceptual information in word access

According to experiential theories of language comprehension, perceptual information plays an essential role when word meanings are accessed. We conducted four experiments to investigate how different types of perceptual information such as colour and shape are combined during word access. One possibility is that the colour and shape of a word's referent are activated independently from one another and are combined in an additive manner. Alternatively, words might activate perceptual representations via a multiplicative integration of colour and shape. Experiment 1 established that participants follow a multiplicative similarity rule when they judge the similarity of schematic pictures to actual fruits and vegetables. In Experiments 2 to 4, participants performed a classification task, a lexical decision task, or a word-naming task on names of fruits and vegetables that were superimposed on a background picture. Responses were facilitated only when both colour and shape of the picture matched the word's referents. Response times were associated negatively with mean similarity ratings and the consistency of these ratings obtained in the first experiment. These results suggest a multiplicative integration of different types of perceptual information during word access.

Hemispheric asymmetries in the perceptual representations of words

The representation of words in sentences can involve the activation and integration of perceptual information. For example, readers who are asked to view pictures of objects relating to a word in a sentence are influenced by perceptual information in the sentence context-readers are faster to respond to a picture of a whole apple after reading, "There is an apple in the bag," than after reading, "There is an apple in the salad." The purpose of this study was to examine how the two cerebral hemispheres use perceptual information about words as a function of sentence context. Patients who had damage to the left or right hemisphere and age-matched control participants read sentences that described, but did not entail, the shape or state of an object. They then made recognition judgments to pictures that either matched or mismatched the perceptual form implied by the sentence. Responses and latencies were examined for a match effect -- faster and more accurate responses to pictures in the match than mismatch condition -- controlling for comprehension ability and lesion size. When comprehension ability and lesion size are properly controlled, left-hemisphere-damaged patients and control participants exhibited the expected match effect, whereas right-hemisphere-damaged participants showed no effect of match condition. These results are consistent with research implicating the right hemisphere in the representation of contextually relevant perceptual information.