The last quarter of a century has seen a dramatic rise of interest in the spatial constraints on multisensory integration. However, until recently, the majority of this research has investigated integration in the space directly in front of the observer. The space around us, however, extends in three spatial dimensions in the front and to the rear beyond such a limited area. The question to be addressed in this review concerns whether multisensory integration operates according to the same rules throughout the whole of three-dimensional space. The results reviewed here not only show that the space around us seems to be divided into distinct functional regions, but they also suggest that multisensory interactions are modulated by the region of space in which stimuli happen to be presented. We highlight a number of key limitations with previous research in this area, including: (1) The focus on only a very narrow region of two-dimensional space in front of the observer; (2) the use of static stimuli in most research; (3) the study of observers who themselves have been mostly static; and (4) the study of isolated observers. All of these factors may change the way in which the senses interact at any given distance, as can the emotional state/personality of the observer. In summarizing these salient issues, we hope to encourage researchers to consider these factors in their own research in order to gain a better understanding of the spatial constraints on multisensory integration as they affect us in our everyday life.
The field of multisensory research has grown steadily over the past decade or so (see Bremner et al., 2012; Murray et al., 2013; Spence and Driver, 2004; Stein, 2012; Stein et al., 2010; Van der Stoep et al., 2015a, for reviews). Advances in our understanding of how the brain processes multisensory information have been made by means of neurophysiological, psychophysical, neuropsychological, neuroimaging, and computational modeling studies of multisensory interactions. Whereas visual, auditory, and tactile information are initially processed independently, they are integrated at various stages of sensory information processing (Calvert and Thesen, 2004 — see Note 1). The conditions under which multisensory interactions occur, and the behavioral benefits (or costs) that accrue as a result of multisensory integration, are becoming somewhat clearer. So, for example, it is now evident that information from the different senses is integrated over a range of spatial and temporal separations (Stein and Meredith, 1990, 1993; Vroomen and Keetels, 2010; though see also Spence, 2013). Furthermore, the circumstances under which attention and multisensory integration interact are now also being more clearly elucidated (see Koelewijn et al., 2010; Talsma, 2015; Talsma et al., 2010, for reviews). Multisensory integration can, for instance, occur before attentional selection (e.g., Bertelson et al., 2000; Soto-Faraco et al., 2004; Spence and Driver, 2000; Vroomen et al., 2001), is modulated by exogenous and endogenous spatial attention (either enhancing or decreasing multisensory integration; e.g., Talsma and Woldorff, 2005; Van der Stoep et al., 2015b; Zou et al., 2012), and attending more to one sense than to the others has also been shown to modulate multisensory integration (e.g., Alsius et al., 2005; Mozolic et al., 2008; Talsma et al., 2007).
As yet, however, one attribute of multisensory signals that has received relatively less attention from the research community is the distance from which information is presented (with the exception of distance-related changes in multisensory temporal perception; e.g., Alais and Carlile, 2005; Arnold et al., 2005; Engel and Dougherty, 1971; Harris et al., 2010; Sugita and Suzuki, 2003; see also below). It is striking that this should be the case because we obviously perceive multisensory information from various distances on a daily basis.
The mere fact that our sense of touch, vision, and audition differ in terms of the distances over which they are able to register information suggests that their interactions may also be distance-dependent. We are able to perceive visual information from a great distance but only in frontal space. Audition allows us to perceive sounds from various distances from all directions. In contrast, our sense of touch allows us to perceive stimulation only when energy reaches the skin. That is, usually only from events happening at close proximity (an exception is our ability to perceive heat, even from far away sources such as the sun). Bodily movements and handheld tools can extend the range in which we can use touch to sense the world, but, even with such extended reach, the maximum distance is still rather limited if compared to audition and vision. The different spatial ranges at which events can stimulate our various sensory systems imply that the way in which the senses interact could depend on the distance from which such events are perceived.
Various studies have demonstrated a distance-dependent modulation of multisensory interactions, which suggests that source distance is indeed a factor worthy of consideration in multisensory research (e.g., Canzoneri et al., 2012; Farnè and Làdavas, 2002; Graziano and Gross, 1994; Graziano et al., 1999; Sambo and Forster, 2009, 2011; Serino et al., 2011; Spence, 2011; Van der Stoep et al., in press).
In this review, we provide an overview of some of the most important factors that need to be taken into account when investigating multisensory interactions. Our goal in writing this review is to underline how crucial it is to be aware of the fact that multisensory interactions are dependent on the region of space in which one presents one’s stimuli. Therefore, our hope is that we may be able to stimulate researchers to more carefully consider the distance or depth from which information is presented when investigating multisensory interactions.
First, we discuss the basis on which the space around the human body has been divided into distinct regions (see Fig. 1). A distinction can be made here between different regions of space based on: (1) The strength of multisensory interactions; (2) the different functions associated with each region (e.g., grasping, communicating, navigation); and (3) an impaired ability to process or integrate information that is presented within a certain region of space as shown by neuropsychological studies of brain-damaged patients (e.g., Bisiach et al., 1986; Halligan and Marshall, 1991; Làdavas and Farnè, 2004; Vuilleumier et al., 1998; see Occelli et al., 2011; Previc, 1998; Van der Stoep et al., 2015a, for reviews).
Furthermore, we highlight the fact that the research that has been conducted to date has been very limited in terms of the regions of space in which multisensory stimuli have been presented, and hence multisensory interactions have been investigated. Not only have the majority of studies presented the stimuli in a relatively small region of frontal space, more distal and lateral locations have been largely ignored (see Fig. 2). Additionally, studies of multisensory interactions in rear space and those studies in which stimuli have been presented at different elevations are surprisingly scarce as well.
Multisensory interactions have primarily been studied in those situations in which not only the observer, but also the stimuli, remained static (though some researchers have been investigating multisensory integration using looming and receding stimuli, e.g., Canzoneri, Magosso and Serino, 2012; Cappe et al., 2012; Cappe, Thut, Romei and Murray, 2009; Cléry et al., 2015a). In everyday life, however, both the observer and the surrounding stimuli often move. This necessitates a neural system that is capable of keeping track of multisensory stimuli and rapidly and continuously updating the representation of the body in space, given that the movement of the observer’s body can change the distance from the stimuli (Note 2).
Distance-dependent modulations of multisensory interactions may, in addition, be further affected by social factors, such as the sheer presence of, trust in, or cooperation with, other individuals or anxiety related to the presence of certain stimuli (Brozzoli et al., 2014; de Vignemont and Iannetti, 2015; Ferri et al., 2015; Heed et al., 2010; Lourenco et al., 2011; Sambo and Iannetti, 2013; Taffou and Viaud-Delmon, 2014; Teneggi et al., 2013).
Last, but by no means least, we will discuss how all these modulations might be explained in the context of the predictive coding framework.
2. Dividing Space
The idea that distinct regions of space relative to the body are represented differently by the brain might at first sound odd, given that we generally perceive the space around us as continuous. Yet support for this notion comes from several social psychological, psychophysiological, neurophysiological and neuropsychological studies of (multisensory) spatial perception. The notion that the brain might process stimuli in different regions of space differently has emerged from neurophysiological studies of a particular kind of multisensory neuron. Several authors have observed that certain bimodal neurons in frontal and parietal areas of the macaque monkey have visual spatial RFs aligned with tactile RFs extending a certain limited distance from the body (e.g., Duhamel et al., 1997; Graziano and Gross, 1994; Graziano et al., 1997; Rizzolatti et al., 1981, 1997).
The visual and tactile RFs of many of those bimodal neurons remained spatially aligned, even after the movement of a limb, such as a hand or arm. This indicates that the position of a body part in space is updated in order to enable relevant multisensory interactions in terms of the spatial layout of events in the environment relative to the body. The depth from which stimuli are presented is, however, not only important when it comes to visuotactile integration, but also for audiotactile integration (Graziano et al., 1999). In particular, bimodal and trimodal neurons in the ventral premotor cortex of the macaque monkey have been shown to be sensitive to sounds presented approximately 10 cm from the head of the animal, but not to the same sounds when presented at a distance of 30 or 50 cm. This spatial region of space has been termed peripersonal space (PPS; of the hand and of the head in these examples; see Cardinali et al., 2010 for a discussion of the distinction between PPS and reachable space). Similar observations have been made in both neurotypical and brain-damaged humans, and in both frontal and rear space (e.g., Farnè and Làdavas, 2002; see Occelli et al., 2011, for a review). These results can therefore be taken to show that the distinction between peri- and extrapersonal space is not only true for frontal space, but for rear space as wel.
Several neuropsychological studies of visuospatial neglect have demonstrated that following stroke, neglect may well be distance-specific (e.g., Halligan and Marshall, 1991). That is, neglect may occur selectively in either peripersonal or extrapersonal space, or in both regions of space (Aimola et al., 2012; Bisiach et al., 1986; Cowey et al., 1994; Halligan and Marshall, 1991; Van der Stoep et al., 2013; Vuilleumier et al., 1998). Such results indicate that a distinction between peripersonal and extrapersonal space can be observed in terms of deficits in visuospatial attention. Depending on the location of the brain lesion, visuospatial attention would seem to be impaired in either peripersonal (dorsal lesions) or extrapersonal space (ventral lesions). There are some indications that damage to brain structures that are connected to both the dorsal and ventral visual processing streams can give rise to neglect in both regions of space (middle temporal cortex, frontal cortex, anterior cingulate cortex, see Aimola et al., 2012).
Furthermore, it has been suggested that different combinations of sensory information might be more or less relevant depending on the distance from which this information happens to be presented (Previc, 1998; Van der Stoep et al., 2015a). For example, touch and vision are dominant in peripersonal space, as they may imply an interaction between the body and the environment (e.g., for grasping or defense), whereas auditory and visual information may be more relevant in extrapersonal space as they provide useful information about farther objects, for spatial orienting, navigation and interaction with others (e.g., during conversation). Space can thus also be divided based on the functions linked to different regions of space, and accordingly to the dominant senses that are associated with those functions.
3. Limited Focus of Previous Research
Research concerning the way in which multisensory interactions are modulated by the distance between the stimuli and the observer has mainly focused on those interactions taking place in frontal peripersonal space. Yet in order to gain a better understanding of how the senses interact in everyday life, it is important to understand how multisensory interactions change as a function of the position of the source of multisensory stimulation in full three-dimensional (3-D) space. Figure 2 (left panel) provides an overview of the distances at which stimuli have been presented in a sample of previous studies of multisensory interactions in different depth-planes (in the online Supplementary Material, see Table S1 for details, and Fig. S1 for a similar figure including studies of temporal perception using distances up to 50 m). What becomes clear when one looks at Fig. 2 (left panel and Fig. S1) is that studies on multisensory interactions in rear space, lateral space, and distances out of reach are underrepresented relative to studies of multisensory interactions in peripersonal and reachable space (Note 3).
Based on the studies that are available, however, it has, in recent years, become increasingly apparent that the way in which sensory information interacts changes as a function of the distance from which that information is presented from the observer. What is perhaps even more striking is that the elevation of stimuli with respect to the observer has almost never been manipulated (see Fig. 2, right panel). The particular spatial alignment of multisensory stimuli in terms of elevation also affects how the senses interact given that there are, for example, differences in the reliability of determining the elevation of auditory and visual information (e.g., Corneil et al., 2002; Frens et al., 1995; Ten Brink et al., 2014).
In addition, several studies have indicated that multisensory interactions are different for multisensory stimuli that are presented in the peripheral as compared to the central visual field. For example, the double-flash illusion is much stronger when sound and light are presented in the periphery as compared to when they are presented at the fovea (Shams et al., 2000, 2001). Furthermore, audiovisual temporal binding windows tend to be larger for those stimuli presented out in the periphery (Stevenson et al., 2012). Interestingly, in line with these findings, it has been observed that the density of direct projections between auditory and visual cortex is larger in the peripheral than in the central visual field, at least in monkeys (Falchier et al., 2002).
Studying multisensory interactions in different regions of 3-D space (i.e., changing the position of stimuli in the front–back, left–right, and up–down dimensions) may therefore be expected to provide a window into how multisensory perception changes in those situations that are more representative of everyday life.
4. Visuotactile and Audiotactile Interactions and the Concept of Peripersonal Space
Those multisensory interactions that are relevant to the body (i.e., multisensory stimulation involving the sense of touch) seem to be especially pronounced in the PPS around different body parts. Various studies have shown that responses to tactile stimulation on the hand, face or trunk are faster when a sound is presented close to the stimulated body-part relative to those situations in which sound is presented in far space (e.g., Canzoneri et al., 2012, 2013a; Farnè and Làdavas, 2000; Galli et al., 2015; Noel et al., 2015a, b; Teneggi et al., 2013). These findings are in line with observations of visual, auditory, and tactile RFs of multisensory neurons that respond to sensory stimuli that are presented within a limited spatial range from the body (see Graziano and Cooke, 2006, for a review). The size of these spatial RFs essentially determines the size of the PPS around different body parts.
Neurophysiology studies have identified neuronal populations with multisensory receptive fields covering the space around the hand, face and trunk, within a network of fronto-parietal areas spanning from the ventral premotor cortex (Fogassi et al., 1996; Graziano et al., 1997), the ventral intraparietal area (Avillac et al., 2005; Duhamel et al., 1997) and parietal area 7 (Hyvarinen, 1981; Leinonen, 1980; see Cléry et al., 2015b, and Graziano and Gross, 1994, for reviews). Human neuroimaging studies have further confirmed that homologous premotor and parietal areas preferentially respond to, and integrate, those stimuli that are presented on, or close to, the hand (Brozzoli et al., 2011; Makin et al., 2007), face (Bremmer et al., 2001; Sereno and Huang, 2006) or trunk (Huang et al., 2012).
Different body-part centered PPS representations might serve different functional roles. The peri-hand space may be mainly involved in hand-object interactions that can be appetitive or defensive in nature: It is mainly represented by premotor and parietal areas, which not only integrate multisensory stimuli around the hand, but also project to the motor system in order to trigger the appropriate responses (Avenanti et al., 2012; Cooke and Graziano, 2004; Cooke et al., 2003; Makin et al., 2009; Serino et al., 2009). The peri-hand space also dynamically updates its size/shape as a function of changes in upper limb function or structure, such as after tool-use (Canzoneri et al., 2013a; Farnè and Làdavas, 2000; Maravita et al., 2001; see Maravita and Iriki, 2004, for a review, and Serino et al., 2015 for a computational account of these effects), amputation and prosthetic implementation (Canzoneri et al., 2013b), and immobilization (Bassolino et al., 2015).
The peri-face space is likely to be more involved in social interactions, as its boundaries are sensitive to the presence of, and interaction with, other people (e.g., Tennegi et al., 2013) and varies depending on individual personality traits, such as anxiety (Sambo and Iannetti, 2013; see also below). Finally, the peri-trunk space may serve as a global representation of the whole-body in space, as it includes the other body-part centered PPSs, shapes during whole body movements (e.g., walking, Noel et al., 2015a) and is centered at the perceived location of the self in space (Noel et al., 2015b). Thus, there would seem to be a series of body-part centered PPS representations, underlying rather specific sensory-motor or cognitive functions, and a whole-body representation of PPS, representing a primary interface between the bodily self and the environment.
5. The Dynamic Observer
In many of the studies of multisensory interactions in peripersonal space that have been conducted to date, the observer has been static (that is, they have had to sit passively on a chair or else, on occasion, they have had to stand still). As such, the presentation of dynamic multisensory stimuli may provide information on the updating of multisensory interactions when objects approach the static observer or during passive movement through the environment (e.g., as when driving; see Moeller et al., in press, for the effects of driving on distance estimation). However, when humans are actively moving through the environment (e.g., while walking) or when interacting with it (e.g., during grasping), the position of the body or a body-part in space will change constantly relative to the stimuli in the environment. Therefore, the representations of peripersonal and extrapersonal space will obviously need to be updated in order to remain functional, enable effective interaction with the environment, and to maintain a proper margin of safety around the body to ensure bodily integrity (see e.g., de Vignemont and Iannetti, 2015; Graziano and Cooke, 2006). Recent findings support the view that the multisensory interactions in space vary depending not only as a consequence of objects moving around the observer, but also depending on the movements of the observer through the environment. We will first review evidence supporting this claim in the case of hand-objects interaction, and then with respect to movement of the body as a whole.
5.1. Multisensory Interactions During Goal-Oriented Actions
Despite the above recalled increase of interest in multisensory perception during the last few decades, virtually nothing is yet known about whether and how information from the different senses interacts when the brain starts transforming the intention to act into a motor program. This gap in our knowledge is even more surprising when it is considered that several studies have demonstrated the involvement of PPS in the guidance of involuntary, defensive movements. In the monkey, electrical stimulation of multisensory areas may evoke rather complex patterns of hand, arm, and head movements such as the withdrawal of the hand, the turning of the head, or the lifting the hand as if to defend the head, which are largely compatible with defensive reactions (e.g., Graziano et al., 2006; Moayedi et al., 2015; see Bufacchi et al., in press, for more on the shape of defensive peripersonal space in humans). It has been suggested that such adaptive responses, possibly evoked by multisensory neurons, are fast and mainly occur outside of the control of top–down voluntary mechanisms.
In line with this prediction, electrophysiological investigations in humans have indeed revealed that the motor cortex is very quickly ‘informed’ when an object appears in the visual field and happens to be approaching one’s hand. As compared to a condition were the object was also falling, but its landing position was far removed from the location of the participant’s hand, hand-approaching objects were able to modify the corticospinal excitability of the visually ‘threatened’ hand within just 70 ms of its appearance (Makin et al., 2009; see also Serino et al., 2009). Importantly, such an automatic hand-centered coding of visual PPS was selective for approaching balls, as static visual distractors did not modulate activity in the motor cortex. Even more important, control conditions for both overt and covert orienting of spatial attention ruled out any major role played by the latter in the hand-centred coding of PPS, a finding that fits well the need for a defensive system to be efficient enough even in unattended conditions (Makin et al., 2009, 2012).
For such a sensorimotor defensive system to be really effective, not only should general information about whether an object is approaching the hand be processed rapidly, but also the more specific information about which hand is approached. This hypothesis was recently tested by probing motor cortex excitability (Makin et al., 2015). The participants’ right hand was concealed while a falling red ball rapidly approached either left- or right-handed dummy hands that were located either in an orientation that was either plausible (egocentric perspective) or implausible (allocentric perspective). It was found that within a very short time-window (i.e., 70 ms from the appearance of the ball in the visual field), the human motor system is already capable of coding not only the proximity of an object to a hand (near vs. far), but also which hand this object potentially threatens. When the ball approached an egocentrically oriented dummy hand, motor evoked potentials were significantly reduced for an anatomically congruent (right) dummy hand, as compared with an incongruent (left) dummy hand. These findings are in line with the neurophysiological evidence gathered in the monkey showing that the tonic activity of certain multisensory neurons is modulated by hand identity. That is, by whether a left or right hand is visually presented in an egocentric perspective (Graziano, Cooke and Taylor, 2000). These findings therefore provide support for the general claim that PPS coding may well serve to perform defensive actions.
These multisensory interfaces might be adaptive not only for defensive, but also for appetitive actions, such as grasping a glass of water (e.g., Gardner et al., 2007; Marzocchi et al., 2008). In this respect, the properties of multisensory neurons that we have recalled previously may allow the brain to represent a target object in a coordinate system centred on the body (e.g., the hand) that, in addition, could be continuously updated during bodily movements. It is worth noting here that some bimodal neurons have been documented that respond when the arm is voluntarily moved within the reaching space of the animal and have been proposed to code goal-directed actions. This question was addressed in a study in neurotypical humans by having participants perform a modified version of the classical Crossmodal Congruency Effect (CCE) paradigm (see Spence et al., 2004, 2008, for reviews). By taking the changes in the magnitude of the CCE as a proxy for changes in PPS, Brozzoli and colleagues (Brozzoli et al., 2009) were able to provide support for the hypothesis that voluntarily acting on objects triggers a hand-centred remapping of multisensory perception (see Brozzoli et al., 2012, for a review). The authors asked healthy participants to discriminate touches on the hand they used to grasp an object that contained task-irrelevant visual distractors. In this case, crossmodal stimulation was applied to either the grasping or the non-grasping hand. There was no cue-target delay between tactile targets and visual distractors thus enhancing the likelihood of multisensory integration instead of crossmodal spatial attention (McDonald et al., 2001; Van der Stoep et al., 2015c). This visuo-tactile task provided a measure of how (much) the visual-tactile interaction varied, in real time, during the execution of the action. When compared to a static condition prior to movement initiation, the start of the grasping action selectively increased the interference exerted by visual inputs originating from the (far) target object on tactile stimuli delivered to the grasping hand. In addition, a further increase in the magnitude of the CCE was observed shortly after (250 ms) the onset of the hand movement. That is, when the hand had initiated its travelling path towards the target object, yet was still far away from it. The increase in CCE was observed both at the start of the movement and during the subsequent unfolding of the action. This not only indicates that PPS can be remapped as a function of the execution of voluntary actions, but also demonstrates that remapping occurs in an on-line fashion.
Such a dynamic, action-dependent modulation of PPS, which — by the way — also reveals that PPS remapping can occur independent of tool-use (see also Serino et al., 2015), was further illustrated by a follow-up study in which different types of actions were performed. Brozzoli and his colleagues assessed the effects of performing different actions towards the same object on the on-line modulations of PPS, as measured by the same variant of the CCE paradigm (Brozzoli et al., 2010). Neurotypical participants had either to grasp or point towards an object while discriminating whether tactile stimuli were delivered on their right index finger or thumb and ignoring visual distractors (from the target object). The strength of visuo-tactile interaction was probed before the movement, as well as at action onset and during action execution. When compared to the static condition, the grasping and the pointing actions had similar effects of increasing the CCE at the action onset (prior to the movement). That is, when these actions were indistinguishable from the point of view of their kinematics. However, the CCE further increased for the grasping action, but not the pointing one, during the execution phase, when the kinematics of these movements started to diverge. These findings therefore suggest that performing voluntary actions induce a continuous remapping of PPS as a function of the on-line contextual demands imposed by their kinematics (Brozzoli et al., 2010). Preliminary unpublished data from the same laboratory suggested that multisensory interactions may actually occur prior to movement onset and may even be sensitive to the hierarchical structures of complex, multistep movements.
Overall, these findings suggest that performing both defensive and appetitive actions towards objects entails a multisensory link between signals from the environment and the body that is functionally related to the action goal. Recently, researchers investigated whether visuotactile interactions are also modulated by the proximity of the hand to an obstacle during a reach-to-grasp movement to a target object (Menger et al., in prep.). This is an especially interesting question as the obstacle is not the goal of the movement, but it can nevertheless still have a tactile consequence that we often want to avoid (e.g., accidently knocking over your own, or worse still, someone else’s, glass of beer; see De Haan et al., 2014). The participants had to grasp a target object in front of them and avoid touching another object that was placed along the trajectory on the outside of the reaching arm. At different positions of the hand along the reaching trajectory, a visual stimulus on the obstacle was simultaneously presented with a tactile stimulus on the hand. The visual and tactile stimuli could either be congruent or incongruent in terms of the side of stimulation. That is, a visual stimulus on the obstacle (on the outside of the arm trajectory) could be simultaneously presented with a tactile stimulus on the index finger (right, congruent with collision) versus on the thumb (opposite side, incongruent with collision). The participants had to respond as rapidly as possible to the onset of the tactile stimulus with their left hand while attempting to grasp a target with their other hand. More specifically, they were required to indicate whether the thumb (left button) or the index finger (right button) was being stimulated. The visuotactile congruency effect increased as the hand that was used to grasp the target (which was also the hand that received tactile stimulation) approached the obstacle. These results could therefore be taken to suggest that representations of peripersonal (hand) space are updated during complex object-oriented actions, depending not only on the goal (object target) of the movement, but also on surrounding potential targets (obstacles).
5.2. Multisensory Interactions During Whole Body Movements and While Driving
If multisensory interactions involving tactile stimuli on the body and visual and/or auditory stimuli related to external objects have a functional role in predicting possible contacts between the body and objects in the environment (Clery et al., 2015a; de Vignemont and Iannetti, 2015; Graziano and Cooke, 2006; Rizzolatti et al., 1997), then one might expect very different spatial modulations of multisensory interactions when the observer is moving through the environment as compared to when he/she is static. Accordingly, Noel et al. (2015a) recently reported that the boundary of peripersonal space extends when participants walk as compared to when they were standing still. The boundary of PPS was assessed by measuring the spatial distance at which a sound, looming towards the participant in frontal-space, significantly speeded-up reactions to tactile stimuli on the participant’s body. The experiment was conducted while the participants were either standing or walking on a treadmill, such that the relative distance between the participant’s body and the sound source was equivalent in the two conditions. Nevertheless, while in the static condition sounds occurring closer than ∼80–90 cm from the participant decreased tactile RTs. This speeding-up of participants’ responses occurred for sounds farther than 2 m in the walking condition, thus suggesting that potential interactions between external stimuli and bodily stimuli are anticipated in the case of a moving perceiver. Compare this case to a situation in which an object is approaching an observer while the observer is moving sideways perpendicular to the approaching object. In this case, when the movement of the observer is taken into account, the object might actually not collide with the body once it gets close, whereas the object would almost certainly hit the observer in a static condition. Predictions concerning multisensory interactions with a stimulus in extrapersonal space may thus be very different depending on whether or not an observer is translating through space, and such effects might be further modulated, depending on whether the observer is passively translated or voluntarily moves.
This issue is directly related to a specific, very interesting case of movement, such as mediated by vehicles as, for instance, while driving. The speed of movement during driving is generally much higher when compared to walking. Such high speeds dramatically change the prediction of the distance at which objects start to become relevant in terms of collision with the body. It is currently unknown whether the boundary of peripersonal space extends even further when we are moving at high speed (either as a driver or a passenger) as compared to when we are walking. Early neurophysiological work on monkeys suggested that this might be the case, as Fogassi et al. (1996) reported that the visual RF of peripersonal neurons in ventral premotor cortex extended in space when it was probed with faster as compared to slower looming visual stimuli. However, there may be limits to the amount that peripersonal space can extend in depth (see Holmes, 2012, for a discussion of the extension vs. projection of peripersonal space after tool-use). In that case, other factors may also play a role in modulating multisensory interactions between stimuli that are present in different depth-planes. For example, visuospatial attention and estimations about the time-to-contact may be important in determining the outcome of multisensory interactions (see, for example, Kandula et al., 2015). The time to contact with an object is obviously dependent on the distance between the object and the body, the speed and direction of movement of both the object and the observer, and the valence of the object (see e.g., Vagnoni et al., 2012). Recently, it was shown that participants underestimated the distance to objects in frontal space when they were sitting in a car as compared to when sitting in a chair with a similar occlusion of the visual field (Moeller et al., in press). The participants in this study underestimated distances even more after driving the car for about ten minutes, but not after walking for the same amount of time. This may not be surprising because the faster one is driving, the shorter the time it takes for an object to reach the car. To maintain a similar safety boundary around the body at higher speeds of movement through the environment, larger distances should be underestimated. The time that is needed to react to events when moving at such high speeds is very important because we are limited in our speed of responding to such events (for example, brake lights illuminating in front of us). Various studies have shown that (multi-)sensory warning signals can save precious time by reduce breaking times significantly (e.g., Ho and Spence, 2005, 2006, 2009, 2014; Ho et al., 2005, 2006, 2007, 2014; Lee et al., 2002; Santangelo et al., 2008; Spence, 2012; see Spence and Ho, 2008, for a review).
The findings from studies of crossmodal exogenous spatial attention in depth and audiovisual integration in near versus far space might be relevant in an applied setting as well. For example, it has been shown that auditory cues presented in far space are more effective in attracting an observer’s attention to the correct lateral location in far space than auditory cues that are presented in near space (Van der Stoep et al., 2014). Whether this would work similarly under conditions of high cognitive load remains to be seen (e.g., while driving), as previous studies have shown that unisensory cues do not capture attention as effectively under such conditions (see Spence and Santangelo, 2009, for a review). Multisensory cues, however, would appear to be relatively unaffected by the amount of cognitive load in terms of their capacity to attract attention. This property of spatially co-located multisensory cues, and the recent observation of enhanced audiovisual integration in far space (Van der Stoep et al., in press), suggest that audiovisual warning signals in far space may be especially effective as a warning signal while driving. However, if the speed of driving extends peripersonal space in such a way that stimuli that are far away are still being considered to be within the peripersonal space, then audiotactile or visuotactile warning signals may be much more effective given their relevance in peripersonal space. In fact, such an extension of peripersonal space does not sound very different from the zone of safe travel first suggested by Gibson and Crooks back in 1938. They refer to this zone as “…the field of possible paths which the car may take unimpeded”, and it is shaped by objects on or features of the road that need to be avoided or that need to be approached (Note 4) (Gibson and Crooks, 1938, p. 454). Interestingly, they also made the prescient suggestion that this zone can dynamically change in shape and size based on the current driving situation. Peripersonal space has been shown to behave similarly as it can flexibly update depending on the situation at hand as has been shown in various of the studies that have been mentioned in this review.
6. Audiovisual Interactions in Different Depth Planes
In the previous sections, we discussed how multisensory interactions involving the sense of touch are dependent on the distance between stimuli and different parts of the observer’s body. Audio- and visuo-tactile interactions are especially enhanced within peripersonal space. In contrast, one might have expected that the distance from which information is presented modulates audiovisual interactions differently when no tactile consequence is anticipated. Unfortunately, however, less is known about how audiovisual integration is affected by changes in distance. Although audiovisual stimuli can obviously have consequences when approaching the body (think only of the sight and sound of a car driving rapidly towards you; we are thinking here, for example, of those studies that have presented looming audiovisual stimuli, see Cappe et al., 2009, 2012), the interaction, or integration, of sound and light per se does not seem to be any more pronounced in peripersonal space than in extrapersonal space (e.g., Van der Stoep et al., 2014). Many of the studies in which audiovisual perception has been investigated in different depth planes have focused on audiovisual temporal perception (e.g., Kopinska and Harris, 2004; Lewald and Guski, 2004; Sugita and Suzuki, 2003; see the next section for further discussion on this matter). Besides the delays in arrival time between sound and light that will be introduced as the distance from the observer increases (Spence and Squire, 2003), certain other factors may also play a role in changing audiovisual interactions as well. So, for example, auditory and visual information may be dominant in far space, and information available in far space is often used for orienting and navigation (Previc, 1998). As a result of this sensory dominance, audiovisual integration might be more pronounced in far space. This was indeed what was observed in a recent study of audiovisual integration in near (∼80 cm) and far space (∼200 cm; Van der Stoep et al., in press). Specifically, multisensory integration was enhanced for audiovisual stimuli that were presented in far as compared to near space. This enhancement was evident from increased multisensory response enhancement (faster responses to multisensory targets relative to the fastest response to unimodal targets), and an increased amount of race model inequality violation (Gondan and Minakata, in press; Miller, 1982, 1986). The increase in multisensory response enhancement in far relative to near space was, however, only present when the stimuli in far space were not corrected for retinal image size and intensity. Importantly, the increase could not be explained by a change in stimulus efficacy because the same decrease in the size of the retinal image and intensity in near space as in far space did not give rise to enhanced multisensory integration. The effect could also not be explained solely based on the region of space in which the stimuli happened to be presented (i.e., audiovisual dominance in far space), because there was no difference in integration between the near and far space condition when the stimuli were corrected for retinal image size and intensity. Thus, it was concluded that both a decrease in retinal image size and intensity, and the region of space in which information was presented contributed to the observed enhancement of audiovisual integration. Interestingly, the far space condition, which resulted in a significant increase in multisensory integration relative to near space, was also the condition that occurs most often in daily life. That is, an increase in the distance between an audiovisual stimulus and an observer going hand-in-hand with a decrease in stimulus effectiveness (i.e., decreased retinal image size and intensity).
It does not seem plausible that this increased benefit of audiovisual integration would hold for those distances that are much greater than 2 m, as differences in arrival times of sound and light will increase and the visual and auditory stimuli will, at some point, eventually become more difficult to perceive. Indeed, this is what was observed when the amount of audiovisual integration was compared between 15 m and 1 m (unpublished data, Van der Stoep and Di Luca, in prep.). At a distance of 1 m, the race model inequality was violated, but not at a distance of 15 m, indicating that audiovisual response enhancement was reduced to the level of statistical facilitation at 15 m.
In a recent study of the Colavita visual dominance effect (see Spence et al., 2011, for a review), visual dominance was shown to be enhanced in far relative to near space (Yue et al., 2015). Although currently unknown, it is likely that visual dominance may increase with distance, contributing to a decrease in multisensory integration at very large distances. The greater reliability of spatial localization of visual information in depth relative to sound may hint that this will indeed be the case (e.g., Agganis et al., 2010; Bowen et al., 2011; Gardner, 1968). However, visual dominance in far space might also depend on the visual angle of stimuli, causing decreased visual dominance for smaller audiovisual stimuli in far space.
On the basis of the studies mentioned above, it could be argued that there is an increase in audiovisual integration from near (<1 m) to far space (∼2 m), and a reduction in audiovisual integration at larger distances from the observer (15 m). An explanation for an increase from near to far space might be found in the reliability of the auditory and visual signals at different distances. Close by, both auditory and visual signals can be perceived clearly. When the same stimuli are presented further away from the observer, however, the reliability of these signals will decrease relative to the same signals when presented close by. The benefit of integrating information that is less reliable might be more pronounced in far space. At very large distances, however, physiological asynchrony increases as a result of differences in arrival times (e.g., Spence and Squire, 2003) and stimulus intensity decreases (e.g., Di Luca, 2014). Hence, the benefit of audiovisual stimulation will once again decrease at large distances. That the reliability of sensory information is taken into account during multisensory integration has been demonstrated in various studies by now (e.g., Alais and Burr, 2004; Ernst and Banks, 2002). It seems that each sense is weighted according to its reliability of, for example, estimating a spatial location. The more reliable an estimate is (i.e., the smaller the variance), the more it will affect the final multisensory estimate. It will be interesting in future research to see how visual and auditory reliability changes as a function of distance and how such changes affect multisensory integration.
Although the previous research suggests an enhancement of audiovisual integration in far space, there is also some support for the lack of such distance-specific enhancements of audiovisual interactions. In a study of crossmodal exogenous spatial attention across and within different depth-planes, the crossmodal cuing effect was found to be dependent on whether the auditory cue and the visual target were presented in the same depth-plane or not (Van der Stoep et al., 2014). However, this effect of crossmodal exogenous spatial attention did not differ for those cues and targets that were presented in the same depth-plane close by (∼1.20 m) as compared to far away (∼2 m). Considering that the visual target stimuli were corrected for the size of the retinal image, these results may not be all that different from those mentioned previously. In the study of audiovisual integration in near and far space, there were no differences in multisensory integration when the stimuli were corrected for retinal image size and intensity (Van der Stoep et al., in press).
These findings therefore indicate that the strength of audiovisual interactions is not only affected by whether spatially aligned multisensory stimuli are presented in near or far space, but also by whether the component auditory and visual stimuli are presented from the same distance or not. Thus, the region of space in which information is presented not only affects multisensory interactions involving touch, but also audiovisual interactions.
6.1. Distance-Related Temporal Modulations of Audiovisual Interactions
The time it takes for signals to reach the sense organs and to be processed by the brain varies between sensory modalities. For example, the transmission delay of sound vibration is much higher than the one of light energy due to the different speed of conduction in air. As a result, the relative asynchrony of stimulation from a physically synchronous audiovisual event increases proportional to the distance of the observer (Spence and Squire, 2003). On the other hand, the neural processing of auditory sensory signals has a lower latency than that of visual information (Corey and Hudspeth, 1979; Pugh and Lamb, 1993; Schnapf et al., 1987). It is often assumed that the effect of distance on perceptual latency is primarily attributable to the lower speed of propagation of sound through air. Given this factor alone, there is a certain distance at which sound and light are expected to reach the sensory cortices simultaneously. With such a synchronous neural signal, it is expected that participants would perceive stimuli to appear simultaneous. This distance has been labeled ‘the horizon of simultaneity’ and exists under the assumption that cortical latency is constant (Pöppel and Artin, 1988; Schroeder and Foxe, 2002, 2004; see Vroomen and Keetels, 2010, for a review on multisensory temporal perception). The horizon of simultaneity has been estimated to lie at a distance between 10–15 m. This reasoning does not consider that light source distance causes decreased retinal size, lower stimulus energy at the retina, and a lower auditory intensity, which might increase the neural latency of vision, but also that of audition. Therefore, the horizon of simultaneity might not be very relevant in a real-life setting. A recent study by Di Luca (2014) examined whether the distance of a light source from an observer would affect perceived simultaneity in an audiovisual temporal order task for audiovisual stimuli presented at arm’s length or at a distance of 16 m. Importantly, whereas sounds had equal loudness at the speaker, in different conditions, the size and intensity of the light source were either maintained constant as a function of distance, or they were corrected to have equal visual angle and stimulus intensity at the observer vantage point. Comparing the results of these two conditions demonstrated that the optical changes due to the distance of the light source affected the perception of audiovisual simultaneity. This effect was greatest when the two distances from which the lights were presented was randomized across trials, presumably causing the participants to divide their attention to several locations in depth and to view the stimuli peripherally. As a result, visual latency increased even further. On the other hand, the perceived simultaneity did not change as a function of visual distance by maintaining foveation of visual stimuli with equal retinal size and perceived luminosity. These results therefore indicate that not only the distance of the auditory source, but also that of the visual source affects perceived simultaneity. Therefore, the horizon of simultaneity is only attainable at the proposed distance of 10–15 m if visual stimuli are compensated for retinal size and energy, whereas in a real-world situation the horizon of simultaneity might be farther than previously thought, or might not exist at all, especially when visual stimuli have limited spatial extension.
7. Social Factors Relevant to Peripersonal Space
A good example of how social factors can affect how we represent the space around us comes from a study of personal space conducted half a century ago by Felipe and Sommer (1966). These researchers observed that there is a certain distance between individuals that is needed to feel comfortable. When this space is ‘invaded’, the participants (patients in this study) tended to increase the distance, make a barrier, or else flee from the situation. In this way, the authors effectively demonstrated that there is a region of personal space that has a certain boundary and that, when invaded, can evoke discomfort (see also Hall, 1966; Hediger, 1955). Thus, in terms of behavioral outcome, the space around these participants was automatically divided into two different regions: a personal, and a non- or extrapersonal region of space. Various factors can affect the distance or size of the personal space, such as the cultural background, the nature of the relationship between people, the status of people, the layout of the environment, etc. (see Burgoon and Jones, 1976, for a review; Gallace and Spence, 2014).
Support for the notion that social factors play a role in coding peripersonal space comes from studies that investigated peripersonal hand space representations of self and others. In the ventral premotor cortex (PMv), the peripersonal hand space does not only seem to represent the space around one’s own hand, but also that of others as seen from a first person perspective (e.g., Brozzoli et al., 2013; Ishida et al., 2010). What is more, whether an action is viewed from a first or third person perspective has also been shown to modulate action representations in PMv (Oosterhof et al., 2012). By contrast, the viewpoint-independent coding of peripersonal hand space was observed in parietal and occipitotemporal cortex. These studies would therefore seem to suggest that peripersonal space representations may also play a role in social interactions.
Teneggi et al. (2013) proposed a link between personal space as studied in the context of social psychology and peripersonal space, as described in neuroscience, by investigating how social factors modulate the size of peripersonal space. In particular, they tested whether peripersonal space changes in size depending on the presence of another individual, and on whether the other individual was cooperative or non-cooperative with the participant (Teneggi et al., 2013). The size of peripersonal space was assessed in terms of the distance at which approaching sounds started to decrease participants’ detection latencies in response to tactile stimuli delivered to the their face. Two main findings were obtained: First, the mere presence of an unknown person shapes peripersonal space representations, since the boundary of peripersonal space was closer to the participants when they faced another individual as compared to when they faced a mannequin instead, as if people automatically and implicitly divide the space between themselves and others (see, for related effect of active presence, Heed et al., 2010). Even more interestingly, the size of peripersonal space was found to increase after playing an economic game with a cooperative individual who was positioned in front of the participant as compared to before playing the economic game (and this was not the case when the game was played with an uncooperative individual). This enlargement of one’s own peripersonal space so as to include the space around the cooperative other was interpreted as a sharing of self-other peripersonal spaces after positive social interactions.
When thinking of the function of maintaining a certain interpersonal distance, not only does the feeling of safety come to mind, but also the maintaining of bodily integrity (see Iannetti and Mouraux, 2010; Melzack, 1999; Moseley et al., 2012). To do so, it is essential to avoid harm, to update a body representation, and monitor potential sources of threat in relation to the body. Several recent studies have investigated how (perceived) threat affects sensory processing in peripersonal space. For example, the size or shape of the defensive peripersonal space has recently been shown to be related to trait anxiety (de Vignemont and Iannetti, 2015; Sambo and Iannetti, 2013). Other researchers, meanwhile, have reported that the size of PPS is correlated with the extent of claustrophobic fear (Lourenco et al., 2011; see Taffou and Viaud-Delmon, 2014, for the relation between cynophobic fear and peripersonal space; see also Dosey and Meisels, 1969). The distance from the body at which auditory stimuli start to affect RTs to tactile targets on the hand is also larger for auditory stimuli with negative compared positive valence. This could be interpreted as an extension of the safety zone or the peripersonal space for threatening or negative stimuli (Ferri et al., 2015).
Interestingly, in these studies, the tactile stimuli that were delivered were not necessarily painful. A recent study investigated whether visual information in peripersonal space could affect the processing of specifically nociceptive stimuli (De Paepe et al., 2014). Unilateral visual cues were presented to the left or the right side of space before the onset of two nociceptive stimuli. Using a temporal order judgment task with nociceptive stimuli delivered to the left and right hand, unilateral visual cues presented in peripersonal space were shown to affect the perceived point of subjective simultaneity more than those visual cues that were situated in extrapersonal space. These results therefore indicate that the interaction between visual and nociceptive stimuli also depends on the region of space in which visual information is presented. The proximity of threat also seems to affect distance estimation of stimuli relative to the body (see Tabor et al., 2015). When participants had to estimate the distance between their body and a switch that was associated with threat (i.e., a nociceptive stimulus delivered to the hand), they generally underestimated the distance as compared to a switch that was associated with relief. Overall, then, the results from these studies clearly indicate that threat and pain perception (or prediction) affect multisensory spatial processing.
8. General Discussion
There has been an enormous increase in studies of multisensory integration in recent years. However, one aspect of multisensory perception that has received less attention than others is the depth or distance from which information is presented. The studies that looked at how the distance at which information is presented affects multisensory integration have primarily investigated multisensory interactions in peripersonal space and multisensory temporal perception. In the present review, we highlight how these studies have contributed to our understanding of multisensory perception, but their scope was limited, as the stimuli used have been presented from a relatively limited region of space. That is, studies of multisensory processing have focused on a narrow region of space in front of the observer (see Fig. 2). Furthermore, it seems we are at a point where we would benefit from studying multisensory interactions in more dynamic situations such as when the observer and the stimuli are moving in relation to each other in a more complex, ecological environment. This will allow us researchers to gain a better understanding of how multisensory integration takes place when those factors, which frequently change in our daily lives, are taken into account (e.g., the distance between stimuli and the body, movement of the body, moving stimuli). A schematic overview of a change in peripersonal space representation by the factors that are discussed in this review can be found in Fig. 3.
A framework that might prove helpful when it comes to thinking about how the brain deals with multisensory interactions that are changed by, for example, movement of the body or movement of stimuli is the predictive coding framework (e.g., Clark, 2013; Friston, 2005; Friston and Kiebel, 2009). According to this framework, predictions about the state of the world through our senses are adjusted via feedback in the form of prediction errors by a constant interaction between bottom–up and top–down information. This idea fits well with the observation that multisensory experience with the environment is essential to the development of multisensory neurons and therefore to multisensory integration (e.g., Wallace and Stein, 2007; Wallace et al., 2004). The brain receives information about the world and the state of the body in that world through multiple senses. Each sense provides information about the world in a different way (e.g., in different reference frames) with different qualities (e.g., different spatial and temporal resolutions). This allows the brain to calibrate information from one sense with information from another sense (e.g., King, 2009; Shams et al., 2011). Such multisensory calibration allows for the maintenance of multisensory spatial representations on a daily basis, given that the world around is always readily available. As such, it is not unthinkable that the brain calibrates in such a way that interactions between the senses become spatially dependent. For example, given that there is a strong correlation between visual and tactile sensations of stimuli near the hand in real life, visual stimuli near the hand interact more strongly with tactile stimulation than visual stimuli far from the hand.
To conclude, in order to gain a better understanding of how humans interact with the abundance of multisensory information in the environment, it seems essential to investigate multisensory integration in three spatial dimensions. The contribution of each of our senses to our perception of and our interactions with the world depends on the region of space in which information is presented (e.g., front vs. rear, peripersonal vs. extrapersonal space). An important next step towards unraveling multisensory integration in everyday situations may be to investigate how multisensory interactions change online in dynamic situations such as while moving (e.g., walking, driving), when perceiving moving stimuli, and their combination in 3-D space.
AF was supported by IHU CeSaMe ANR-10-IBHU-0003, FRC (Fédération pour la Recherche sur le Cerveau, Neurodon) and the James S. McDonnell Scholar Award. MDL was supported a Marie Curie Career Integration Grant 304235 ‘TICS’. CS would like to thank the AHRC for the Rethinking the Senses Grant (AH/ L007053/1).
- 1.We will not discuss the chemical senses in this review. See Spence (2015) for more on multisensory flavour perception.
- 2.There may be an important role for crossmodal spatial remapping and integration with body posture here (Spence and Driver, 2004).
- 3.This becomes all the more clear when one thinks of all the studies of multisensory integration in which depth was of no particular interest. In such studies, stimuli are often presented at a fixed distance in frontal space (∼60–80 cm from the body).
- 4.This may remind the reader of deviation away and towards distractors during eye-movements and reaching and grasping (e.g., Tipper et al., 1997; see Van der Stigchel et al., 2007, for a review on eye-movement trajectories).
Agganis B. T., Muday J. A., Schirillo J. A. (2010). Visual biasing of auditory localization in azimuth and depth, Percept. Mot. Skills 111, 872–892.
Aimola L., Schindler I., Simone A. M., Venneri A. (2012). Near and far space neglect: task sensitivity and anatomical substrates, Neuropsychologia 50, 1115–1123.
Alais D., Carlile S. (2005). Synchronizing to real events: subjective audiovisual alignment scales with perceived auditory depth and speed of sound, Proc. Natl Acad. Sci. USA 102, 2244–2247.
Alsius A., Navarra J., Campbell R., Soto-Faraco S. (2005). Audiovisual integration of speech falters under high attention demands, Curr. Biol. 15, 839–843.
Avenanti A., Annela L., Serino A. (2012). Suppression of premotor cortex disrupts motor coding of peripersonal space, Neuroimage 63, 281–288.
Avillac M., Deneve S., Olivier E., Pouget A., Duhamel J. R. (2005). Reference frames for representing visual and tactile locations in parietal cortex, Nat. Neurosci. 8, 941–949.
Bassolino M., Finisguerra A., Canzoneri E., Serino A., Pozzo T. (2015). Dissociating effect of upper limb non-use and overuse on space and body representations, Neuropsychologia 70, 385–392.
Bertelson P., Vroomen J., De Gelder B., Driver J. (2000). The ventriloquist effect does not depend on the direction of deliberate visual attention, Percept. Psychophys. 62, 321–332.
Bowen A. L., Ramachandran R., Muday J. A., Schirillo J. A. (2011). Visual signals bias auditory targets in azimuth and depth, Exp. Brain Res. 214, 403–414.
Bremmer F., Schlack A., Shah N. J., Zafiris O., Kubischik M., Hoffmann K., Zilles K., Fink G. R. (2001). Polymodal motion processing in posterior parietal and premotor cortex: a human fMRI study strongly implies equivalencies between humans and monkeys, Neuron 29, 287–296.
Brozzoli C., Pavani F., Urquizar C., Cardinali L., Farne A. (2009). Grasping actions remap peripersonal space, Neuroreport 20, 913–917.
Brozzoli C., Cardinali L., Pavani F., Farnè A. (2010). Action-specific remapping of peripersonal space, Neuropsychologia 48, 796–802.
Brozzoli C., Gentile G., Petkova V. I., Ehrsson H. H. (2011). FMRI adaptation reveals a cortical mechanism for the coding of space near the hand, J. Neurosci. 31, 9023–9031.
Brozzoli C., Makin T. R., Cardinali L., Holmes N. P., Farnè A. (2012). Peripersonal space: a multisensory interface for body-object interactions, in: The Neural Bases of Multisensory Processes, Murray M. M., Wallace M. T. (Eds), pp. 447–464. CRC Press, Boca Raton, FL, USA.
- Search Google Scholar
- Export Citation
( Brozzoli C. Makin T. R. Cardinali L. Holmes N. P. Farnè A. 2012). Peripersonal space: a multisensory interface for body-object interactions, in: The Neural Bases of Multisensory Processes, (Eds), pp. Murray M. M. Wallace M. T. 447– 464. CRC Press, Boca Raton, FL, USA.
Brozzoli C., Gentile G., Bergouignan L., Ehrsson H. H. (2013). A shared representation of the space near oneself and others in the human premotor cortex, Curr. Biol. 23, 1764–1768.
Brozzoli C., Ehrsson H. H., Farnè A. (2014). Multisensory representation of the space near the hand from perception to action and interindividual interactions, Neuroscientist 20, 122–135.
Bufacchi R. J., Liang M., Griffin L. D. & Iannetti G. D. (in press). A geometric model of defensive peripersonal space, J. Neurophysiol. DOI:10.1152/jn.00691.2015.
Burgoon J. K., Jones S. B. (1976). Toward a theory of personal space expectations and their violations, Hum. Commun. Res. 2, 131–146.
Calvert G. A., Thesen T. (2004). Multisensory integration: methodological approaches and emerging principles in the human brain, J. Physiol. Paris 98, 191–205.
Canzoneri E., Magosso E., Serino A. (2012). Dynamic sounds capture the boundaries of peripersonal space representation in humans, PLoS One 7, e44306. DOI:10.1371/journal.pone.0044306.
Canzoneri E., Ubaldi S., Rastelli V., Finisguerra A., Bassolino M., Serino A. (2013a). Tool-use reshapes the boundaries of body and peripersonal space representations, Exp. Brain Res. 228, 25–42.
Canzoneri E., Marzolla M., Amoresano A., Verni G., Serino A. (2013b). Amputation and prosthesis implantation shape body and peripersonal space representations, Sci. Rep. 3, 2844. DOI:10.1038/srep02844.
Cappe C., Thut G., Romei V., Murray M. M. (2009). Selective integration of auditory–visual looming cues by humans, Neuropsychologia 47, 1045–1052.
Cappe C., Thelen A., Romei V., Thut G., Murray M. M. (2012). Looming signals reveal synergistic principles of multisensory integration, J. Neurosci. 32, 1171–1182.
Cardinali L., Brozzoli C., Farnè A. (2010). Peripersonal space and body schema, in: Encyclopedia of Behavioural Neuroscience, Koob G. G., Le Moal M., Thompson R. F. (Eds), pp. 40–46. Academic Press, Oxford, UK.
Clark A. (2013). Whatever next? Predictive brains, situated agents, and the future of cognitive science, Behav. Brain Sci. 36, 181–204.
Cléry J., Guipponi O., Odouard S., Wardak C., Hamed S. B. (2015a). Impact prediction by looming visual stimuli enhances tactile detection, J. Neurosci. 35, 4179–4189.
Cléry J., Guipponi O., Wardak C., Hamed S. B. (2015b). Neuronal bases of peripersonal and extrapersonal spaces, their plasticity and their dynamics: knowns and unknowns, Neuropsychologia 70, 313–326.
Cooke D. F., Graziano M. S. (2004). Sensorimotor integration in the precentral gyrus: polysensory neurons and defensive movements, J. Neurophysiol. 91, 1648–1660.
Cooke D. F., Taylor C. S., Moore T., Graziano M. S. (2003). Complex movements evoked by microstimulation of the ventral intraparietal area, Proc. Natl Acad. Sci. USA 100, 6163–6168.
Corneil B. D., Van Wanrooij M., Munoz D. P., Van Opstal A. J. (2002). Auditory–visual interactions subserving goal-directed saccades in a complex scene, J. Neurophysiol. 88, 438–454.
Cowey A., Small M., Ellis S. (1994). Left visuo–spatial neglect can be worse in far than in near space, Neuropsychologia 32, 1059–1066.
De Haan A. M., Van der Stigchel S., Nijnens C. M., Dijkerman H. C. (2014). The influence of object identity on obstacle avoidance reaching behaviour, Acta Psychol. 150, 94–99.
De Paepe A. L., Crombez G., Spence C., Legrain V. (2014). Mapping nociceptive stimuli in a peripersonal frame of reference: evidence from a temporal order judgment task, Neuropsychologia 56, 219–228.
Duhamel J. R., Bremmer F., BenHamed S., Graf W. (1997). Spatial invariance of visual receptive fields in parietal cortex neurons, Nature 389(6653), 845–848.
Ernst M. O., Banks M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion, Nature 415(6870), 429–433.
Falchier A., Clavagnier S., Barone P., Kennedy H. (2002). Anatomical evidence of multimodal integration in primate striate cortex, J. Neurosci. 22, 5749–5759.
Ferri F., Tajadura-Jiménez A., Väljamäe A., Vastano R., Costantini M. (2015). Motion-inducing approaching sounds shape the boundaries of multisensory peripersonal space, Neuropsychologia 70, 468–475.
Fogassi L., Gallese V., Fadiga L., Luppino G., Matelli M., Rizzolatti G. (1996). Coding of peripersonal space in inferior premotor cortex (area F4), J. Neurophysiol. 76, 141–157.
Frens M. A., Van Opstal A. J., Van der Willigen R. F. (1995). Spatial and temporal factors determine auditory–visual interactions in human saccadic eye movements, Percept. Psychophys. 57, 802–816.
Friston K., Kiebel S. (2009). Predictive coding under the free-energy principle, Phil. Trans. R. Soc. B Biol. Sci. 364(1521), 1211–1221.
Gallace A., Spence C. (2014). In Touch With the Future: the Sense of Touch From Cognitive Neuroscience to Virtual Reality. Oxford University Press, Oxford, UK.
Galli G., Noel J. P., Canzoneri E., Blanke O., Serino A. (2015). The wheelchair as a full-body tool extending the peripersonal space, Front. Psychol. 6, 639. DOI:10.3389/fpsyg.2015.00639.
Gardner E. P., Babu K. S., Reitzen S. D., Ghosh S., Brown A. S., Chen J., Hall A. L., Herzlinger M. D., Kohlenstein J. B., Ro J. Y. (2007). Neurophysiology of prehension. I. Posterior parietal cortex and object-oriented hand behaviors, J. Neurophysiol. 97, 387–406.
Gondan M. & Minakata K. (in press). A tutorial on testing the race model inequality, Atten. Percept. Psychophys. DOI:10.3758/s13414-015-1018-y.
Graziano M. S., Cooke D. F. (2006). Parieto-frontal interactions, personal space, and defensive behavior, Neuropsychologia 44, 845–859.
Graziano M. S. A., Gross C. G. (1994). The representation of extrapersonal space: a possible role for bimodal visual–tactile neurons, in: The Cognitive Neurosciences, Gazzaniga M. S. (Ed.), pp. 1021–1034. MIT Press, Cambridge, MA, USA.
Graziano M. S., Reiss L. A., Gross C. G. (1999). A neuronal representation of the location of nearby sounds, Nature 397(6718), 428–430.
Harris L., Harrar V., Jaekl P., Kopinska A. (2010). Mechanisms of simultaneity constancy, in: Space and Time in Perception and Action, Nijhawan R., Khurana B. (Eds), pp. 232–253. Cambridge University Press, Cambridge, UK.
Hediger H. (1955). Studies of the Psychology and Behaviour of Captive Animals in Zoos and Circuses. Criterion Books, New York, NY, USA.
Heed T., Habets B., Sebanz N., Knoblich G. (2010). Others’ actions reduce crossmodal integration in peripersonal space, Curr. Biol. 20, 1345–1349.
Ho C., Spence C. (2005). Assessing the effectiveness of various auditory cues in capturing a driver’s visual attention, J. Exp. Psychol. Appl. 11, 157–174.
Ho C., Spence C. (2006). Verbal interface design: do verbal directional cues automatically orient visual spatial attention? Comput. Human Behav. 22, 733–748.
Ho C., Spence C. (2014). Effectively responding to tactile stimulation: do homologous cue and effector locations really matter? Acta Psychol. 151, 32–39.
Ho C., Tan H. Z., Spence C. (2005). Using spatial vibrotactile cues to direct visual attention in driving scenes, Transp. Res. Part F: Traffic Psychol. Behav. 8, 397–412.
Ho C., Tan H. Z., Spence C. (2006). The differential effect of vibrotactile and auditory cues on visual spatial attention, Ergonomics 49, 724–738.
Huang R. S., Chen C. F., Tran A. T., Holstein K. L., Sereno M. I. (2012). Mapping multisensory parietal face and body areas in humans, Proc. Natl Acad. Sci. USA 109, 18114–18119.
Ishida H., Nakajima K., Inase M., Murata A. (2010). Shared mapping of own and others’ bodies in visuotactile bimodal area of monkey parietal cortex, J. Cogn. Neurosci. 22, 83–96.
Kandula M., Hofman D., Dijkerman H. C. (2015). Visuo-tactile interactions are dependent on the predictive value of the visual stimulus, Neuropsychologia 70, 358–366.
Koelewijn T., Bronkhorst A., Theeuwes J. (2010). Attention and the multiple stages of multisensory integration: a review of audiovisual studies, Acta Psychol. 134, 372–384.
Lee J. D., McGehee D. V., Brown T. L., Reyes M. L. (2002). Collision warning timing, driver distraction, and driver response to imminent rear-end collisions in a high-fidelity driving simulator, Hum. Factors 44, 314–334.
Leinonen L. (1980). Functional properties of neurones in the posterior part of area 7 in awake monkey, Acta Physiol. Scand. 108, 301–308.
Lewald J., Guski R. (2004). Auditory–visual temporal integration as a function of distance: no compensation for sound-transmission time in human perception, Neurosci. Lett. 357, 119–122.
Makin T. R., Holmes N. P., Zohary E. (2007). Is that near my hand? Multisensory representation of peripersonal space in human intraparietal sulcus, J. Neurosci. 27, 731–740.
Makin T. R., Holmes N. P., Brozzoli C., Rossetti Y., Farne A. (2009). Coding of visual space during motor preparation: approaching objects rapidly modulate corticospinal excitability in hand-centered coordinates, J. Neurosci. 29, 11841–11851.
Makin T. R., Holmes N. P., Brozzoli C., Farnè A. (2012). Keeping the world at hand: rapid visuomotor processing for hand–object interactions, Exp. Brain Res. 219, 421–428.
Makin T. R., Brozzoli C., Cardinali L., Holmes N. P., Farnè A. (2015). Left or right? Rapid visuomotor coding of hand laterality during motor decisions, Cortex 64, 289–292.
Maravita A., Husain M., Clarke K., Driver J. (2001). Reaching with a tool extends visual–tactile interactions into far space: evidence from cross-modal extinction, Neuropsychologia 39, 580–585.
Marzocchi N., Breveglieri R., Galletti C., Fattori P. (2008). Reaching activity in parietal area V6A of macaque: eye influence on arm activity or retinocentric coding of reaching movements? Eur. J. Neurosci. 27, 775–789.
McDonald J. J., Teder-Sälejärvi W. A., Ward L. M. (2001). Multisensory integration and crossmodal attention effects in the human brain, Science 292(5523), 1791.
Moayedi M., Liang M., Sim A. L., Hu L., Haggard P., Iannetti G. D. (2015). Laser-evoked vertex potentials predict defensive motor actions, Cereb. Cortex 25(12), 4789–4798.
Moeller B., Zoppke H. & Frings C. (in press). What a car does to your perception: distance evaluations differ from within and outside of a car, Psychonom. Bull. Rev. DOI:10.3758/s13423-015-0954-9.
Moseley G. L., Gallace A., Spence C. (2012). Bodily illusions in health and disease: physiological and clinical perspectives and the concept of a cortical ‘body matrix’, Neurosci. Biobehav. Rev. 36, 34–46.
Mozolic J. L., Hugenschmidt C. E., Peiffer A. M., Laurienti P. J. (2008). Modality-specific selective attention attenuates multisensory integration, Exp. Brain Res. 184, 39–52.
Murray M., Spence C., Harris L. (2013). International Multisensory Research Forum 2012 meeting special issue, Multisens. Res. 26, 287–289.
Noel J. P., Grivaz P., Marmaroli P., Lissek H., Blanke O., Serino A. (2015a). Full body action remapping of peripersonal space: the case of walking, Neuropsychologia 70, 375–384.
Oosterhof N. N., Tipper S. P., Downing P. E. (2012). Viewpoint (in) dependence of action representations: an MVPA study, J. Cogn. Neurosci. 24, 975–989.
Pugh E. N., Lamb T. D. (1993). Amplification and kinetics of the activation steps in phototransduction, Biochim. Biophys. Acta 1141, 111–149.
Rizzolatti G., Scandolara C., Matelli M., Gentilucci M. (1981). Afferent properties of periarcuate neurons in macaque monkeys. II. Visual responses, Behav. Brain Res. 2, 147–163.
Sambo C. F., Forster B. (2009). An ERP investigation on visuotactile interactions in peripersonal and extrapersonal space: evidence for the spatial rule, J. Cogn. Neurosci. 21, 1550–1559.
Sambo C. F., Forster B. (2011). When far is near: ERP correlates of crossmodal spatial interactions between tactile and mirror-reflected visual stimuli, Neurosci. Lett. 500, 10–15.
Sambo C. F., Iannetti G. D. (2013). Better safe than sorry? The safety margin surrounding the body is increased by anxiety, J. Neurosci. 33, 14225–14230.
Schroeder C. E., Foxe J. J. (2002). The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex, Brain Res.: Cogn. Brain Res. 14, 187–198.
Schroeder C. E., Foxe J. J. (2004). Multisensory convergence in early cortical processing, in: The Handbook of Multisensory Processes, Calvert G. A., Spence C., Stein B. E. (Eds), pp. 295–309. MIT Press, Cambridge, MA, USA.
Sereno M. I., Huang R. S. (2006). A human parietal face area contains aligned head-centered visual and tactile maps, Nat. Neurosci. 9, 1337–1343.
Serino A., Annella L., Avenanti A. (2009). Motor properties of peripersonal space in humans, PLoS One 4, e6582. DOI:10.1371/journal.pone.0006582.
Serino A., Canzoneri E., Avenanti A. (2011). Fronto-parietal areas necessary for a multisensory representation of peripersonal space in humans: an rTMS study, J. Cogn. Neurosci. 23, 2956–2967.
Serino A., Canzoneri E., Marzolla M., Di Pellegrino G., Magosso E. (2015). Extending peripersonal space representation without tool-use: evidence from a combined behavioral-computational approach, Front. Behav. Neurosci. 9, 4. DOI:10.3389/fnbeh.2015.00004.
Shams L., Wozny D. R., Kim R., Seitz A. (2011). Influences of multisensory experience on subsequent unisensory processing, Front. Psychol. 2, 264. DOI:10.3389/fpsyg.2011.00264.
Soto-Faraco S., Navarra J., Alsius A. (2004). Assessing automaticity in audiovisual speech integration: evidence from the speeded classification task, Cognition 92, B13–B23.
Spence C. (2011). Assessing the consequences of tool-use for the representation of peripersonal space in humans, in: Tool Use and Causal Cognition, McCormack T., Hoerl C., Butterfill S. (Eds), pp. 220–247. Oxford University Press, Oxford, UK.
Spence C. (2013). Just how important is spatial coincidence to multisensory integration? Evaluating the spatial rule, Ann. NY Acad. Sci. 1296, 31–49.
Spence C., Driver J. (2000). Attracting attention to the illusory location of a sound: reflexive crossmodal orienting and ventriloquism, Neuroreport 11, 2057–2061.
Spence C., Ho C. (2008). Multisensory warning signals for event perception and safe driving, Theor. Issues Ergon. Sci. 9, 523–554.
Spence C., Pavani F., Maravita A., Holmes N. (2004). Multisensory contributions to the 3-D representation of visuotactile peripersonal space in humans: evidence from the crossmodal congruency task, J. Physiol. Paris 98, 171–189.
Spence C., Pavani F., Maravita A., Holmes N. P. (2008). Multi-sensory interactions, in: Haptic Rendering: Foundations, Algorithms, and Applications, Lin M. C., Otaduy M. A. (Eds), pp. 21–52. AK Peters, Wellesley, MA, USA.
Spence C., Parise C., Chen Y.-C. (2011). The Colavita visual dominance effect, in: Frontiers in the Neural Bases of Multisensory Processes, Murray M. M., Wallace M. (Eds), pp. 523–550. CRC Press, Boca Raton, FL, USA.
Stein B. E., Burr D., Constantinidis C., Laurienti P. J., Meredith M. A., Perrault T. J., Ramachandran R., Röder B., Rowland B. A., Sathian K., Schroeder C. E., Shams L., Stanford T. R., Wallace M. T., Yu L., Lewkowicz D. J. (2010). Semantic confusion regarding the development of multisensory integration: a practical solution, Eur. J. Neurosci. 31, 1713–1720.
- Search Google Scholar
- Export Citation
( Stein B. E. Burr D. Constantinidis C. Laurienti P. J. Meredith M. A. Perrault T. J. Ramachandran R. Röder B. Rowland B. A. Sathian K. Schroeder C. E. Shams L. Stanford T. R. Wallace M. T. Yu L. Lewkowicz D. J. 2010). Semantic confusion regarding the development of multisensory integration: a practical solution, Eur. J. Neurosci. 31, 1713– 1720.
Stevenson R. A., Fister J. K., Barnett Z. P., Nidiffer A. R., Wallace M. T. (2012). Interactions between the spatial and temporal stimulus factors that influence multisensory integration in human performance, Exp. Brain Res. 219, 121–137.
Tabor A., Catley M. J., Gandevia S. C., Thacker M. A., Spence C., Moseley G. L. (2015). The close proximity of threat: altered distance perception in the anticipation of pain, Front. Psychol. 6, 626. DOI:10.3389/fpsyg.2015.00626.
Taffou M., Viaud-Delmon I. (2014). Cynophobic fear adaptively extends peri-personal space, Front. Psychiatry 5, 122. DOI:10.3389/fpsyt.2014.00122.
Talsma D. (2015). Predictive coding and multisensory integration: an attentional account of the multisensory mind, Front. Integr. Neurosci. 9, 19. DOI:10.3389/fnint.2015.00019.
Talsma D., Woldorff M. G. (2005). Selective attention and multisensory integration: multiple phases of effects on the evoked brain activity, J. Cogn. Neurosci. 17, 1098–1114.
Talsma D., Doty T. J., Woldorff M. G. (2007). Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration? Cereb. Cortex 17, 679–690.
Talsma D., Senkowski D., Soto-Faraco S., Woldorff M. G. (2010). The multifaceted interplay between attention and multisensory integration, Trends Cogn. Sci. 14, 400–410.
Ten Brink A. F., Nijboer T. C. W., Van der Stoep N., Van der Stigchel S. (2014). The influence of vertically and horizontally aligned visual distractors on aurally guided saccadic eye movements, Exp. Brain Res. 232, 1357–1366.
Teneggi C., Canzoneri E., di Pellegrino G., Serino A. (2013). Social modulation of peripersonal space boundaries, Curr. Biol. 23, 406–411.
Tipper S. P., Howard L. A., Jackson S. R. (1997). Selective reaching to grasp: evidence for distractor interference effects, Vis. Cogn. 4, 1–38.
Van der Stigchel S., Meeter M., Theeuwes J. (2006). Eye movement trajectories and what they tell us, Neurosci. Biobehav. Rev. 30, 666–679.
Van der Stoep N., Visser-Meily J. M. A., Kappelle L. J., De Kort P. L. M., Huisman K. D., Eijsackers A. L., Kouwenhoven M., Van der Stigchel S., Nijboer T. C. W. (2013). Exploring near and far regions of space: distance specific visuospatial neglect after stroke, J. Clin. Exp. Neuropsychol. 35, 799–811.
- Search Google Scholar
- Export Citation
( Van der Stoep N. Visser-Meily J. M. A. Kappelle L. J. De Kort P. L. M. Huisman K. D. Eijsackers A. L. Kouwenhoven M. Van der Stigchel S. Nijboer T. C. W. 2013). Exploring near and far regions of space: distance specific visuospatial neglect after stroke, J. Clin. Exp. Neuropsychol. 35, 799– 811.
Van der Stoep N., Nijboer T. C. W., Van der Stigchel S. (2014). Exogenous orienting of crossmodal attention in 3-D space: support for a depth-aware crossmodal attentional system, Psychonom. Bull. Rev. 21, 708–714.
Van der Stoep N., Nijboer T. C. W., Van der Stigchel S., Spence C. (2015a). Multisensory interactions in the depth-plane in front and rear space: a review, Neuropsychologia 70, 335–349.
Van der Stoep N., Van der Stigchel S., Nijboer T. C. W. (2015b). Exogenous spatial attention decreases audiovisual integration, Atten. Percept. Psychophys. 77, 464–482.
Van der Stoep N., Spence C., Nijboer T. C. W., Van der Stigchel S. (2015c). On the relative contributions of multisensory integration and crossmodal exogenous spatial attention to multisensory response enhancement, Acta Psychol. 162, 20–28.
Van der Stoep N., Van der Stigchel S., Nijboer T. C. W. & Van der Smagt M. J. (in press). Audiovisual integration in near and far space: effects of changes in distance and stimulus effectiveness, Exp. Brain Res. DOI:10.1007/s00221-015-4248-2.
Vroomen J., Bertelson P., De Gelder B. (2001). The ventriloquist effect does not depend on the direction of automatic visual attention, Percept. Psychophys. 63, 651–659.
Vuilleumier P., Valenza N., Mayer E., Reverdin A., Landis T. (1998). Near and far visual space in unilateral neglect, Ann. Neurol. 43, 406–410.
Wallace M. T., Perrault T. J., Hairston W. D., Stein B. E. (2004). Visual experience is necessary for the development of multisensory integration, J. Neurosci. 24, 9580–9584.
Zou H., Müller H. J., Shi Z. (2012). Non-spatial sounds regulate eye movements and enhance visual search, J. Vis. 12, 2. DOI:10.1167/12.5.2.