The role attention plays in our experience of a coherent, multisensory world is still controversial. On the one hand, a subset of inputs may be selected for detailed processing and multisensory integration in a top-down manner, i.e., guidance of multisensory integration by attention. On the other hand, stimuli may be integrated in a bottom-up fashion according to low-level properties such as spatial coincidence, thereby capturing attention. Moreover, attention itself is multifaceted and can be described via both top-down and bottom-up mechanisms. Thus, the interaction between attention and multisensory integration is complex and situation-dependent. The authors of this opinion paper are researchers who have contributed to this discussion from behavioural, computational and neurophysiological perspectives. We posed a series of questions, the goal of which was to illustrate the interplay between bottom-up and top-down processes in various multisensory scenarios in order to clarify the standpoint taken by each author and with the hope of reaching a consensus. Although divergence of viewpoint emerges in the current responses, there is also considerable overlap: In general, it can be concluded that the amount of influence that attention exerts on MSI depends on the current task as well as prior knowledge and expectations of the observer. Moreover stimulus properties such as the reliability and salience also determine how open the processing is to influences of attention.
Purchase
Buy instant access (PDF download and unlimited online access):
Institutional Login
Log in with Open Athens, Shibboleth, or your institutional credentials
Personal login
Log in with your brill.com account
Adam R., Noppeney U. (2014). A phonologically congruent sound boosts a visual target into perceptual awareness, Front. Integr. Neurosci. 8, 70. DOI:10.3389/fnint.2014.00070.
Adam R., Schönfelder S., Forneck J., Wessa M. (2014). Regulating the blink: cognitive reappraisal modulates attention, Front. Psychol. 5, 143. DOI:10.3389/fpsyg.2014.00143.
Alais D., Burr D. (2003). The ‘flash-lag’ effect occurs in audition and cross-modally, Curr. Biol. 13, 59–63.
Aller M., Giani A., Conrad V., Watanabe M., Noppeney U. (2015). A spatially collocated sound thrusts a flash into awareness, Front. Integr. Neurosci. 9, 16. DOI:10.3389/fnint.2015.00016.
Alsius A., Navarra J., Campbell R., Soto-Faraco S. (2005). Audiovisual integration of speech falters under high attention demands, Curr. Biol. 15, 839–843.
Alsius A., Navarra J., Soto-Faraco S. (2007). Attention to touch weakens audiovisual speech integration, Exp. Brain Res. 183, 399–404.
Bertelson P., Vroomen J., de Gelder B. D., Driver J. (2000). The ventriloquist effect does not depend on the direction of deliberate visual attention, Percept. Psychophys. 62, 321–332.
Buchtel H. A., Butter C. M. (1988). Spatial attentional shifts: implications for the role of polysensory mechanisms, Neuropsychologia 26, 499–509.
Busse L., Roberts K. C., Crist R. E., Weissman D. H., Woldorff M. G. (2005). The spread of attention across modalities and space in a multisensory object, Proc. Natl Acad. Sci. USA 102, 18751–18756.
Calvert G. A., Campbell R., Brammer M. J. (2000). Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex, Curr. Biol. 10, 649–657.
Cappe C., Thelen A., Romei V., Thut G., Murray M. M. (2012). Looming signals reveal synergistic principles of multisensory integration, J. Neurosci. 32, 1171–1182.
Corbetta M., Shulman G. L. (2002). Control of goal-directed and stimulus-driven attention in the brain, Nat. Rev. Neurosci. 3, 201–215.
Corbetta M., Patel G., Shulman G. L. (2008). The reorienting system of the human brain: from environment to theory of mind, Neuron 58, 306–324.
Doehrmann O., Naumer M. J. (2008). Semantics and the multisensory brain: how meaning modulates processes of audio-visual integration, Brain Res. 1242, 136–150.
Donohue S. E., Roberts K. C., Grent-‘t-Jong T., Woldorff M. G. (2011). The cross-modal spread of attention reveals differential constraints for the temporal and spatial linking of visual and auditory stimulus events, J. Neurosci. 31, 7982–7990.
Donohue S. E., Green J. J., Woldorff M. G. (2015). The effects of attention on the temporal integration of multisensory stimuli, Front. Integr. Neurosci. 9, 32. DOI:10.3389/fnint.2015.00032.
Downar J., Crawley A. P., Mikulis D. J., Davis K. D. (2000). A multimodal cortical network for the detection of changes in the sensory environment, Nat. Neurosci. 3, 277–283.
Driver J. (1996). Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading, Nature 381(6577), 66–68.
Eimer M., Driver J. (2001). Crossmodal links in endogenous and exogenous spatial attention: evidence from event-related brain potential studies, Neurosci. Biobehav. Rev. 25, 497–511. DOI:10.1016/S0149-7634(01)00029-X.
Ernst M. O., Banks M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion, Nature 415(6870), 429–433.
Ernst M. O., Bülthoff H. H. (2004). Merging the senses into a robust percept, Trends Cogn. Sci. 8, 162–169.
Fairhall S. L., Macaluso E. (2009). Spatial attention can modulate audiovisual integration at multiple cortical and subcortical sites, Eur. J. Neurosci. 29, 1247–1257.
Farah M. J., Wong A. B., Monheit M. A., Morrow L. A. (1989). Parietal lobe mechanisms of spatial attention: modality-specific or supramodal? Neuropsychologia 27, 461–470.
Fischer R., Plessow F. (2015). Efficient multitasking: parallel versus serial processing of multiple tasks, Front. Psychol. 6, 1366. DOI:10.3389/fpsyg.2015.01366.
Friston K. (2005). A theory of cortical responses, Phil. Trans. R. Soc. Lond. B Biol. Sci. 360, 815–836.
Giani A. S., Belardinelli P., Ortiz E., Kleiner M., Noppeney U. (2015). Detecting tones in complex auditory scenes, NeuroImage 122, 203–213.
Hecht D., Reiner M. (2008). Sensory dominance in combinations of audio, visual and haptic stimuli, Exp. Brain Res. 193, 307–314.
Helbig H. B., Ernst M. O. (2008). Visual-haptic cue weighting is independent of modality-specific attention, J. Vis. 8, 21. DOI:10.1167/8.1.21.
Hill K. T., Miller L. M. (2010). Auditory attentional control and selection during cocktail party listening, Cereb. Cortex 20, 583–590.
Hillyard S. A., Vogel E. K., Luck S. J. (1998). Sensory gain control (amplification) as a mechanism of selective attention: electrophysiological and neuroimaging evidence, Phil. Trans. R. Soc. Lond. B Biol. Sci. 353, 1257–1270.
Hink R. F., Van Voorhis S. T., Hillyard S. A., Smith T. S. (1977). The division of attention and the human auditory evoked potential, Neuropsychologia 15, 597–605.
Holmes N. P. (2009). The principle of inverse effectiveness in multisensory integration: some statistical considerations, Brain Topogr. 21, 168–176.
Jack B. N., O’Shea R. P., Cottrell D., Ritter W. (2013). Does the ventriloquist illusion assist selective listening? J. Exp. Psychol. Hum. Percept. Perform. 39, 1496–1502.
Karns C. M., Knight R. T. (2009). Intermodal auditory, visual, and tactile attention modulates early stages of neural processing, J. Cogn. Neurosci. 21, 669–683.
Kastner S., Pinsk M. A., De Weerd P., Desimone R., Ungerleider L. G. (1999). Increased activity in human visual cortex during directed attention in the absence of visual stimulation, Neuron 22, 751–761.
Kayser C., Petkov C. I., Augath M., Logothetis N. K. (2005). Integration of touch and sound in auditory cortex, Neuron 48, 373–384.
Klemen J., Chambers C. D. (2012). Current perspectives and methods in studying neural mechanisms of multisensory interactions, Neurosci. Biobehav. Rev. 36, 111–133.
Koelewijn T., Bronkhorst A., Theeuwes J. (2009). Auditory and visual capture during focused visual attention, J. Exp. Psychol. Hum. Percept. Perform. 35, 1303–1315.
Körding K. P., Beierholm U., Ma W. J., Quartz S., Tenenbaum J. B., Shams L. (2007). Causal inference in multisensory perception, PLoS ONE 2, e943. DOI:10.1371/journal.pone.0000943.
Krumbholz K., Nobis E. A., Weatheritt R. J., Fink G. R. (2009). Executive control of spatial attention shifts in the auditory compared to the visual modality, Hum. Brain Mapp. 30, 1457–1469.
Langers D. R. M., Backes W. H., Van Dijk P. (2007). Representation of lateralization and tonotopy in primary versus secondary human auditory cortex, NeuroImage 34, 264–273.
Lee H., Noppeney U. (2014). Temporal prediction errors in visual and auditory cortices, Curr. Biol. 24, R309–R310.
Lewis R., Noppeney U. (2010). Audiovisual synchrony improves motion discrimination via enhanced connectivity between early visual and auditory areas, J. Neurosci. 30, 12329–12339.
Li W., Piëch V., Gilbert C. D. (2004). Perceptual learning and top-down influences in primary visual cortex, Nat. Neurosci. 7, 651–657.
Luck S. J., Hillyard S. A., Mouloua M., Woldorff M. G., Clark V. P., Hawkins H. L. (1994). Effects of spatial cuing on luminance detectability: psychophysical and electrophysiological evidence for early selection, J. Exp. Psychol. Hum. Percept. Perform. 20, 887–904.
Lueck C. J., Zeki S., Friston K. J., Deiber M.-P., Cope P., Cunningham V. J., Lammertsma A. A., Kennard C., Frackowiak R. S. J. (1989). The colour centre in the cerebral cortex of man, Nature 340(6232), 386–389.
Macaluso E., Driver J. (2005). Multisensory spatial interactions: a window onto functional integration in the human brain, Trends Neurosci. 28, 264–271.
Macaluso E., Frith C. D., Driver J. (2002a). Supramodal effects of covert spatial orienting triggered by visual or tactile events, J. Cogn. Neurosci. 14, 389–401.
Macaluso E., Frith C. D., Driver J. (2002b). Directing attention to locations and to sensory modalities: multiple levels of selective processing revealed with PET, Cereb. Cortex 12, 357–368.
Macaluso E., Eimer M., Frith C. D., Driver J. (2003). Preparatory states in crossmodal spatial attention: spatial specificity and possible control mechanisms, Exp. Brain Res. 149, 62–74.
Marois R., Ivanoff J. (2005). Capacity limits of information processing in the brain, Trends Cogn. Sci. 9, 296–305.
Matusz P. J., Broadbent H., Ferrari J., Forrest B., Merkley R., Scerif G. (2015). Multi-modal distraction: insights from children’s limited attention, Cognition 136, 156–165.
Mazza V., Turatto M., Rossi M., Umiltà C. (2007). How automatic are audiovisual links in exogenous spatial attention? Neuropsychologia 45, 514–522.
McDonald J. J., Ward L. M. (1999). Spatial relevance determines facilitatory and inhibitory effects of auditory covert spatial orienting, J. Exp. Psychol. Hum. Percept. Perform. 25, 1234–1252.
McDonald J. J., Teder-Sälejärvi W. A., Russo F. D., Hillyard S. A. (2003). Neural substrates of perceptual enhancement by cross-modal spatial attention, J. Cogn. Neurosci. 15, 10–19.
McGurk H., Macdonald J. (1976). Hearing lips and seeing voices, Nature 264(5588), 746–748.
Molholm S., Ritter W., Murray M. M., Javitt D. C., Schroeder C. E., Foxe J. J. (2002). Multisensory auditory–visual interactions during early sensory processing in humans: a high-density electrical mapping study, Cogn. Brain Res. 14, 115–128.
Molholm S., Martinez A., Shpaner M., Foxe J. J. (2007). Object-based attention is multisensory: co-activation of an object’s representations in ignored sensory modalities, Eur. J. Neurosci. 26, 499–509.
Motter B. C. (1993). Focal attention produces spatially selective processing in visual cortical areas V1, V2, and V4 in the presence of competing stimuli, J. Neurophysiol. 70, 909–919.
Munhall K. G., MacDonald E. N., Byrne S. K., Johnsrude I. (2009). Talkers alter vowel production in response to real-time formant perturbation even when instructed not to compensate, J. Acoust. Soc. Am. 125, 384–390.
Näätänen R. (1982). Processing negativity: an evoked-potential reflection, Psychol. Bull. 92, 605–640.
Noesselt T., Tyll S., Boehler C. N., Budinger E., Heinze H.-J., Driver J. (2010). Sound-induced enhancement of low-intensity vision: multisensory influences on human sensory-specific cortices and thalamic bodies relate to perceptual enhancement of visual detection sensitivity, J. Neurosci. 30, 13609–13623.
Noppeney U., Ostwald D., Werner S. (2010). Perceptual decisions formed by accumulation of audiovisual evidence in prefrontal cortex, J. Neurosci. 30, 7434–7446.
Öhman A., Flykt A., Esteves F. (2001). Emotion drives attention: detecting the snake in the grass, J. Exp. Psychol. Gen. 130, 466–478.
Olivers C. N. L., Van der Burg E. (2008). Bleeping you out of the blink: sound saves vision from oblivion, Brain Res. 1242, 191–199.
Oruc I., Sinnett S., Bischof W. F., Soto-Faraco S., Lock K., Kingstone A. (2008). The effect of attention on the illusory capture of motion in bimodal stimuli, Brain Res. 1242, 200–208.
Picton T. W., Hillyard S. A. (1974). Human auditory evoked potentials. II: Effects of attention, Electroencephalogr. Clin. Neurophysiol. 36, 191–200.
Purves D., Wojtach W. T., Lotto R. B. (2011). Understanding vision in wholly empirical terms, Proc. Natl Acad. Sci. USA 108(Suppl. 3), 15588–15595.
Rach S., Diederich A., Colonius H. (2010). On quantifying multisensory interaction effects in reaction time and detection rate, Psychol. Res. 75, 77–94.
Recanzone G. H. (2009). Interactions of auditory and visual stimuli in space and time, Hearing Res. 258, 89–99.
Rohe T., Noppeney U. (2015a). Cortical hierarchies perform Bayesian causal inference in multisensory perception, PLoS Biol. 13, e1002073. DOI:10.1371/journal.pbio.1002073.
Rohe T., Noppeney U. (2015b). Sensory reliability shapes perceptual inference via two mechanisms, J. Vis. 15, 22. DOI:10.1167/15.5.22.
Santangelo V., Spence C. (2008). Is the exogenous orienting of spatial attention truly automatic? Evidence from unimodal and multisensory studies, Consc. Cogn. 17, 989–1015.
Santangelo V., Olivetti Belardinelli M., Spence C. (2007). The suppression of reflexive visual and auditory orienting when attention is otherwise engaged, J. Exp. Psychol. Hum. Percept. Perform. 33, 137–148.
Santangelo V., Belardinelli M. O., Spence C., Macaluso E. (2009). Interactions between voluntary and stimulus-driven spatial attention mechanisms across sensory modalities, J. Cogn. Neurosci. 21, 2384–2397.
Senkowski D., Talsma D., Herrmann C. S., Woldorff M. G. (2005). Multisensory processing and oscillatory gamma responses: effects of spatial selective attention, Exp. Brain Res. 166, 411–426.
Senkowski D., Saint-Amour D., Gruber T., Foxe J. J. (2008). Look who’s talking: the deployment of visuo-spatial attention during multisensory speech processing under noisy environmental conditions, NeuroImage 43, 379–387.
Shams L., Beierholm U. R. (2010). Causal inference in perception, Trends Cogn. Sci. 14, 425–432.
Spence C. (2010). Crossmodal spatial attention, Ann. N. Y. Acad. Sci. 1191, 182–200.
Spence C. (2013). Just how important is spatial coincidence to multisensory integration? Evaluating the spatial rule, Ann. N. Y. Acad. Sci. 1296, 31–49.
Spence C. J., Driver J. (1994). Covert spatial orienting in audition: exogenous and endogenous mechanisms, J. Exp. Psychol. Hum. Percept. Perform. 20, 555–574.
Spence C., Driver J. (1996). Audiovisual links in endogenous covert spatial attention, J. Exp. Psychol. Hum. Percept. Perform. 22, 1005–1030.
Spence C., Driver J. (2004). Crossmodal Space and Crossmodal Attention. Oxford University Press, Oxford, UK.
Stanford T. R., Quessy S., Stein B. E. (2005). Evaluating the operations underlying multisensory integration in the cat superior colliculus, J. Neurosci. 25, 6499–6508.
Staufenbiel S. M., van der Lubbe R. H. J., Talsma D. (2011). Spatially uninformative sounds increase sensitivity for visual motion change, Exp. Brain Res. 213, 457–464.
Stein B. E., Meredith M. A. (1993). The Merging of the Senses. MIT Press, Cambridge, MA, USA.
Stein B. E., Stanford T. R. (2008). Multisensory integration: current issues from the perspective of the single neuron, Nat. Rev. Neurosci. 9, 255–266.
Stein B. E., Jiang W., Stanford T. R. (2004). Multisensory integration in single neurons of the midbrain, in: The Handbook of Multisensory Processes, Vol. 15, Calvert G. A., Spence C., Stein B. E. (Eds), pp. 243–264. MIT Press, Cambridge, MA, USA.
Stekelenburg J. J., Vroomen J., de Gelder B. (2004). Illusory sound shifts induced by the ventriloquist illusion evoke the mismatch negativity, Neurosci. Lett. 357, 163–166.
Talsma D. (2015). Predictive coding and multisensory integration: an attentional account of the multisensory mind, Front. Integr. Neurosci. 9, 19. DOI:10.3389/fnint.2015.00019.
Talsma D., Woldorff M. (2005). Selective attention and multisensory integration: multiple phases of effects on the evoked brain activity, J. Cogn. Neurosci. 17, 1098–1114.
Talsma D., Doty T. J., Woldorff M. G. (2007). Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration? Cereb. Cortex 17, 679–690.
Talsma D., Senkowski D., Soto-Faraco S., Woldorff M. G. (2010). The multifaceted interplay between attention and multisensory integration, Trends Cogn. Sci. 14, 400–410. DOI:10.1016/j.tics.2010.06.008.
Tuomainen J., Andersen T. S., Tiippana K., Sams M. (2005). Audio-visual speech perception is special, Cognition 96, B13–B22.
Van der Burg E., Olivers C. N. L., Bronkhorst A. W., Theeuwes J. (2008). Pip and pop: nonspatial auditory signals improve spatial visual search, J. Exp. Psychol. Hum. Percept. Perform. 34, 1053–1065.
Van der Burg E., Cass J., Olivers C. N. L., Theeuwes J., Alais D. (2010). Efficient visual search from synchronized auditory signals requires transient audiovisual events, PLoS ONE 5, e10664. DOI:10.1371/journal.pone.0010664.
Van der Burg E., Talsma D., Olivers C. N. L., Hickey C., Theeuwes J. (2011). Early multisensory interactions affect the competition among multiple visual objects, NeuroImage 55, 1208–1218.
Van der Lubbe R. H. J., Postma A. (2005). Interruption from irrelevant auditory and visual onsets even when attention is in a focused state, Exp. Brain Res. 164, 464–471.
Van Wassenhove V., Grant K. W., Poeppel D. (2007). Temporal window of integration in auditory-visual speech perception, Neuropsychologia 45, 598–607.
Vatakis A., Spence C. (2007). Crossmodal binding: evaluating the ‘unity assumption’ using audiovisual speech stimuli, Percept. Psychophys. 69, 744–756.
Vatakis A., Ghazanfar A. A., Spence C. (2008). Facilitation of multisensory integration by the ‘unity effect’ reveals that speech is special, J. Vis. 8, 14. DOI:10.1167/8.9.14.
Vercillo T., Gori M. (2015). Attention to sound improves auditory reliability in audio-tactile spatial optimal integration, Front. Integr. Neurosci. 9, 34. DOI:10.3389/fnint.2015.00034.
Vroomen J., Bertelson P., de Gelder B. D. (2001a). The ventriloquist effect does not depend on the direction of automatic visual attention, Percept. Psychophys. 63, 651–659.
Vroomen J., Bertelson P., de Gelder B. (2001b). Directing spatial attention towards the illusory location of a ventriloquized sound, Acta Psychol. (Amst.) 108, 21–33.
Ward L. M., McDonald J. J., Lin D. (2000). On asymmetries in cross-modal spatial attention orienting, Percept. Psychophys. 62, 1258–1564.
Welch R. B., Warren D. H. (1980). Immediate perceptual response to intersensory discrepancy, Psychol. Bull. 88, 638–667.
Werner S., Noppeney U. (2010). Distinct functional contributions of primary sensory and association areas to audiovisual integration in object categorization, J. Neurosci. 30, 2662–2675.
Wolfe J. M., Butcher S. J., Lee C., Hyle M. (2003). Changing your mind: on the contributions of top-down and bottom-up guidance in visual search for feature singletons, J. Exp. Psychol. Hum. Percept. Perform. 29, 483–502.
Yantis S., Schwarzbach J., Serences J. T., Carlson R. L., Steinmetz M. A., Pekar J. J., Courtney S. M. (2002). Transient neural activity in human parietal cortex during spatial attention shifts, Nat. Neurosci. 5, 995–1002.
Zhang X., Zhaoping L., Zhou T., Fang F. (2012). Neural activities in V1 create a bottom-up saliency map, Neuron 73, 183–192.
Zimmer U., Macaluso E. (2007). Processing of multisensory spatial congruency can be dissociated from working memory and visuo-spatial attention, Eur. J. Neurosci. 26, 1681–1691.
All Time | Past 365 days | Past 30 Days | |
---|---|---|---|
Abstract Views | 3511 | 624 | 49 |
Full Text Views | 690 | 45 | 6 |
PDF Views & Downloads | 617 | 75 | 12 |
The role attention plays in our experience of a coherent, multisensory world is still controversial. On the one hand, a subset of inputs may be selected for detailed processing and multisensory integration in a top-down manner, i.e., guidance of multisensory integration by attention. On the other hand, stimuli may be integrated in a bottom-up fashion according to low-level properties such as spatial coincidence, thereby capturing attention. Moreover, attention itself is multifaceted and can be described via both top-down and bottom-up mechanisms. Thus, the interaction between attention and multisensory integration is complex and situation-dependent. The authors of this opinion paper are researchers who have contributed to this discussion from behavioural, computational and neurophysiological perspectives. We posed a series of questions, the goal of which was to illustrate the interplay between bottom-up and top-down processes in various multisensory scenarios in order to clarify the standpoint taken by each author and with the hope of reaching a consensus. Although divergence of viewpoint emerges in the current responses, there is also considerable overlap: In general, it can be concluded that the amount of influence that attention exerts on MSI depends on the current task as well as prior knowledge and expectations of the observer. Moreover stimulus properties such as the reliability and salience also determine how open the processing is to influences of attention.
All Time | Past 365 days | Past 30 Days | |
---|---|---|---|
Abstract Views | 3511 | 624 | 49 |
Full Text Views | 690 | 45 | 6 |
PDF Views & Downloads | 617 | 75 | 12 |