The Complex Interplay Between Multisensory Integration and Perceptual Awareness
In: Multisensory ResearchSearch for other papers by O. Deroy in
Current site
Google Scholar
Search for other papers by N. Faivre in
Current site
Google Scholar
Search for other papers by C. Lunghi in
Current site
Google Scholar
Search for other papers by C. Spence in
Current site
Google Scholar
Search for other papers by M. Aller in
Current site
Google Scholar
Search for other papers by U. Noppeney in
Current site
Google Scholar
The integration of information has been considered a hallmark of human consciousness, as it requires information being globally available via widespread neural interactions. Yet the complex interdependencies between multisensory integration and perceptual awareness, or consciousness, remain to be defined. While perceptual awareness has traditionally been studied in a single sense, in recent years we have witnessed a surge of interest in the role of multisensory integration in perceptual awareness. Based on a recent IMRF symposium on multisensory awareness, this review discusses three key questions from conceptual, methodological and experimental perspectives: (1) What do we study when we study multisensory awareness? (2) What is the relationship between multisensory integration and perceptual awareness? (3) Which experimental approaches are most promising to characterize multisensory awareness? We hope that this review paper will provoke lively discussions, novel experiments, and conceptual considerations to advance our understanding of the multifaceted interplay between multisensory integration and consciousness.
Purchase
Buy instant access (PDF download and unlimited online access):
Institutional Login
Log in with Open Athens, Shibboleth, or your institutional credentials
Personal login
Log in with your brill.com account
-
Adam R., Noppeney U. (2014). A phonologically congruent sound boosts a visual target into perceptual awareness, Front. Integr. Neurosci. 8, 70. DOI:10.3389/fnint.2014.00070.
-
Alais D., Burr D. (2004). Ventriloquist effect results from rear-optimal bimodal integration, Curr. Biol. 14, 257–262.
-
Allen J., Kraus N., Bradlow A. (2000). Neural representation of consciously imperceptible speech sound differences, Percept. Psychophys. 62, 1383–1393.
-
Aller M., Giani A., Conrad V., Watanabe M., Noppeney U. (2015). A spatially collocated sound thrusts a flash into awareness, Front. Integr. Neurosci. 9, 16. DOI:10.3389/fnint.2015.00016.
-
Alsius A., Munhall K. G. (2013). Detection of audiovisual speech correspondences without visual awareness, Psychol. Sci. 24, 423–431.
-
Andersen T. S., Tiippana K., Sams M. (2004). Factors influencing audiovisual fission and fusion illusions, Cogn. Brain Res. 21, 301–308.
-
Arzi A., Shedlesky L., Ben-Shaul M., Nasser K., Oksenberg A., Hairston I. S., Sobel N. (2012). Humans can learn new information during sleep, Nat. Neurosci. 15, 1460–1465.
-
Astle D. E., Nobre A. C., Scerif G. (2010). Subliminally presented and stored objects capture spatial attention, J. Neurosci. 30, 3567–3571.
-
Balduzzi D., Tononi G. (2008). Integrated information in discrete dynamical systems: motivation and theoretical framework, PLoS Comput. Biol. 4, e1000091. DOI:10.1371/journal.pcbi.1000091.
-
Bekinschtein T. A., Dehaene S., Rohaut B., Tadel F., Cohen L., Naccache L. (2009). Neural signature of the conscious processing of auditory regularities, Proc. Natl Acad. Sci. USA 106, 1672–1677.
-
Bertelson P., Aschersleben G. (1998). Automatic visual bias of perceived auditory location, Psychonom. Bull. Rev. 5, 482–489.
-
Blake R., Logothetis N. K. (2002). Visual competition, Nat. Rev. Neurosci. 3, 13–21.
-
Blake R., Sobel K. V., James T. W. (2004). Neural synergy between kinetic vision and touch, Psychol. Sci. 15, 397–402.
-
Blanke O. (2012). Multisensory brain mechanisms of bodily self-consciousness, Nat. Rev. Neurosci. 13, 556–571.
-
Boring E. G. (1930). A new ambiguous figure, Am. J. Psychol. 42, 444–445.
-
Bruno N., Jacomuzzi A., Bertamini M., Meyer G. (2007). A visual-haptic Necker cube reveals temporal constraints on intersensory merging during perceptual exploration, Neuropsychologia 45, 469–475.
-
Chalmers D., Bayne T. (2003). What is the unity of consciousness?, in: The Unity of Consciousness: Binding, Integration, and Dissociation, Cleeremans A., Frith C. (Eds), pp. 23–58. Oxford University Press, Oxford, UK.
-
Chen Y.-C., Spence C. (2010). When hearing the bark helps to identify the dog: semantically-congruent sounds modulate the identification of masked pictures, Cognition 114, 389–404.
-
Chen Y.-C., Spence C. (2011a). Crossmodal semantic priming by naturalistic sounds and spoken words enhances visual sensitivity, J. Exp. Psychol. Hum. Percept. Perform. 37, 1554–1568.
-
Chen Y.-C., Spence C. (2011b). The crossmodal facilitation of visual object representations by sound: evidence from the backward masking paradigm, J. Exp. Psychol. Hum. Percept. Perform. 37, 1784–1802.
-
Chen Y.-C., Yeh S.-L., Spence C. (2011). Crossmodal constraints on human perceptual awareness: auditory semantic modulation of binocular rivalry, Front. Psychol. 2, 212. DOI:10.3389/fpsyg.2011.00212.
-
Conrad V., Bartels A., Kleiner M., Noppeney U. (2010). Audiovisual interactions in binocular rivalry, J. Vis. 10, 27. DOI:10.1167/10.10.27.
-
Conrad V., Vitello M. P., Noppeney U. (2012). Interactions between apparent motion rivalry in vision and touch, Psychol. Sci. 23, 940–948.
-
Conrad V., Kleiner M., Bartels A., Hartcher O’Brien J., Bülthoff H. H., Noppeney U. (2013). Naturalistic stimulus structure determines the integration of audiovisual looming signals in binocular rivalry, PLoS ONE 8, e70710. DOI:10.1371/journal.pone.0070710.
-
Critchley H. D., Harrison N. A. (2013). Visceral influences on brain and behavior, Neuron 77, 624–638.
-
Dayan P. (1998). A hierarchical model of binocular rivalry, Neural Comput. 10, 1119–1135.
-
De Graaf T. A., Hsieh P. J., Sack A. T. (2012). The “correlates” in neural correlates of consciousness, Neurosci. Biobehav. Rev. 36, 191–197.
-
De Meo R., Murray M. M., Clarke S., Matusz P. J. (2015). Top-down control and early multisensory processes: chicken vs. egg, Front. Integr. Neurosci. 9, 17. DOI:10.3389/fnint.2015.00017.
-
Dehaene S. (2001). Towards a cognitive neuroscience of consciousness: basic evidence and a workspace framework, Cognition 79, 1–37.
-
Dehaene S., Changeux J.-P. (2011). Experimental and theoretical approaches to conscious processing, Neuron 70, 200–227.
-
Deroy O. (2014). The unity assumption and the many unities of consciousness, in: Sensory Integration and the Unity of Consciousness, Bennett D., Hill C. (Eds), pp. 105–124. MIT Press, Cambridge, MA, USA.
-
Deroy O., Chen Y.-C., Spence C. (2014). Multisensory constraints on awareness, Phil. Trans. R. Soc. Lond. B, Biol. Sci. 369(1641), 20130207. DOI:10.1098/rstb.2013.0207.
-
Di Luca M., Ernst M. O., Backus B. T. (2010). Learning to use an invisible visual signal for perception, Curr. Biol. 20, 1860–1863.
-
Di Pace E., Saracini C. (2014). Action imitation changes perceptual alternations in binocular rivalry, PLoS ONE 9, e98305. DOI:10.1371/journal.pone.0098305.
-
Doner J., Lappin J. S., Perfetto G. (1984). Detection of three-dimensional structure in moving optical patterns, J. Exp. Psychol. Hum. Percept. Perform. 10, 1–11.
-
Ernst M. O., Banks M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion, Nature 415(6870), 429–433.
-
Faivre N., Mudrik L., Schwartz N., Koch C. (2014). Multisensory integration in complete unawareness: evidence from audiovisual congruency priming, Psychol. Sci. 25, 2006–2016.
-
Faivre N., Salomon R., Blanke O. (2015). Visual consciousness and bodily self-consciousness, Curr. Opin. Neurobiol. 28, 23–28.
-
Gallace A., Spence C. (2008). The cognitive and neural correlates of “tactile consciousness”: a multisensory perspective, Conscious. Cogn. 17, 370–407.
-
Gallace A., Spence C. (2014). In Touch With the Future. Oxford University Press, Oxford, UK.
-
Gau R., Noppeney U. (2016). How prior expectations shape multisensory perception, NeuroImage 124, 876–886.
-
Ghazanfar A. A., Schroeder C. E. (2006). Is neocortex essentially multisensory? Trends Cogn. Sci. 10, 278–285.
-
Ghose G. M., Maunsell J. (1999). Specialized representations in visual cortex: a role for binding? Neuron 24, 79–85.
-
Giani A. S., Belardinelli P., Ortiz E., Kleiner M., Noppeney U. (2015). Detecting tones in complex auditory scenes, NeuroImage 122, 203–213.
-
Gómez C., Argandoña E. D., Solier R. G., Angulo J. C., Vázquez M. (1995). Timing and competition in networks representing ambiguous figures, Brain Cogn. 29, 103–114.
-
Gutschalk A., Micheyl C., Oxenham A. J. (2008). Neural correlates of auditory perceptual awareness under informational masking, PLoS Biol. 6, 1156–1165.
-
Guzman-Martinez E., Ortega L., Grabowecky M., Mossbridge J., Suzuki S. (2012). Interactive coding of visual spatial frequency and auditory amplitude-modulation rate, Curr. Biol. 22, 383–388.
-
Haynes J. D., Driver J., Rees G. (2005). Visibility reflects dynamic changes of effective connectivity between V1 and fusiform cortex, Neuron 46, 811–821.
-
Heffner R. S., Heffner H. E. (1992a). Evolution of sound localization in mammals, in: The Evolutionary Biology of Hearing, Webster D. B., Popper A. N., Fay R. R. (Eds), pp. 691–715. Springer Verlag, New York, NY, USA.
-
Heffner R. S., Heffner H. E. (1992b). Visual factors in sound localization in mammals, J. Comp. Neurol. 317, 219–232.
-
Hillis J. M., Ernst M. O., Banks M. S., Landy M. S. (2002). Combining sensory information: mandatory fusion within, but not between, senses, Science 298(5598), 1627–1630.
-
Horlitz K. L., O’Leary A. (1993). Satiation or availability? Effects of attention, memory, and imagery on the perception of ambiguous figures, Percept. Psychophys. 53, 668–681.
-
Hsiao J. Y., Chen Y.-C., Spence C., Yeh S. L. (2012). Assessing the effects of audiovisual semantic congruency on the perception of a bistable figure, Conscious. Cogn. 21, 775–787.
-
Hsieh P.-J., Colas J. T. (2012). Awareness is necessary for extracting patterns in working memory but not for directing spatial attention, J. Exp. Psychol. Hum. Percept. Perform. 38, 1085–1090.
-
Hsieh P.-J., Colas J. T., Kanwisher N. (2011). Pop-out without awareness: unseen feature singletons capture attention only when top-down attention is available, Psychol. Sci. 22, 1220–1226.
-
Jackson C. V. (1953). Visual factors in auditory localization, Q. J. Exp. Psychol. 5, 52–65.
-
Kang M.-S., Blake R. (2005). Perceptual synergy between seeing and hearing revealed during binocular rivalry, Psychologija 32, 7–15.
-
Kayser C., Logothetis N. K., Panzeri S. (2010). Visual enhancement of the information representation in auditory cortex, Curr. Biol. 20, 19–24.
-
Knill D. C., Pouget A. (2004). The Bayesian brain: the role of uncertainty in neural coding and computation, Trends Neurosci. 27, 712–719.
-
Körding K. P., Beierholm U., Ma W. J., Quartz S., Tenenbaum J. B., Shams L. (2007). Causal inference in multisensory perception, PLoS ONE 2, e943. DOI:10.1371/journal.pone.0000943.
-
Kruger H., Collins T., Cavanagh P. (2014). Similar effects of saccades on auditory and visual localization suggest common spatial map, J. Vis. 14, 1232.
-
Kuang S., Zhang T. (2014). Smelling directions: olfaction modulates ambiguous visual motion perception, Sci. Rep. 4, 5796. DOI:10.1038/srep05796.
-
Lamme V. A. F. (2006). Towards a true neural stance on consciousness, Trends Cogn. Sci. 10, 494–501.
-
Lamme V. A. F., Roelfsema P. R. (2000). The distinct modes of vision offered by feedforward and recurrent processing, Trends Neurosci. 23, 571–579.
-
Lamy D., Alon L., Carmel T., Shalev N. (2015). The role of conscious perception in attentional capture and object-file updating, Psychol. Sci. 26, 48–57.
-
Lee H., Noppeney U. (2011). Long-term music training tunes how the brain temporally binds signals from multiple senses, Proc. Natl Acad. Sci. USA 108, E1441–E1450.
-
Lee H., Noppeney U. (2014). Temporal prediction errors in visual and auditory cortices, Curr. Biol. 24, R309–R310.
-
Lee M., Blake R., Kim S., Kim C.-Y. (2015). Melodic sound enhances visual awareness of congruent musical notes, but only if you can read music, Proc. Natl Acad. Sci. USA 112, 8493–8498.
-
Levelt W. J. (1965). On Binocular Rivalry. Institute for Perception, Soesterberg, Netherlands.
-
Liang M., Mouraux A., Hu L., Iannetti G. D. (2013). Primary sensory cortices contain distinguishable spatial patterns of activity for each sense, Nat. Commun. 4, 1979. DOI:10.1038/ncomms2979.
-
Liebert R. M., Burk B. (1985). Voluntary control of reversible figures, Percept. Mot. Skills 61, 1307–1310.
-
Lunghi C., Alais D. (2013). Touch interacts with vision during binocular rivalry with a tight orientation tuning, PLoS ONE 8, e58754. DOI:10.1371/journal.pone.0058754.
-
Lunghi C., Alais D. (2015). Congruent tactile stimulation reduces the strength of visual suppression during binocular rivalry, Sci. Rep. 5, 9413. DOI:10.1038/srep09413.
-
Lunghi C., Morrone M. C. (2013). Early interaction between vision and touch during binocular rivalry, Multisens. Res. 26, 291–306.
-
Lunghi C., Binda P., Morrone M. C. (2010). Touch disambiguates rivalrous perception at early stages of visual analysis, Curr. Biol. 20, R143–R144.
-
Lunghi C., Morrone M. C., Alais D. (2014). Auditory and tactile signals combine to influence vision during binocular rivalry, J. Neurosci. 34, 784–792.
-
Maeda F., Kanai R., Shimojo S. (2004). Changing pitch induced visual motion illusion, Curr. Biol. 14, R990–R991.
-
Maruya K., Yang E., Blake R. (2007). Voluntary action influences visual competition, Psychol. Sci. 18, 1090–1098.
-
McGurk H., MacDonald J. (1976). Hearing lips and seeing voices, Nature 264(5588), 746–748.
-
Meng M., Tong F. (2004). Can attention selectively bias bistable perception? Differences between binocular rivalry and ambiguous figures, J. Vis. 4, 539–551.
-
Meredith M. A., Stein B. E. (1986). Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration, J. Neurophysiol. 56, 640–662.
-
Meyer K., Kaplan J. T., Essex R., Webber C., Damasio H., Damasio A. (2010). Predicting visual stimuli on the basis of activity in auditory cortices, Nat. Neurosci. 13, 667–668.
-
Mudrik L., Faivre N., Koch C. (2014). Information integration without awareness, Trends Cogn. Sci. 18, 488–496.
-
Munhall K. G., Gribble P., Sacco L., Ward M. (1996). Temporal constraints on the McGurk effect, Percept. Psychophys. 58, 351–362.
-
Munhall K. G., ten Hove M. W., Brammer M., Paré M. (2009). Audiovisual integration of speech in a bistable illusion, Curr. Biol. 19, 735–739.
-
Necker L. A. (1832). Observations on some remarkable optical phænomena seen in Switzerland; and on an optical phænomenon which occurs on viewing a figure of a crystal or geometrical solid, Philos. Mag. Ser. 3 1, 329–337.
-
Noel J.-P., Wallace M., Blake R. (2015). Cognitive neuroscience: integration of sight and sound outside of awareness? Curr. Biol. 25, R157–R159.
-
Olivers C. N. L., Van der Burg E. (2008). Bleeping you out of the blink: sound saves vision from oblivion, Brain Res. 1242, 191–199.
-
Paffen C. L. E., Alais D. (2011). Attentional modulation of binocular rivalry, Front. Hum. Neurosci. 5, 105. DOI:10.3389/fnhum.2011.00105.
-
Palmer T. D., Ramsey A. K. (2012). The function of consciousness in multisensory integration, Cognition 125, 353–364.
-
Partan S., Marler P. (1999). Communication goes multimodal, Science 283(5406), 1272–1273.
-
Ro T., Breitmeyer B., Burton P., Singhal N. S., Lane D. (2003). Feedback contributions to visual awareness in human occipital cortex, Curr. Biol. 13, 1038–1041.
-
Rohe T., Noppeney U. (2015). Cortical hierarchies perform Bayesian causal inference in multisensory perception, PLoS Biol. 13, e1002073. DOI:10.1371/journal.pbio.1002073.
-
Rohe T., Noppeney U. (2016). Distinct computational principles govern multisensory integration in primary sensory and association cortices, Curr. Biol. 1, 1–6.
-
Romei V., Murray M. M., Cappe C., Thut G. (2009). Preperceptual and stimulus-selective enhancement of low-level human visual cortex excitability by sounds, Curr. Biol. 19, 1799–1805.
-
Roskies A. L. (1999). The binding problem, Neuron 24, 7–9.
-
Rubin E. (1915). Synsoplevede Figurer [Visually experienced Figures]. Studier i Psykologisk Analyse. Gyldendal, Copenhagen, Denmark.
-
Salomon R., Lim M., Herbelin B., Hesselmann G., Blanke O. (2013). Posing for awareness: proprioception modulates access to visual consciousness in a continuous flash suppression task, J. Vis. 13, 2.
-
Salomon R., Kaliuzhna M., Herbelin B., Blanke O. (2015). Balancing awareness: vestibular signals modulate visual consciousness in the absence of awareness, Conscious. Cogn. 36, 289–297.
-
Schroeder C. E., Foxe J. (2005). Multisensory contributions to low-level, “unisensory” processing, Curr. Opin. Neurobiol. 15, 454–458.
-
Shams L., Kamitani Y., Shimojo S. (2000). Illusions. What you see is what you hear, Nature 408(6814), 788.
-
Spence C., Bayne T. (2015). Is consciousness multisensory?, in: Perception and Its Modalities, Stokes D., Matthen M., Biggs S. (Eds), pp. 95–132. Oxford University Press, Oxford, UK.
-
Spence C., Smith B., Auvray M. (2015). Confusing tastes and flavours, in: Perception and Its Modalities, Stokes D., Matthen M., Biggs S. (Eds), pp. 247–274. Oxford University Press, Oxford, UK.
-
Stanford T. R. (2005). Evaluating the operations underlying multisensory integration in the cat superior colliculus, J. Neurosci. 25, 6499–6508.
-
Stein B. E., Meredith M. A. (1993). The Merging of the Senses. MIT Press, Cambridge, MA, USA.
-
Stein B. E., Stanford T. R. (2008). Multisensory integration: current issues from the perspective of the single neuron, Nat. Rev. Neurosci. 9, 255–266.
-
Stein B. E., Stanford T. R., Ramachandran R., Perrault T. J., Rowland B. A. (2009). Challenges in quantifying multisensory integration: alternative criteria, models, and inverse effectiveness, Exp. Brain Res. 198, 113–126.
-
Stevenson R. J., Attuquayefio T. (2013). Human olfactory consciousness and cognition: its unusual features may not result from unusual functions but from limited neocortical processing resources, Front. Psychol. 4, 819. DOI:10.3389/fpsyg.2013.00819.
-
Tsuchiya N., Koch C. (2005). Continuous flash suppression reduces negative afterimages, Nat. Neurosci. 8, 1096–1101.
-
Tsuchiya N., Koch C., Gilroy L. A., Blake R. (2006). Depth of interocular suppression associated with continuous flash suppression, flash suppression, and binocular rivalry, J. Vis. 6, 1068–1078.
-
Van Ee R., Van Boxtel J. J., Parker A. L., Alais D. (2009). Multisensory congruency as a mechanism for attentional control over perceptual selection, J. Neurosci. 29, 11641–11649.
-
Vetter P., Smith F. W., Muckli L. (2014). Decoding sound and imagery content in early visual cortex, Curr. Biol. 24, 1256–1262.
-
Vidal M., Barrès V. (2014). Hearing (rivaling) lips and seeing voices: how audiovisual interactions modulate perceptual stabilization in binocular rivalry, Front. Hum. Neurosci. 8, 677. DOI:10.3389/fnhum.2014.00677.
-
von Saldern S., Noppeney U. (2013). Sensory and striatal areas integrate auditory and visual signals into behavioral benefits during motion discrimination, J. Neurosci. 33, 8841–8849.
-
Vroomen J., De Gelder B. (2004). Perceptual effects of cross-modal stimulation: ventriloquism and the freezing phenomenon, in: The Handbook of Multisensory Processes, Calvert G. A., Spence C., Stein B. E. (Eds), pp. 141–146. MIT Press, Cambridge, MA, USA.
-
Werner S., Noppeney U. (2010a). Distinct functional contributions of primary sensory and association areas to audiovisual integration in object categorization, J. Neurosci. 30, 2662–2675.
-
Werner S., Noppeney U. (2010b). Superadditive responses in superior temporal sulcus predict audiovisual benefits in object categorization, Cereb. Cortex 20, 1829–1842.
-
Wolfe J. M., Cave K. R. (1999). The psychophysical evidence for a binding problem in human vision, Neuron 24, 11–17.
-
Yang Y. H., Yeh S. L. (2014). Unmasking the dichoptic mask by sound: spatial congruency matters, Exp. Brain Res. 232, 1109–1116.
-
Zhang X., Fang F. (2012). Object-based attention guided by an invisible object, Exp. Brain Res. 223, 397–404.
-
Zhou W., Jiang Y., He S., Chen D. (2010). Olfaction modulates visual perception in binocular rivalry, Curr. Biol. 20, 1356–1358.
| All Time | Past 365 days | Past 30 Days | |
|---|---|---|---|
| Abstract Views | 2876 | 506 | 53 |
| Full Text Views | 462 | 40 | 5 |
| PDF Views & Downloads | 401 | 75 | 9 |
The Complex Interplay Between Multisensory Integration and Perceptual Awareness
In: Multisensory ResearchSearch for other papers by O. Deroy in
Current site
Google Scholar
PubMed
Search for other papers by N. Faivre in
Current site
Google Scholar
PubMed
Search for other papers by C. Lunghi in
Current site
Google Scholar
PubMed
Search for other papers by C. Spence in
Current site
Google Scholar
PubMed
Search for other papers by M. Aller in
Current site
Google Scholar
PubMed
Search for other papers by U. Noppeney in
Current site
Google Scholar
PubMed
The integration of information has been considered a hallmark of human consciousness, as it requires information being globally available via widespread neural interactions. Yet the complex interdependencies between multisensory integration and perceptual awareness, or consciousness, remain to be defined. While perceptual awareness has traditionally been studied in a single sense, in recent years we have witnessed a surge of interest in the role of multisensory integration in perceptual awareness. Based on a recent IMRF symposium on multisensory awareness, this review discusses three key questions from conceptual, methodological and experimental perspectives: (1) What do we study when we study multisensory awareness? (2) What is the relationship between multisensory integration and perceptual awareness? (3) Which experimental approaches are most promising to characterize multisensory awareness? We hope that this review paper will provoke lively discussions, novel experiments, and conceptual considerations to advance our understanding of the multifaceted interplay between multisensory integration and consciousness.
| All Time | Past 365 days | Past 30 Days | |
|---|---|---|---|
| Abstract Views | 2876 | 506 | 53 |
| Full Text Views | 462 | 40 | 5 |
| PDF Views & Downloads | 401 | 75 | 9 |