The Complex Interplay Between Multisensory Integration and Perceptual Awareness

in Multisensory Research
Restricted Access
Get Access to Full Text
Rent on DeepDyve

Have an Access Token?

Enter your access token to activate and access content online.

Please login and go to your personal user account to enter your access token.


Have Institutional Access?

Access content through your institution. Any other coaching guidance?


The integration of information has been considered a hallmark of human consciousness, as it requires information being globally available via widespread neural interactions. Yet the complex interdependencies between multisensory integration and perceptual awareness, or consciousness, remain to be defined. While perceptual awareness has traditionally been studied in a single sense, in recent years we have witnessed a surge of interest in the role of multisensory integration in perceptual awareness. Based on a recent IMRF symposium on multisensory awareness, this review discusses three key questions from conceptual, methodological and experimental perspectives: (1) What do we study when we study multisensory awareness? (2) What is the relationship between multisensory integration and perceptual awareness? (3) Which experimental approaches are most promising to characterize multisensory awareness? We hope that this review paper will provoke lively discussions, novel experiments, and conceptual considerations to advance our understanding of the multifaceted interplay between multisensory integration and consciousness.

The Complex Interplay Between Multisensory Integration and Perceptual Awareness

in Multisensory Research



AdamR.NoppeneyU. (2014). A phonologically congruent sound boosts a visual target into perceptual awarenessFront. Integr. Neurosci. 870. DOI:10.3389/fnint.2014.00070.

AlaisD.BurrD. (2004). Ventriloquist effect results from rear-optimal bimodal integrationCurr. Biol. 14257262.

AllenJ.KrausN.BradlowA. (2000). Neural representation of consciously imperceptible speech sound differencesPercept. Psychophys. 6213831393.

AllerM.GianiA.ConradV.WatanabeM.NoppeneyU. (2015). A spatially collocated sound thrusts a flash into awarenessFront. Integr. Neurosci. 916. DOI:10.3389/fnint.2015.00016.

AlsiusA.MunhallK. G. (2013). Detection of audiovisual speech correspondences without visual awarenessPsychol. Sci. 24423431.

AndersenT. S.TiippanaK.SamsM. (2004). Factors influencing audiovisual fission and fusion illusionsCogn. Brain Res. 21301308.

ArziA.ShedleskyL.Ben-ShaulM.NasserK.OksenbergA.HairstonI. S.SobelN. (2012). Humans can learn new information during sleepNat. Neurosci. 1514601465.

AstleD. E.NobreA. C.ScerifG. (2010). Subliminally presented and stored objects capture spatial attentionJ. Neurosci. 3035673571.

BalduzziD.TononiG. (2008). Integrated information in discrete dynamical systems: motivation and theoretical frameworkPLoS Comput. Biol. 4e1000091. DOI:10.1371/journal.pcbi.1000091.

BekinschteinT. A.DehaeneS.RohautB.TadelF.CohenL.NaccacheL. (2009). Neural signature of the conscious processing of auditory regularitiesProc. Natl Acad. Sci. USA 10616721677.

BertelsonP.AscherslebenG. (1998). Automatic visual bias of perceived auditory locationPsychonom. Bull. Rev. 5482489.

BlakeR.LogothetisN. K. (2002). Visual competitionNat. Rev. Neurosci. 31321.

BlakeR.SobelK. V.JamesT. W. (2004). Neural synergy between kinetic vision and touchPsychol. Sci. 15397402.

BlankeO. (2012). Multisensory brain mechanisms of bodily self-consciousnessNat. Rev. Neurosci. 13556571.

BoringE. G. (1930). A new ambiguous figureAm. J. Psychol. 42444445.

BrunoN.JacomuzziA.BertaminiM.MeyerG. (2007). A visual-haptic Necker cube reveals temporal constraints on intersensory merging during perceptual explorationNeuropsychologia 45469475.

ChalmersD.BayneT. (2003). What is the unity of consciousness? in: The Unity of Consciousness: Binding Integration and DissociationCleeremansA.FrithC. (Eds) pp.  2358. Oxford University PressOxford, UK.

ChenY.-C.SpenceC. (2010). When hearing the bark helps to identify the dog: semantically-congruent sounds modulate the identification of masked picturesCognition 114389404.

ChenY.-C.SpenceC. (2011a). Crossmodal semantic priming by naturalistic sounds and spoken words enhances visual sensitivityJ. Exp. Psychol. Hum. Percept. Perform. 3715541568.

ChenY.-C.SpenceC. (2011b). The crossmodal facilitation of visual object representations by sound: evidence from the backward masking paradigmJ. Exp. Psychol. Hum. Percept. Perform. 3717841802.

ChenY.-C.YehS.-L.SpenceC. (2011). Crossmodal constraints on human perceptual awareness: auditory semantic modulation of binocular rivalryFront. Psychol. 2212. DOI:10.3389/fpsyg.2011.00212.

ConradV.BartelsA.KleinerM.NoppeneyU. (2010). Audiovisual interactions in binocular rivalryJ. Vis. 1027. DOI:10.1167/10.10.27.

ConradV.VitelloM. P.NoppeneyU. (2012). Interactions between apparent motion rivalry in vision and touchPsychol. Sci. 23940948.

ConradV.KleinerM.BartelsA.Hartcher O’BrienJ.BülthoffH. H.NoppeneyU. (2013). Naturalistic stimulus structure determines the integration of audiovisual looming signals in binocular rivalryPLoS ONE 8e70710. DOI:10.1371/journal.pone.0070710.

CritchleyH. D.HarrisonN. A. (2013). Visceral influences on brain and behaviorNeuron 77624638.

DayanP. (1998). A hierarchical model of binocular rivalryNeural Comput. 1011191135.

De GraafT. A.HsiehP. J.SackA. T. (2012). The “correlates” in neural correlates of consciousnessNeurosci. Biobehav. Rev. 36191197.

De MeoR.MurrayM. M.ClarkeS.MatuszP. J. (2015). Top-down control and early multisensory processes: chicken vs. eggFront. Integr. Neurosci. 917. DOI:10.3389/fnint.2015.00017.

DehaeneS. (2001). Towards a cognitive neuroscience of consciousness: basic evidence and a workspace frameworkCognition 79137.

DehaeneS.ChangeuxJ.-P. (2011). Experimental and theoretical approaches to conscious processingNeuron 70200227.

DeroyO. (2014). The unity assumption and the many unities of consciousness in: Sensory Integration and the Unity of ConsciousnessBennettD.HillC. (Eds) pp.  105124. MIT PressCambridge, MA, USA.

DeroyO.ChenY.-C.SpenceC. (2014). Multisensory constraints on awarenessPhil. Trans. R. Soc. Lond. B Biol. Sci. 369(1641) 20130207. DOI:10.1098/rstb.2013.0207.

Di LucaM.ErnstM. O.BackusB. T. (2010). Learning to use an invisible visual signal for perceptionCurr. Biol. 2018601863.

Di PaceE.SaraciniC. (2014). Action imitation changes perceptual alternations in binocular rivalryPLoS ONE 9e98305. DOI:10.1371/journal.pone.0098305.

DonerJ.LappinJ. S.PerfettoG. (1984). Detection of three-dimensional structure in moving optical patternsJ. Exp. Psychol. Hum. Percept. Perform. 10111.

ErnstM. O.BanksM. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashionNature 415(6870) 429433.

FaivreN.MudrikL.SchwartzN.KochC. (2014). Multisensory integration in complete unawareness: evidence from audiovisual congruency primingPsychol. Sci. 2520062016.

FaivreN.SalomonR.BlankeO. (2015). Visual consciousness and bodily self-consciousnessCurr. Opin. Neurobiol. 282328.

GallaceA.SpenceC. (2008). The cognitive and neural correlates of “tactile consciousness”: a multisensory perspectiveConscious. Cogn. 17370407.

GallaceA.SpenceC. (2014). In Touch With the Future. Oxford University PressOxford, UK.

GauR.NoppeneyU. (2016). How prior expectations shape multisensory perceptionNeuroImage 124876886.

GhazanfarA. A.SchroederC. E. (2006). Is neocortex essentially multisensory? Trends Cogn. Sci. 10278285.

GhoseG. M.MaunsellJ. (1999). Specialized representations in visual cortex: a role for binding? Neuron 247985.

GianiA. S.BelardinelliP.OrtizE.KleinerM.NoppeneyU. (2015). Detecting tones in complex auditory scenesNeuroImage 122203213.

GómezC.ArgandoñaE. D.SolierR. G.AnguloJ. C.VázquezM. (1995). Timing and competition in networks representing ambiguous figuresBrain Cogn. 29103114.

GutschalkA.MicheylC.OxenhamA. J. (2008). Neural correlates of auditory perceptual awareness under informational maskingPLoS Biol. 611561165.

Guzman-MartinezE.OrtegaL.GraboweckyM.MossbridgeJ.SuzukiS. (2012). Interactive coding of visual spatial frequency and auditory amplitude-modulation rateCurr. Biol. 22383388.

HaynesJ. D.DriverJ.ReesG. (2005). Visibility reflects dynamic changes of effective connectivity between V1 and fusiform cortexNeuron 46811821.

HeffnerR. S.HeffnerH. E. (1992a). Evolution of sound localization in mammals in: The Evolutionary Biology of HearingWebsterD. B.PopperA. N.FayR. R. (Eds) pp.  691715. Springer VerlagNew York, NY, USA.

HeffnerR. S.HeffnerH. E. (1992b). Visual factors in sound localization in mammalsJ. Comp. Neurol. 317219232.

HillisJ. M.ErnstM. O.BanksM. S.LandyM. S. (2002). Combining sensory information: mandatory fusion within, but not between, sensesScience 298(5598) 16271630.

HorlitzK. L.O’LearyA. (1993). Satiation or availability? Effects of attention, memory, and imagery on the perception of ambiguous figuresPercept. Psychophys. 53668681.

HsiaoJ. Y.ChenY.-C.SpenceC.YehS. L. (2012). Assessing the effects of audiovisual semantic congruency on the perception of a bistable figureConscious. Cogn. 21775787.

HsiehP.-J.ColasJ. T. (2012). Awareness is necessary for extracting patterns in working memory but not for directing spatial attentionJ. Exp. Psychol. Hum. Percept. Perform. 3810851090.

HsiehP.-J.ColasJ. T.KanwisherN. (2011). Pop-out without awareness: unseen feature singletons capture attention only when top-down attention is availablePsychol. Sci. 2212201226.

JacksonC. V. (1953). Visual factors in auditory localizationQ. J. Exp. Psychol. 55265.

KangM.-S.BlakeR. (2005). Perceptual synergy between seeing and hearing revealed during binocular rivalryPsychologija 32715.

KayserC.LogothetisN. K.PanzeriS. (2010). Visual enhancement of the information representation in auditory cortexCurr. Biol. 201924.

KnillD. C.PougetA. (2004). The Bayesian brain: the role of uncertainty in neural coding and computationTrends Neurosci. 27712719.

KördingK. P.BeierholmU.MaW. J.QuartzS.TenenbaumJ. B.ShamsL. (2007). Causal inference in multisensory perceptionPLoS ONE 2e943. DOI:10.1371/journal.pone.0000943.

KrugerH.CollinsT.CavanaghP. (2014). Similar effects of saccades on auditory and visual localization suggest common spatial mapJ. Vis. 141232.

KuangS.ZhangT. (2014). Smelling directions: olfaction modulates ambiguous visual motion perceptionSci. Rep. 45796. DOI:10.1038/srep05796.

LammeV. A. F. (2006). Towards a true neural stance on consciousnessTrends Cogn. Sci. 10494501.

LammeV. A. F.RoelfsemaP. R. (2000). The distinct modes of vision offered by feedforward and recurrent processingTrends Neurosci. 23571579.

LamyD.AlonL.CarmelT.ShalevN. (2015). The role of conscious perception in attentional capture and object-file updatingPsychol. Sci. 264857.

LeeH.NoppeneyU. (2011). Long-term music training tunes how the brain temporally binds signals from multiple sensesProc. Natl Acad. Sci. USA 108E1441E1450.

LeeH.NoppeneyU. (2014). Temporal prediction errors in visual and auditory corticesCurr. Biol. 24R309R310.

LeeM.BlakeR.KimS.KimC.-Y. (2015). Melodic sound enhances visual awareness of congruent musical notes, but only if you can read musicProc. Natl Acad. Sci. USA 11284938498.

LeveltW. J. (1965). On Binocular Rivalry. Institute for PerceptionSoesterberg, Netherlands.

LiangM.MourauxA.HuL.IannettiG. D. (2013). Primary sensory cortices contain distinguishable spatial patterns of activity for each senseNat. Commun. 41979. DOI:10.1038/ncomms2979.

LiebertR. M.BurkB. (1985). Voluntary control of reversible figuresPercept. Mot. Skills 6113071310.

LunghiC.AlaisD. (2013). Touch interacts with vision during binocular rivalry with a tight orientation tuningPLoS ONE 8e58754. DOI:10.1371/journal.pone.0058754.

LunghiC.AlaisD. (2015). Congruent tactile stimulation reduces the strength of visual suppression during binocular rivalrySci. Rep. 59413. DOI:10.1038/srep09413.

LunghiC.MorroneM. C. (2013). Early interaction between vision and touch during binocular rivalryMultisens. Res. 26291306.

LunghiC.BindaP.MorroneM. C. (2010). Touch disambiguates rivalrous perception at early stages of visual analysisCurr. Biol. 20R143R144.

LunghiC.MorroneM. C.AlaisD. (2014). Auditory and tactile signals combine to influence vision during binocular rivalryJ. Neurosci. 34784792.

MaedaF.KanaiR.ShimojoS. (2004). Changing pitch induced visual motion illusionCurr. Biol. 14R990R991.

MaruyaK.YangE.BlakeR. (2007). Voluntary action influences visual competitionPsychol. Sci. 1810901098.

McGurkH.MacDonaldJ. (1976). Hearing lips and seeing voicesNature 264(5588) 746748.

MengM.TongF. (2004). Can attention selectively bias bistable perception? Differences between binocular rivalry and ambiguous figuresJ. Vis. 4539551.

MeredithM. A.SteinB. E. (1986). Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integrationJ. Neurophysiol. 56640662.

MeyerK.KaplanJ. T.EssexR.WebberC.DamasioH.DamasioA. (2010). Predicting visual stimuli on the basis of activity in auditory corticesNat. Neurosci. 13667668.

MudrikL.FaivreN.KochC. (2014). Information integration without awarenessTrends Cogn. Sci. 18488496.

MunhallK. G.GribbleP.SaccoL.WardM. (1996). Temporal constraints on the McGurk effectPercept. Psychophys. 58351362.

MunhallK. G.ten HoveM. W.BrammerM.ParéM. (2009). Audiovisual integration of speech in a bistable illusionCurr. Biol. 19735739.

NeckerL. A. (1832). Observations on some remarkable optical phænomena seen in Switzerland; and on an optical phænomenon which occurs on viewing a figure of a crystal or geometrical solidPhilos. Mag. Ser. 3 1329337.

NoelJ.-P.WallaceM.BlakeR. (2015). Cognitive neuroscience: integration of sight and sound outside of awareness? Curr. Biol. 25R157R159.

OliversC. N. L.Van der BurgE. (2008). Bleeping you out of the blink: sound saves vision from oblivionBrain Res. 1242191199.

PaffenC. L. E.AlaisD. (2011). Attentional modulation of binocular rivalryFront. Hum. Neurosci. 5105. DOI:10.3389/fnhum.2011.00105.

PalmerT. D.RamseyA. K. (2012). The function of consciousness in multisensory integrationCognition 125353364.

PartanS.MarlerP. (1999). Communication goes multimodalScience 283(5406) 12721273.

RoT.BreitmeyerB.BurtonP.SinghalN. S.LaneD. (2003). Feedback contributions to visual awareness in human occipital cortexCurr. Biol. 1310381041.

RoheT.NoppeneyU. (2015). Cortical hierarchies perform Bayesian causal inference in multisensory perceptionPLoS Biol. 13e1002073. DOI:10.1371/journal.pbio.1002073.

RoheT.NoppeneyU. (2016). Distinct computational principles govern multisensory integration in primary sensory and association corticesCurr. Biol. 116.

RomeiV.MurrayM. M.CappeC.ThutG. (2009). Preperceptual and stimulus-selective enhancement of low-level human visual cortex excitability by soundsCurr. Biol. 1917991805.

RoskiesA. L. (1999). The binding problemNeuron 2479.

RubinE. (1915). Synsoplevede Figurer [Visually experienced Figures]. Studier i Psykologisk Analyse. GyldendalCopenhagen, Denmark.

SalomonR.LimM.HerbelinB.HesselmannG.BlankeO. (2013). Posing for awareness: proprioception modulates access to visual consciousness in a continuous flash suppression taskJ. Vis. 132.

SalomonR.KaliuzhnaM.HerbelinB.BlankeO. (2015). Balancing awareness: vestibular signals modulate visual consciousness in the absence of awarenessConscious. Cogn. 36289297.

SchroederC. E.FoxeJ. (2005). Multisensory contributions to low-level, “unisensory” processingCurr. Opin. Neurobiol. 15454458.

ShamsL.KamitaniY.ShimojoS. (2000). Illusions. What you see is what you hearNature 408(6814) 788.

SpenceC.BayneT. (2015). Is consciousness multisensory? in: Perception and Its ModalitiesStokesD.MatthenM.BiggsS. (Eds) pp.  95132. Oxford University PressOxford, UK.

SpenceC.SmithB.AuvrayM. (2015). Confusing tastes and flavours in: Perception and Its ModalitiesStokesD.MatthenM.BiggsS. (Eds) pp.  247274. Oxford University PressOxford, UK.

StanfordT. R. (2005). Evaluating the operations underlying multisensory integration in the cat superior colliculusJ. Neurosci. 2564996508.

SteinB. E.MeredithM. A. (1993). The Merging of the Senses. MIT PressCambridge, MA, USA.

SteinB. E.StanfordT. R. (2008). Multisensory integration: current issues from the perspective of the single neuronNat. Rev. Neurosci. 9255266.

SteinB. E.StanfordT. R.RamachandranR.PerraultT. J.RowlandB. A. (2009). Challenges in quantifying multisensory integration: alternative criteria, models, and inverse effectivenessExp. Brain Res. 198113126.

StevensonR. J.AttuquayefioT. (2013). Human olfactory consciousness and cognition: its unusual features may not result from unusual functions but from limited neocortical processing resourcesFront. Psychol. 4819. DOI:10.3389/fpsyg.2013.00819.

TsuchiyaN.KochC. (2005). Continuous flash suppression reduces negative afterimagesNat. Neurosci. 810961101.

TsuchiyaN.KochC.GilroyL. A.BlakeR. (2006). Depth of interocular suppression associated with continuous flash suppression, flash suppression, and binocular rivalryJ. Vis. 610681078.

Van EeR.Van BoxtelJ. J.ParkerA. L.AlaisD. (2009). Multisensory congruency as a mechanism for attentional control over perceptual selectionJ. Neurosci. 291164111649.

VetterP.SmithF. W.MuckliL. (2014). Decoding sound and imagery content in early visual cortexCurr. Biol. 2412561262.

VidalM.BarrèsV. (2014). Hearing (rivaling) lips and seeing voices: how audiovisual interactions modulate perceptual stabilization in binocular rivalryFront. Hum. Neurosci. 8677. DOI:10.3389/fnhum.2014.00677.

von SaldernS.NoppeneyU. (2013). Sensory and striatal areas integrate auditory and visual signals into behavioral benefits during motion discriminationJ. Neurosci. 3388418849.

VroomenJ.De GelderB. (2004). Perceptual effects of cross-modal stimulation: ventriloquism and the freezing phenomenon in: The Handbook of Multisensory ProcessesCalvertG. A.SpenceC.SteinB. E. (Eds) pp.  141146. MIT PressCambridge, MA, USA.

WernerS.NoppeneyU. (2010a). Distinct functional contributions of primary sensory and association areas to audiovisual integration in object categorizationJ. Neurosci. 3026622675.

WernerS.NoppeneyU. (2010b). Superadditive responses in superior temporal sulcus predict audiovisual benefits in object categorizationCereb. Cortex 2018291842.

WolfeJ. M.CaveK. R. (1999). The psychophysical evidence for a binding problem in human visionNeuron 241117.

YangY. H.YehS. L. (2014). Unmasking the dichoptic mask by sound: spatial congruency mattersExp. Brain Res. 23211091116.

ZhangX.FangF. (2012). Object-based attention guided by an invisible objectExp. Brain Res. 223397404.

ZhouW.JiangY.HeS.ChenD. (2010). Olfaction modulates visual perception in binocular rivalryCurr. Biol. 2013561358.


  • View in gallery

    Diagram of a binocular rivalry display and possible effects of crossmodal stimulation on rivalrous visual perception. (A) An example of dichoptic stimulation in which orthogonal gratings are separately presented to the eyes, the resulting conscious perception (B) is dominated by one of the two monocular images until a perceptual switch occurs in favour of the other visual stimulus. Normally dominance duration of the rivalrous stimuli is balanced. (C) Example of crossmodal stimulation prolonging dominance of the congruent visual stimulus during binocular rivalry: if the observer touches a haptic grating parallel to the visual grating dominating rivalrous perception, the switch towards the orthogonal (incongruent) visual grating is delayed as compared to visual only stimulation. (D) Example of crossmodal stimulation shortening the suppression of the congruent visual stimulus during binocular rivalry: if the observer touches a haptic grating orthogonal to the visual grating dominating rivalrous perception, the switch towards the parallel (congruent) visual grating occurs earlier compared to visual only stimulation.

Index Card

Content Metrics

Content Metrics

All Time Past Year Past 30 Days
Abstract Views 22 22 15
Full Text Views 7 7 7
PDF Downloads 2 2 2
EPUB Downloads 0 0 0