We are continually bombarded by information arriving to each of our senses; however, the brain seems to effortlessly integrate this separate information into a unified percept. Although multisensory integration has been researched extensively using simple computer tasks and stimuli, much less is known about how multisensory integration functions in real-world contexts. Additionally, several recent studies have demonstrated that multisensory integration varies tremendously across naturalistic stimuli. Virtual reality can be used to study multisensory integration in realistic settings because it combines realism with precise control over the environment and stimulus presentation. In the current study, we investigated whether multisensory integration as measured by the redundant signals effects (RSE) is observable in naturalistic environments using virtual reality and whether it differs as a function of target and/or environment cue-richness. Participants detected auditory, visual, and audiovisual targets which varied in cue-richness within three distinct virtual worlds that also varied in cue-richness. We demonstrated integrative effects in each environment-by-target pairing and further showed a modest effect on multisensory integration as a function of target cue-richness but only in the cue-rich environment. Our study is the first to definitively show that minimal and more naturalistic tasks elicit comparable redundant signals effects. Our results also suggest that multisensory integration may function differently depending on the features of the environment. The results of this study have important implications in the design of virtual multisensory environments that are currently being used for training, educational, and entertainment purposes.
BrandweinA. B.FoxeJ. J.RussoN. N.AltschulerT. S.GomesH. and MolholmS. (2011). The development of audiovisual multisensory integration across childhood and early adolescence: a high-density electrical mapping studyCereb. Cortex211042–1055.
BrandweinA. B.FoxeJ. J.ButlerJ. S.RussoN. N.AltschulerT. S.GomesH. and MolholmS. (2013). The development of multisensory integration in high-functioning autism: high-density electrical mapping and psychophysical measures reveal impairments in the processing of audiovisual inputsCereb. Cortex231329–1341.
FengM.DeyA. and LindemanR. W. (2016). The effect of multi-sensory cues on performance and experience during walking in immersive virtual environments in: 2016 IEEE Virtual Reality (VR) pp. 173–174. Greenville, SC, USA.
FoxeJ. J.MoroczI. A.MurrayM. M.HigginsB. A.JavittD. C. and SchroederC. E. (2000). Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mappingBrain Res. Cogn. Brain Res.1077–83.
JuanC.CappeC.AlricB.RobyB.GilardeauS.BaroneP. and GirardP. (2017). The variability of multisensory processes of natural stimuli in human and non-human primates in a detection taskPLoS One12e0172480. DOI:10.1371/journal.pone.0172480.
LovelaceC. T.SteinB. E. and WallaceM. T. (2003). An irrelevant light enhances auditory detection in humans: a psychophysical analysis of multisensory integration in stimulus detectionBrain Res. Cogn. Brain Res.17447–453.
MeyerG. F.ShaoF.WhiteM. D.HopkinsC. and RobothamA. J. (2013). Modulation of visually evoked postural responses by contextual visual, haptic and auditory information: a “virtual reality check”PLoS One8e67651. DOI:10.1371/journal.pone.0067651.
MolholmS.RitterW.MurrayM. M.JavittD. C.SchroederC. E. and FoxeJ. J. (2002). Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping studyBrain Res. Cogn. Brain Res.14115–128.
MolholmS.SehatpourP.MehtaA. D.ShpanerM.Gomez-RamirezM.OrtigueS.DykeJ. P.SchwartzT. H. and FoxeJ. J. (2006). Audio-visual multisensory integration in superior parietal lobule revealed by human intracranial recordingsJ. Neurophysiol.96721–729.
MurrayM. M.MolholmS.MichelC. M.HeslenfeldD. J.RitterW.JavittD. C.SchroederC. E. and FoxeJ. J. (2005). Grabbing your ear: rapid auditory-somatosensory multisensory interactions in low-level sensory cortices are not constrained by stimulus alignmentCereb. Cortex15963–974.
NormandJ.-M.GiannopoulosE.SpanlangB. and SlaterM. (2011). Multisensory stimulation can induce an illusion of larger belly size in immersive virtual realityPLoS One6e16128. DOI:10.1371/journal.pone.0016128.
RussoN.MottronL.BurackJ. A. and JemelB. (2012). Parameters of semantic multisensory integration depend on timing and modality order among people on the autism spectrum: evidence from event-related potentialsNeuropsychologia502131–2141.
StevensonR. A.GhoseD.FisterJ. K.SarkoD. K.AltieriN. A.NidifferA. R.KurelaL. R.SiemannJ. K.JamesT. W. and WallaceM. T. (2014). Identifying and quantifying multisensory integration: a tutorial reviewBrain Topogr.27707–730.
Van der StoepN.Van der StigchelS.NijboerT. C. W. and Van der SmagtM. J. (2016). Audiovisual integration in near and far space: effects of changes in distance and stimulus effectivenessExp. Brain Res.2341175–1188.