Cross-correlation between Auditory and Visual Signals Promotes Multisensory Integration

In: Multisensory Research

Humans are equipped with multiple sensory channels that provide both redundant and complementary information about the objects and events in the world around them. A primary challenge for the brain is therefore to solve the ‘correspondence problem’, that is, to bind those signals that likely originate from the same environmental source, while keeping separate those unisensory inputs that likely belong to different objects/events. Whether multiple signals have a common origin or not must, however, be inferred from the signals themselves through a causal inference process.

Recent studies have demonstrated that cross-correlation, that is, the similarity in temporal structure between unimodal signals, represents a powerful cue for solving the correspondence problem in humans. Here we provide further evidence for the role of the temporal correlation between auditory and visual signals in multisensory integration. Capitalizing on the well-known fact that sensitivity to crossmodal conflict is inversely related to the strength of coupling between the signals, we measured sensitivity to crossmodal spatial conflicts as a function of the cross-correlation between the temporal structures of the audiovisual signals. Observers’ performance was systematically modulated by the cross-correlation, with lower sensitivity to crossmodal conflict being measured for correlated as compared to uncorrelated audiovisual signals. These results therefore provide support for the claim that cross-correlation promotes multisensory integration. A Bayesian framework is proposed to interpret the present results, whereby stimulus correlation is represented on the prior distribution of expected crossmodal co-occurrence.

  • AlaisD.BlakeR.LeeS. H. (1998). Visual features that vary together over time group together over spaceNat. Neurosci. 1160164.

  • BrescianiJ. P.DammeierF.ErnstM. (2006). Vision and touch are automatically integrated for the perception of sequences of eventsJ. Vision 6554564.

    • Search Google Scholar
    • Export Citation
  • BrescianiJ. P.ErnstM. O.DrewingK.BouyerG.MauryV.KheddarA. (2005). Feeling what you hear: auditory signals can modulate tactile tap perceptionExp. Brain Res. 162172180.

    • Search Google Scholar
    • Export Citation
  • ErnstM. O. (2005). A Bayesian view on multimodal cue integration, in: Perception of the Human Body from the Inside OutKnoblichG.ThorntonI.GrosejanM.ShiffrarM. (Eds), pp.  105131. Oxford University PressNew York, NY, USA.

    • Search Google Scholar
    • Export Citation
  • ErnstM. O. (2007). Learning to integrate arbitrary signals from vision and touchJ. Vision 7114.

  • ErnstM. O.BanksM. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashionNature 415429433.

    • Search Google Scholar
    • Export Citation
  • ErnstM. O.BülthoffH. H. (2004). Merging the senses into a robust perceptTrends Cogn. Sci. 8162169.

  • ErnstM. O.Di LucaM. (2011). Multisensory perception: from integration to remapping, in: Sensory Cue IntegrationTrommershäuserJ.LandyM.KördingK. (Eds), pp.  224250. Oxford University PressNew York, NY, USA.

    • Search Google Scholar
    • Export Citation
  • FujisakiW.NishidaS. (2005). Temporal frequency characteristics of synchrony–asynchrony discrimination of audio-visual signalsExp. Brain Res. 166455464.

    • Search Google Scholar
    • Export Citation
  • FujisakiW.NishidaS. (2010). A common perceptual temporal limit of binding synchronous inputs across different sensory attributes and modalitiesProc. Royal Soc. B: Biol. Sci. 27722812290.

    • Search Google Scholar
    • Export Citation
  • GepshteinS.BurgeJ.ErnstM. O.BanksM. S. (2005). The combination of vision and touch depends on spatial proximityJ. Vision 510131023.

    • Search Google Scholar
    • Export Citation
  • HillisJ.ErnstM. O.BanksM.LandyM. (2002). Combining sensory information: mandatory fusion within, but not between, sensesScience 29816271630.

    • Search Google Scholar
    • Export Citation
  • KleinerM.BrainardD.PelliD.InglingA.MurrayR.BroussardC. (2007). What’s new in Psychtoolbox-3? Perception 361.116.

  • KnillD. C. (2007). Robust cue integration: a Bayesian model and evidence from cue-conflict studies with stereoscopic and figure cues to slantJ. Vision 7124.

    • Search Google Scholar
    • Export Citation
  • KördingK. P.BeierholmU.MaW. J.QuartzS.TenenbaumJ. B.ShamsL. (2007). Causal inference in multisensory perceptionPLoS ONE 2943.

  • LeeS. H.BlakeR. (1999). Visual form created solely from temporal structureScience 28411651168.

  • PariseC.SpenceC. (2009). When birds of a feather flock together: synesthetic correspondences modulate audiovisual integration in non-synesthetesPLoS ONE 4e5664.

    • Search Google Scholar
    • Export Citation
  • PariseC. V.SpenceC.ErnstM. O. (2012). When correlation implies causation in multisensory integrationCurr. Biol. 224649.

  • RaposoD.SheppardJ. P.SchraterP. R.ChurchlandA. K. (2012). Multisensory decision-making in rats and humansJ. Neurosci. 3237263735.

    • Search Google Scholar
    • Export Citation
  • ShamsL.BeierholmU. R. (2010). Causal inference in perceptionTrends Cogn. Sci. 14425432.

  • SpenceC. (in press). Cross-modal perceptual organization, in: The Oxford Handbook of Perceptual OrganizationWagemansJ. (Ed.). Oxford University PressOxford, UK.

    • Search Google Scholar
    • Export Citation
  • VatakisA.SpenceC. (2007). Crossmodal binding: evaluating the ‘unity assumption’ using audiovisual speech stimuliPercep. Psychophys. 69744756.

    • Search Google Scholar
    • Export Citation
  • WatsonA.PelliD. (1983). QUEST — A Bayesian adaptive psychometric methodPercept. Psychophys. 33113120.

  • WelchR.WarrenD. (1980). Immediate perceptual response to intersensory discrepancyPsycholog. Bull. 88638667.

  • WertheimerM. (1923). Untersuchungen zur Lehre von der Gestalt (Studies on the science of Gestalt). IIPsychol. Res. 4301350.

  • WichmannF.HillN. (2001). The psychometric function: I. Fitting, sampling and goodness of fitPercept. Psychophys. 6312931313.

Content Metrics

All Time Past Year Past 30 Days
Abstract Views 199 137 5
Full Text Views 180 16 0
PDF Downloads 16 5 0