Cross-correlation between Auditory and Visual Signals Promotes Multisensory Integration

In: Multisensory Research
View More View Less
  • 1 Max Planck Institute for Biological Cybernetics and Bernstein Center for Computational Neuroscience, Tübingen, Germany
  • 2 Cognitive Neuroscience Department and Cognitive Interaction Technology — Center of Excellence, Bielefeld University, Germany
  • 3 Department of Experimental Psychology, University of Oxford, UK

Humans are equipped with multiple sensory channels that provide both redundant and complementary information about the objects and events in the world around them. A primary challenge for the brain is therefore to solve the ‘correspondence problem’, that is, to bind those signals that likely originate from the same environmental source, while keeping separate those unisensory inputs that likely belong to different objects/events. Whether multiple signals have a common origin or not must, however, be inferred from the signals themselves through a causal inference process.

Recent studies have demonstrated that cross-correlation, that is, the similarity in temporal structure between unimodal signals, represents a powerful cue for solving the correspondence problem in humans. Here we provide further evidence for the role of the temporal correlation between auditory and visual signals in multisensory integration. Capitalizing on the well-known fact that sensitivity to crossmodal conflict is inversely related to the strength of coupling between the signals, we measured sensitivity to crossmodal spatial conflicts as a function of the cross-correlation between the temporal structures of the audiovisual signals. Observers’ performance was systematically modulated by the cross-correlation, with lower sensitivity to crossmodal conflict being measured for correlated as compared to uncorrelated audiovisual signals. These results therefore provide support for the claim that cross-correlation promotes multisensory integration. A Bayesian framework is proposed to interpret the present results, whereby stimulus correlation is represented on the prior distribution of expected crossmodal co-occurrence.

  • Alais D., Blake R., Lee S. H. (1998). Visual features that vary together over time group together over space, Nat. Neurosci. 1, 160164.

  • Bresciani J. P., Dammeier F., Ernst M. (2006). Vision and touch are automatically integrated for the perception of sequences of events, J. Vision 6, 554564.

    • Search Google Scholar
    • Export Citation
  • Bresciani J. P., Ernst M. O., Drewing K., Bouyer G., Maury V., Kheddar A. (2005). Feeling what you hear: auditory signals can modulate tactile tap perception, Exp. Brain Res. 162, 172180.

    • Search Google Scholar
    • Export Citation
  • Ernst M. O. (2005). A Bayesian view on multimodal cue integration, in: Perception of the Human Body from the Inside Out, Knoblich G., Thornton I., Grosejan M., Shiffrar M. (Eds), pp.  105131. Oxford University Press, New York, NY, USA.

    • Search Google Scholar
    • Export Citation
  • Ernst M. O. (2007). Learning to integrate arbitrary signals from vision and touch, J. Vision 7, 114.

  • Ernst M. O., Banks M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion, Nature 415, 429433.

    • Search Google Scholar
    • Export Citation
  • Ernst M. O., Bülthoff H. H. (2004). Merging the senses into a robust percept, Trends Cogn. Sci. 8, 162169.

  • Ernst M. O., Di Luca M. (2011). Multisensory perception: from integration to remapping, in: Sensory Cue Integration, Trommershäuser J., Landy M., Körding K. (Eds), pp.  224250. Oxford University Press, New York, NY, USA.

    • Search Google Scholar
    • Export Citation
  • Fujisaki W., Nishida S. (2005). Temporal frequency characteristics of synchrony–asynchrony discrimination of audio-visual signals, Exp. Brain Res. 166, 455464.

    • Search Google Scholar
    • Export Citation
  • Fujisaki W., Nishida S. (2010). A common perceptual temporal limit of binding synchronous inputs across different sensory attributes and modalities, Proc. Royal Soc. B: Biol. Sci. 277, 22812290.

    • Search Google Scholar
    • Export Citation
  • Gepshtein S., Burge J., Ernst M. O., Banks M. S. (2005). The combination of vision and touch depends on spatial proximity, J. Vision 5, 10131023.

    • Search Google Scholar
    • Export Citation
  • Hillis J., Ernst M. O., Banks M., Landy M. (2002). Combining sensory information: mandatory fusion within, but not between, senses, Science 298, 16271630.

    • Search Google Scholar
    • Export Citation
  • Kleiner M., Brainard D., Pelli D., Ingling A., Murray R., Broussard C. (2007). What’s new in Psychtoolbox-3? Perception 36, 1.116.

  • Knill D. C. (2007). Robust cue integration: a Bayesian model and evidence from cue-conflict studies with stereoscopic and figure cues to slant, J. Vision 7, 124.

    • Search Google Scholar
    • Export Citation
  • Körding K. P., Beierholm U., Ma W. J., Quartz S., Tenenbaum J. B., Shams L. (2007). Causal inference in multisensory perception, PLoS ONE 2, 943.

  • Lee S. H., Blake R. (1999). Visual form created solely from temporal structure, Science 284, 11651168.

  • Parise C., Spence C. (2009). When birds of a feather flock together: synesthetic correspondences modulate audiovisual integration in non-synesthetes, PLoS ONE 4, e5664.

    • Search Google Scholar
    • Export Citation
  • Parise C. V., Spence C., Ernst M. O. (2012). When correlation implies causation in multisensory integration, Curr. Biol. 22, 4649.

  • Raposo D., Sheppard J. P., Schrater P. R., Churchland A. K. (2012). Multisensory decision-making in rats and humans, J. Neurosci. 32, 37263735.

    • Search Google Scholar
    • Export Citation
  • Shams L., Beierholm U. R. (2010). Causal inference in perception, Trends Cogn. Sci. 14, 425432.

  • Spence C. (in press). Cross-modal perceptual organization, in: The Oxford Handbook of Perceptual Organization, Wagemans J. (Ed.). Oxford University Press, Oxford, UK.

    • Search Google Scholar
    • Export Citation
  • Vatakis A., Spence C. (2007). Crossmodal binding: evaluating the ‘unity assumption’ using audiovisual speech stimuli, Percep. Psychophys. 69, 744756.

    • Search Google Scholar
    • Export Citation
  • Watson A., Pelli D. (1983). QUEST — A Bayesian adaptive psychometric method, Percept. Psychophys. 33, 113120.

  • Welch R., Warren D. (1980). Immediate perceptual response to intersensory discrepancy, Psycholog. Bull. 88, 638667.

  • Wertheimer M. (1923). Untersuchungen zur Lehre von der Gestalt (Studies on the science of Gestalt). II, Psychol. Res. 4, 301350.

  • Wichmann F., Hill N. (2001). The psychometric function: I. Fitting, sampling and goodness of fit, Percept. Psychophys. 63, 12931313.

Content Metrics

All Time Past Year Past 30 Days
Abstract Views 214 130 8
Full Text Views 186 16 5
PDF Downloads 19 6 2