The Efficacy of Single-Trial Multisensory Memories

In: Multisensory Research
View More View Less
  • 1 The Functional Electrical Neuroimaging Laboratory, Neuropsychology and Neurorehabilitation Service, Department of Clinical Neurosciences, Centre Hospitalier Universitaire Vaudois and University of Lausanne, Switzerland
  • 2 Electroencephalography Brain Mapping Core, Center for Biomedical Imaging (CIBM), Lausanne, Switzerland
  • 3 Radiology Department, Centre Hospitalier Universitaire Vaudois and University of Lausanne, Switzerland
  • 4 Department of Hearing and Speech Sciences, Vanderbilt University, Nashville, TN, USA

Purchase instant access (PDF download and unlimited online access):

€29.95$34.95

This review article summarizes evidence that multisensory experiences at one point in time have long-lasting effects on subsequent unisensory visual and auditory object recognition. The efficacy of single-trial exposure to task-irrelevant multisensory events is its ability to modulate memory performance and brain activity to unisensory components of these events presented later in time. Object recognition (either visual or auditory) is enhanced if the initial multisensory experience had been semantically congruent and can be impaired if this multisensory pairing was either semantically incongruent or entailed meaningless information in the task-irrelevant modality, when compared to objects encountered exclusively in a unisensory context. Processes active during encoding cannot straightforwardly explain these effects; performance on all initial presentations was indistinguishable despite leading to opposing effects with stimulus repetitions. Brain responses to unisensory stimulus repetitions differ during early processing stages (∼100 ms post-stimulus onset) according to whether or not they had been initially paired in a multisensory context. Plus, the network exhibiting differential responses varies according to whether or not memory performance is enhanced or impaired. The collective findings we review indicate that multisensory associations formed via single-trial learning exert influences on later unisensory processing to promote distinct object representations that manifest as differentiable brain networks whose activity is correlated with memory performance. These influences occur incidentally, despite many intervening stimuli, and are distinguishable from the encoding/learning processes during the formation of the multisensory associations. The consequences of multisensory interactions thus persist over time to impact memory retrieval and object discrimination.

  • Amedi A., von Kriegstein K., van Atteveldt N. M., Beauchamp M. S., Naumer M. J. (2005). Functional imaging of human crossmodal identification and object reocognition, Exp. Brain Res. 166, 559571.

    • Search Google Scholar
    • Export Citation
  • Beer A. L., Plank T., Meyer G., Greenlee M. W. (2013). Combined diffusion-weighted and functional magnetic resonance imaging reveals a temporal-occipital network involved in auditory–visual object processing, Front. Integr. Neurosci. 7, 5.

    • Search Google Scholar
    • Export Citation
  • Butler A. J., James K. H. (2011). Cross-modal versus within-modal recall: Differences in behavioral and brain responses, Behav. Brain Res. 224, 387396.

    • Search Google Scholar
    • Export Citation
  • Cappe C., Rouiller E. M., Barone P. (2009). Multisensory anatomical pathways, Hearing Res. 258, 2836.

  • Cappe C., Thut G., Romei V., Murray M. M. (2010). Auditory–visual multisensory interactions in humans: timing, topography, directionality, and sources, J. Neurosci. 30, 1257212580.

    • Search Google Scholar
    • Export Citation
  • Cappe C., Thelen A., Romei V., Thut G., Murray M. M. (2012). Looming signals reveal synergistic principles of multisensory interactions, J. Neurosci. 32, 11711182.

    • Search Google Scholar
    • Export Citation
  • Chen Y. C., Yeh S. L. (2008). Visual events modulated by sound in repetition blindness, Psychon. B. Rev. 15, 404408.

  • Colombo M., Gross C. G. (1994). Responses of inferior temporal cortex and hippocampal neurons during delayed matching to sample in monkeys (Macaca fascicularis), Behav. Neurosci. 108, 443455.

    • Search Google Scholar
    • Export Citation
  • Ghazanfar A. A., Schroeder C. E. (2006). Is neocortex essentially multisensory? Tr. Cogn. Sci. 10, 278285.

  • Gibson J. R., Maunsell J. H. R. (1997). Sensory modality specificity of neural activity related to memory in visual cortex, J. Neurophysiol. 78, 12631275.

    • Search Google Scholar
    • Export Citation
  • Gottfried J. A., Smith A. P. R., Rugg M. D., Dolan R. J. (2004). Remembrance of odors past: human olfactory cortex in cross-modal recognition memory, Neuron 42, 687695.

    • Search Google Scholar
    • Export Citation
  • Guo J., Guo A. (2005). Crossmodal interactions between olfactory and visual learning in Drosophila, Science 309, 307310.

  • Haenny P. E., Maunsell J. H. R., Schiler P. H. (1988). State dependent activity in monkey visual cortex: II. Retinal and extraretinal factors in V4, Exp. Brain Res. 69, 245259.

    • Search Google Scholar
    • Export Citation
  • Hamilton W. (1859). Lectures on Metaphysics and Logic. Gould & Lincoln, Boston, MA, USA.

  • James T. W., Humphrey G. K., Gati J. S., Servos P., Menon R. S., Goodale M. A. (2002). Haptic study of three-dimensional objects activates extrastriate visual areas, Neuropsychologia 40, 17061714.

    • Search Google Scholar
    • Export Citation
  • Johansson B. B. (2012). Multisensory stimulation in stroke rehabilitation, Front. Hum. Neurosci. 6, 60.

  • Lehmann S., Murray M. M. (2005). The role of multisensory memories in unisensory object discrimination, Cogn. Brain Res. 24, 326334.

  • Maunsell J. H. R., Sclar G., Nealey T. A., Depriest D. D. (1991). Extraretinal representations in area V4 in the macaque monkey, Visual Neurosci. 7, 561573.

    • Search Google Scholar
    • Export Citation
  • Meylan R. V., Murray M. M. (2007). Auditory–visual multisensory interactions attenuate subsequent visual responses in humans, Neuroimage 35, 244254.

    • Search Google Scholar
    • Export Citation
  • Michel C. M., Murray M. M. (2012). Towards the utilization of EEG as a brain imaging tool, Neuroimage 61, 371385.

  • Michel C. M., Murray M. M., Lantz G., Gonzalez S., Spinelli L., Grave de Peralta R. (2004). EEG source imaging, Clin. Neurophysiol. 115, 21952222.

    • Search Google Scholar
    • Export Citation
  • Murray E. A., Bussey T. J. (1999). Perceptual–mnemonic functions of the perirhinal cortex, Tr. Cogn. Sci. 3, 142151.

  • Murray E. A., Gaffan D. (1994). Removal of the amygdala plus subjacent cortex disrupts the retention of both intramodal and crossmodal associative memories in monkeys, Behav. Neurosci. 108, 494500.

    • Search Google Scholar
    • Export Citation
  • Murray M. M., Cappe C., Romei V., Martuzzi R., Thut G. (2012). Auditory–visual multisensory interactions in humans: a synthesis of findings from behavior, ERPs, fMRI, and TMS, in: The New Handbook of Multisensory Processes, Stein B. E. (Ed.), pp.  223238. MIT Press, Cambridge, MA, USA.

    • Search Google Scholar
    • Export Citation
  • Murray M. M., Sperdin H. F. (2010). Single-trial multisensory learning and memory retrieval, in: Multisensory Object Perception in the Primate Brain, Kaiser J., Naumer M. J. (Eds), pp.  191208. Springer, Heidelberg, Germany.

    • Search Google Scholar
    • Export Citation
  • Murray M. M., Brunet D., Michel C. M. (2008). Topographic ERP analyses: a step-by-step tutorial review, Brain Topogr. 20, 249264.

  • Murray M. M., Foxe J. J., Wylie G. R. (2005). The brain uses single-trial multisensory memories to discriminate without awareness, Neuroimage 27, 473478.

    • Search Google Scholar
    • Export Citation
  • Murray M. M., Michel C. M., Grave de Peralta R., Ortigue S., Brunet D., Andino S. G., Schnider A. (2004). Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging, Neuroimage 21, 125135.

    • Search Google Scholar
    • Export Citation
  • Naci L., Taylor K. I., Cusack R., Tyler L. K. (2012). Are the senses enough for sense? Early high-level feedback shapes our comprehension of multisensory objects, Front. Integr. Neurosci. 6, 82.

    • Search Google Scholar
    • Export Citation
  • Naue N., Rach S., Struber D., Huster R. J., Zaehle T., Korner U., Herrmann C. S. (2011). Auditory event-related response in visual cortex modulates subsequent visual responses in humans, J. Neurosci. 31, 77297736.

    • Search Google Scholar
    • Export Citation
  • Naumer M. J., Doehrmann O., Müller N. G., Muckli L., Kaiser J., Hein G. (2009). Cortical plasticity of audio-visual object representations, Cereb. Cortex 19, 16411653.

    • Search Google Scholar
    • Export Citation
  • Nyberg L., Habib R., McIntosh A. R., Tulving E. (2000). Reactivation of encoding-related brain activity during memory retrieval, Proc. Natl Acad. Sci. USA 97, 1112011124.

    • Search Google Scholar
    • Export Citation
  • Ranganath C., Rainer G. (2003). Neural mechanisms for detecting and remembering novel events, Nat. Rev. Neurosci. 4, 193202.

  • Shams L., Wozny D. R., Kim R., Seitz A. (2011). Influences of multisensory experience on subsequent unisensory processing, Front. Psychol. 2, 264.

    • Search Google Scholar
    • Export Citation
  • Shams L., Seitz A. R. (2008). Benefits of multisensory learning, Tr. Cogn. Sci. 12, 411417.

  • Spence C., Nicholls M. E., Driver J. (2001). The cost of expecting events in the wrong sensory modality, Percept. Psychophys. 63, 330336.

    • Search Google Scholar
    • Export Citation
  • Tanabe H. C., Honda M., Sadato N. (2005). Functionally segregated neural substrates for arbitrary audiovisual paired-association learning, J. Neurosci. 25, 64096418.

    • Search Google Scholar
    • Export Citation
  • Taylor K. I., Moss H. E., Stamatakis E. A., Tyler L. K. (2006). Binding crossmodal object features in perirhinal cortex, Proc. Natl Acad. Sci. USA 103, 82398244.

    • Search Google Scholar
    • Export Citation
  • Taylor K. I., Stamatakis E. A., Tyler L. K. (2009). Crossmodal integration of object features: voxel-based correlations in brain-damaged patients, Brain 132, 671683.

    • Search Google Scholar
    • Export Citation
  • Thelen A., Cappe C., Murray M. M. (2012). Electrical neuroimaging of memory discrimination based on single-trial multisensory learning, Neuroimage 62, 14781488.

    • Search Google Scholar
    • Export Citation
  • Thelen A., Murray M. M. (2013). Predicting individual differences in the impact of multisensory, single-trial exposure upon subsequent object recognition. Neuroscience 2013 Abstracts, Program No. 765.15483. Society for Neuroscience, San Diego, California, USA.

  • Thelen A., Talsma D., Murray M. M. (submitted). The efficacy of single-trial multisensory memories for visual and auditory object recognition, J. Cognitive Neurosci.

    • Search Google Scholar
    • Export Citation
  • Tsivilis D., Otten L. J., Rugg M. D. (2001). Context effects on the neural correlates of recognition memory: an electrophysiological study, Neuron 31, 497505.

    • Search Google Scholar
    • Export Citation
  • Tzovara A., Murray M. M., Plomp G., Herzog M., Michel C. M., De Lucia M. (2012a). Decoding stimulus-related information from single-trial EEG responses based on voltage topographies, Pattern Recogn. 45, 21092122.

    • Search Google Scholar
    • Export Citation
  • Tzovara A., Murray M. M., Michel C. M., De Lucia M. (2012b). A tutorial review of electrical neuroimaging from group-average to single-trial event-related potentials, Dev. Neuropsychol. 37, 518544.

    • Search Google Scholar
    • Export Citation
  • van der Linden M., van Turennout M., Indefrey P. (2010). Formation of category representations in superior temporal sulcus, J. Cognitive Neurosci. 22, 12701282.

    • Search Google Scholar
    • Export Citation
  • Von Kriegstein K., Giraud A. L. (2006). Implicit multisensory associations influence voice recognition, PLoS Biol. 4, e326.

  • Wheeler M. E., Petersen S. E., Buckner R. L. (2000). Memory’s echo: vivid remembering reactivates sensory-specific cortex, Proc. Natl Acad. Sci. USA 97, 1112511129.

    • Search Google Scholar
    • Export Citation
  • Wozny D. R., Shams L. (2011). Computational characterization of visually induced auditory spatial adaptation, Front. Integr. Neurosci. 5, 75.

    • Search Google Scholar
    • Export Citation
  • Zangenehpour S., Zatorre R. J. (2010). Crossmodal recruitment of primary visual cortex following brief exposure to bimodal audiovisual stimuli, Neuropsychologia 48, 591600.

    • Search Google Scholar
    • Export Citation

Content Metrics

All Time Past Year Past 30 Days
Abstract Views 381 159 17
Full Text Views 217 7 1
PDF Views & Downloads 35 15 2