The Efficacy of Single-Trial Multisensory Memories

in Multisensory Research
Restricted Access
Get Access to Full Text
Rent on DeepDyve

Have an Access Token?

Enter your access token to activate and access content online.

Please login and go to your personal user account to enter your access token.


Have Institutional Access?

Access content through your institution. Any other coaching guidance?


This review article summarizes evidence that multisensory experiences at one point in time have long-lasting effects on subsequent unisensory visual and auditory object recognition. The efficacy of single-trial exposure to task-irrelevant multisensory events is its ability to modulate memory performance and brain activity to unisensory components of these events presented later in time. Object recognition (either visual or auditory) is enhanced if the initial multisensory experience had been semantically congruent and can be impaired if this multisensory pairing was either semantically incongruent or entailed meaningless information in the task-irrelevant modality, when compared to objects encountered exclusively in a unisensory context. Processes active during encoding cannot straightforwardly explain these effects; performance on all initial presentations was indistinguishable despite leading to opposing effects with stimulus repetitions. Brain responses to unisensory stimulus repetitions differ during early processing stages (∼100 ms post-stimulus onset) according to whether or not they had been initially paired in a multisensory context. Plus, the network exhibiting differential responses varies according to whether or not memory performance is enhanced or impaired. The collective findings we review indicate that multisensory associations formed via single-trial learning exert influences on later unisensory processing to promote distinct object representations that manifest as differentiable brain networks whose activity is correlated with memory performance. These influences occur incidentally, despite many intervening stimuli, and are distinguishable from the encoding/learning processes during the formation of the multisensory associations. The consequences of multisensory interactions thus persist over time to impact memory retrieval and object discrimination.

The Efficacy of Single-Trial Multisensory Memories

in Multisensory Research



AmediA.von KriegsteinK.van AtteveldtN. M.BeauchampM. S.NaumerM. J. (2005). Functional imaging of human crossmodal identification and object reocognitionExp. Brain Res. 166559571.

BeerA. L.PlankT.MeyerG.GreenleeM. W. (2013). Combined diffusion-weighted and functional magnetic resonance imaging reveals a temporal-occipital network involved in auditory–visual object processingFront. Integr. Neurosci. 75.

ButlerA. J.JamesK. H. (2011). Cross-modal versus within-modal recall: Differences in behavioral and brain responsesBehav. Brain Res. 224387396.

CappeC.RouillerE. M.BaroneP. (2009). Multisensory anatomical pathwaysHearing Res. 2582836.

CappeC.ThutG.RomeiV.MurrayM. M. (2010). Auditory–visual multisensory interactions in humans: timing, topography, directionality, and sourcesJ. Neurosci. 301257212580.

CappeC.ThelenA.RomeiV.ThutG.MurrayM. M. (2012). Looming signals reveal synergistic principles of multisensory interactionsJ. Neurosci. 3211711182.

ChenY. C.YehS. L. (2008). Visual events modulated by sound in repetition blindnessPsychon. B. Rev. 15404408.

ColomboM.GrossC. G. (1994). Responses of inferior temporal cortex and hippocampal neurons during delayed matching to sample in monkeys (Macaca fascicularis)Behav. Neurosci. 108443455.

GhazanfarA. A.SchroederC. E. (2006). Is neocortex essentially multisensory? Tr. Cogn. Sci. 10278285.

GibsonJ. R.MaunsellJ. H. R. (1997). Sensory modality specificity of neural activity related to memory in visual cortexJ. Neurophysiol. 7812631275.

GottfriedJ. A.SmithA. P. R.RuggM. D.DolanR. J. (2004). Remembrance of odors past: human olfactory cortex in cross-modal recognition memoryNeuron 42687695.

GuoJ.GuoA. (2005). Crossmodal interactions between olfactory and visual learning in DrosophilaScience 309307310.

HaennyP. E.MaunsellJ. H. R.SchilerP. H. (1988). State dependent activity in monkey visual cortex: II. Retinal and extraretinal factors in V4Exp. Brain Res. 69245259.

HamiltonW. (1859). Lectures on Metaphysics and Logic. Gould & LincolnBoston, MA, USA.

JamesT. W.HumphreyG. K.GatiJ. S.ServosP.MenonR. S.GoodaleM. A. (2002). Haptic study of three-dimensional objects activates extrastriate visual areasNeuropsychologia 4017061714.

JohanssonB. B. (2012). Multisensory stimulation in stroke rehabilitationFront. Hum. Neurosci. 660.

LehmannS.MurrayM. M. (2005). The role of multisensory memories in unisensory object discriminationCogn. Brain Res. 24326334.

MaunsellJ. H. R.SclarG.NealeyT. A.DepriestD. D. (1991). Extraretinal representations in area V4 in the macaque monkeyVisual Neurosci. 7561573.

MeylanR. V.MurrayM. M. (2007). Auditory–visual multisensory interactions attenuate subsequent visual responses in humansNeuroimage 35244254.

MichelC. M.MurrayM. M. (2012). Towards the utilization of EEG as a brain imaging toolNeuroimage 61371385.

MichelC. M.MurrayM. M.LantzG.GonzalezS.SpinelliL.Grave de PeraltaR. (2004). EEG source imagingClin. Neurophysiol. 11521952222.

MurrayE. A.BusseyT. J. (1999). Perceptual–mnemonic functions of the perirhinal cortexTr. Cogn. Sci. 3142151.

MurrayE. A.GaffanD. (1994). Removal of the amygdala plus subjacent cortex disrupts the retention of both intramodal and crossmodal associative memories in monkeysBehav. Neurosci. 108494500.

MurrayM. M.CappeC.RomeiV.MartuzziR.ThutG. (2012). Auditory–visual multisensory interactions in humans: a synthesis of findings from behavior, ERPs, fMRI, and TMS in: The New Handbook of Multisensory ProcessesSteinB. E. (Ed.) pp.  223238. MIT PressCambridge, MA, USA.

MurrayM. M.SperdinH. F. (2010). Single-trial multisensory learning and memory retrieval in: Multisensory Object Perception in the Primate BrainKaiserJ.NaumerM. J. (Eds) pp.  191208. SpringerHeidelberg, Germany.

MurrayM. M.BrunetD.MichelC. M. (2008). Topographic ERP analyses: a step-by-step tutorial reviewBrain Topogr. 20249264.

MurrayM. M.FoxeJ. J.WylieG. R. (2005). The brain uses single-trial multisensory memories to discriminate without awarenessNeuroimage 27473478.

MurrayM. M.MichelC. M.Grave de PeraltaR.OrtigueS.BrunetD.AndinoS. G.SchniderA. (2004). Rapid discrimination of visual and multisensory memories revealed by electrical neuroimagingNeuroimage 21125135.

NaciL.TaylorK. I.CusackR.TylerL. K. (2012). Are the senses enough for sense? Early high-level feedback shapes our comprehension of multisensory objectsFront. Integr. Neurosci. 682.

NaueN.RachS.StruberD.HusterR. J.ZaehleT.KornerU.HerrmannC. S. (2011). Auditory event-related response in visual cortex modulates subsequent visual responses in humansJ. Neurosci. 3177297736.

NaumerM. J.DoehrmannO.MüllerN. G.MuckliL.KaiserJ.HeinG. (2009). Cortical plasticity of audio-visual object representationsCereb. Cortex 1916411653.

NybergL.HabibR.McIntoshA. R.TulvingE. (2000). Reactivation of encoding-related brain activity during memory retrievalProc. Natl Acad. Sci. USA 971112011124.

RanganathC.RainerG. (2003). Neural mechanisms for detecting and remembering novel eventsNat. Rev. Neurosci. 4193202.

ShamsL.WoznyD. R.KimR.SeitzA. (2011). Influences of multisensory experience on subsequent unisensory processingFront. Psychol. 2264.

ShamsL.SeitzA. R. (2008). Benefits of multisensory learningTr. Cogn. Sci. 12411417.

SpenceC.NichollsM. E.DriverJ. (2001). The cost of expecting events in the wrong sensory modalityPercept. Psychophys. 63330336.

TanabeH. C.HondaM.SadatoN. (2005). Functionally segregated neural substrates for arbitrary audiovisual paired-association learningJ. Neurosci. 2564096418.

TaylorK. I.MossH. E.StamatakisE. A.TylerL. K. (2006). Binding crossmodal object features in perirhinal cortexProc. Natl Acad. Sci. USA 10382398244.

TaylorK. I.StamatakisE. A.TylerL. K. (2009). Crossmodal integration of object features: voxel-based correlations in brain-damaged patientsBrain 132671683.

ThelenA.CappeC.MurrayM. M. (2012). Electrical neuroimaging of memory discrimination based on single-trial multisensory learningNeuroimage 6214781488.

ThelenA.MurrayM. M. (2013). Predicting individual differences in the impact of multisensory single-trial exposure upon subsequent object recognition. Neuroscience 2013 Abstracts Program No. 765.15483. Society for Neuroscience San Diego California USA.

ThelenA.TalsmaD.MurrayM. M. (submitted). The efficacy of single-trial multisensory memories for visual and auditory object recognitionJ. Cognitive Neurosci.

TsivilisD.OttenL. J.RuggM. D. (2001). Context effects on the neural correlates of recognition memory: an electrophysiological studyNeuron 31497505.

TzovaraA.MurrayM. M.PlompG.HerzogM.MichelC. M.De LuciaM. (2012a). Decoding stimulus-related information from single-trial EEG responses based on voltage topographiesPattern Recogn. 4521092122.

TzovaraA.MurrayM. M.MichelC. M.De LuciaM. (2012b). A tutorial review of electrical neuroimaging from group-average to single-trial event-related potentialsDev. Neuropsychol. 37518544.

van der LindenM.van TurennoutM.IndefreyP. (2010). Formation of category representations in superior temporal sulcusJ. Cognitive Neurosci. 2212701282.

Von KriegsteinK.GiraudA. L. (2006). Implicit multisensory associations influence voice recognitionPLoS Biol. 4e326.

WheelerM. E.PetersenS. E.BucknerR. L. (2000). Memory’s echo: vivid remembering reactivates sensory-specific cortexProc. Natl Acad. Sci. USA 971112511129.

WoznyD. R.ShamsL. (2011). Computational characterization of visually induced auditory spatial adaptationFront. Integr. Neurosci. 575.

ZangenehpourS.ZatorreR. J. (2010). Crossmodal recruitment of primary visual cortex following brief exposure to bimodal audiovisual stimuliNeuropsychologia 48591600.


  • View in gallery

    Illustration of the continuous recognition task used in our studies. In this paradigm participants indicate whether each image is being presented for the first or repeated time. Stimuli are presented for 500 milliseconds. Initial presentations are divided between those containing only images (V condition) and those presented with sounds (AV condition). Repeated presentations consist only of images, but can be divided between those that had been initially presented as images only (V− condition) and those that had been initially presented with sounds (V+ condition). In this way, contrasting performance and/or brain activity from the V− and V+ conditions reveals effects of past multisensory experiences on current unisensory (visual) processing.

  • View in gallery

    Psychophysical results. Panel A: The top set of bar graphs displays the mean (s.e.m. indicated) accuracy rates on the continuous recognition task for each experimental condition. The bottom set of bar graphs displays the mean (s.e.m. indicated) reaction times. An asterisk indicates a significant difference (p<0.05) either for repeated presentations in the case of accuracy (see Table 1 for details) or initial presentations in the case of reaction times (details available in original publications). Panel B: The bar graphs display the mean (s.e.m. indicated) accuracy rates from Experiments 1 and 2 in Thelen et al. (submitted). In Experiment 1, one-tailed post-hoc comparisons were warranted, while in Experiment 2 two-tailed post-hoc comparisons were used. An asterisk indicates a significant difference vs. all other conditions (p<0.05; see Table 1 for details). The same shade/filling across histograms refers to the same condition from different experiments.

  • View in gallery

    Brain imaging results. Panel A displays group-averaged event-related potential waveforms from an exemplar posterior scalp site from the data of Murray et al., 2004 (left) and Thelen et al., 2012 (right). The asterisk highlights differences observed at ∼100 ms post-stimulus onset. The greyscale topographic maps accounting best for each condition are displayed below the waveform plots with loci of peak positive potential indicated. The nasion is positioned upward and left hemiscalp on the left. Although subtle, topographic differences in each study were statistically reliable. Panel B displays the results of statistical analyses of source estimations in Murray et al. (2004) and Thelen et al. (2012) during the earliest period of event-related potential differences as well as the results of statistical contrasts in the fMRI study of Murray et al. (2005).


Content Metrics

Content Metrics

All Time Past Year Past 30 Days
Abstract Views 33 33 9
Full Text Views 68 68 56
PDF Downloads 5 5 1
EPUB Downloads 0 0 0