Attentional Interactions Between Vision and Hearing in Event-Related Responses to Crossmodal and Conjunct Oddballs

In: Multisensory Research

Abstract

Are alternation and co-occurrence of stimuli of different sensory modalities conspicuous? In a novel audio-visual oddball paradigm, the P300 was used as an index of the allocation of attention to investigate stimulus- and task-related interactions between modalities. Specifically, we assessed effects of modality alternation and the salience of conjunct oddball stimuli that were defined by the co-occurrence of both modalities. We presented (a) crossmodal audio-visual oddball sequences, where both oddballs and standards were unimodal, but of a different modality (i.e., visual oddball with auditory standard, or vice versa), and (b) oddball sequences where standards were randomly of either modality while the oddballs were a combination of both modalities (conjunct stimuli). Subjects were instructed to attend to one of the modalities (whether part of a conjunct stimulus or not). In addition, we also tested specific attention to the conjunct stimuli. P300-like responses occurred even when the oddball was of the unattended modality. The pattern of event-related potential (ERP) responses obtained with the two crossmodal oddball sequences switched symmetrically between stimulus modalities when the task modality was switched. Conjunct oddballs elicited no oddball response if only one modality was attended. However, when conjunctness was specifically attended, an oddball response was obtained. Crossmodal oddballs capture sufficient attention even when not attended. Conjunct oddballs, however, are not sufficiently salient to attract attention when the task is unimodal. Even when specifically attended, the processing of conjunctness appears to involve additional steps that delay the oddball response.

  • AchimA. (1995). Signal detection in averaged evoked potentials: Monte Carlo comparison of the sensitivity of different methodsElectroencephalogr. Clin. Neurophysiol. 96574584.

    • Search Google Scholar
    • Export Citation
  • AlhoK. (1995). Cerebral generators of mismatch negativity (MMN) and its magnetic counterpart (MMNm) elicited by sound changesEar Hear. 163851.

    • Search Google Scholar
    • Export Citation
  • American Clinical Neurophysiology Society (2006). Guideline 5: guidelines for standard electrode position nomenclatureJ. Clin. Neurophysiol. 23107110.

    • Search Google Scholar
    • Export Citation
  • BatesD.MächlerM.BolkerB. and WalkerS. (2015). Fitting linear mixed-effects models using lme4J. Stat. Softw. 67148.

  • BeckerD. E. and ShapiroD. (1980). Directing attention toward stimuli affects the P300 but not the orienting responsePsychophysiology 17385389.

    • Search Google Scholar
    • Export Citation
  • BendixenA.GrimmS.DeouellL. Y.WetzelN.MädebachA. and SchrögerE. (2010). The time-course of auditory and visual distraction effects in a new crossmodal paradigmNeuropsychologia 4821302139.

    • Search Google Scholar
    • Export Citation
  • BrennanA. R. and ArnstenA. F. (2008). Neuronal mechanisms underlying attention deficit hyperactivity disorderAnn. N. Y. Acad. Sci. 1129236245.

    • Search Google Scholar
    • Export Citation
  • BrownC. R.ClarkeA. R. and BarryR. J. (2006). Inter-modal attention: ERPs to auditory targets in an inter-modal oddball taskInt. J. Psychophysiol. 627786.

    • Search Google Scholar
    • Export Citation
  • BrownC. R.ClarkeA. R. and BarryR. J. (2007). Auditory processing in an inter-modal oddball task: effects of a combined auditory/visual standard on auditory target ERPsInt. J. Psychophysiol. 65122131.

    • Search Google Scholar
    • Export Citation
  • CappeC.ThutG.RomeiV. and MurrayM. M. (2010). Auditory–visual multisensory interactions in humans: timing, topography, directionality, and sourcesJ. Neurosci. 301257212580.

    • Search Google Scholar
    • Export Citation
  • CornuL. and BianchiL. (1981). Visually emitted potential and attentional processesBoll. Soc. Ital. Biol. Sper. 5720672073.

  • DehaeneS. and ChangeuxJ.-P. (2011). Experimental and theoretical approaches to conscious processingNeuron 70200227.

  • DuncanC. C.BarryR. J.ConnollyJ. F.FischerC.MichieP. T.NäätänenR.PolichJ.ReinvangI. and Van PettenC. (2009). Event-related potentials in clinical research: guidelines for eliciting, recording, and quantifying mismatch negativity, P300, and N400Clin. Neurophysiol. 12018831908.

    • Search Google Scholar
    • Export Citation
  • Duncan-JohnsonC. C. and DonchinE. (1977). On quantifying surprise: the variation of event-related potentials with subjective probabilityPsychophysiology 14456467.

    • Search Google Scholar
    • Export Citation
  • FengT.QiuY.ZhuY. and TongS. (2008). Attention rivalry under irrelevant audiovisual stimulationNeurosci. Lett. 43869.

  • FortA.DelpuechC.PernierJ. and GiardM.-H. (2002a). Dynamics of cortico-subcortical cross-modal operations involved in audio-visual object detection in humansCereb. Cortex 1210311039.

    • Search Google Scholar
    • Export Citation
  • FortA.DelpuechC.PernierJ. and GiardM. H. (2002b). Early auditory–visual interactions in human cortex during nonredundant target identificationBrain Res. Cogn. Brain Res. 142030.

    • Search Google Scholar
    • Export Citation
  • FriedmanD.CycowiczY. M. and GaetaH. (2001). The novelty P3: an event-related brain potential (ERP) sign of the brains evaluation of noveltyNeurosc. Biobehav. Rev. 25355373.

    • Search Google Scholar
    • Export Citation
  • GarridoM. I.KilnerJ. M.StephanK. E. and FristonK. J. (2009). The mismatch negativity: a review of underlying mechanismsClin. Neurophysiol. 120453463.

    • Search Google Scholar
    • Export Citation
  • GonsalvezC. J.BarryR. J.RushbyJ. A. and PolichJ. (2007). Target-to-target interval, intensity, and P300 from an auditory single-stimulus taskPsychophysiology 44245250.

    • Search Google Scholar
    • Export Citation
  • GonsalvezC. L. and PolichJ. (2002). P300 amplitude is determined by target-to-target intervalPsychophysiology 39388396.

  • GoodP. I. (2005). Resampling Methods: a Practical Guide to Data Analysis3rd edn.. BirkhäuserBoston, MA, USA.

  • GrayH. M.AmbadyN.LowenthalW. T. and DeldinP. (2004). P300 as an index of attention to self-relevant stimuliJ. Exp. Soc. Psychol. 40216224.

    • Search Google Scholar
    • Export Citation
  • HeinrichS. P. and BachM. (2008). Signal and noise in P300 recordings to visual stimuliDoc. Ophthalmol. 1177383.

  • HeinrichS. P.AertsenA. and BachM. (2008). Oblique effects beyond low-level visual processingVision Res. 48809818.

  • HeinrichS. P.MellD. and BachM. (2009). Frequency-domain analysis of fast oddball responses to visual stimuli: a feasibility studyInt. J. Psychophysiol. 73287293.

    • Search Google Scholar
    • Export Citation
  • HeinzeH. J.LuckS. J.MangunG. R. and HillyardS. A. (1990). Visual event-related potentials index focused attention within bilateral stimulus arrays. I. Evidence for early selectionElectroencephalogr. Clin. Neurophysiol. 75511527.

    • Search Google Scholar
    • Export Citation
  • HolmS. (1979). A simple sequentially rejective multiple test procedureScand. J. Stat. 66570.

  • JiJ.PorjeszB.BegleiterH. and ChorlianD. (1999). P300: the similarities and differences in the scalp distribution of visual and auditory modalityBrain Topogr. 11315327.

    • Search Google Scholar
    • Export Citation
  • Johnson JrR. (1989). Developmental evidence for modality-dependent P300 generators: a normative studyPsychophysiology 26651667.

    • Search Google Scholar
    • Export Citation
  • KastnerS. and UngerleiderL. G. (2000). Mechanisms of visual attention in the human cortexAnnu. Rev. Neurosci. 23315341.

  • KatayamaJ. and PolichJ. (1996). P300, probability, and the three-tone paradigmElectroencephalogr. Clin. Neurophysiol. 100555562.

    • Search Google Scholar
    • Export Citation
  • KayserC. and LogothetisN. K. (2007). Do early sensory cortices integrate cross-modal information?Brain Struct. Funct. 212121132.

    • Search Google Scholar
    • Export Citation
  • KeilA.BradleyM. M.JunghöferM.RussmannT.LowenthalW. and LangP. J. (2007). Cross-modal attention capture by affective stimuli: evidence from event-related potentialsCogn. Affect. Behav. Neurosci. 71824.

    • Search Google Scholar
    • Export Citation
  • KeilA.DebenerS.GrattonG.JunghöferM.KappenmanE. S.LuckS. J.LuuP.MillerG. A. and YeeC. M. (2014). Committee report: publication guidelines and recommendations for studies using electroencephalography and magnetoencephalographyPsychophysiology 51121.

    • Search Google Scholar
    • Export Citation
  • KeshavanM. S.NasrallahH. A. and TandonR. (2011). Schizophrenia, “Just the Facts” 6. Moving ahead with the schizophrenia concept: from the elephant to the mouseSchizophrenia Res. 127313.

    • Search Google Scholar
    • Export Citation
  • KokA. (2001). On the utility of P3 amplitude as a measure of processing capacityPsychophysiology 38557577.

  • LenthR. (2018). emmeans: Estimated Marginal Means, aka Least-Squares Means. R package version 1.3.0. https://CRAN.R-project.org/package=emmeans. Retrieved 15 September 2018.

  • LindenD. E. J.PrvulovicD.FormisanoE.VöllingerM.ZanellaF. E.GoebelR. and DierksT. (1999). The functional neuroanatomy of target detection: an fMRI study of visual and auditory oddball tasksCereb. Cortex 9815823.

    • Search Google Scholar
    • Export Citation
  • LuckS. J. (2004). Ten simple rules for designing and interpreting ERP experiments, in: Event-Related Potentials: a Methods HandbookT. C. Handy (Ed.), pp. 1732. MIT PressCambridge, MA, USA.

    • Search Google Scholar
    • Export Citation
  • LuckS. J. and GaspelinN. (2017). How to get statistically significant effects in any ERP experiment (and why you shouldn’t)Psychophysiology 54146157.

    • Search Google Scholar
    • Export Citation
  • MacalusoE. (2006). Multisensory processing in sensory-specific cortical areasNeuroscientist 12327338.

  • MayP. J. C. and TiitinenH. (2010). Mismatch negativity (MMN), the deviance-elicited auditory deflection, explainedPsychophysiology 4766122.

    • Search Google Scholar
    • Export Citation
  • MillnerJ. (1982). Divided attention: evidence for coactivation with redundand signalsCogn. Psychol. 14247279.

  • NäätänenR.PaavilainenP.RinneT. and AlhoK. (2007). The mismatch negativity (MMN) in basic research of central auditory processing: a reviewClin. Neurophysiol. 11825442590.

    • Search Google Scholar
    • Export Citation
  • OrayS.LuZ.-L. and DawsonM. E. (2002). Modification of sudden onset auditory ERP by involuntary attention to visual stimuliInt. J. Psychophysiol. 43213224.

    • Search Google Scholar
    • Export Citation
  • Pazo-AlvarezP.CadaveiraF. and AmenedoE. (2003). MMN in the visual modality: a reviewBiol. Psychol. 63199236.

  • PennartzC. M. A. (2009). Identification and integration of sensory modalities: neural basis and relation to consciousnessConscious. Cogn. 18718739.

    • Search Google Scholar
    • Export Citation
  • PictonT. W. (1992). The P300 wave of the human event-related potentialJ. Clin. Neurophysiol. 9456479.

  • PictonT. W.AlainC.OttenL.RitterW. and AchimA. (2000). Mismatch negativity: different water in the same riverAudiol. Neurotol. 5111139.

    • Search Google Scholar
    • Export Citation
  • PolichJ. (2003). Theoretical overview of P3a and P3b, in: Detection of Change: Event-Related Potential and fMRI FindingsJ. Polich (Ed.), pp. 8398. Kluwer Academic PressBoston, MA USA.

    • Search Google Scholar
    • Export Citation
  • PolichJ. (2004). Neuropsychology of P3a and P3b: a theoretical overview, in: Brainwaves and Mind: Recent DevelopmentsN. C. Moore and K. Arikan (Eds), pp. 1529. KjellbergWheaton, IL, USA.

    • Search Google Scholar
    • Export Citation
  • PolichJ. (2007). Updating P300: an integrative theory of P3a and P3bClin Neurophysiol 11821282148.

  • PolichJ. and KokA. (1995). Cognitive and biological determinants of P300: an integrative reviewBiol. Psychol. 41103146.

  • PomperU.KeilJ.FoxeJ. J. and SenkowskiD. (2015). Intersensory selective attention and temporal orienting operate in parallel and are instantiated in spatially distinct sensory and motor cortices: human intersensory and temporal attentionHum. Brain Mapp. 3632463259.

    • Search Google Scholar
    • Export Citation
  • R Core Team (2018). R: a Language and Environment for Statistical Computing. R Foundation for Statistical ComputingVienna, Austria.

  • ReynoldsJ. H. and ChelazziL. (2004). Attentional modulation of visual processingAnnu. Rev. Neurosci. 27611647.

  • RogersR. L.PapanicolaouA. C.BaumannS. B. and EisenbergH. M. (1992). Late magnetic fields and positive evoked potentials following infrequent and unpredictable omissions of visual stimuliElectroencephalogr. Clin. Neurophysiol. 83146152.

    • Search Google Scholar
    • Export Citation
  • RosenfeldJ. P.BiroschakJ. R.KleschenM. J. and SmithK. M. (2005). Subjective and objective probability effects on P300 amplitude revisitedPsychophysiology 42356359.

    • Search Google Scholar
    • Export Citation
  • RoyM.DilloW.EmrichH. M. and OhlmeierM. D. (2009). Aspergers syndrome in adulthoodDtsch. Arztebl. Int. 1065964.

  • RuchkinD. S. and SuttonS. (1973). Visual evoked and emitted potentials and stimulus significanceBull. Psychon. Soc. 2144146.

    • Search Google Scholar
    • Export Citation
  • RüsselerJ.AltenmüllerE.NagerW.KohlmetzC. and MünteT. F. (2001). Event-related brain potentials to sound omissions differ in musicians and non-musiciansNeurosci. Lett. 3083336.

    • Search Google Scholar
    • Export Citation
  • SaevarssonS.KristjánssonA.BachM. and HeinrichS. P. (2012). P300 in neglectClin. Neurophysiol. 123496506.

  • SchröderE.KajoschH.VerbanckP.KornreichC. and CampanellaS. (2016). Methodological considerations about the use of bimodal oddball P300 in psychiatry: topography and reference effectFront. Psychol. 71387. DOI:10.3389/fpsyg.2016.01387.

    • Search Google Scholar
    • Export Citation
  • SerencesJ. T. and YantisS. (2006). Selective visual attention and perceptual coherenceTrends Cogn. Sci. 103845.

  • ShimojoS. and ShamsL. (2001). Sensory modalities are not separate modalities: plasticity and interactionsCurr. Opin. Neurobiol. 11505509.

    • Search Google Scholar
    • Export Citation
  • SimonJ. R. and OvermeyerS. P. (1984). The effect of redundant cues on retrieval timeHum. Fact. 26315321.

  • SimonsR. F.GrahamF. K.MilesM. A. and ChenX. (2001). On the relationship of P3a and the novelty-P3Biol. Psychol. 56207218.

    • Search Google Scholar
    • Export Citation
  • SoltaniM. and KnightR. T. (2000). Neural origins of the P300Crit. Rev. Neurobiol. 14199224.

  • StefanicsG.StavrinouM.SestieriC.CiancettaL.BelardinelliP.CianfloneF.BernáthL.HernádiI.PizzellaV. and RomaniG. L. (2005). Cross-modal visual–auditory–somatosensory integration in a multimodal object recognition task in humansInt. Congr. Ser. 1278163166.

    • Search Google Scholar
    • Export Citation
  • SussmanE. S. (2007). A new view on the MMN and attention debateJ. Psychophysiol. 21164175.

  • SuttonS.BrarenM.ZubinJ. and JohnE. R. (1965). Evoked-potential correlates of stimulus uncertaintyScience 15011871188.

    • Search Google Scholar
    • Export Citation
  • TalsmaD. (2015). Predictive coding and multisensory integration: an attentional account of the multisensory mindFront. Integr. Neurosci. 919. DOI:10.3389/fnint.2015.00019.

    • Search Google Scholar
    • Export Citation
  • TalsmaD.DotyT. J. and WoldorffM. G. (2007). Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration?Cereb. Cortex 17679690.

    • Search Google Scholar
    • Export Citation
  • TalsmaD.SenkowskiD.Soto-FaracoS. and WoldorffM. G. (2010). The multifaceted interplay between attention and multisensory integrationTrends Cogn. Sci. 14400410.

    • Search Google Scholar
    • Export Citation
  • Teder-SälejärviW. A.McDonaldJ. J.Di RussoF. and HillyardS. A. (2002). An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordingsCogn. Brain Res. 14106114.

    • Search Google Scholar
    • Export Citation
  • WangW. Y.HuL.ValentiniE.XieX. B.CuiH. Y. and HuY. (2012). Dynamic characteristics of multisensory facilitation and inhibitionCogn. Neurodyn. 6409419.

    • Search Google Scholar
    • Export Citation
  • WickensC.KramerA.VanasseL. and DonchinE. (1983). Performance of concurrent tasks: a psychophysiological analysis of the reciprocity of information-processing resourcesScience 22110801082.

    • Search Google Scholar
    • Export Citation
  • WoodmanG. F. (2010). A brief introduction to the use of event-related potentials in studies of perception and attentionAtten. Percept. Psychophys. 7220312046.

    • Search Google Scholar
    • Export Citation
  • YabeH.TervaniemiM.ReinikainenK. and NäätänenR. (1997). Temporal window of integration revealed by MMN to sound omissionNeuroReport 819711974.

    • Search Google Scholar
    • Export Citation
  • ZmigrodS. and HommelB. (2013). Feature integration across multimodal perception and action: a reviewMultisens. Res. 26143157.

    • Search Google Scholar
    • Export Citation

Content Metrics

All Time Past Year Past 30 Days
Abstract Views 118 118 69
Full Text Views 8 8 4
PDF Downloads 8 8 5