Attentional Interactions Between Vision and Hearing in Event-Related Responses to Crossmodal and Conjunct Oddballs

In: Multisensory Research

Abstract

Are alternation and co-occurrence of stimuli of different sensory modalities conspicuous? In a novel audio-visual oddball paradigm, the P300 was used as an index of the allocation of attention to investigate stimulus- and task-related interactions between modalities. Specifically, we assessed effects of modality alternation and the salience of conjunct oddball stimuli that were defined by the co-occurrence of both modalities. We presented (a) crossmodal audio-visual oddball sequences, where both oddballs and standards were unimodal, but of a different modality (i.e., visual oddball with auditory standard, or vice versa), and (b) oddball sequences where standards were randomly of either modality while the oddballs were a combination of both modalities (conjunct stimuli). Subjects were instructed to attend to one of the modalities (whether part of a conjunct stimulus or not). In addition, we also tested specific attention to the conjunct stimuli. P300-like responses occurred even when the oddball was of the unattended modality. The pattern of event-related potential (ERP) responses obtained with the two crossmodal oddball sequences switched symmetrically between stimulus modalities when the task modality was switched. Conjunct oddballs elicited no oddball response if only one modality was attended. However, when conjunctness was specifically attended, an oddball response was obtained. Crossmodal oddballs capture sufficient attention even when not attended. Conjunct oddballs, however, are not sufficiently salient to attract attention when the task is unimodal. Even when specifically attended, the processing of conjunctness appears to involve additional steps that delay the oddball response.

  • Achim, A. (1995). Signal detection in averaged evoked potentials: Monte Carlo comparison of the sensitivity of different methods, Electroencephalogr. Clin. Neurophysiol. 96, 574–584.

    • Search Google Scholar
    • Export Citation
  • Alho, K. (1995). Cerebral generators of mismatch negativity (MMN) and its magnetic counterpart (MMNm) elicited by sound changes, Ear Hear. 16, 38–51.

    • Search Google Scholar
    • Export Citation
  • American Clinical Neurophysiology Society (2006). Guideline 5: guidelines for standard electrode position nomenclature, J. Clin. Neurophysiol. 23, 107–110.

    • Search Google Scholar
    • Export Citation
  • Bates, D., Mächler, M., Bolker, B. and Walker, S. (2015). Fitting linear mixed-effects models using lme4, J. Stat. Softw. 67, 1–48.

  • Becker, D. E. and Shapiro, D. (1980). Directing attention toward stimuli affects the P300 but not the orienting response, Psychophysiology 17, 385–389.

    • Search Google Scholar
    • Export Citation
  • Bendixen, A., Grimm, S., Deouell, L. Y., Wetzel, N., Mädebach, A. and Schröger, E. (2010). The time-course of auditory and visual distraction effects in a new crossmodal paradigm, Neuropsychologia 48, 2130–2139.

    • Search Google Scholar
    • Export Citation
  • Brennan, A. R. and Arnsten, A. F. (2008). Neuronal mechanisms underlying attention deficit hyperactivity disorder, Ann. N. Y. Acad. Sci. 1129, 236–245.

    • Search Google Scholar
    • Export Citation
  • Brown, C. R., Clarke, A. R. and Barry, R. J. (2006). Inter-modal attention: ERPs to auditory targets in an inter-modal oddball task, Int. J. Psychophysiol. 62, 77–86.

    • Search Google Scholar
    • Export Citation
  • Brown, C. R., Clarke, A. R. and Barry, R. J. (2007). Auditory processing in an inter-modal oddball task: effects of a combined auditory/visual standard on auditory target ERPs, Int. J. Psychophysiol. 65, 122–131.

    • Search Google Scholar
    • Export Citation
  • Cappe, C., Thut, G., Romei, V. and Murray, M. M. (2010). Auditory–visual multisensory interactions in humans: timing, topography, directionality, and sources, J. Neurosci. 30, 12572–12580.

    • Search Google Scholar
    • Export Citation
  • Cornu, L. and Bianchi, L. (1981). Visually emitted potential and attentional processes, Boll. Soc. Ital. Biol. Sper. 57, 2067–2073.

  • Dehaene, S. and Changeux, J.-P. (2011). Experimental and theoretical approaches to conscious processing, Neuron 70, 200–227.

  • Duncan, C. C., Barry, R. J., Connolly, J. F., Fischer, C., Michie, P. T., Näätänen, R., Polich, J., Reinvang, I. and Van Petten, C. (2009). Event-related potentials in clinical research: guidelines for eliciting, recording, and quantifying mismatch negativity, P300, and N400, Clin. Neurophysiol. 120, 1883–1908.

    • Search Google Scholar
    • Export Citation
  • Duncan-Johnson, C. C. and Donchin, E. (1977). On quantifying surprise: the variation of event-related potentials with subjective probability, Psychophysiology 14, 456–467.

    • Search Google Scholar
    • Export Citation
  • Feng, T., Qiu, Y., Zhu, Y. and Tong, S. (2008). Attention rivalry under irrelevant audiovisual stimulation, Neurosci. Lett. 438, 6–9.

  • Fort, A., Delpuech, C., Pernier, J. and Giard, M.-H. (2002a). Dynamics of cortico-subcortical cross-modal operations involved in audio-visual object detection in humans, Cereb. Cortex 12, 1031–1039.

    • Search Google Scholar
    • Export Citation
  • Fort, A., Delpuech, C., Pernier, J. and Giard, M. H. (2002b). Early auditory–visual interactions in human cortex during nonredundant target identification, Brain Res. Cogn. Brain Res. 14, 20–30.

    • Search Google Scholar
    • Export Citation
  • Friedman, D., Cycowicz, Y. M. and Gaeta, H. (2001). The novelty P3: an event-related brain potential (ERP) sign of the brains evaluation of novelty, Neurosc. Biobehav. Rev. 25, 355–373.

    • Search Google Scholar
    • Export Citation
  • Garrido, M. I., Kilner, J. M., Stephan, K. E. and Friston, K. J. (2009). The mismatch negativity: a review of underlying mechanisms, Clin. Neurophysiol. 120, 453–463.

    • Search Google Scholar
    • Export Citation
  • Gonsalvez, C. J., Barry, R. J., Rushby, J. A. and Polich, J. (2007). Target-to-target interval, intensity, and P300 from an auditory single-stimulus task, Psychophysiology 44, 245–250.

    • Search Google Scholar
    • Export Citation
  • Gonsalvez, C. L. and Polich, J. (2002). P300 amplitude is determined by target-to-target interval, Psychophysiology 39, 388–396.

  • Good, P. I. (2005). Resampling Methods: a Practical Guide to Data Analysis, 3rd edn.. Birkhäuser, Boston, MA, USA.

  • Gray, H. M., Ambady, N., Lowenthal, W. T. and Deldin, P. (2004). P300 as an index of attention to self-relevant stimuli, J. Exp. Soc. Psychol. 40, 216–224.

    • Search Google Scholar
    • Export Citation
  • Heinrich, S. P. and Bach, M. (2008). Signal and noise in P300 recordings to visual stimuli, Doc. Ophthalmol. 117, 73–83.

  • Heinrich, S. P., Aertsen, A. and Bach, M. (2008). Oblique effects beyond low-level visual processing, Vision Res. 48, 809–818.

  • Heinrich, S. P., Mell, D. and Bach, M. (2009). Frequency-domain analysis of fast oddball responses to visual stimuli: a feasibility study, Int. J. Psychophysiol. 73, 287–293.

    • Search Google Scholar
    • Export Citation
  • Heinze, H. J., Luck, S. J., Mangun, G. R. and Hillyard, S. A. (1990). Visual event-related potentials index focused attention within bilateral stimulus arrays. I. Evidence for early selection, Electroencephalogr. Clin. Neurophysiol. 75, 511–527.

    • Search Google Scholar
    • Export Citation
  • Holm, S. (1979). A simple sequentially rejective multiple test procedure, Scand. J. Stat. 6, 65–70.

  • Ji, J., Porjesz, B., Begleiter, H. and Chorlian, D. (1999). P300: the similarities and differences in the scalp distribution of visual and auditory modality, Brain Topogr. 11, 315–327.

    • Search Google Scholar
    • Export Citation
  • Johnson Jr, R. (1989). Developmental evidence for modality-dependent P300 generators: a normative study, Psychophysiology 26, 651–667.

    • Search Google Scholar
    • Export Citation
  • Kastner, S. and Ungerleider, L. G. (2000). Mechanisms of visual attention in the human cortex, Annu. Rev. Neurosci. 23, 315–341.

  • Katayama, J. and Polich, J. (1996). P300, probability, and the three-tone paradigm, Electroencephalogr. Clin. Neurophysiol. 100, 555–562.

    • Search Google Scholar
    • Export Citation
  • Kayser, C. and Logothetis, N. K. (2007). Do early sensory cortices integrate cross-modal information?, Brain Struct. Funct. 212, 121–132.

    • Search Google Scholar
    • Export Citation
  • Keil, A., Bradley, M. M., Junghöfer, M., Russmann, T., Lowenthal, W. and Lang, P. J. (2007). Cross-modal attention capture by affective stimuli: evidence from event-related potentials, Cogn. Affect. Behav. Neurosci. 7, 18–24.

    • Search Google Scholar
    • Export Citation
  • Keil, A., Debener, S., Gratton, G., Junghöfer, M., Kappenman, E. S., Luck, S. J., Luu, P., Miller, G. A. and Yee, C. M. (2014). Committee report: publication guidelines and recommendations for studies using electroencephalography and magnetoencephalography, Psychophysiology 51, 1–21.

    • Search Google Scholar
    • Export Citation
  • Keshavan, M. S., Nasrallah, H. A. and Tandon, R. (2011). Schizophrenia, “Just the Facts” 6. Moving ahead with the schizophrenia concept: from the elephant to the mouse, Schizophrenia Res. 127, 3–13.

    • Search Google Scholar
    • Export Citation
  • Kok, A. (2001). On the utility of P3 amplitude as a measure of processing capacity, Psychophysiology 38, 557–577.

  • Lenth, R. (2018). emmeans: Estimated Marginal Means, aka Least-Squares Means. R package version 1.3.0. https://CRAN.R-project.org/package=emmeans. Retrieved 15 September 2018.

  • Linden, D. E. J., Prvulovic, D., Formisano, E., Völlinger, M., Zanella, F. E., Goebel, R. and Dierks, T. (1999). The functional neuroanatomy of target detection: an fMRI study of visual and auditory oddball tasks, Cereb. Cortex 9, 815–823.

    • Search Google Scholar
    • Export Citation
  • Luck, S. J. (2004). Ten simple rules for designing and interpreting ERP experiments, in: Event-Related Potentials: a Methods Handbook, T. C. Handy (Ed.), pp. 17–32. MIT Press, Cambridge, MA, USA.

    • Search Google Scholar
    • Export Citation
  • Luck, S. J. and Gaspelin, N. (2017). How to get statistically significant effects in any ERP experiment (and why you shouldn’t), Psychophysiology 54, 146–157.

    • Search Google Scholar
    • Export Citation
  • Macaluso, E. (2006). Multisensory processing in sensory-specific cortical areas, Neuroscientist 12, 327–338.

  • May, P. J. C. and Tiitinen, H. (2010). Mismatch negativity (MMN), the deviance-elicited auditory deflection, explained, Psychophysiology 47, 66–122.

    • Search Google Scholar
    • Export Citation
  • Millner, J. (1982). Divided attention: evidence for coactivation with redundand signals, Cogn. Psychol. 14, 247–279.

  • Näätänen, R., Paavilainen, P., Rinne, T. and Alho, K. (2007). The mismatch negativity (MMN) in basic research of central auditory processing: a review, Clin. Neurophysiol. 118, 2544–2590.

    • Search Google Scholar
    • Export Citation
  • Oray, S., Lu, Z.-L. and Dawson, M. E. (2002). Modification of sudden onset auditory ERP by involuntary attention to visual stimuli, Int. J. Psychophysiol. 43, 213–224.

    • Search Google Scholar
    • Export Citation
  • Pazo-Alvarez, P., Cadaveira, F. and Amenedo, E. (2003). MMN in the visual modality: a review, Biol. Psychol. 63, 199–236.

  • Pennartz, C. M. A. (2009). Identification and integration of sensory modalities: neural basis and relation to consciousness, Conscious. Cogn. 18, 718–739.

    • Search Google Scholar
    • Export Citation
  • Picton, T. W. (1992). The P300 wave of the human event-related potential, J. Clin. Neurophysiol. 9, 456–479.

  • Picton, T. W., Alain, C., Otten, L., Ritter, W. and Achim, A. (2000). Mismatch negativity: different water in the same river, Audiol. Neurotol. 5, 111–139.

    • Search Google Scholar
    • Export Citation
  • Polich, J. (2003). Theoretical overview of P3a and P3b, in: Detection of Change: Event-Related Potential and fMRI Findings, J. Polich (Ed.), pp. 83–98. Kluwer Academic Press, Boston, MA USA.

    • Search Google Scholar
    • Export Citation
  • Polich, J. (2004). Neuropsychology of P3a and P3b: a theoretical overview, in: Brainwaves and Mind: Recent Developments, N. C. Moore and K. Arikan (Eds), pp. 15–29. Kjellberg, Wheaton, IL, USA.

    • Search Google Scholar
    • Export Citation
  • Polich, J. (2007). Updating P300: an integrative theory of P3a and P3b, Clin Neurophysiol 118, 2128–2148.

  • Polich, J. and Kok, A. (1995). Cognitive and biological determinants of P300: an integrative review, Biol. Psychol. 41, 103–146.

  • Pomper, U., Keil, J., Foxe, J. J. and Senkowski, D. (2015). Intersensory selective attention and temporal orienting operate in parallel and are instantiated in spatially distinct sensory and motor cortices: human intersensory and temporal attention, Hum. Brain Mapp. 36, 3246–3259.

    • Search Google Scholar
    • Export Citation
  • R Core Team (2018). R: a Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria.

  • Reynolds, J. H. and Chelazzi, L. (2004). Attentional modulation of visual processing, Annu. Rev. Neurosci. 27, 611–647.

  • Rogers, R. L., Papanicolaou, A. C., Baumann, S. B. and Eisenberg, H. M. (1992). Late magnetic fields and positive evoked potentials following infrequent and unpredictable omissions of visual stimuli, Electroencephalogr. Clin. Neurophysiol. 83, 146–152.

    • Search Google Scholar
    • Export Citation
  • Rosenfeld, J. P., Biroschak, J. R., Kleschen, M. J. and Smith, K. M. (2005). Subjective and objective probability effects on P300 amplitude revisited, Psychophysiology 42, 356–359.

    • Search Google Scholar
    • Export Citation
  • Roy, M., Dillo, W., Emrich, H. M. and Ohlmeier, M. D. (2009). Aspergers syndrome in adulthood, Dtsch. Arztebl. Int. 106, 59–64.

  • Ruchkin, D. S. and Sutton, S. (1973). Visual evoked and emitted potentials and stimulus significance, Bull. Psychon. Soc. 2, 144–146.

    • Search Google Scholar
    • Export Citation
  • Rüsseler, J., Altenmüller, E., Nager, W., Kohlmetz, C. and Münte, T. F. (2001). Event-related brain potentials to sound omissions differ in musicians and non-musicians, Neurosci. Lett. 308, 33–36.

    • Search Google Scholar
    • Export Citation
  • Saevarsson, S., Kristjánsson, A., Bach, M. and Heinrich, S. P. (2012). P300 in neglect, Clin. Neurophysiol. 123, 496–506.

  • Schröder, E., Kajosch, H., Verbanck, P., Kornreich, C. and Campanella, S. (2016). Methodological considerations about the use of bimodal oddball P300 in psychiatry: topography and reference effect, Front. Psychol. 7, 1387. DOI:10.3389/fpsyg.2016.01387.

    • Search Google Scholar
    • Export Citation
  • Serences, J. T. and Yantis, S. (2006). Selective visual attention and perceptual coherence, Trends Cogn. Sci. 10, 38–45.

  • Shimojo, S. and Shams, L. (2001). Sensory modalities are not separate modalities: plasticity and interactions, Curr. Opin. Neurobiol. 11, 505–509.

    • Search Google Scholar
    • Export Citation
  • Simon, J. R. and Overmeyer, S. P. (1984). The effect of redundant cues on retrieval time, Hum. Fact. 26, 315–321.

  • Simons, R. F., Graham, F. K., Miles, M. A. and Chen, X. (2001). On the relationship of P3a and the novelty-P3, Biol. Psychol. 56, 207–218.

    • Search Google Scholar
    • Export Citation
  • Soltani, M. and Knight, R. T. (2000). Neural origins of the P300, Crit. Rev. Neurobiol. 14, 199–224.

  • Stefanics, G., Stavrinou, M., Sestieri, C., Ciancetta, L., Belardinelli, P., Cianflone, F., Bernáth, L., Hernádi, I., Pizzella, V. and Romani, G. L. (2005). Cross-modal visual–auditory–somatosensory integration in a multimodal object recognition task in humans, Int. Congr. Ser. 1278, 163–166.

    • Search Google Scholar
    • Export Citation
  • Sussman, E. S. (2007). A new view on the MMN and attention debate, J. Psychophysiol. 21, 164–175.

  • Sutton, S., Braren, M., Zubin, J. and John, E. R. (1965). Evoked-potential correlates of stimulus uncertainty, Science 150, 1187–1188.

    • Search Google Scholar
    • Export Citation
  • Talsma, D. (2015). Predictive coding and multisensory integration: an attentional account of the multisensory mind, Front. Integr. Neurosci. 9, 19. DOI:10.3389/fnint.2015.00019.

    • Search Google Scholar
    • Export Citation
  • Talsma, D., Doty, T. J. and Woldorff, M. G. (2007). Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration?, Cereb. Cortex 17, 679–690.

    • Search Google Scholar
    • Export Citation
  • Talsma, D., Senkowski, D., Soto-Faraco, S. and Woldorff, M. G. (2010). The multifaceted interplay between attention and multisensory integration, Trends Cogn. Sci. 14, 400–410.

    • Search Google Scholar
    • Export Citation
  • Teder-Sälejärvi, W. A., McDonald, J. J., Di Russo, F. and Hillyard, S. A. (2002). An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings, Cogn. Brain Res. 14, 106–114.

    • Search Google Scholar
    • Export Citation
  • Wang, W. Y., Hu, L., Valentini, E., Xie, X. B., Cui, H. Y. and Hu, Y. (2012). Dynamic characteristics of multisensory facilitation and inhibition, Cogn. Neurodyn. 6, 409–419.

    • Search Google Scholar
    • Export Citation
  • Wickens, C., Kramer, A., Vanasse, L. and Donchin, E. (1983). Performance of concurrent tasks: a psychophysiological analysis of the reciprocity of information-processing resources, Science 221, 1080–1082.

    • Search Google Scholar
    • Export Citation
  • Woodman, G. F. (2010). A brief introduction to the use of event-related potentials in studies of perception and attention, Atten. Percept. Psychophys. 72, 2031–2046.

    • Search Google Scholar
    • Export Citation
  • Yabe, H., Tervaniemi, M., Reinikainen, K. and Näätänen, R. (1997). Temporal window of integration revealed by MMN to sound omission, NeuroReport 8, 1971–1974.

    • Search Google Scholar
    • Export Citation
  • Zmigrod, S. and Hommel, B. (2013). Feature integration across multimodal perception and action: a review, Multisens. Res. 26, 143–157.

    • Search Google Scholar
    • Export Citation

Content Metrics

All Time Past Year Past 30 Days
Abstract Views 137 137 137
Full Text Views 8 8 8
PDF Downloads 5 5 5