Attentional Interactions Between Vision and Hearing in Event-Related Responses to Crossmodal and Conjunct Oddballs
In: Multisensory ResearchFaculty of Medicine, University of Freiburg, Germany
Search for other papers by Evelyn B. N. Friedel in
Current site
Google Scholar
Faculty of Medicine, University of Freiburg, Germany
Search for other papers by Michael Bach in
Current site
Google Scholar
Faculty of Medicine, University of Freiburg, Germany
Search for other papers by Sven P. Heinrich in
Current site
Google Scholar
Abstract
Are alternation and co-occurrence of stimuli of different sensory modalities conspicuous? In a novel audio-visual oddball paradigm, the P300 was used as an index of the allocation of attention to investigate stimulus- and task-related interactions between modalities. Specifically, we assessed effects of modality alternation and the salience of conjunct oddball stimuli that were defined by the co-occurrence of both modalities. We presented (a) crossmodal audio-visual oddball sequences, where both oddballs and standards were unimodal, but of a different modality (i.e., visual oddball with auditory standard, or vice versa), and (b) oddball sequences where standards were randomly of either modality while the oddballs were a combination of both modalities (conjunct stimuli). Subjects were instructed to attend to one of the modalities (whether part of a conjunct stimulus or not). In addition, we also tested specific attention to the conjunct stimuli. P300-like responses occurred even when the oddball was of the unattended modality. The pattern of event-related potential (ERP) responses obtained with the two crossmodal oddball sequences switched symmetrically between stimulus modalities when the task modality was switched. Conjunct oddballs elicited no oddball response if only one modality was attended. However, when conjunctness was specifically attended, an oddball response was obtained. Crossmodal oddballs capture sufficient attention even when not attended. Conjunct oddballs, however, are not sufficiently salient to attract attention when the task is unimodal. Even when specifically attended, the processing of conjunctness appears to involve additional steps that delay the oddball response.
Purchase
Buy instant access (PDF download and unlimited online access):
Institutional Login
Log in with Open Athens, Shibboleth, or your institutional credentials
Personal login
Log in with your brill.com account
-
Achim, A. (1995). Signal detection in averaged evoked potentials: Monte Carlo comparison of the sensitivity of different methods, Electroencephalogr. Clin. Neurophysiol. 96, 574–584.
-
Alho, K. (1995). Cerebral generators of mismatch negativity (MMN) and its magnetic counterpart (MMNm) elicited by sound changes, Ear Hear. 16, 38–51.
-
American Clinical Neurophysiology Society (2006). Guideline 5: guidelines for standard electrode position nomenclature, J. Clin. Neurophysiol. 23, 107–110.
-
Bates, D., Mächler, M., Bolker, B. and Walker, S. (2015). Fitting linear mixed-effects models using lme4, J. Stat. Softw. 67, 1–48.
-
Becker, D. E. and Shapiro, D. (1980). Directing attention toward stimuli affects the P300 but not the orienting response, Psychophysiology 17, 385–389.
-
Bendixen, A., Grimm, S., Deouell, L. Y., Wetzel, N., Mädebach, A. and Schröger, E. (2010). The time-course of auditory and visual distraction effects in a new crossmodal paradigm, Neuropsychologia 48, 2130–2139.
-
Brennan, A. R. and Arnsten, A. F. (2008). Neuronal mechanisms underlying attention deficit hyperactivity disorder, Ann. N. Y. Acad. Sci. 1129, 236–245.
-
Brown, C. R., Clarke, A. R. and Barry, R. J. (2006). Inter-modal attention: ERPs to auditory targets in an inter-modal oddball task, Int. J. Psychophysiol. 62, 77–86.
-
Brown, C. R., Clarke, A. R. and Barry, R. J. (2007). Auditory processing in an inter-modal oddball task: effects of a combined auditory/visual standard on auditory target ERPs, Int. J. Psychophysiol. 65, 122–131.
-
Cappe, C., Thut, G., Romei, V. and Murray, M. M. (2010). Auditory–visual multisensory interactions in humans: timing, topography, directionality, and sources, J. Neurosci. 30, 12572–12580.
-
Cornu, L. and Bianchi, L. (1981). Visually emitted potential and attentional processes, Boll. Soc. Ital. Biol. Sper. 57, 2067–2073.
-
Dehaene, S. and Changeux, J.-P. (2011). Experimental and theoretical approaches to conscious processing, Neuron 70, 200–227.
-
Duncan, C. C., Barry, R. J., Connolly, J. F., Fischer, C., Michie, P. T., Näätänen, R., Polich, J., Reinvang, I. and Van Petten, C. (2009). Event-related potentials in clinical research: guidelines for eliciting, recording, and quantifying mismatch negativity, P300, and N400, Clin. Neurophysiol. 120, 1883–1908.
-
Duncan-Johnson, C. C. and Donchin, E. (1977). On quantifying surprise: the variation of event-related potentials with subjective probability, Psychophysiology 14, 456–467.
-
Feng, T., Qiu, Y., Zhu, Y. and Tong, S. (2008). Attention rivalry under irrelevant audiovisual stimulation, Neurosci. Lett. 438, 6–9.
-
Fort, A., Delpuech, C., Pernier, J. and Giard, M.-H. (2002a). Dynamics of cortico-subcortical cross-modal operations involved in audio-visual object detection in humans, Cereb. Cortex 12, 1031–1039.
-
Fort, A., Delpuech, C., Pernier, J. and Giard, M. H. (2002b). Early auditory–visual interactions in human cortex during nonredundant target identification, Brain Res. Cogn. Brain Res. 14, 20–30.
-
Friedman, D., Cycowicz, Y. M. and Gaeta, H. (2001). The novelty P3: an event-related brain potential (ERP) sign of the brains evaluation of novelty, Neurosc. Biobehav. Rev. 25, 355–373.
-
Garrido, M. I., Kilner, J. M., Stephan, K. E. and Friston, K. J. (2009). The mismatch negativity: a review of underlying mechanisms, Clin. Neurophysiol. 120, 453–463.
-
Gonsalvez, C. J., Barry, R. J., Rushby, J. A. and Polich, J. (2007). Target-to-target interval, intensity, and P300 from an auditory single-stimulus task, Psychophysiology 44, 245–250.
-
Gonsalvez, C. L. and Polich, J. (2002). P300 amplitude is determined by target-to-target interval, Psychophysiology 39, 388–396.
-
Good, P. I. (2005). Resampling Methods: a Practical Guide to Data Analysis, 3rd edn.. Birkhäuser, Boston, MA, USA.
-
Gray, H. M., Ambady, N., Lowenthal, W. T. and Deldin, P. (2004). P300 as an index of attention to self-relevant stimuli, J. Exp. Soc. Psychol. 40, 216–224.
-
Heinrich, S. P. and Bach, M. (2008). Signal and noise in P300 recordings to visual stimuli, Doc. Ophthalmol. 117, 73–83.
-
Heinrich, S. P., Aertsen, A. and Bach, M. (2008). Oblique effects beyond low-level visual processing, Vision Res. 48, 809–818.
-
Heinrich, S. P., Mell, D. and Bach, M. (2009). Frequency-domain analysis of fast oddball responses to visual stimuli: a feasibility study, Int. J. Psychophysiol. 73, 287–293.
-
Heinze, H. J., Luck, S. J., Mangun, G. R. and Hillyard, S. A. (1990). Visual event-related potentials index focused attention within bilateral stimulus arrays. I. Evidence for early selection, Electroencephalogr. Clin. Neurophysiol. 75, 511–527.
-
Holm, S. (1979). A simple sequentially rejective multiple test procedure, Scand. J. Stat. 6, 65–70.
-
Ji, J., Porjesz, B., Begleiter, H. and Chorlian, D. (1999). P300: the similarities and differences in the scalp distribution of visual and auditory modality, Brain Topogr. 11, 315–327.
-
Johnson Jr, R. (1989). Developmental evidence for modality-dependent P300 generators: a normative study, Psychophysiology 26, 651–667.
-
Kastner, S. and Ungerleider, L. G. (2000). Mechanisms of visual attention in the human cortex, Annu. Rev. Neurosci. 23, 315–341.
-
Katayama, J. and Polich, J. (1996). P300, probability, and the three-tone paradigm, Electroencephalogr. Clin. Neurophysiol. 100, 555–562.
-
Kayser, C. and Logothetis, N. K. (2007). Do early sensory cortices integrate cross-modal information?, Brain Struct. Funct. 212, 121–132.
-
Keil, A., Bradley, M. M., Junghöfer, M., Russmann, T., Lowenthal, W. and Lang, P. J. (2007). Cross-modal attention capture by affective stimuli: evidence from event-related potentials, Cogn. Affect. Behav. Neurosci. 7, 18–24.
-
Keil, A., Debener, S., Gratton, G., Junghöfer, M., Kappenman, E. S., Luck, S. J., Luu, P., Miller, G. A. and Yee, C. M. (2014). Committee report: publication guidelines and recommendations for studies using electroencephalography and magnetoencephalography, Psychophysiology 51, 1–21.
-
Keshavan, M. S., Nasrallah, H. A. and Tandon, R. (2011). Schizophrenia, “Just the Facts” 6. Moving ahead with the schizophrenia concept: from the elephant to the mouse, Schizophrenia Res. 127, 3–13.
-
Kok, A. (2001). On the utility of P3 amplitude as a measure of processing capacity, Psychophysiology 38, 557–577.
-
Lenth, R. (2018). emmeans: Estimated Marginal Means, aka Least-Squares Means. R package version 1.3.0. https://CRAN.R-project.org/package=emmeans. Retrieved 15 September 2018.
-
Linden, D. E. J., Prvulovic, D., Formisano, E., Völlinger, M., Zanella, F. E., Goebel, R. and Dierks, T. (1999). The functional neuroanatomy of target detection: an fMRI study of visual and auditory oddball tasks, Cereb. Cortex 9, 815–823.
-
Luck, S. J. (2004). Ten simple rules for designing and interpreting ERP experiments, in: Event-Related Potentials: a Methods Handbook, T. C. Handy (Ed.), pp. 17–32. MIT Press, Cambridge, MA, USA.
-
Luck, S. J. and Gaspelin, N. (2017). How to get statistically significant effects in any ERP experiment (and why you shouldn’t), Psychophysiology 54, 146–157.
-
Macaluso, E. (2006). Multisensory processing in sensory-specific cortical areas, Neuroscientist 12, 327–338.
-
May, P. J. C. and Tiitinen, H. (2010). Mismatch negativity (MMN), the deviance-elicited auditory deflection, explained, Psychophysiology 47, 66–122.
-
Millner, J. (1982). Divided attention: evidence for coactivation with redundand signals, Cogn. Psychol. 14, 247–279.
-
Näätänen, R., Paavilainen, P., Rinne, T. and Alho, K. (2007). The mismatch negativity (MMN) in basic research of central auditory processing: a review, Clin. Neurophysiol. 118, 2544–2590.
-
Oray, S., Lu, Z.-L. and Dawson, M. E. (2002). Modification of sudden onset auditory ERP by involuntary attention to visual stimuli, Int. J. Psychophysiol. 43, 213–224.
-
Pazo-Alvarez, P., Cadaveira, F. and Amenedo, E. (2003). MMN in the visual modality: a review, Biol. Psychol. 63, 199–236.
-
Pennartz, C. M. A. (2009). Identification and integration of sensory modalities: neural basis and relation to consciousness, Conscious. Cogn. 18, 718–739.
-
Picton, T. W. (1992). The P300 wave of the human event-related potential, J. Clin. Neurophysiol. 9, 456–479.
-
Picton, T. W., Alain, C., Otten, L., Ritter, W. and Achim, A. (2000). Mismatch negativity: different water in the same river, Audiol. Neurotol. 5, 111–139.
-
Polich, J. (2003). Theoretical overview of P3a and P3b, in: Detection of Change: Event-Related Potential and fMRI Findings, J. Polich (Ed.), pp. 83–98. Kluwer Academic Press, Boston, MA USA.
-
Polich, J. (2004). Neuropsychology of P3a and P3b: a theoretical overview, in: Brainwaves and Mind: Recent Developments, N. C. Moore and K. Arikan (Eds), pp. 15–29. Kjellberg, Wheaton, IL, USA.
-
Polich, J. (2007). Updating P300: an integrative theory of P3a and P3b, Clin Neurophysiol 118, 2128–2148.
-
Polich, J. and Kok, A. (1995). Cognitive and biological determinants of P300: an integrative review, Biol. Psychol. 41, 103–146.
-
Pomper, U., Keil, J., Foxe, J. J. and Senkowski, D. (2015). Intersensory selective attention and temporal orienting operate in parallel and are instantiated in spatially distinct sensory and motor cortices: human intersensory and temporal attention, Hum. Brain Mapp. 36, 3246–3259.
-
R Core Team (2018). R: a Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria.
-
Reynolds, J. H. and Chelazzi, L. (2004). Attentional modulation of visual processing, Annu. Rev. Neurosci. 27, 611–647.
-
Rogers, R. L., Papanicolaou, A. C., Baumann, S. B. and Eisenberg, H. M. (1992). Late magnetic fields and positive evoked potentials following infrequent and unpredictable omissions of visual stimuli, Electroencephalogr. Clin. Neurophysiol. 83, 146–152.
-
Rosenfeld, J. P., Biroschak, J. R., Kleschen, M. J. and Smith, K. M. (2005). Subjective and objective probability effects on P300 amplitude revisited, Psychophysiology 42, 356–359.
-
Roy, M., Dillo, W., Emrich, H. M. and Ohlmeier, M. D. (2009). Aspergers syndrome in adulthood, Dtsch. Arztebl. Int. 106, 59–64.
-
Ruchkin, D. S. and Sutton, S. (1973). Visual evoked and emitted potentials and stimulus significance, Bull. Psychon. Soc. 2, 144–146.
-
Rüsseler, J., Altenmüller, E., Nager, W., Kohlmetz, C. and Münte, T. F. (2001). Event-related brain potentials to sound omissions differ in musicians and non-musicians, Neurosci. Lett. 308, 33–36.
-
Saevarsson, S., Kristjánsson, A., Bach, M. and Heinrich, S. P. (2012). P300 in neglect, Clin. Neurophysiol. 123, 496–506.
-
Schröder, E., Kajosch, H., Verbanck, P., Kornreich, C. and Campanella, S. (2016). Methodological considerations about the use of bimodal oddball P300 in psychiatry: topography and reference effect, Front. Psychol. 7, 1387. DOI:10.3389/fpsyg.2016.01387.
-
Serences, J. T. and Yantis, S. (2006). Selective visual attention and perceptual coherence, Trends Cogn. Sci. 10, 38–45.
-
Shimojo, S. and Shams, L. (2001). Sensory modalities are not separate modalities: plasticity and interactions, Curr. Opin. Neurobiol. 11, 505–509.
-
Simon, J. R. and Overmeyer, S. P. (1984). The effect of redundant cues on retrieval time, Hum. Fact. 26, 315–321.
-
Simons, R. F., Graham, F. K., Miles, M. A. and Chen, X. (2001). On the relationship of P3a and the novelty-P3, Biol. Psychol. 56, 207–218.
-
Soltani, M. and Knight, R. T. (2000). Neural origins of the P300, Crit. Rev. Neurobiol. 14, 199–224.
-
Stefanics, G., Stavrinou, M., Sestieri, C., Ciancetta, L., Belardinelli, P., Cianflone, F., Bernáth, L., Hernádi, I., Pizzella, V. and Romani, G. L. (2005). Cross-modal visual–auditory–somatosensory integration in a multimodal object recognition task in humans, Int. Congr. Ser. 1278, 163–166.
-
Sussman, E. S. (2007). A new view on the MMN and attention debate, J. Psychophysiol. 21, 164–175.
-
Sutton, S., Braren, M., Zubin, J. and John, E. R. (1965). Evoked-potential correlates of stimulus uncertainty, Science 150, 1187–1188.
-
Talsma, D. (2015). Predictive coding and multisensory integration: an attentional account of the multisensory mind, Front. Integr. Neurosci. 9, 19. DOI:10.3389/fnint.2015.00019.
-
Talsma, D., Doty, T. J. and Woldorff, M. G. (2007). Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration?, Cereb. Cortex 17, 679–690.
-
Talsma, D., Senkowski, D., Soto-Faraco, S. and Woldorff, M. G. (2010). The multifaceted interplay between attention and multisensory integration, Trends Cogn. Sci. 14, 400–410.
-
Teder-Sälejärvi, W. A., McDonald, J. J., Di Russo, F. and Hillyard, S. A. (2002). An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings, Cogn. Brain Res. 14, 106–114.
-
Wang, W. Y., Hu, L., Valentini, E., Xie, X. B., Cui, H. Y. and Hu, Y. (2012). Dynamic characteristics of multisensory facilitation and inhibition, Cogn. Neurodyn. 6, 409–419.
-
Wickens, C., Kramer, A., Vanasse, L. and Donchin, E. (1983). Performance of concurrent tasks: a psychophysiological analysis of the reciprocity of information-processing resources, Science 221, 1080–1082.
-
Woodman, G. F. (2010). A brief introduction to the use of event-related potentials in studies of perception and attention, Atten. Percept. Psychophys. 72, 2031–2046.
-
Yabe, H., Tervaniemi, M., Reinikainen, K. and Näätänen, R. (1997). Temporal window of integration revealed by MMN to sound omission, NeuroReport 8, 1971–1974.
-
Zmigrod, S. and Hommel, B. (2013). Feature integration across multimodal perception and action: a review, Multisens. Res. 26, 143–157.
| All Time | Past 365 days | Past 30 Days | |
|---|---|---|---|
| Abstract Views | 1420 | 220 | 9 |
| Full Text Views | 72 | 5 | 1 |
| PDF Views & Downloads | 105 | 12 | 0 |
Attentional Interactions Between Vision and Hearing in Event-Related Responses to Crossmodal and Conjunct Oddballs
In: Multisensory ResearchFaculty of Medicine, University of Freiburg, Germany
Search for other papers by Evelyn B. N. Friedel in
Current site
Google Scholar
PubMed
Faculty of Medicine, University of Freiburg, Germany
Search for other papers by Michael Bach in
Current site
Google Scholar
PubMed
Faculty of Medicine, University of Freiburg, Germany
Search for other papers by Sven P. Heinrich in
Current site
Google Scholar
PubMed
Abstract
Are alternation and co-occurrence of stimuli of different sensory modalities conspicuous? In a novel audio-visual oddball paradigm, the P300 was used as an index of the allocation of attention to investigate stimulus- and task-related interactions between modalities. Specifically, we assessed effects of modality alternation and the salience of conjunct oddball stimuli that were defined by the co-occurrence of both modalities. We presented (a) crossmodal audio-visual oddball sequences, where both oddballs and standards were unimodal, but of a different modality (i.e., visual oddball with auditory standard, or vice versa), and (b) oddball sequences where standards were randomly of either modality while the oddballs were a combination of both modalities (conjunct stimuli). Subjects were instructed to attend to one of the modalities (whether part of a conjunct stimulus or not). In addition, we also tested specific attention to the conjunct stimuli. P300-like responses occurred even when the oddball was of the unattended modality. The pattern of event-related potential (ERP) responses obtained with the two crossmodal oddball sequences switched symmetrically between stimulus modalities when the task modality was switched. Conjunct oddballs elicited no oddball response if only one modality was attended. However, when conjunctness was specifically attended, an oddball response was obtained. Crossmodal oddballs capture sufficient attention even when not attended. Conjunct oddballs, however, are not sufficiently salient to attract attention when the task is unimodal. Even when specifically attended, the processing of conjunctness appears to involve additional steps that delay the oddball response.
| All Time | Past 365 days | Past 30 Days | |
|---|---|---|---|
| Abstract Views | 1420 | 220 | 9 |
| Full Text Views | 72 | 5 | 1 |
| PDF Views & Downloads | 105 | 12 | 0 |
